Diagnosis and treatment image data desensitization method based on cross-modal adversarial example generation

Abstract

The invention provides a diagnosis and treatment image data desensitization method based on cross-modal adversarial example generation, and relates to the technical field of image data processing, and the method comprises the steps: obtaining a target category label of a given attacker classifier, and generating a fine-grained text description based on the target category label; converting the text description into specific visual representation by using a cross-modal method; forming an adversarial example image by adding perturbations to the visual representation using different adversarial example generation methods; and covering the diagnosis and treatment image data with the adversarial example image to enable the diagnosis and treatment image data to avoid classification of the classifier in the absence of clean samples. According to the invention, the problem that the existing diagnosis and treatment image data desensitization method has strong dependence on source data and cannot meet fine-grained semantic requirements is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a method for desensitizing diagnostic image data based on cross-modal adversarial example generation. Background Art

[0002] With the rapid development of big data and Internet technology, the efficiency of data acquisition and processing has changed dramatically, making the use of data resources more convenient, but also increasing security risks such as privacy leakage. Among them, image diagnosis and treatment data, as an important part of data resources, contains a large amount of patient user privacy information and medical technology information, and should have a high level of protection.

[0003] Traditional image diagnosis and treatment data desensitization methods combine diagnosis and treatment images with clean samples that have been identified by classifiers or generate adversarial examples to add a masking layer to make the classifier's attack on diagnosis and treatment images ineffective, and require a source data set as a basis. However, as sensitive data, diagnosis and treatment image data is identified by the classifier, which means there is a risk of data leakage. It is difficult to obtain clean samples that have been identified as source data, and traditional diagnosis and treatment image desensitization schemes based on adversarial example generation are difficult to implement. Second, existing adversarial example generation methods cannot perform customized adversarial attacks or meet fine-grained semantic requirements. For example, adversarial example images of "cats" are required, and they also need to have additional fine-grained semantic details, such as "orange eyes" and "white body". In addition, existing methods all add unstructured perturbations to the pixel space of diagnosis and treatment images, which cannot handle scenes with semantic information, and create difficulties for cross-modal diagnosis and treatment data desensitization. Finally, these unstructured pixel perturbations affect the visual effect of adversarial examples, resulting in poor quality of generated images. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for desensitizing medical image data based on cross-modal adversarial example generation, which solves the problems of strong dependence on source data and inability to meet fine-grained semantic requirements in existing diagnostic image data desensitization methods, and effectively realizes an efficient method for desensitizing medical image data based on semantic cross-modal generation of adversarial examples.

[0005] In order to achieve the above objectives, the technical solution adopted by the present invention is: a method for desensitizing diagnostic image data based on cross-modal adversarial example generation, comprising the following steps:

[0006] S1. Obtain the target category label of the given attacker classifier and generate a fine-grained text description of each category based on the target category label;

[0007] S2, using a cross-modal approach to transform textual descriptions into concrete visual representations;

[0008] S3, generating adversarial perturbations by using different adversarial example generation methods and adding them to specific visual representations to generate adversarial example images;

[0009] S4. Overlay the generated adversarial example image onto the diagnosis and treatment image data, so that the diagnosis and treatment data can avoid classification by the classifier in the absence of clean samples, thereby achieving desensitization of the diagnosis and treatment image data.

[0010] The beneficial effects of the present invention are as follows: the present invention provides a method for desensitizing medical image data that combines a subtitle generation module GPT (Generative Pre-Trained Transformer), a cross-modal generation model, and an adversarial example generation method. The present invention solves the problems of strong dependence on source data and inability to meet fine-grained semantic requirements in existing methods for desensitizing medical image data, and effectively implements an efficient method for desensitizing medical image data based on semantic cross-modal generation of adversarial examples. Specifically, 1) the present invention effectively utilizes the advantage of natural language being easy to acquire, and desensitization only requires category semantic labels. 2) The present invention can well handle the problem of strong dependence on source data that requires clean samples in traditional methods. 3) The adversarial perturbations of the present invention are generated by three methods, and there are multiple perturbation reserves, which ensure that the generated adversarial examples can deceive the model, and can overcome the problems of low generation efficiency and poor portability of other attack algorithms.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S101, obtaining a target category label of a given attacking classifier;

[0013] S102, embedding the target category label into the general input template of the subtitle generation module GPT;

[0014] S103. Construct a semantic description of the attacker in combination with the general template, and generate a fine-grained text description for each category.

[0015] The beneficial effect of the above further scheme is that the present invention combines the general template and the GPT subtitle generation module to automatically generate different and innovative text description tags, overcoming the challenge of needing to collect semantic descriptions of attack examples.

[0016] Furthermore, the cross-modal method is used to convert the text description into a specific visual representation, which specifically includes: utilizing the image generation capability of the stable diffusion model to convert the fine-grained text description into a specific visual representation that can be recognized by the classifier.

[0017] Furthermore, the stable diffusion model includes:

[0018] U-Net structure, which is used to denoise the latent image iteratively through the cross-attention mechanism with text embedding as the condition to obtain the conditional latent image representation;

[0019] A decoder is used to perform decoding operations on the conditional latent image representation to generate a concrete visual representation that reflects the attacker's semantic requirements.

[0020] The beneficial effect of the above further scheme is that the present invention utilizes a stable diffusion model based on a latent diffusion model and having an attention mechanism to improve the synthesis capability of images, thereby generating realistic high-quality images.

[0021] Furthermore, step S2 includes the following steps:

[0022] S201, inputting the fine-grained text description generated by the subtitle generation module GPT into a text encoder with frozen parameters to convert it into a text embedding;

[0023] S202, in text embedding, a random latent image generated in combination with a latent seed is used as an input of the U-Net structure;

[0024] S203, in the U-Net structure, using the text embedding as a condition, the latent image is denoised iteratively through the cross attention mechanism to obtain a conditional latent image representation;

[0025] S204: Input the conditional latent image representation into the decoder in the autoencoder for decoding operation, generate a specific visual representation that reflects the attacker's semantic requirements, and complete the conversion of fine-grained text description.

[0026] The beneficial effect of the above further scheme is: the present invention synthesizes images corresponding to text descriptions through a diffusion model, thereby solving the problem of unavailable or lacking clean samples, and cross-modally generates clean image samples to provide source data for subsequent generation of adversarial examples to complete data desensitization.

[0027] Furthermore, step S3 includes the following steps:

[0028] S301, combining the text description and the specific visual representation to generate a label-title-image triplet;

[0029] S302, generating adversarial perturbations using different adversarial example generation methods according to the label-title-image triplet;

[0030] S303: Add the generated adversarial perturbation to the specific visual representation to generate an adversarial example image.

[0031] The beneficial effect of the above further scheme is: the present invention combines three different adversarial example generation methods to generate adversarial perturbations to obtain adversarial examples, which can overcome the problems of low generation efficiency and poor portability of other attack algorithms.

[0032] Furthermore, the step S302 is specifically as follows:

[0033] According to the label-title-image triplet, the first adversarial perturbation is generated using the fast gradient sign method FGSM;

[0034] According to the label-title-image triplet, the projected gradient descent method PGD is used to generate the second adversarial perturbation;

[0035] Based on the label-title-image triplet, the attack algorithm CW generates the third adversarial perturbation.

[0036] The beneficial effect of the above further scheme is: the adversarial perturbation fast gradient sign method FGSM, the projected gradient descent method PGD and the three generated by the attack algorithm of the present invention are selected to ensure that the generated adversarial samples can deceive the model.

[0037] Furthermore, the expression of the first anti-disturbance is as follows:

[0038]

[0039] Among them, η FGSM represents the first adversarial perturbation, ε represents the adjustment coefficient used to control the size of the adversarial perturbation, sign() represents the sign function, represents the partial derivative of the loss function J with respect to x, x represents the image of the triplet input to the classifier, θ represents the weight parameter of the classifier, and y represents the category label;

[0040] The expression of the second adversarial perturbation is as follows:

[0041]

[0042] Among them, η PGD represents the second adversarial disturbance, α represents the adjustment coefficient used to control the amplitude of the iteratively generated adversarial disturbance;

[0043] The expression of the third counter-perturbation is as follows:

[0044]

[0045] Among them, η CW represents the third adversarial disturbance, and ω represents a variable.

[0046] The beneficial effects of the above further scheme are: the fast gradient sign method FGSM is a single-step attack that can quickly generate adversarial examples, and the projected gradient descent method PGD subdivides the process of calculating adversarial perturbations into iterative calculations and adds randomization processing to solve the problem that the adversarial perturbations calculated by the fast gradient sign FGSM may be inaccurate. The CW attack algorithm customizes different objective functions to solve the problems of low generation efficiency and poor portability of the shadow gradient descent method PGD. The present invention utilizes the characteristics of the three complementing each other and selects the best perturbation among the three according to actual conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of the method of the present invention.

[0048] Figure 2 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0049] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0050] Example

[0051] The basic idea of ​​the present invention is to generate adversarial examples and combine them with medical images according to the category labels of the attacking white-box classifier in the absence of clean samples, so as to avoid the successful classification of the classifier and achieve the effect of desensitizing the medical image data. The generative pre-trained Transformer has the advantages of powerful natural language understanding, and the diffusion model has excellent cross-modal image generation ability. The adversarial overlay is generated by the adversarial example generation method of the fast gradient sign method (FGSM) and the projected gradient descent method (PGD), which solves the problems of strong dependency of source data and fine-grained semantic requirements in data desensitization.

[0052] In order to solve the problem that adversarial examples need fine-grained semantic description in practice, the present invention introduces a subtitle generation module with GPT as the core, supplements the semantic description of relevant category labels, constructs text information that can be effectively recognized and generated by the diffusion model, and then uses the adversarial example generation method to add unstructured perturbations to the image, and finally covers it on the diagnosis and treatment image, effectively shielding the recognition of the other party's classifier, and achieving the desensitization effect of diagnosis and treatment image data. Figure 1 and Figure 2 As shown, the present invention provides a method for desensitizing diagnostic image data based on cross-modal adversarial examples, and its implementation method is as follows:

[0053] S1. Get the target category label of the given attacker classifier and generate a fine-grained text description of each category based on the target category label. The implementation method is as follows:

[0054] S101, obtaining a target category label of a given attacking classifier;

[0055] S102, embedding the target category label into the general input template of the subtitle generation module GPT;

[0056] S103. Construct a semantic description of the attacker in combination with the general template, and generate a fine-grained text description for each category.

[0057] In this embodiment, for a given attacking classifier, its target category label is obtained. The class label is then used as a condition and combined with the generative model to construct a semantic description of the attacker to achieve fine-grained semantic generation. Specifically, given the category label of the opponent's classifier, a set of prompts is used as a general template input to GPT to generate a title corresponding to each [LABEL]. By prompting "Please generate a simple title for [cardiac diagnosis and treatment imaging]", a large number of concise descriptions of [LABEL] can be obtained, which not only enriches the relevant information of the object, but also can be used as a title for standard customization requirements.

[0058] S2. Use a cross-modal approach to convert text descriptions into concrete visual representations. Specifically, the image generation capability of the stable diffusion model is used to convert fine-grained text descriptions into concrete visual representations that can be recognized by the classifier.

[0059] The stable diffusion model includes:

[0060] U-Net structure, which is used to denoise the latent image iteratively through the cross-attention mechanism with text embedding as the condition to obtain the conditional latent image representation;

[0061] A decoder, which is used to perform a decoding operation on the conditional latent image representation to generate a concrete visual representation that reflects the attacker's semantic requirements;

[0062] The implementation method of step S2 is as follows:

[0063] S201, inputting the fine-grained text description generated by the subtitle generation module GPT into a text encoder with frozen parameters to convert it into a text embedding;

[0064] S202, in text embedding, a random latent image generated in combination with a latent seed is used as an input of the U-Net structure;

[0065] S203, in the U-Net structure, using the text embedding as a condition, the latent image is denoised iteratively through the cross attention mechanism to obtain a conditional latent image representation;

[0066] S204: Input the conditional latent image representation into the decoder in the autoencoder for decoding operation, generate a specific visual representation that reflects the attacker's semantic requirements, and complete the conversion of fine-grained text description.

[0067] In this embodiment, the image generation capability of the stable diffusion model composed of the U-Net structure and the VAE decoder is utilized to transform the fine-grained text description into a specific visual representation that can be recognized by the classifier.

[0068] In this embodiment, the subtitles generated by the subtitle generation module GPT in step S1 are input to the CLIP text encoder with frozen parameters to convert them into text embeddings. Then, a random latent image generated by a latent seed is combined with the text embedding as the input of the U-Net architecture. The U-Net structure uses the text embedding as a condition, iteratively denoises the latent image through a cross-attention mechanism, and outputs a conditional latent image representation. Finally, the output is sent to the decoder in the variational autoencoder for decoding, thereby generating a visually coherent image that accurately reflects the attacker's semantic requirements.

[0069] In this embodiment, the subtitle generation module GPT and labels are used to synthesize images corresponding to the text description through a stable diffusion model, which solves the problem of unavailable or lacking clean samples. The basic idea is to use the available category labels to generate clean image samples across modalities to provide source data for subsequent generation of adversarial examples to complete data desensitization. Using a pre-trained semantic generation model, relevant fine-grained semantic descriptions can be generated with only one label. The large-scale contrast language-image pre-training model CLIP can generate realistic high-quality images when combined with a diffusion model with an attention mechanism, making it possible to autonomously generate effective diagnostic and treatment images, effectively solving the problem of unavailable samples for the other party's classifier.

[0070] S3. Generate adversarial perturbations by using different adversarial example generation methods and add them to specific visual representations to generate adversarial example images. The implementation method is as follows:

[0071] S301, combining the text description and the specific visual representation to generate a label-title-image triple;

[0072] S302: Generate adversarial perturbations using different adversarial example generation methods according to the label-title-image triplet, which is specifically:

[0073] According to the label-title-image triplet, the first adversarial perturbation is generated using the fast gradient sign method FGSM;

[0074] According to the label-title-image triplet, the projected gradient descent method PGD is used to generate the second adversarial perturbation;

[0075] Based on the label-title-image triplet, the attack algorithm CW generates the third adversarial perturbation;

[0076] S303: Add the generated adversarial perturbation to the specific visual representation to generate an adversarial example image.

[0077] In this embodiment, the category label, the text description generated by the caption generation module GPT in step S1, and the image generated in step S2 are combined into a triple. Then, the corresponding adversarial examples are generated by using different adversarial example generation methods based on the label-title-image triple to generate adversarial perturbations added to the image in step S2. The adversarial perturbations are generated by three attack algorithms: fast gradient sign method FGSM, projected gradient descent method PGD, and CW attack method, to ensure that the generated adversarial examples can deceive the model.

[0078] In this embodiment, the adversarial perturbation is composed of the fast gradient sign method FGSM, the projected gradient descent method PGD and the CW attack algorithm, which are gradient-based and optimization-based white-box attack methods. The fast gradient sign method FGSM adds perturbations in the opposite direction along the gradient of the classifier loss function, and the projected gradient descent method PGD subdivides the process of calculating the adversarial perturbation into iterative calculations and adds randomization processing; while the CW attack algorithm can customize different objective functions and select the optimal one through empirical data to perform adversarial attacks, which can overcome the problems of low generation efficiency and poor portability of other attack algorithms. The algorithm combining the three is as follows:

[0079] Let the label-title-image triplet be (y, t, x), the generated adversarial perturbation be η, and the generated adversarial example be x adv . Then the process of generating adversarial examples can be formalized as:

[0080] x adv =x+η

[0081] Among them, η∈{η FGSM , η PGD , η CW}. They correspond to the disturbances generated by the fast gradient sign method FGSM, the projected gradient descent method PGD and the CW attack algorithm. The three adversarial disturbance generation methods are as follows:

[0082] For the fast gradient sign method FGSM, the first adversarial perturbation is generated as follows:

[0083]

[0084] Among them, η FGSM represents the first adversarial perturbation, ε represents the adjustment coefficient used to control the size of the adversarial perturbation, sign() represents the sign function, represents the partial derivative of the loss function J with respect to x, x represents the partial derivative of the loss function J with respect to x, x represents the image input to the classifier in the triplet, θ represents the weight parameter of the classifier, and y represents the category label.

[0085] Similarly, the second adversarial perturbation of projected gradient descent PGD is:

[0086]

[0087] Among them, η PGD represents the second adversarial disturbance, and α represents the adjustment coefficient used to control the amplitude of the iteratively generated adversarial disturbance.

[0088] When the projected gradient descent method PGD executes the fast gradient sign method FGSM with a smaller step size and projects the examples after multiple iterations into a specified range, an adversarial example can be generated. The corresponding process of generating adversarial examples can be expressed as:

[0089]

[0090] Among them, CLIP x,ε {*} function Projecting to the interval [-ε,ε], we get a new adversarial example represents the adversarial sample generated in step N, Represents the adversarial sample generated in step N+1, CLIP x,ε Represents the projection function, x represents the image input, and ε represents the projection interval width.

[0091] In addition, CW introduces a new variable ω, and the solution to minimize the following objective function is found as:

[0092]

[0093] where c is a balancing hyperparameter and f is defined as:

[0094] f(x)=max(Z(x) y -max{Z(x) i :i≠y},-κ)

[0095] Among them, Z(x) yrepresents the predicted probability of image x as label y determined by the classifier, κ is an artificially set classification confidence threshold used to control the desired confidence level of adversarial examples, i represents other categories different from y, y represents the category label, and Z(x) i Represents the predicted probability that the classifier determines that the image x is of label i.

[0096] Finally, the adversarial perturbation corresponding to the CW attack method is:

[0097]

[0098] Among them, η CW represents the third adversarial perturbation, ω represents the variable, and x represents the input image.

[0099] In this embodiment, the classifier is for most white-box classification models, and this framework can be used to avoid attacks.

[0100] S4. Overlay the generated adversarial example image onto the diagnosis and treatment image data, so that the diagnosis and treatment data can avoid classification by the classifier in the absence of clean samples, thereby achieving desensitization of the diagnosis and treatment image data.

[0101] In this embodiment, the generated adversarial examples that successfully shield the classifier are overlaid on the patient's diagnosis and treatment images, ultimately allowing the patient's diagnosis and treatment image data to avoid classification by the classifier in the absence of clean samples, thereby achieving the effect of data desensitization.

[0102] In summary, the present invention can effectively shield the diagnosis and treatment images from the recognition of the white box classifier.

[0103] The above-mentioned examples only express the implementation methods of the present invention, but it cannot be understood that there is no limitation on the patent scope of the present invention. It should be noted that for those skilled in the art, any equivalent replacement or change based on the technical solution and inventive concept of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for desensitizing diagnostic image data based on cross-modal adversarial example generation, characterized in that: The following steps are involved: S1. Obtain the target category label of the given attacker classifier and generate a fine-grained text description of each category based on the target category label; S2, using a cross-modal approach to transform textual descriptions into concrete visual representations; S3, generating adversarial perturbations by using different adversarial example generation methods and adding them to specific visual representations to generate adversarial example images; S4. Overlay the generated adversarial example image onto the diagnosis and treatment image data, so that the diagnosis and treatment data can avoid classification by the classifier in the absence of clean samples, thereby achieving desensitization of the diagnosis and treatment image data.

2. The method for desensitizing diagnostic image data based on cross-modal adversarial examples according to claim 1, characterized in that: The step S1 comprises the following steps: S101, obtaining a target category label of a given attacking classifier; S102, embedding the target category label into the general input template of the subtitle generation module GPT; S103. Construct a semantic description of the attacker in combination with the general template, and generate a fine-grained text description for each category.

3. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 2, characterized in that: The cross-modal method is used to convert text descriptions into specific visual representations, which specifically includes: utilizing the image generation capability of the stable diffusion model to convert fine-grained text descriptions into specific visual representations that can be recognized by a classifier.

4. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 3, characterized in that: The stable diffusion model includes: U-Net structure, which is used to denoise the latent image iteratively through the cross-attention mechanism with text embedding as the condition to obtain the conditional latent image representation; A decoder is used to perform decoding operations on the conditional latent image representation to generate a concrete visual representation that reflects the attacker's semantic requirements.

5. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 4, characterized in that: The step S2 comprises the following steps: S201, inputting the fine-grained text description generated by the subtitle generation module GPT into a text encoder with frozen parameters to convert it into a text embedding; S202, in text embedding, a random latent image generated in combination with a latent seed is used as an input of the U-Net structure; S203, in the U-Net structure, using the text embedding as a condition, the latent image is denoised iteratively through the cross attention mechanism to obtain a conditional latent image representation; S204: Input the conditional latent image representation into the decoder in the autoencoder for decoding operation, generate a specific visual representation that reflects the attacker's semantic requirements, and complete the conversion of fine-grained text description.

6. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 1, characterized in that: The step S3 comprises the following steps: S301, combining the text description and the specific visual representation to generate a label-title-image triplet; S302, generating adversarial perturbations using different adversarial example generation methods according to the label-title-image triplet; S303: Add the generated adversarial perturbation to the specific visual representation to generate an adversarial example image.

7. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 6, characterized in that: The step S302 is specifically as follows: According to the label-title-image triplet, the first adversarial perturbation is generated using the fast gradient sign method FGSM; According to the label-title-image triplet, the projected gradient descent method PGD is used to generate the second adversarial perturbation; Based on the label-title-image triplet, the attack algorithm CW generates the third adversarial perturbation.

8. The method for desensitizing diagnostic image data based on cross-modal adversarial example generation according to claim 7, characterized in that: The expression of the first adversarial perturbation is as follows: Among them, η FGSM represents the first adversarial perturbation, ε represents the adjustment coefficient used to control the size of the adversarial perturbation, sign() represents the sign function, represents the partial derivative of the loss function J with respect to x, x represents the image of the triplet input to the classifier, θ represents the weight parameter of the classifier, and y represents the category label; The expression of the second adversarial perturbation is as follows: Among them, η PGD represents the second adversarial disturbance, α represents the adjustment coefficient used to control the amplitude of the iteratively generated adversarial disturbance; The expression of the third counter-perturbation is as follows: Among them, η CW represents the third adversarial disturbance, and ω represents a variable.

Images