A method, device and equipment for generating an adversarial medical image and a storage medium

Abstract

The application discloses a kind of generation method, device and equipment of counteracting medical image and storage medium, the method includes: based on convolution parameter data, the last medical image obtained and the last disturbance noise corresponding to the last medical image, determine current medical image;Determine current disturbance noise based on current medical image, reference neural network model and the last disturbance noise;Current medical image is used as the last medical image and current disturbance noise is used as the last disturbance noise, and return executes based on the last medical image, the last disturbance noise and convolution parameter data, determine the operation of current medical image;In the case where current iteration number meets preset number threshold, current disturbance noise is used as target disturbance noise, and based on target disturbance noise and the original medical image corresponding to test neural network model, determine counteracting medical image.The application embodiment solves the problem that disturbance noise determined based on existing counterattack method is large.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a method and device for generating an adversarial medical image, an apparatus, and a storage medium. BACKGROUND

[0002] Models based on deep neural networks have good performance on various tasks, especially in the field of medical images, there are many computer-aided diagnosis systems based on deep neural networks. For computer-aided diagnosis systems, the application object is real patients. If the model outputs an incorrect result, it may lead to misdiagnosis and affect subsequent treatment, causing loss to the patient's economy, spirit, etc. Therefore, adversarial attack technology has important practical significance for training neural network models in the medical field.

[0003] The principle of adversarial attack technology is to add unperceivable disturbance noise to the input image as much as possible according to the characteristics of the trained neural network model, generate an adversarial image, input the adversarial image into the neural network model, so as to mislead the neural network model and make the neural network model produce an incorrect output.

[0004] The goal of adversarial attack technology is to make the added disturbance as small as possible while ensuring that the neural network model can produce an incorrect output. However, the existing adversarial attack technology adds a large amount of disturbance noise, which cannot meet the high quality requirements of the medical field for adversarial medical images. SUMMARY

[0005] The embodiments of the present application provide a method and device for generating an adversarial medical image, an apparatus, and a storage medium to solve the problem of non-smooth disturbance noise determined based on the existing adversarial attack method, improve the image quality of the generated adversarial medical image, and meet the high quality requirements of the medical field for adversarial medical images.

[0006] According to an embodiment of the present application, a method for generating an adversarial medical image is provided, which comprises:

[0007] In response to obtaining a previous medical image and a previous disturbance noise corresponding to the previous medical image, determining a current medical image based on the previous medical image, the previous disturbance noise, and convolution parameter data;

[0008] Determining a current disturbance noise based on the current medical image, a reference neural network model, and the previous disturbance noise;

[0009] Taking the current medical image as a previous medical image and taking the current disturbance noise as a previous disturbance noise, and returning to perform the operation of determining a current medical image based on the previous medical image, the previous disturbance noise, and convolution parameter data;

[0010] In a case where the current iteration number meets a preset number threshold, the current perturbation noise is taken as a target perturbation noise, and an adversarial medical image is determined based on the target perturbation noise and an original medical image corresponding to the test neural network model.

[0011] According to another embodiment of the present application, an apparatus for generating an adversarial medical image is provided, which comprises:

[0012] A current medical image determination module is configured to, in response to obtaining a previous medical image and a previous perturbation noise corresponding to the previous medical image, determine a current medical image based on the previous medical image, the previous perturbation noise and convolution parameter data.

[0013] A current perturbation noise determination module is configured to determine a current perturbation noise based on the current medical image, a reference neural network model and the previous perturbation noise.

[0014] A return execution module is configured to take the current medical image as the previous medical image and the current perturbation noise as the previous perturbation noise, and return to execute an operation of determining a smoothed perturbation noise based on the previous perturbation noise and the convolution parameter data.

[0015] An adversarial medical image determination module is configured to, in a case where the current iteration number meets a preset number threshold, take the current perturbation noise as a target perturbation noise, and determine an adversarial medical image based on the target perturbation noise and an original medical image corresponding to a test neural network model.

[0016] According to another embodiment of the present application, an electronic device is provided, which comprises:

[0017] at least one processor; and

[0018] a memory connected with the at least one processor in communication; wherein

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for generating an adversarial medical image according to any one of the embodiments of the present application.

[0020] According to another embodiment of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the method for generating an adversarial medical image according to any one of the embodiments of the present application.

[0021] The technical scheme of the embodiment of the present application determines the current medical image based on the convolution parameter data, the obtained previous medical image and the previous disturbance noise corresponding to the previous medical image, determines the current disturbance noise based on the current medical image, the reference neural network model and the previous disturbance noise, takes the current medical image as the previous medical image and takes the current disturbance noise as the previous disturbance noise, and returns to perform the operation of determining the current medical image based on the previous medical image, the previous disturbance noise and the convolution parameter data, and in the case that the current iteration number meets the preset number threshold, takes the current disturbance noise as the target disturbance noise, and determines the adversarial medical image based on the target disturbance noise and the original medical image corresponding to the test neural network model. The embodiment of the present application iterates from the perspective of the disturbance noise, and smoothes the previous disturbance noise based on the convolution parameter data, solves the problem of large disturbance noise determined based on the existing adversarial attack method, improves the image quality of the generated adversarial medical image, and meets the high-quality requirement of the medical field for the adversarial medical image.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained from these drawings by those skilled in the art without creative labor.

[0024] Figure 1 A flowchart of a method for generating an adversarial medical image provided by the first embodiment of the present application;

[0025] Figure 2 A flowchart of a method for generating an adversarial medical image provided by the second embodiment of the present application;

[0026] Figure 3 A schematic diagram of a specific example of a method for generating a current medical image provided by the second embodiment of the present application;

[0027] Figure 4 A structural schematic diagram of a device for generating an adversarial medical image provided by the third embodiment of the present application;

[0028] Figure 5 A structural schematic diagram of an electronic device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0030] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover not exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment one

[0032] Figure 1 A flowchart of a method for generating an adversarial medical image is provided in the first embodiment of the present application. The present embodiment can be applied to the generation of an adversarial medical image in the context of a neural network model. The method can be performed by an adversarial medical image generation device, which can be realized in the form of hardware and / or software, and can be configured in a terminal device. As shown in Figure 1 The method comprises the following steps:

[0033] S110, in response to obtaining a previous medical image and a previous perturbation noise corresponding to the previous medical image, determining a current medical image based on the previous medical image, the previous perturbation noise and convolution parameter data.

[0034] In examples, the acquisition device corresponding to the medical image in the embodiments of the present application includes, but is not limited to, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device or an ultrasound device, and the like, and the acquisition site corresponding to the medical image includes, but is not limited to, an eye fundus, a lung or a chest, and the like. The acquisition device and the acquisition site corresponding to the medical image are not limited herein.

[0035] In specific, the process of determining the target perturbation noise is an iterative process, the previous medical image is a medical image obtained through a previous iteration operation, and the previous perturbation noise is a perturbation noise obtained through the previous iteration operation. In an embodiment, when the current iteration operation is the first iteration operation, the previous medical image is an initial medical image corresponding to the reference neural network model, wherein the initial medical image is a real existing medical image corresponding to the reference neural network model, and the previous perturbation noise is 0.

[0036] In an embodiment, the convolution parameter data includes at least one preset convolution parameter, and correspondingly, the determination of the current medical image based on the previous medical image, the previous perturbation noise and the convolution parameter data includes: performing a convolution operation on the previous perturbation noise based on the at least one preset convolution parameter to obtain at least one corrected perturbation noise, and performing a mean operation on each corrected perturbation noise to obtain a smoothed perturbation noise; and determining the current medical image based on the smoothed perturbation noise and the previous medical image.

[0037] In specific, the convolution operation can fuse the surrounding information of the previous perturbation noise, so as to achieve the purpose of smoothing the previous perturbation noise.

[0038] In an embodiment, in specific, the preset convolution parameter includes a convolution kernel size and a convolution kernel weight, and when the number of preset convolution parameters is at least two, the convolution kernel size and / or the convolution kernel weight in each preset convolution parameter is different.

[0039] In specific, the convolution kernel refers to that when image processing is performed, a small region in an input image is weighted and averaged to become each object pixel in an output image, wherein the weight is defined by a function, and the function is called a convolution kernel. In specific, the convolution kernel size can be used to represent the size of the region, and the weight of the convolution kernel can be used to represent the parameter value of each matrix element in the convolution kernel.

[0040] In an embodiment, when the preset convolution parameter is at least two, different convolution kernel sizes can achieve different range smoothing of the perturbation noise, and different convolution kernel weights can achieve different magnitude smoothing of the perturbation noise.

[0041] The advantage of such setting is that the smoothing effect of the previous perturbation noise can be improved.

[0042] In an embodiment, the determination of the current medical image based on the smoothed perturbation noise and the previous medical image includes: adding the smoothed perturbation noise to the previous medical image to obtain the current medical image. In specific, the current medical image X t satisfies the formula:

[0043]

[0044] Specifically, X t-1 This represents the previous medical image, m represents the number of preset convolution parameters included in the convolution parameter data, and η represents the... t-1 K represents the previous disturbance noise. i Let represent the i-th preset convolution parameter, where η0 = 0 and t ≥ 1.

[0045] S120. Based on the current medical image, the reference neural network model, and the previous perturbation noise, determine the current perturbation noise.

[0046] In one embodiment, specifically, when the attack scenario is a white-box attack, the reference neural network model is the attacked neural network model; when the attack scenario is a black-box attack, the reference neural network model is a pre-built neural network model with a network architecture similar to or the same as the attacked neural network model. For example, when the attacked neural network model is an image classification model, the reference neural network model can be a ResNet50 classification model.

[0047] In one embodiment, optionally, determining the current perturbation noise based on the current medical image, the reference neural network model, and the previous perturbation noise includes: inputting the current medical image into the reference neural network model to obtain the current output result, and determining the current loss gradient corresponding to the reference neural network model based on the current output result and the standard result; and determining the current perturbation noise based on the previous perturbation noise and the current loss gradient.

[0048] Specifically, the current disturbance noise η t Satisfying the formula:

[0049]

[0050] Where ε is the adjustment range of the perturbation noise, used to control the attack amplitude of the medical image. The larger ε is, the greater the attack intensity, and the easier it is for the perturbation noise in the modified medical image to be observed by the naked eye. clip represents the clipping operation, used to control the current perturbation noise within the range of ε, and α represents the preset coefficient.

[0051] Here, sign(x) is the sign function, which takes the value 1 when x is greater than 0, takes the value 0 when x is equal to 0, and takes the value -1 when x is less than 0. This means finding L(f(X)). t The current loss gradient of L(f(X),y), t ),y) represents the reference neural network model when the input is the current medical image X. t The current output is f(X) tthe current loss function when the standard output result is y.

[0052] During the training of the neural network model, the update of the model parameters of each iteration is to subtract the calculated current loss gradient from the model parameters, so that the function value of the current loss function is smaller and smaller, that is, the probability of the prediction pair of the neural network model is larger and larger. The purpose of the adversarial attack is to hope that the neural network model can be classified incorrectly, so that only the function value of the current loss function is larger and larger. In the case that the model parameters of the neural network model are fixed and unchanged, adding the perturbation noise calculated based on the loss gradient to the input medical image can achieve the above purpose.

[0053] S130, judge whether the current iteration number meets the preset number threshold, if yes, execute S150, if no, execute S140.

[0054] S140 takes the current medical image as the last medical image and takes the current perturbation noise as the last perturbation noise, and executes S110.

[0055] Specifically, in the case that the current iteration number does not meet the preset number threshold, the iteration operation is continued. Exemplarily, the preset number threshold can be 1 or 10, and the preset number threshold is not limited here.

[0056] S150, takes the current perturbation noise as the target perturbation noise, and determines the adversarial medical image based on the target perturbation noise and the original medical image corresponding to the test neural network model.

[0057] Specifically, the test neural network model is the attacked neural network model. In an embodiment, specifically, when the attack scenario is white-box attack, the reference neural network model and the test neural network model are the same neural network model, and when the attack scenario is black-box attack, the reference neural network model is a neural network model similar or identical to the network architecture of the test neural network model.

[0058] Specifically, the target perturbation noise is added to the original medical image corresponding to the test neural network model to obtain the adversarial medical image corresponding to the test neural network model. Exemplarily, the attack operation can be performed on the test neural network model based on the adversarial medical image to determine the performance score of the test neural network model. Specifically, the original medical image is a real existing medical image corresponding to the test neural network model.

[0059] The technical scheme of the embodiment determines the current medical image based on the convolution parameter data, the obtained previous medical image and the previous perturbation noise corresponding to the previous medical image, determines the current perturbation noise based on the current medical image, the reference neural network model and the previous perturbation noise, takes the current medical image as the previous medical image and takes the current perturbation noise as the previous perturbation noise, and returns to perform the operation of determining the current medical image based on the previous medical image, the previous perturbation noise and the convolution parameter data, and in the case that the current iteration number meets the preset number threshold, takes the current perturbation noise as the target perturbation noise, and determines the adversarial medical image based on the target perturbation noise and the original medical image corresponding to the test neural network model. The embodiment of the application iterates from the perspective of the perturbation noise, and smoothes the previous perturbation noise based on the convolution parameter data, solves the problem that the perturbation noise determined based on the existing adversarial attack method is not smooth and cannot protect the medical semantic information, improves the image quality of the generated adversarial medical image, and meets the high-quality requirement of the medical field for the adversarial medical image.

[0060] Embodiment two

[0061] Figure 2 A flowchart of a method for generating an adversarial medical image is provided in Embodiment Two of the application. In this embodiment, the technical feature of "determining a current medical image based on a smoothed perturbation noise and a previous medical image" in the above-mentioned embodiment is further refined. As shown in Figure 2 The method comprises:

[0062] S210, inputting an initial medical image corresponding to a reference neural network model into a pre-trained target image segmentation model to obtain an output region of interest mask.

[0063] S220, in response to obtaining a previous medical image and a previous perturbation noise corresponding to the previous medical image, performing a convolution operation on the previous perturbation noise based on at least one preset convolution parameter to obtain at least one corrected perturbation noise.

[0064] The initial medical image is a real existing medical image, and the target image segmentation model is used to perform an image segmentation operation on the input initial medical image to output a region of interest mask corresponding to the initial medical image. The network architecture of the target image segmentation model includes but is not limited to FCN (Fully Convolution Netwoks), SegNet, U-Net (encoder-decoder), PSPNet (fusion multi-feature improved network) or Deeplab, etc., and the network architecture of the target image segmentation model is not limited here.

[0065] In an embodiment, specifically, the network architecture of the target image segmentation model is Attention U-Net. In an embodiment, specifically, in the region-of-interest mask, the pixel value corresponding to the region of interest is 1, and the pixel value of other regions except the region of interest is 0. The region-of-interest mask is stored in correspondence with the initial medical image corresponding thereto, and the region-of-interest mask can be a lesion mask.

[0066] S230, performing a mean operation on each modified perturbation noise to obtain a smoothed perturbation noise.

[0067] S240, determining whether the current iteration number is included in the preset iteration set, if yes, performing S250, and if no, performing S260.

[0068] In the embodiment, the preset iteration set includes at least one preset iteration number. For example, the preset iteration set includes at least one of the first iteration, the second iteration, and the Nth iteration, where N is the same as the preset number threshold.

[0069] In an embodiment, the preset iteration set includes the first iteration, the second iteration, or the Nth iteration. The advantage of this setting is that since the embodiment only performs the subsequent point-by-point multiplication operation in a certain iteration operation in the entire iteration process, the number of overall execution steps is reduced, thereby controlling the protection strength of the region of interest.

[0070] Specifically, when the preset iteration set only includes the Nth iteration, the final target perturbation noise obtained may have abrupt noise near other regions except the region of interest, resulting in that the smoothness of the generated target perturbation noise cannot meet the requirements. In another embodiment, the preset iteration set includes the first iteration, the second iteration, and the Nth iteration. The advantage of this setting is that it can improve the smoothness of the generated target perturbation noise.

[0071] S250, determining a reference attack noise based on the region-of-interest mask and the smoothed perturbation noise, adding the reference attack noise to the previous medical image to obtain a current medical image, and performing S270.

[0072] Specifically, the region-of-interest mask and the target perturbation noise are subjected to point-by-point multiplication operation to obtain the reference attack noise.

[0073] Figure 3FIG. 2 is a schematic diagram of a specific example of the method for generating a current medical image according to the second embodiment of the present application. Specifically, the previous medical image is input into the reference neural network model to obtain an output result, and based on the output result and the previous standard result, a previous loss gradient corresponding to the reference neural network model is determined. Based on the previous previous perturbation noise and the previous loss gradient, a previous perturbation noise is determined. At least one convolution operation is performed on the previous perturbation noise based on at least one preset convolution parameter to obtain at least one corrected perturbation noise, and a mean operation is performed on each corrected perturbation noise to obtain a smoothed perturbation noise. The smoothed perturbation noise and the region of interest mask corresponding to the initial medical image are multiplied to obtain a reference attack noise, and the reference attack noise is added to the previous medical image to obtain the current medical image.

[0074] S260, the smoothed perturbation noise is added to the previous medical image to obtain the current medical image.

[0075] S270, based on the current medical image, the reference neural network model and the previous perturbation noise, a current perturbation noise is determined.

[0076] S280, it is judged whether the current iteration number meets the preset number threshold, if yes, S291 is performed, and if not, S290 is performed.

[0077] S290, the current medical image is taken as the previous medical image, and the current perturbation noise is taken as the previous perturbation noise, and S220 is performed.

[0078] S291, the current perturbation noise is taken as the target perturbation noise, and based on the target perturbation noise and the original medical image corresponding to the test neural network model, an adversarial medical image is determined.

[0079] In the present embodiment, specifically, based on the target perturbation noise and the original medical image corresponding to the test neural network model, an adversarial medical image is determined, which includes: inputting the original medical image corresponding to the test neural network model into the target image segmentation model to obtain an output result of the original region of interest mask, based on the original region of interest mask and the target perturbation noise, a test attack noise is determined, and the test attack noise is added to the original medical image to obtain the adversarial medical image.

[0080] The medical image itself has particularity. The medical images of the same part of different patients are similar, but the positions and distributions of the regions of interest can be different. In the existing adversarial attack technology, when generating an adversarial medical image, the attack range of the perturbation noise is the whole medical image, so that the generated adversarial medical image destroys the key medical information in the original medical image, and cannot meet the image quality requirements of the generated adversarial medical image in the medical field. The technical scheme of the embodiment, by inputting the initial medical image corresponding to the reference neural network model into the pre-trained target image segmentation model, obtaining the output region of interest mask; in the case that the current iteration number is contained in the preset iteration set, based on the region of interest mask and the smooth perturbation noise, the reference attack noise is determined, and the reference attack noise is added to the last medical image to obtain the current medical image; wherein, the preset iteration set contains at least one preset iteration number, solves the problem that the generated adversarial medical image based on the existing method cannot protect the key medical information, improves the image quality of the generated adversarial medical image, and further meets the image quality requirements of the generated adversarial medical image in the medical field.

[0081] On the basis of the above-mentioned embodiment, optionally, the method further comprises: inputting the obtained at least one original medical image and the adversarial medical image corresponding to each original medical image into a test neural network model to obtain output first prediction results corresponding to each original medical image and second prediction results corresponding to each adversarial medical image; based on the first prediction results and the second prediction results, determining a current performance score of the test neural network model to test the robustness of the model; in the case that the current performance score does not meet a preset performance condition, continuing to train the test neural network model until the current performance score meets the preset performance condition, and taking the test neural network model as a target neural network model.

[0082] Specifically, the medical image dataset is divided into a training dataset and a test dataset D based on a preset ratio. For example, the preset ratio can be 8:2. The training dataset contains a plurality of training medical images, and the test dataset D contains a plurality of original medical images. The plurality of original medical images and their corresponding adversarial medical images constitute an attack test dataset D a .

[0083] Specifically, the test neural network model is trained based on the training dataset. The medical images in the test dataset D a are input into the test neural network model to obtain output first prediction results corresponding to each original medical image in the test dataset D a and second prediction results corresponding to each adversarial medical image in the test dataset D athe second prediction result corresponding to each pair of adversarial medical images in the second set of adversarial medical images, and determining a current performance score of the test neural network model based on the first prediction result, the first standard result corresponding to the first prediction result, the second prediction result, and the second standard result corresponding to the second prediction result.

[0084] wherein, for example, the performance parameter corresponding to the current performance score includes, but is not limited to, a structural similarity (SSIM), a success rate (SR), and an AUC loss rate (ALR), and the like.

[0085] wherein, specifically, the SSIM is an index for measuring the similarity between two images, and the mean in the SSIM method is used as an estimate of the image brightness, the standard deviation is used as an estimate of the image contrast, and the variance is used as an estimate of the structural similarity of the image. Specifically, the SSIM satisfies the formula:

[0086]

[0087] wherein, μ x represents the image mean of the original generated image, μ y represents the image mean of the adversarial medical image, σ xy represents the covariance of the original generated image and the adversarial medical image corresponding thereto, represents the image variance of the original generated image, represents the image variance of the adversarial medical image, and c1 and c2 are constants used to maintain stability.

[0088] wherein, specifically, the success ratio (SR) can be used to represent the ratio between the number n of medical images whose prediction is changed from correct to incorrect after being attacked and the number N of medical images whose prediction is correct before being attacked.

[0089] wherein, specifically, the AUC loss ratio (ALR) can be used to describe the influence of the adversarial medical image on the AUC (Area Under Curve) index, and can be used to represent the decline rate of the AUC index. Specifically, the ALR satisfies the formula:

[0090]

[0091] wherein, specifically, AUC1 represents the AUC parameter value determined based on the first prediction result corresponding to the original medical image and the first standard result, and AUC2 represents the AUC parameter value determined based on the second prediction result corresponding to the adversarial medical image and the second standard result.

[0092] The preset performance condition may be, for example, that each of the current performance scores meets a corresponding threshold requirement, or that a preset proportion of the current performance parameters meets a corresponding threshold requirement.

[0093] In this way, the performance of the test neural network model can be evaluated in the case of an adversarial attack, thereby improving the accuracy of the performance evaluation of the test neural network model and improving the robustness of the target neural network model finally trained.

[0094] On the basis of the above embodiment, the original medical images may include training original medical images and test original medical images, and the corresponding adversarial medical images may include training adversarial medical images corresponding to the training original medical images and test adversarial medical images corresponding to the test original medical images. The method further includes: training the test neural network model based on at least one training original medical image and a training adversarial medical image corresponding to each training original medical image to obtain an enhanced neural network model; determining an enhanced performance score corresponding to the enhanced neural network model and a test performance score of the test neural network model based on at least one test original medical image and a test adversarial medical image corresponding to each test original medical image; and outputting the enhanced performance score and the test performance score.

[0095] Specifically, the plurality of training original medical images constitute an original training set T, the plurality of test original medical images constitute an original test set B, and the plurality of test original medical images and the test adversarial medical images corresponding to each test original medical image constitute an adversarial test set B. a .

[0096] In one embodiment, specifically, the test neural network model is trained based on the original training set T to obtain a training-completed test neural network model, and the adversarial training set T is determined based on the adversarial medical images generated by the embodiment of the application and the original training set T. a The adversarial training set T is determined based on the adversarial medical images generated by the embodiment of the application and the original training set T. a The test neural network model is trained based on the adversarial training set T to obtain an enhanced neural network model.

[0097] In one embodiment, specifically, the adversarial training set T is determined based on the adversarial medical images generated by the embodiment of the application and the original training set T. a The adversarial training set T is determined based on the adversarial medical images generated by the embodiment of the application and the original training set T. a In this way, all samples can be used to the maximum extent.

[0098] In another embodiment, specifically, the adversarial training set T is determined based on the adversarial medical images generated by the embodiment of the application and the original training set T. a, including: adding half of the training original medical images in the original training set T to the adversarial training set T a , and adding adversarial medical images corresponding to the other half of the training original medical images in the original training set T to the adversarial training set T a . In this embodiment, the data amount of the adversarial training set T a is the same as that of the original training set T. The advantage of such setting is that the training time for training the test neural network model based on the adversarial training set T a can be saved.

[0099] , and determining the enhancement performance score of the enhanced neural network model and the test performance score of the test neural network model based on the original test set B and the adversarial test set B a . According to the enhancement performance score and the test performance score, the enhancement effect of the test neural network model under the data enhancement operation can be determined.

[0100] Embodiment three

[0101] Figure 4 is a structural schematic diagram of an adversarial medical image generation device provided by Embodiment Three of the present application. As shown in Figure 4 , the device includes a current medical image determination module 310, a current perturbation noise determination module 320, a return execution module 330, and an adversarial medical image determination module 340.

[0102] The current medical image determination module 310 is configured to, in response to obtaining a previous medical image and a previous perturbation noise corresponding to the previous medical image, determine a current medical image based on the previous medical image, the previous perturbation noise, and convolution parameter data.

[0103] The current perturbation noise determination module 320 is configured to determine a current perturbation noise based on the current medical image, a reference neural network model, and the previous perturbation noise.

[0104] The return execution module 330 is configured to take the current medical image as the previous medical image and take the current perturbation noise as the previous perturbation noise, and return to execute an operation of determining a smoothed perturbation noise based on the previous perturbation noise and the convolution parameter data.

[0105] The adversarial medical image determination module 340 is configured to, in a case where the current iteration number meets a preset number threshold, take the current perturbation noise as a target perturbation noise, and determine an adversarial medical image based on the target perturbation noise and an original medical image corresponding to the test neural network model.

[0106] The technical scheme of the embodiment determines the current medical image based on the convolution parameter data, the obtained previous medical image and the previous perturbation noise corresponding to the previous medical image, determines the current perturbation noise based on the current medical image, the reference neural network model and the previous perturbation noise, takes the current medical image as the previous medical image and takes the current perturbation noise as the previous perturbation noise, and returns to perform the operation of determining the current medical image based on the previous medical image, the previous perturbation noise and the convolution parameter data, and in the case that the current iteration number meets the preset number threshold, takes the current perturbation noise as the target perturbation noise, and determines the adversarial medical image based on the target perturbation noise and the original medical image corresponding to the test neural network model. The embodiment of the application iterates from the perspective of the perturbation noise, and smoothes the previous perturbation noise based on the convolution parameter data, solves the problem of large perturbation noise determined based on the existing adversarial attack method, improves the image quality of the generated adversarial medical image, and meets the high-quality requirement of the medical field for the adversarial medical image.

[0107] On the basis of the above-mentioned embodiment, optionally, the convolution parameter data includes at least one preset convolution parameter, and the current medical image determination module 310 includes:

[0108] The smoothing perturbation noise determination unit is configured to perform a convolution operation on the previous perturbation noise based on the at least one preset convolution parameter respectively to obtain at least one corrected perturbation noise, and perform a mean operation on each corrected perturbation noise to obtain a smoothing perturbation noise.

[0109] The current medical image determination unit is configured to determine a current medical image based on the smoothing perturbation noise and the previous medical image.

[0110] On the basis of the above-mentioned embodiment, optionally, the preset convolution parameter includes a convolution kernel size and a convolution kernel weight, and when the number of preset convolution parameters is at least two, the convolution kernel size and / or the convolution kernel weight in each preset convolution parameter are different.

[0111] On the basis of the above-mentioned embodiment, optionally, the device further includes:

[0112] The region of interest mask determination module is configured to input an initial medical image corresponding to the reference neural network model into a pre-trained target image segmentation model to obtain an output region of interest mask.

[0113] Correspondingly, the current medical image determination unit is specifically configured to:

[0114] In a case that the current iteration number is contained in the preset iteration set, a reference attack noise is determined based on the region of interest mask and the smoothed perturbation noise, and the reference attack noise is added to the previous medical image to obtain the current medical image; wherein the preset iteration set contains at least one preset iteration number.

[0115] On the basis of the above-mentioned embodiments, optionally, the current perturbation noise determination module 320 is specifically configured to:

[0116] input the current medical image into the reference neural network model to obtain an output current output result, and determine a current loss gradient corresponding to the reference neural network model based on the current output result and the current standard result;

[0117] determine the current perturbation noise based on the previous perturbation noise and the current loss gradient.

[0118] On the basis of the above-mentioned embodiments, optionally, the device further comprises:

[0119] a target neural network model determination module configured to input the obtained at least one original medical image and the adversarial medical images corresponding to the original medical images respectively into the test neural network model to obtain output first prediction results corresponding to the original medical images and second prediction results corresponding to the adversarial medical images;

[0120] determine a current performance score of the test neural network model based on the first prediction results and the second prediction results;

[0121] in a case that the current performance score does not satisfy a preset performance condition, continue to train the test neural network model until the current performance score satisfies the preset performance condition, and take the test neural network model as the target neural network model.

[0122] On the basis of the above-mentioned embodiments, optionally, the original medical images include training original medical images and test original medical images, and correspondingly, the adversarial medical images include training adversarial medical images corresponding to the training original medical images and test adversarial medical images corresponding to the test original medical images, and the device further comprises:

[0123] an enhanced neural network model training module configured to train the test neural network model based on the at least one training original medical image and the training adversarial medical images corresponding to the training original medical images respectively to obtain an enhanced neural network model;

[0124] determine an enhanced performance score corresponding to the enhanced neural network model and a test performance score of the test neural network model based on the at least one test original medical image and the test adversarial medical images corresponding to the test original medical images respectively;

[0125] The enhanced performance score and the test performance score are output.

[0126] The generation device for the adversarial medical image provided in the embodiments of the present application can execute the generation method for the adversarial medical image provided in any of the embodiments of the present application, and has the function modules and beneficial effects corresponding to the execution method.

[0127] Embodiment four

[0128] Figure 5 A structure schematic diagram of an electronic device provided in Embodiment four of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0129] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0131] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the generation method of an adversarial medical image.

[0132] In some embodiments, the generation method of an adversarial medical image can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded onto and / or installed into the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the generation method of an adversarial medical image described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the generation method of an adversarial medical image by any other suitable means, such as by means of firmware.

[0133] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0134] Computer programs used to implement the generation method of an adversarial medical image of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor produces the functions / operations specified in the flowcharts and / or the block diagrams. The computer program can be entirely executed on a machine, partially executed on a machine, partially executed on a machine as a stand-alone software package and partially executed on a remote machine, or entirely executed on a remote machine or server.

[0135] Embodiment five

[0136] Embodiment five of the present application also provides a computer readable storage medium, which stores computer instructions for causing a processor to execute a generation method of an adversarial medical image, the method comprising:

[0137] In response to obtaining a previous medical image and a previous perturbation noise corresponding to the previous medical image, determining a current medical image based on the previous medical image, the previous perturbation noise and convolution parameter data;

[0138] Determining a current perturbation noise based on the current medical image, a reference neural network model and the previous perturbation noise;

[0139] Taking the current medical image as the previous medical image and taking the current perturbation noise as the previous perturbation noise, and returning to execute the operation of determining the current medical image based on the previous medical image, the previous perturbation noise and the convolution parameter data;

[0140] In a case where the current iteration number meets a preset number threshold, taking the current perturbation noise as a target perturbation noise, and determining the adversarial medical image based on the target perturbation noise and an original medical image corresponding to the test neural network model.

[0141] In the context of the present application, the computer readable storage medium can be a tangible medium, which can contain or store a computer program for use by or in connection with an instruction execution system, apparatus or device. The computer readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. Alternatively, the computer readable storage medium can be a machine readable signal medium. More specific examples of the machine readable storage medium will include one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fibers, compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0142] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0143] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0144] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0145] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0146] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A method for generating adversarial medical images, characterized in that, include: In response to acquiring the previous medical image and the previous perturbation noise corresponding to the previous medical image, the current medical image is determined based on the previous medical image, the previous perturbation noise, and the convolution parameter data; Based on the current medical image, the reference neural network model, and the previous perturbation noise, the current perturbation noise is determined; The current medical image is used as the previous medical image and the current perturbation noise is used as the previous perturbation noise. The operation of determining the current medical image based on the previous medical image, the previous perturbation noise, and the convolution parameter data is returned. If the current iteration count meets the preset threshold, the current perturbation noise is taken as the target perturbation noise, and the adversarial medical image is determined based on the target perturbation noise and the original medical image corresponding to the test neural network model. The convolution parameter data includes at least one preset convolution parameter. Correspondingly, determining the current medical image based on the previous medical image, the previous perturbation noise, and the convolution parameter data includes: Based on at least one preset convolution parameter, convolution operations are performed on the previous perturbation noise to obtain at least one corrected perturbation noise, and the mean operation is performed on each of the corrected perturbation noises to obtain smooth perturbation noise; wherein, the convolution operation integrates the surrounding information of the previous perturbation noise; The current medical image is determined based on the smoothed perturbation noise and the previous medical image; The preset convolution parameters include kernel size and kernel weight. When there are at least two preset convolution parameters, the kernel size and / or kernel weight of each preset convolution parameter are different.

2. The method according to claim 1, characterized in that, The method further includes: inputting an initial medical image corresponding to the reference neural network model into a pre-trained target image segmentation model to obtain an output region of interest mask; Accordingly, determining the current medical image based on the smoothed perturbation noise and the previous medical image includes: If the current iteration number is included in a preset iteration set, a reference attack noise is determined based on the region of interest mask and the smoothing perturbation noise, and the reference attack noise is added to the previous medical image to obtain the current medical image; wherein, the preset iteration set contains at least one preset iteration number.

3. The method according to claim 1, characterized in that, The step of determining the current perturbation noise based on the current medical image, the reference neural network model, and the previous perturbation noise includes: The current medical image is input into the reference neural network model to obtain the current output result. Based on the current output result and the current standard result, the current loss gradient corresponding to the reference neural network model is determined. The current perturbation noise is determined based on the previous perturbation noise and the current loss gradient.

4. The method according to claim 1, characterized in that, The method further includes: The acquired original medical image and the adversarial medical image corresponding to each of the original medical images are input into the test neural network model to obtain the output first prediction result corresponding to each of the original medical images and the second prediction result corresponding to each of the adversarial medical images. Based on the first prediction result and the second prediction result, the current performance score of the test neural network model is determined; If the current performance score does not meet the preset performance conditions, the test neural network model continues to be trained until the current performance score meets the preset performance conditions, at which point the test neural network model is used as the target neural network model.

5. The method according to claim 1, characterized in that, The original medical images include training original medical images and test original medical images. Correspondingly, the adversarial medical images include training adversarial medical images corresponding to the training original medical images and test adversarial medical images corresponding to the test original medical images. The method further includes: The test neural network model is trained based on at least one training original medical image and training adversarial medical images corresponding to each of the training original medical images to obtain an enhanced neural network model. Based on at least one test original medical image and test adversarial medical images corresponding to each of the test original medical images, the enhancement performance score corresponding to the enhancement neural network model and the test performance score of the test neural network model are determined. The enhanced performance score and the test performance score are output.

6. A device for generating medical images, characterized in that, include: The current medical image determination module is used to determine the current medical image in response to acquiring the previous medical image and the previous perturbation noise corresponding to the previous medical image, based on the previous medical image, the previous perturbation noise and convolution parameter data; The current disturbance noise determination module is used to determine the current disturbance noise based on the current medical image, the reference neural network model, and the previous disturbance noise; The execution module is returned to use the current medical image as the previous medical image and the current perturbation noise as the previous perturbation noise, and then returns to perform the operation of determining the smoothing perturbation noise based on the previous perturbation noise and convolution parameter data. The adversarial medical image determination module is used to determine the adversarial medical image based on the target perturbation noise and the original medical image corresponding to the test neural network model, when the current iteration number meets a preset threshold number. The convolution parameter data includes at least one preset convolution parameter, and the current medical image determination module includes: A smooth perturbation noise determination unit is configured to perform convolution operations on the previous perturbation noise based on at least one preset convolution parameter to obtain at least one corrected perturbation noise, and perform an averaging operation on each of the corrected perturbation noises to obtain smooth perturbation noise; wherein the convolution operation integrates the surrounding information of the previous perturbation noise. The current medical image determination unit is used to determine the current medical image based on the smoothed perturbation noise and the previous medical image; The preset convolution parameters include kernel size and kernel weight. When there are at least two preset convolution parameters, the kernel size and / or kernel weight of each preset convolution parameter are different.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the method for generating anti-medical images according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for generating adversarial medical images as described in any one of claims 1-5.

Images