Method and system for anonymizing image

Abstract

Provided are an image anonymization method and system capable of achieving high image analysis performance, such as classification, while anonymizing an image. An anonymization model 2 and an analysis model 3 are connected to simultaneously train the anonymization model 2 and the analysis model 3. In the simultaneous training, an analysis error of the analysis model 3 is obtained using a loss function, and the analysis error is propagated back to the analysis model and the anonymization model to correct the weight of the analysis model and correct the weight of the anonymization model. The anonymized image is output from the anonymization model by using the simultaneously trained anonymization model and analysis model.

Description

Image anonymization method and system

[0001] The present invention relates to a method and system for anonymizing an image.

[0002] One concern with sending and receiving information over the Internet is the risk of the image itself being leaked. Therefore, efforts have been made to prevent information leakage by sending anonymized images instead of transmitting the images themselves over the Internet. For example, Japanese Patent Publication No. 2021-530792 (Patent Document 1) discloses a conventional anonymization technique entitled "Method and System for Generating Synthetically Anonymized Data for a Predetermined Task." The invention described in Patent Document 1 creates synthetically anonymized data by mixing a first sample guaranteed to be distant from the data to be anonymized with a second sample guaranteed to occur near task features. This conventional technique uses an adversarial learning mixture model for unsupervised data clustering.

[0003] Special Publication No. 2021-530792

[0004] The anonymization method of the technology described in Patent Document 1, for example, when the given task is a lesion classification task, anonymizes image data regardless of the classifier, so it is unclear whether classification accuracy can be guaranteed and there is no guarantee that it can be used in combination with a classifier. As a result, it is predicted that the lesion classification performance of the AI ​​model will be significantly reduced. Therefore, this conventional technology is unsuitable for use in image diagnosis support systems, etc., which are required to achieve high classification performance while anonymizing images.

[0005] An object of the present invention is to provide an image anonymization method and system that can improve image analysis performance such as classification while anonymizing images.

[0006] Another object of the present invention is to provide an image diagnosis support system using an image anonymization method and system that can anonymize images using an anonymization model and an analytical model while improving the analytical performance of the analytical model.

[0007] The image anonymization method of the present invention connects an anonymization model that compresses input image data, creates latent variables that retain only important features, and then restores the original dimensions to an analytical model that analyzes the output of the anonymization model, and calculates the analysis error of the analytical model using a loss function. The analytical error is then back-propagated to the analytical model and the anonymization model, and the weights of the analytical model and the anonymization model are modified. An anonymized image is output from the anonymization model using the simultaneously trained anonymization model and analytical model. According to the present invention, the output of the analytical model can be an output specialized for analysis (anonymized image) rather than an image similar to the original image. Furthermore, the performance of the simultaneously trained analytical model is comparable in accuracy rate to that of an anonymization model and an analytical model trained separately, providing an image anonymization method and system.

[0008] When a variational autoencoder that uses a probability distribution for the latent variable is used as the anonymization model, the weights of the variational autoencoder are corrected by backpropagating the analysis error of the analytical model, and the variational autoencoder is trained so as to aggregate only the information necessary for the analysis task by the analytical model as the latent variable of the variational autoencoder. In this way, an anonymization model can be constructed using a variational autoencoder without evaluating the state of anonymization.

[0009] The image diagnosis support method for supporting image diagnosis using diagnostic images anonymized using the image anonymization method of the present invention has the advantage of not reducing classification accuracy while protecting patient privacy.

[0010] The image anonymization system of the present invention, which implements the method of the present invention, includes an anonymization model that compresses input image data, creates latent variables that leave only important features, and then restores the original dimensions, and an analytical model that analyzes the output of the anonymization model.The anonymization model and analytical model are simultaneously trained by calculating the analytical error of the analytical model using a loss function, back-propagating the analytical error to the analytical model and the anonymization model, and correcting the weights of the analytical model and the anonymization model.

[0011] FIG. 1 is a diagram illustrating the configuration of an example of an embodiment of an image anonymization system of the present invention that implements the image anonymization method of the present invention. FIG. 1 is a diagram used to explain learning in the embodiment of FIG. 1. FIG. 2 is a diagram illustrating an overview of the configuration when a variational autoencoder is used as an anonymization model. FIG. 3 is a diagram used to explain learning of a classification model as an analysis model. FIG. 4 is a diagram used to explain learning of a variational autoencoder used as an anonymization model. FIG. 5 is a diagram illustrating a general configuration using a variational autoencoder and a classification model. FIG. 6 is a diagram illustrating the results of calculating and comparing accuracy rates in 10-class classification using MNIST, F-MNIST, and CIFAR-10 as test data. FIG. 7 is a diagram showing experimental images of MNIST, F-MNIST, and CIFAR-10 in the order of original image, followed by images after simultaneous learning using the variational autoencoder (after anonymization). FIG. 8 is a diagram illustrating experimental results of a method without image anonymization and a method with image anonymization of the present embodiment, using medical images. FIG. 9 is a diagram illustrating experimental results of a method without image anonymization and a method with image anonymization of the present embodiment, for medical image anonymization. FIG. 10 is a diagram illustrating ROC curves without image anonymization and with image anonymization of the present embodiment, for medical image anonymization. FIG. 10 is a diagram showing an outline of another embodiment in which the present invention is applied to a diagnostic support service that determines whether a bladder endoscope image is abnormal or normal via a network, for example.

[0012] A preferred embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a diagram showing the configuration of an embodiment of an image anonymization system of the present invention that implements the image anonymization method of the present invention, and FIG. 2 is a diagram used to explain simultaneous learning in the embodiment of FIG. 1. Note that FIG. 2 shows an example of binary classification of medical images based on the presence or absence of a lesion. The image anonymization system 1 of the embodiment shown in FIG. 2 uses a variational auto-encoder (VAE) as the anonymization model 2 and a classification model, ResNet-18, as the analysis model 3. ResNet-18 is disclosed in detail in "HE, Kaiming, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. pp. 770-778."

[0013] In this embodiment, an anonymization system 1 is configured in which an anonymization model 2 and an analysis model 3 are connected. In this embodiment, as shown in FIG. 2 , the anonymization model 2 and the analysis model 3 are connected and simultaneously trained. In simultaneous training, the error of the analysis model 3 is calculated using a loss function, and the analysis error is back-propagated to the analysis model 3 and the anonymization model 2, and the weights of the analysis model 3 and the anonymization model 2 are corrected for simultaneous training. By correcting the weights of the anonymization model 2 and the analysis model 3 through simultaneous training, the output of the anonymization model 2 is not an image similar to the original image, but an output specialized for analysis. As a result, in this embodiment, high analysis performance is achieved while anonymizing the image.

[0014] In this embodiment, the outline of the configuration when a variational autoencoder is used as the anonymization model 2 is as shown in FIG. 3 . An input image x is compressed by an encoder 2A, and a latent variable Z is created using the mean μ and variance σ of the encoder 2A's output. The decoder 2B restores the image based on the latent variable Z and outputs it as x'. The variational autoencoder learns, and the encoder 2A aggregates the information to be restored and output by the decoder 2B into the latent variable Z defined by the center μ and σ. In this embodiment, the weights are corrected (learned) so that the variational autoencoder outputs an image that is easy for the analysis model 3 to analyze. As a result, the information in the original image is aggregated to only the information necessary for analysis. As a result, information unnecessary for restoring an image that is easy to analyze is filtered out from the information in the original image by the encoder 2A, and is not transmitted to the decoder 2B as the latent variable Z. It is common to use the loss function of a variational autoencoder as the reconstruction error (error between input and output) and the latent error (error due to a normal distribution), but in this embodiment, the analysis error is calculated using the loss function, and the analysis error is back-propagated to the analysis model 3 and the anonymization model 2, and the weights of the analysis model as the analysis model 3 and the weights of the variational autoencoder as the anonymization model are corrected. That is, in this embodiment, the mechanism of the variational autoencoder is effectively used to extract only the information necessary for the analysis (target task) from the medical image as latent variables, thereby anonymizing image information that can be understood by humans from the original image.

[0015] When the task of analysis model 3 is classification, the error between output values ​​y1 and y2 and target values ​​t1 and t2 is calculated using a loss function, as shown in FIG. 4. Specifically, the output value of the fully connected layer of the deep learning model that constitutes analysis model 3 is compared with the target value (category in the case of classification), the error is calculated using a loss function, and the weights of analysis model 3 are reversely corrected (by backpropagation) to reduce this error. Furthermore, the training of anonymization model 2 using a variational autoencoder is performed by reversely correcting the weights of anonymization model 2 (variational autoencoder) using the error transmitted from analysis model 3 (classification model), as shown in FIG. 5. That is, in the case of image classification, training is performed so that anonymization model 2 outputs images that reduce the error between the output value of analysis model 3 and the target value.

[0016] As a comparative example, Figure 6 shows a general configuration using a variational autoencoder that does not simultaneously train and a classification model as an analytical model. In the general configuration shown in Figure 6, the variational autoencoder trains using the error between the output image and the original image to correct the weights, and the classification model trains based on the error between the classification result and the label to correct the weights. That is, when training for the purpose of anonymization using the configuration shown in Figure 6, the error indicating whether the output image of the variational autoencoder has been anonymized is calculated, and the variational autoencoder is trained to reduce this error. The classification model is then trained to reduce the classification error. However, even with this training, the classification model used as the analytical model receives the output image of the variational autoencoder trained to achieve the purpose of anonymization, so there is a high possibility that information necessary for classification has been filtered out. This makes it impossible to simultaneously achieve anonymization and high-precision classification, as in this embodiment. Note that the error loss functions typically used in binary classification are squared error and cross-entropy error.

[0017] (Verification Experiment) The results of an experiment conducted to verify the effects of the above embodiment will be described.

[0018] (Experimental Overview) In order to demonstrate the effectiveness of the above embodiment, we compared the anonymization state and classification accuracy when training was performed using only an analysis model consisting of a classification model (no image anonymization) and when training was performed simultaneously using an anonymized model and an analysis model (the method of the present invention).

[0019] (Experimental Data and Training Conditions) In this experiment, we used 60,000 images from the 10-class MNIST dataset (handwritten digit dataset), 60,000 images from the F-MNIST dataset (clothing dataset), 50,000 images from the CIFAR-10 dataset (natural image dataset), and 7,050 images from two classes of cystoscopy images with diagnostic labels (6,095 normal, 985 abnormal) provided by the University of Tsukuba Hospital. A training dataset with a training / validation ratio of 8:2 was randomly selected. The model training conditions were: optimization algorithm: RAdam, learning rate: 0.001 (10 epochs, 0.1x for 40 epochs), maximum number of epochs: 50. Because there is a large bias in the number of normal and abnormal cystoscopy images, the cross-entropy (loss function) was weighted by the ratio of the number of images when training the cystoscopy images.

[0020] (Evaluation Method) To verify the effectiveness of the above-described embodiment, 10,000 images of MNIST, F-MNIST, and CIFAR-10 were used as test data to calculate and compare the accuracy rates (proportion of correctly classified images) in 10-class classification ( FIG. 7 ). 422 cystoscope images (335 normal images, 87 abnormal images) were used to calculate the accuracy rate, sensitivity (proportion of images that were actually abnormal correctly classified as abnormal), specificity (proportion of images that were actually normal correctly classified as normal), precision (proportion of images that were actually classified as abnormal), and F-measure (harmonic mean of sensitivity and precision) in a two-class classification experiment of cystoscope images. AUC (Area Under the Curve), a performance evaluation index, was calculated from the ROC (Receiver Operating Characteristics) curves shown in FIGS. 9 and 11 and compared.

[0021] (Verification Experiment Results) Figure 7 shows the experimental results for MNIST, F-MNIST, and CIFAR-10. As can be seen from Figure 7, the accuracy rate for MNIST decreased by approximately 0.36%, the accuracy rate for F-MNIST decreased by approximately 1.54%, and the accuracy rate for CIFAR-10 decreased by approximately 16.59%. The accuracy rates for MNIST and F-MNIST were almost unchanged, and 10-class classification was achieved. The significant deterioration in accuracy for CIFAR-10 is likely due to the fact that, unlike the other two datasets, it contains natural images and is therefore more difficult to classify. Figure 8 shows the experimental images for MNIST, F-MNIST, and CIFAR-10, displayed in the order of original image and output image (anonymized image) after simultaneous training using a variational autoencoder. As can be seen from Figure 8, it is believed that the original image cannot be inferred from the anonymized output image, confirming that the images are anonymized using this embodiment. Since the anonymized images contain classification information, classification can be performed with a practical accuracy rate (probability) from the anonymized images.

[0022] Figure 9 also shows experimental results for lesion classification in medical images, comparing a method without image anonymization and a method with image anonymization according to this embodiment. Comparing the method without image anonymization with the method with image anonymization (the method according to this embodiment), the accuracy rate was approximately 1.42%, sensitivity was approximately 14.94%, F-value was approximately 0.0759, and AUC was approximately 6.82% worse, while specificity was approximately 2.09% and accuracy was approximately 3.78%, exceeding 80%. Although sensitivity and AUC significantly deteriorated, other evaluation indices showed little decline, and specificity and accuracy improved. Figure 10 shows the anonymized state of medical images. As can be seen from Figure 10, it is believed that the state of the original medical image cannot be observed or estimated from the anonymized image, confirming that sufficient anonymization was achieved even for medical images. Figure 11 shows the ROC curves for the method without image anonymization and the method with image anonymization according to this embodiment. The ROC curve is an index showing the relationship between sensitivity and specificity, and in the case of a classification task, the overall evaluation of the method can be evaluated by the AUC, which is the area under the curve. With this embodiment, even though the classification is based on images whose original image state cannot be estimated, it is clear that the AUC value can be kept to a 7% decrease, maintaining a certain level of classification accuracy.

[0023] The above experimental results confirmed that image anonymization was possible while maintaining classification performance at a practical level, although it decreased somewhat depending on the task, and that the effectiveness of this embodiment was confirmed.

[0024] (Other) FIG. 12 shows an overview of another embodiment of the present invention, in which the present invention is applied to a diagnostic support service that, for example, determines whether a cystoscope image is abnormal (has a lesion) or normal (has no lesion) via a network (Internet). When data is transmitted via a network (Internet), anonymization models E (variational autoencoder encoder 2A) and F (variational autoencoder decoder 2B), which are obtained by dividing anonymization model 2, are placed on the transmitting communication device and the receiving server, respectively, and analysis model 3 (classification model) is placed on the receiving side. In this manner, the amount of data transmitted can be minimized by transmitting latent variables between anonymization models E and F. Note that anonymization model E used here has the function of converting latent variables into transmittable data using variational autoencoder encoder 2A, and anonymization model F has the function of restoring the transmitted latent variables using variational autoencoder decoder 2B.

[0025] Furthermore, since the analytical performance of an anonymization system depends heavily on the performance of the analytical model, it is believed that, depending on the analytical model, better results than the classification results of this verification experiment can be obtained, and the system may be applicable to more complex analytical tasks, etc. Therefore, the present invention is not limited to the use of the image classification model used in the embodiment, and can naturally also be used when an anomaly detection model or a region segmentation model is used as the analytical model.

[0026] According to the present invention, it is possible to provide an image anonymization method and system that can improve the analysis performance of an analyzer while anonymizing an image.

[0027] 1. Anonymization system 2. Anonymization model 3. Analysis model

Claims

1. A method for anonymizing an image, comprising: connecting an anonymization model that compresses input image data, creates latent variables that leave only important features, and then restores the data to the original dimensions; and connecting an analysis model that analyzes the output of the anonymization model; determining an analysis error of the analysis model using a loss function; back-propagating the analysis error to the analysis model and the anonymization model; and outputting an anonymized image from the anonymization model using the anonymization model and the analysis model that have been simultaneously trained by modifying the weights of the analysis model and the anonymization model.

2. The image anonymization method described in claim 1, wherein the anonymization model is a variational autoencoder that uses a probability distribution for the latent variables, and the backpropagation trains the variational autoencoder to aggregate only information necessary for the analysis task by the analysis model as the latent variables.

3. The method of claim 1, wherein the analysis model is a classification model.

4. A method for supporting image diagnosis, which supports image diagnosis using diagnostic images anonymized using the image anonymization method according to claim 3.

5. An image anonymization system comprising: an anonymization model that compresses input image data, creates latent variables that leave only important features, and then restores the data to the original dimensions; and an analysis model that analyzes the output of the anonymization model, wherein the anonymization model and the analysis model are simultaneously trained by determining an analysis error of the analysis model using a loss function, back-propagating the analysis error to the analysis model and the anonymization model, and correcting the weights of the analysis model and the anonymization model.

6. The image anonymization system of claim 5, wherein the anonymization model is a variational autoencoder that uses a probability distribution for the latent variables, and is trained by the backpropagation to output only information necessary for the classification task by the classification model using the variational autoencoder as the latent variables.

7. The image anonymization system of claim 5, wherein the analysis model is a classification model.

8. An image diagnosis support system that supports image diagnosis using diagnostic images that have been anonymized using the image anonymization system according to claim 5 or 6.

Images