An Unsupervised Domain Adaptation System and Method Based on Generative Adversarial Networks

Abstract

The present invention discloses an unsupervised domain adaptation system, method and storage medium based on a generative adversarial network. The system includes: a feature extraction module for extracting high-dimensional feature maps of a first sample and a second sample, where the first sample and the second sample are slice images with different staining styles; a dimensionality reduction module for reducing the dimensionality of the high-dimensional feature maps of the first sample and the second sample and converting them into low-dimensional feature maps; a discriminator for receiving the low-dimensional feature maps of the first sample and the second sample and making the low-dimensional feature maps output by the dimensionality reduction module consistent through adversarial training; a dimensionality increase module for increasing the dimensionality of the low-dimensional feature maps of the first sample and the second sample to regenerate the high-dimensional feature maps of the first sample and the second sample; and a classification module for receiving the regenerated high-dimensional feature maps of the first sample and the second sample and outputting an image analysis result. The present invention has good analysis ability for slice images with different staining styles.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical cytopathological image analysis, and more specifically, relates to an unsupervised domain adaptation system, method and storage medium based on a generative adversarial network. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly, and the combination of artificial intelligence and medicine can alleviate the problem of shortage of doctor resources. In the field of medical cytopathology, the large accumulation of pathological section data provides a big data background for the analysis of medical cytopathological images. In the processing of big data samples, since the analysis and processing capabilities of deep learning algorithms are generally stronger than other traditional analysis algorithms, deep learning is widely used in the field of big data medical cytopathological image analysis.

[0003] Using deep learning for the analysis of medical cytopathological images requires training a deep model with classification, recognition or segmentation effects through learning a large amount of labeled data. However, in reality, due to differences in imaging instruments, instrument parameters, pathological section staining methods, etc., the staining styles of the resulting pathological sections vary greatly. For example, there are differences in image attributes such as hue, saturation, and brightness. The staining style differences will cause some problems for the model, such as: a deep model trained under a single staining style is difficult to have the same or similar performance in data with another staining style; when using pathological sections with different staining styles as data to train a deep model, it will cause the model to be difficult to converge.

[0004] The solution of retraining a deep model by separately labeling data for pathological sections with a new staining style is not feasible due to the high cost. Therefore, the staining style differences of medical cytopathological images require that the deep model needs to have good generalization ability and be able to have good adaptability to data with different staining styles. Existing methods improve the generalization ability of the model by data augmentation to expand the training data and adding noise to the data, etc. However, the applicable range of the models trained by these methods is always limited, and it cannot be guaranteed that the model can perform well on data with any staining style. There are also some methods that match the distributions between data with different staining styles by analyzing color and spatial information or by learning a mapping function through a deep model. The normalization of these methods can only narrow the differences between different staining styles to a certain extent and cannot truly achieve consistent staining styles.

[0005] In summary, there are differences in the staining styles of medical cytopathological images from different sources, and the deep model needs to have good generalization ability in practical applications. Although methods such as data augmentation and normalization can improve the generalization ability of the model to a certain extent, they still cannot meet the requirements of the deep model for data with different staining styles. Summary of the Invention

[0006] In view of at least one defect or improvement requirement of the prior art, the present invention provides an unsupervised domain adaptation system, method and storage medium based on a generative adversarial network, which has good analysis capabilities for samples with different data distribution characteristics.

[0007] To achieve the above object, according to the first aspect of the present invention, there is provided an unsupervised domain adaptation system based on a generative adversarial network, including:

[0008] A feature extraction module, configured to extract high-dimensional feature maps of a first sample and a second sample, where the first sample is a first stained-style slice image and the second sample is a second stained-style slice image;

[0009] A dimensionality reduction module, configured to receive the high-dimensional feature maps of the first sample and the second sample and convert them into low-dimensional feature maps of the first sample and the second sample through dimensionality reduction;

[0010] A discriminator, configured to receive the low-dimensional feature maps of the first sample and the second sample and make the low-dimensional feature maps of the first sample and the second sample output by the dimensionality reduction module have consistency through adversarial training;

[0011] A dimensionality increase module, configured to receive the low-dimensional feature maps of the first sample and the second sample and regenerate the high-dimensional feature maps of the first sample and the second sample through dimensionality increase;

[0012] A classification module, configured to receive the regenerated high-dimensional feature maps of the first sample and the second sample and output an image analysis result.

[0013] Preferably, the low-dimensional features of the first sample and the second sample are both three-dimensional features, and the three-dimensional features are features with physical meanings.

[0014] Preferably, the dimensionality reduction module uses convolutional kernel filtering for dimensionality reduction, and the dimensionality increase module uses convolutional kernel filtering for dimensionality increase.

[0015] Preferably, the feature extraction module is a feature extraction model stacked with residual modules as basic modules, and the residual module is a basic component constructed by residual connection of a convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0016] Preferably, the discriminator uses a classification network stacked with convolutional modules as basic modules, and the convolutional module is a basic component constructed by a convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0017] Preferably, the feature extraction module and the dimensionality reduction module serve as a generator in generative adversarial training, and the generator and the discriminator use a generative adversarial loss to update network parameters during training.

[0018] According to the second aspect of the present invention, an unsupervised domain adaptation method based on a generative adversarial network is provided, including the steps of:

[0019] Extract the high-dimensional feature maps of the first sample and the second sample, where the first sample is a first stained-style slice image, and the second sample is a second stained-style slice image;

[0020] Reduce the dimensions of the high-dimensional feature maps of the first sample and the second sample to convert them into low-dimensional feature maps of the first sample and the second sample;

[0021] Receive the low-dimensional feature maps of the first sample and the second sample, and through adversarial training, make the low-dimensional feature maps of the first sample and the second sample output by the dimensionality reduction have consistency;

[0022] Increase the dimensions of the low-dimensional feature maps of the first sample and the second sample to regenerate the high-dimensional feature maps of the first sample and the second sample;

[0023] Output an image analysis result according to the regenerated high-dimensional feature maps of the first sample and the second sample.

[0024] According to the third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned any method is implemented.

[0025] Generally speaking, compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The deep learning model proposed by the present invention has good analysis capabilities for samples with different data distribution characteristics. In particular, it has an adaptive ability for cervical cell pathological sections from different sources, effectively solving the problem that a deep learning model trained under a single staining style is difficult to have the same or similar performance in data of another staining style: through the supervision of the discriminator, the generator can output a low-dimensional feature structure diagram with the same distribution as the input of pathological images with different staining styles as the feature map for analysis in the second half of the deep learning model, so that the deep learning model has the same or similar analysis capabilities for data of different staining styles.

[0027] (2) The present invention effectively solves the problem that it is difficult for the model to converge when using pathological sections with different staining styles as data to train a deep learning model. Based on the generative adversarial network, through the supervision of the discriminator, the generator learns a mapping function that maps pathological images with different staining styles to a structural feature map with the same structural distribution. When training based on the same structural distribution, the deep model will not have the problem of difficult convergence.

[0028] (3) The present invention embeds the dimension - raising module and the dimension - lowering module inside the deep - learning model, enabling the structured feature maps output by the generator to have actual pathological meanings. This structural design allows the convolutional kernel parameters of the dimension - lowering and dimension - raising modules to be learned according to the supervision information provided by the input image labels during the training process of the deep - learning model. Therefore, the feature maps output by the generator not only need to ensure the same distribution for inputs presenting different staining styles but also need to be suitable for pathological image analysis. That is to say, the structured feature maps generated by the generator have actual pathological meanings.

[0029] (4) The present invention performs dimension - raising and dimension - lowering by means of convolutional kernel filtering to extract high - dimensional or low - dimensional feature maps, and can be placed at different positions in the deep - learning model according to the different importance degrees of structural information and semantic information in different image analysis tasks. When the dimension - lowering and dimension - raising module is placed relatively early, the generated low - dimensional feature maps focus more on structural information; when placed relatively late, the generated low - dimensional feature maps focus more on semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of an unsupervised domain adaptation system based on a generative adversarial network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] An unsupervised domain adaptation system based on a generative adversarial network according to an embodiment of the present invention, as Figure 1 shown, the unsupervised domain adaptation system is essentially a deep - learning model based on a generative adversarial network. The first sample set and the second sample set are input into the deep - learning model based on the generative adversarial network for training. By utilizing the characteristics of adversarial training of the generative adversarial network, the features extracted by the deep - learning model from slices with different staining styles are made consistent, thereby reducing the difference in the analysis capabilities of the deep - learning model for slices from different sources, that is, strengthening the adaptive ability of the deep - learning model for slices from different sources.

[0033] The deep - learning model includes a feature extraction module, a dimension - lowering module, a discriminator, a dimension - raising module, and a classification module.

[0034] A feature extraction module for extracting high-dimensional feature maps of a first sample and a second sample, where the first sample is a first stained-style slice image and the second sample is a second stained-style slice image. The feature extraction module is a feature extraction model formed by stacking residual modules as basic modules, and the residual module is a basic component constructed by residual connection of a convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0035] A dimensionality reduction module for receiving the high-dimensional feature maps of the first sample and the second sample and transforming them into low-dimensional feature maps of the first sample and the second sample through dimensionality reduction. The dimensionality reduction module can use convolutional kernel filtering for dimensionality reduction.

[0036] A discriminator for receiving the low-dimensional feature maps of the first sample and the second sample and making the low-dimensional features of the first sample and the second sample output by the dimensionality reduction module consistent through adversarial training. The feature extraction module and the dimensionality reduction module constitute the generator in the generative adversarial training, and the generator and the discriminator use the generative adversarial loss to update the network parameters during training. Through the adversarial training of the generator and the discriminator, the generator can finally extract consistent low-dimensional feature maps for sample sets with different staining styles. The discriminator adopts a classification network formed by stacking convolutional modules as basic modules, and the convolutional module is a basic component constructed by a convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0037] An upsampling module for receiving the low-dimensional feature maps of the first sample and the second sample and regenerating the high-dimensional feature maps of the first sample and the second sample through upsampling. In this way, the feature maps output by the generator have actual pathological meanings and can further improve the analysis ability of the deep learning model. The upsampling module can use convolutional kernel filtering for upsampling.

[0038] A classification module for receiving the regenerated high-dimensional feature maps of the first sample and the second sample and outputting an image analysis result.

[0039] In the embodiment of the present invention, the feature extraction module and the classification module constitute a reference network, and the reference network can be a model trained according to a first sample set and has good analysis and processing capabilities for the first sample set. A dimensionality reduction module and an upsampling module are added in the middle of the reference network, that is, the dimensionality reduction module and the upsampling module are connected to a certain intermediate convolutional layer of the reference network. The reference network before the connection position of the dimensionality reduction module and the upsampling module is the feature extraction module, and the reference network after the connection position of the dimensionality reduction module and the upsampling module is the classification module.

[0040] The dimensionality reduction module and the feature extraction module are used as generators of low-dimensional feature maps. The generators and discriminators update network parameters using generative adversarial loss during training. Through the adversarial training of the generators and discriminators, the sensitivity of the baseline network to staining style differences can be effectively reduced, that is, forcing the generators composed of the first half to output similar or identical low-dimensional feature maps for sliced images of different staining styles, so that the high-dimensional feature maps reconstructed by the second half have consistency. Through this consistency, the image analysis ability of the deep model for sliced images of the second staining style is improved.

[0041] The dimensionality reduction module converts the high-dimensional feature maps extracted from the middle part of the deep model into low-dimensional feature maps through multiple convolutional layer operations. And the supervision information of the learnable parameters in this dimensionality reduction module comes from the gradient information backpropagated from the task labels, that is, it has actual pathological meanings. The learnable parameters are the parameters of the convolutional kernels, the coefficients in the normalization layer, etc. Similarly, the learnable parameters in the dimensionality increase module are also affected by the task labels. The advantage of incorporating the dimensionality reduction and increase modules into the deep learning model rather than being independent of the deep learning model is that the low-dimensional feature maps converted by the dimensionality reduction module and the high-dimensional information restored by the dimensionality increase module both have actual pathological meanings.

[0042] Preferably, the low-dimensional features of the first sample and the second sample output by the dimensionality reduction module are both three-dimensional features. The three-dimensional features are features with physical meanings and include nuclear features, cytoplasmic features, and background image features. These three features are also the most important semantic information for cervical cell pathological image analysis.

[0043] Preferably, the connection position between the dimensionality reduction module and the baseline network can be changed, that is, the dimensionality reduction module can be connected to convolutional layers at different positions of the baseline network based on different image analysis tasks. According to the different importance degrees of structural information and semantic information in different image analysis tasks, it is connected to different positions of the baseline network. When the dimensionality reduction module and the dimensionality increase module are placed relatively forward, the generated low-dimensional feature maps focus more on structural information; when the dimensionality reduction module and the dimensionality increase module are placed relatively backward, the generated low-dimensional feature maps focus more on semantic information. Specifically, when the dimensionality reduction module is connected to a shallower convolutional layer network of the baseline network, the pathological features extracted by the dimensionality reduction module focus on low-level semantic information, such as texture, contour, etc.; when the dimensionality reduction module is connected to a deeper convolutional layer network of the baseline network, the proposed pathological features focus on pathological meanings, such as whether the nucleus is abnormal, whether it is deeply stained, etc.

[0044] An unsupervised domain adaptation method based on a generative adversarial network according to an embodiment of the present invention includes the steps:

[0045] (1) Extract high-dimensional feature maps of the first sample and the second sample respectively, where the first sample is a sliced image of the first staining style, and the second sample is a sliced image of the second staining style.

[0046] (2) The high-dimensional feature maps of the first sample and the second sample are respectively converted into low-dimensional feature maps by dimensionality reduction.

[0047] (3) receiving low-dimensional feature maps of the first sample and the second sample, and making the low-dimensional feature maps of the first sample and the second sample output by dimensionality reduction consistent through adversarial training, thereby effectively reducing the sensitivity to the difference in dyeing styles.

[0048] (4) The low-dimensional feature maps of the first sample and the second sample are regenerated by dimensionality increase to generate high-dimensional feature maps of the first sample and the second sample and input into the benchmark network. Since the low-dimensional feature maps after dimensionality reduction are consistent, the high-dimensional feature maps regenerated by dimensionality increase are also consistent, thereby improving the image analysis ability of the deep model for the second dyeing style slice

[0049] (5) Output the image analysis result according to the regenerated high-dimensional feature map of the first sample and the second sample.

[0050] The unsupervised domain adaptation method based on generative adversarial networks has the same principle as the unsupervised domain adaptation system based on generative adversarial networks mentioned above. The following uses two pathological sections with very different staining styles A and B as examples to illustrate the details.

[0051] Suppose there are two slices with very different staining styles, A and B, where style A has a large number of image analysis task labels, such as lesion type labels, and style B has no or only a small number of labels. Based on the large number of task labels on staining style A, a baseline network is trained that can have excellent analysis performance on staining style A. In one experiment, the average accuracy of the baseline network for positive and negative image blocks in the interpretation of cervical cell lesions reached 95%+. Because deep models trained under a single staining style are difficult to have the same or similar performance in images presenting another staining style. In one experiment, the baseline network's image analysis results for staining style B were poor, with an average accuracy of only 65%+ for positive and negative images.

[0052] The unsupervised domain adaptation method based on a generative adversarial network for pathological sections with staining style A and pathological sections with staining style B includes the following steps:

[0053] (1) On the first sample set of staining style A with a large number of task tags, a benchmark network that can have excellent pathological image analysis performance on staining style A is trained according to the existing tags, called the task network (abbreviated as T). For pathological images with the same staining style, the task network can often output low-dimensional feature maps with the same or similar numerical distributions, such as the same or similar peaks, means, and histogram distributions; while for pathological images with different staining styles, the low-dimensional feature maps output by the task network without adversarial training always have large differences in numerical distributions. That is, due to the large staining differences between staining styles A and B visually, this difference is reflected in the numerical distribution in digital pathological images. The convolutional neural network for feature extraction based on numerical distribution is very sensitive to this numerical difference, specifically manifested as the poor image analysis ability of the task network on staining style B.

[0054] Taking image classification as an example, the specific image analysis objective function is as follows:

[0055]

[0056]

[0057] Among them, I A is the cervical cell pathological image presenting staining style A, Y A is the task label of image I A , T(I A ) is the actual output result of the task network for image I A , is the mathematical expectation calculation, arg min T L Task is to find the benchmark network that makes the value of L Task the smallest, and T * is the optimal benchmark network.

[0058] However, due to the large staining differences between staining styles A and B visually, this difference is reflected in the numerical distribution in digital pathological images. The convolutional neural network for feature extraction based on numerical distribution is very sensitive to this numerical difference, specifically manifested as the poor image analysis ability of the task network on staining style B.

[0059] (2) Input the cervical cell pathological image I A presenting staining style A into the task network trained in step (1), and the low-dimensional feature map i A and the image analysis result T(I A ) can be obtained. Input the cervical cell pathological image I B presenting staining style BInput the task network, which can obtain the low-dimensional feature map i B and the image analysis result T(I B ). For pathological images with the same staining style, the task network can often output low-dimensional feature maps with the same or similar numerical distributions (such as the same or similar peaks, means, and histogram distributions); while for pathological images with different staining styles, the low-dimensional feature maps output by the task network without adversarial training always have large differences in numerical distributions.

[0060] The objective function of the training process is as follows:

[0061]

[0062]

[0063]

[0064] Among them, T G (I) is the low-dimensional feature map transformed by the feature extraction module and the dimensionality reduction module, and D(T G (I)) is the discrimination result output by the discriminator for the low-dimensional feature map. is to obtain the reference network T and the dimensionality increase and reduction modules that minimize the value of L, and T G * is the optimal generator.

[0065] (3) Step (2) effectively improves the image analysis performance of the deep model on the staining style B by the sensitivity of the dimensionality reduction model to the difference between the unlabeled staining style B and the actual training data of the deep model. In addition, for the staining style B with a small amount of labels, the method proposed in the present invention can also utilize its small amount of labels, combine transfer learning with the method proposed in step (2), and effectively improve the performance of the deep model on B.

[0066] For the staining style B with a small amount of labeled data, the objective function of the deep model is as follows:

[0067]

[0068]

[0069] Among them, I B is the cervical cell pathological image presenting the staining style B, Y B is the task label of the image I B and T(I B ) is the task network for the image I BFor the actual output result, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement any of the above-mentioned unsupervised domain adaptation methods based on a generative adversarial network. The implementation principle and technical effect are similar to those of the above method, and will not be elaborated here.

[0070] It must be noted that in any of the above embodiments, the methods do not necessarily need to be executed in the order of the serial numbers. As long as it cannot be inferred from the execution logic that they must be executed in a certain order, it means that they can be executed in any other possible order.

[0071] Those skilled in the art can easily understand that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An unsupervised domain adaptation system based on a generative adversarial network, characterized in that, Comprising: A feature extraction module, configured to extract high-dimensional feature maps of a first sample and a second sample, wherein the first sample is a first stained-style slice image, and the second sample is a second stained-style slice image; A dimensionality reduction module, configured to receive the high-dimensional feature maps of the first sample and the second sample, and convert them into low-dimensional feature maps of the first sample and the second sample through dimensionality reduction; A discriminator, configured to receive the low-dimensional feature maps of the first sample and the second sample, and through adversarial training, make the low-dimensional feature maps of the first sample and the second sample output by the dimensionality reduction module have consistency; A dimensionality increase module, configured to receive the low-dimensional feature maps of the first sample and the second sample with consistency output by the discriminator, and regenerate the high-dimensional feature maps of the first sample and the second sample through dimensionality increase; A classification module, configured to receive the regenerated high-dimensional feature maps of the first sample and the second sample and output an image analysis result; the low-dimensional features of the first sample and the second sample are both three-dimensional features, and the three-dimensional features are features with physical meanings; The feature extraction module and the dimensionality reduction module serve as a generator in generative adversarial training, and the generator and the discriminator use a generative adversarial loss to update network parameters during training.

2. The unsupervised domain adaptation system based on a generative adversarial network according to claim 1, characterized in that, The dimensionality reduction module uses convolutional kernel filtering for dimensionality reduction, and the dimensionality increase module uses convolutional kernel filtering for dimensionality increase.

3. The unsupervised domain adaptation system based on a generative adversarial network according to claim 1, characterized in that, The feature extraction module is a feature extraction model formed by stacking residual modules as basic modules, and the residual module is a basic component constructed by residual connection of a convolutional layer, a batch normalization layer, and a ReLU activation layer.

4. The unsupervised domain adaptation system based on a generative adversarial network according to claim 1, characterized in that, The discriminator uses a classification network formed by stacking convolutional modules as basic modules, and the convolutional module is a basic component constructed by a convolutional layer, a batch normalization layer, and a ReLU activation layer.

5. An unsupervised domain adaptation method based on a generative adversarial network, characterized in that, Including steps: Extracting high-dimensional feature maps of a first sample and a second sample, wherein the first sample is a first stained-style slice image, and the second sample is a second stained-style slice image; Converting the high-dimensional feature maps of the first sample and the second sample into low-dimensional feature maps of the first sample and the second sample through dimensionality reduction; Receiving the low-dimensional feature maps of the first sample and the second sample, and through adversarial training, making the low-dimensional feature maps of the first sample and the second sample output by dimensionality reduction have consistency; a generative adversarial loss is used to update network parameters during the training; Regenerating the high-dimensional feature maps of the first sample and the second sample through dimensionality increase for the low-dimensional feature maps of the first sample and the second sample with consistency; Outputting an image analysis result according to the regenerated high-dimensional feature maps of the first sample and the second sample; the low-dimensional features of the first sample and the second sample are both three-dimensional features, and the three-dimensional features are features with physical meanings.

6. The unsupervised domain adaptation method based on a generative adversarial network according to claim 5, characterized in that, Using convolutional kernel filtering for dimensionality reduction and using convolutional kernel filtering for dimensionality increase.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 5 to 6.

Images