Passive domain medical image deep learning model progressive adaptive method
By constructing a domain-adaptive framework for multi-scale image reconstruction and style feature alignment, the problems of domain offset and pseudo-label noise in deep learning models in medical image analysis are solved, achieving improved business performance and enhanced task versatility in both the target and source domains.
Patent Information
- Application Number
- CN202410818553.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-08
AI Technical Summary
Existing deep learning models suffer from domain shift issues in medical image analysis, making it difficult to adapt to changes in the target domain. Furthermore, domain adaptation algorithms in the absence of a source domain suffer from problems such as pseudo-label noise, loss of image semantic information, multi-step error propagation, and poor task versatility. Transfer learning, on the other hand, is prone to forgetting the distribution of source domain data.
We employ a progressive adaptive approach for a passive domain medical image deep learning model. By constructing an adaptive framework for both the first and second domains, and utilizing a multi-scale image reconstruction module, a style feature alignment module, and a continuous learning module, combined with unsupervised and supervised datasets, we perform end-to-end training to generate pseudo-labels and correct them, ensuring the conversion and preservation of image style and semantic information.
The system systematically solves the business performance problem of deep learning models in the target and source domains under passive domain conditions, improves the adaptability and stability of the model under different tasks, reduces error propagation, and enhances the generality of tasks and applicability to clinical applications.
Smart Images

Figure CN118918410A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and more specifically, to a progressive adaptive method for a passive domain medical image deep learning model. Background Art
[0002] Medical images have become an important reference for clinicians to make diagnoses and treatments, but the acquisition and generation of medical images is a complex process, including image acquisition, storage, processing and other links. The stability and consistency of these links directly affect the visual effect of the final medical image. When the image generation process changes, the medical image will also change accordingly. Although changes in the appearance of the image may affect the doctor's interpretation, relying on rich clinical experience, doctors can still adapt to these changes, so as to correctly understand the image and make an accurate diagnosis.
[0003] In recent years, deep learning algorithms have been increasingly used in medical image analysis, but the limitations of their generalization ability have gradually emerged. The core problem is that deep learning models have too strong a fit to the training data, but poor adaptability to changes in data distribution. Specifically, the dataset used to train the model can be regarded as the source domain, and the new dataset that the model needs to make predictions on is the target domain. There may be differences in the visual representation of the two, that is, two different distributions, which is the so-called domain shift phenomenon. When domain shift occurs, the model trained on the source domain dataset is difficult to adapt to new changes in the image in the target domain, and its performance will be significantly reduced. This fragility of over-reliance on the distribution of training data limits the generalization ability of deep learning models and makes it difficult to promote and apply them in clinical practice.
[0004] Due to the inability to share image data between multiple medical institutions due to privacy protection, as well as the high labor cost of accurately annotating medical images, there are many difficulties in building a large-scale medical image dataset across medical centers to train deep learning models for clinical applications. In addition, existing algorithms based on source domain adaptation need to obtain source domain datasets to adapt to the target domain, but different medical institutions cannot share their central data for privacy reasons, resulting in the loss of source domain data support for such algorithms, making them difficult to apply in practice. Considering the above limitations, the scenarios for deploying the model to a new medical center can be simplified into two situations: "Scenario 1" is when the target domain only has unannotated medical images; "Scenario 2" refers to when the target domain contains annotated medical images.
[0005] The domain adaptation algorithm under sourceless domain data is proposed to solve the "scenario 1" problem. However, the existing technical solutions to "scenario 1" still have some problems. On the technical level: the solution based on pseudo-label self-training is highly dependent on the quality of the generated pseudo-labels. Since the pseudo-labels are generated by the model itself, it is inevitable to introduce certain noise, which may mislead the model training. The technical solution based on image translation lacks constraints on the semantic content of the image during the image translation process, so it is difficult to completely keep the semantic content unchanged, and some subtle semantic information is easily lost during the conversion process. The solution of constructing a super target domain and aligning features relies on obtaining high-quality super target domain images. When the distribution range of the super target domain images can more completely cover the distribution of the source domain images, better performance will be achieved. In addition, most of the existing technical solutions only focus on the adaptability of the algorithm to the target domain images, ignoring the performance of the source domain data. And some algorithms have multiple steps. The disadvantage of this method is that if a large error occurs in one of the steps, it will affect the effectiveness of the overall algorithm, and the steps cannot be independent. In terms of practical application, most existing technical solutions are aimed at one image analysis task (image classification, image segmentation or target detection, etc.), and are poor in task versatility. In addition, existing inventions only focus on the adaptation of target domain images in unsupervised scenarios, but the performance of the model after unsupervised adaptation is still far behind the performance of the model trained under supervised learning, which is still unacceptable in real clinical scenarios.
[0006] Transfer learning is a common way to solve the problem in "Scenario 2". Transfer learning refers to fine-tuning the model of the source domain data with the labeled data of the target domain, so that the model can better adapt to the performance of the target domain data. However, this solution still has certain disadvantages, that is, after the fine-tuned model adapts to the new target domain central data, it will "forget" the distribution of the source domain data, resulting in reduced performance of the model on the source domain data and reduced generalization. Summary of the invention
[0007] One of the purposes of the present invention is to provide a method for progressively adapting a deep learning model for medical images in a passive domain, so as to improve the service performance of the deep learning model on the target domain image under "scenario one"; One of the purposes of the present invention is to provide a method for progressively adapting a deep learning model for medical images in a passive domain, so as to improve the service performance of the deep learning model on the target domain image under "scenario two"
[0008] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0009] A first aspect of the present invention provides a method for progressively adapting a deep learning model of a passive domain medical image. When the collected target domain image is unlabeled, the method comprises the following steps:
[0010] Obtaining the first deep learning model that has been applied to medical image analysis tasks for source domain images;
[0011] According to the first deep learning model, a first domain adaptation framework and a second domain adaptation framework are respectively constructed, wherein the first domain adaptation framework and the second domain adaptation framework respectively convert the target domain image style into the source domain image style, and then adjust the parameters of the first deep learning module;
[0012] Use unlabeled target domain images to build an unsupervised target domain image dataset;
[0013] Collecting a public dataset of the same type as the unlabeled target domain image, and merging the public dataset with the unsupervised target domain image dataset into an enhanced dataset;
[0014] Using the unsupervised target domain image dataset and the enhanced dataset to train the first domain adaptation framework, obtain a trained first domain adaptation framework, and obtain a first image generator and a second deep learning model;
[0015] Processing the unlabeled target domain image using a first image generator and a second deep learning model to obtain a target domain image with a pseudo label;
[0016] Using the target domain images with pseudo labels, and checking and correcting the pseudo labels, to construct a supervised target domain image dataset;
[0017] Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model;
[0018] After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
[0019] Furthermore, the first domain adaptation framework includes a first multi-scale image reconstruction module, the first multi-scale image reconstruction module is used to transform the target domain image style into the source domain image style, the first multi-scale image reconstruction module includes a first image generator and a second image generator, wherein:
[0020] The input of the first image generator is a target domain image, and the output of the first image generator is a target domain image having the style of the source domain image;
[0021] The input of the second image generator is the target domain image having the style of the source domain image, and the output of the second image generator is the reconstructed target domain image having the style of the target domain image.
[0022] Furthermore, the first domain adaptation block further includes a first style feature alignment module, and the first style feature alignment module is based on the first deep learning model, wherein:
[0023] Obtaining source domain image style information from the Batch Normalization layer in the first deep learning model;
[0024] Inputting the output image of the first image generator into the first deep learning model to obtain image style feature information of the output image of the first image generator;
[0025] An image style loss is calculated according to the source domain image style information and the image style feature information, and the image style loss is used to constrain the first image generator.
[0026] Further, using the unsupervised target domain image dataset and the enhanced dataset to train the first domain adaptation framework includes using the unsupervised target domain image dataset to train the first image generator and the second image generator, including:
[0027] Whole image reconstruction: The target domain image Input to the image first image generator G t→n In the image first image generator G t→n The output is And Do the following processing to get the image
[0028]
[0029] Will Input to the first image generator G t→n A second image generator G of the same type n→r In the example, the second image generator G n→r The output is an image Calculate the reconstruction loss L rebuild_f For images Constrain the image The visual appearance of the target domain image The difference between is less than the preset threshold, and the reconstruction loss L rebuild_f is the L1 loss function, calculated as:
[0030]
[0031] Patch image reconstruction: The target domain image is randomly cropped to obtain a patch image. Input to the first image generator G t→n After getting the image Then the image Input to the second image generator G n→r Get the image Using reconstruction loss L rebuild_p For images Constraints are calculated as follows:
[0032]
[0033] Loss function L for multi-scale image reconstruction rebuild for:
[0034] L rebuild =L rebuild_f +L rebuild_p
[0035] Image Input into the first deep learning model to calculate the image style loss L wassertauin :
[0036]
[0037] In the formula, and Respectively represent the image After input into the first deep learning model, the mean and standard deviation of the feature map before the mth batch normalization layer are calculated; t represents the number of iterations of the current training, t-1 represents the number of iterations of the previous training, and α is used to balance the statistics when the number of iterations is t and The statistic when the number of iterations is t-1 and Hyperparameters between ; and are the mean and standard deviation stored in the mth batch normalization layer in the first deep learning model trained on source domain data; ||·||2 represents the L2 loss function;
[0038] Using the loss function L rebuild and L wassertauin The first image generator and the second image generator are trained.
[0039] Furthermore, training the first image generator and the second image generator also includes calculating an improved total variation loss function Loss tv ,as follows:
[0040] Loss tv =Loss channel +Loss spatial
[0041] In the formula, Loss spatial is the loss function in image space, Loss channel is the loss function of the image channel, where:
[0042]
[0043] In the formula, h and w represent the coordinates of the image on the y-axis and x-axis respectively, sum represents the summation function, and || represents the absolute value. Indicated in the image The matrix area taken from 0 to h-1 on the y axis and from 0 to w on the x axis; Indicated in the image The matrix area taken from 1 to h on the y axis and 0 to w on the x axis; Indicated in the image The matrix area taken from 0 to h on the y axis and from 0 to w-1 on the x axis; Indicated in the image The matrix area taken from 0 to h-1 on the y axis and 1 to w on the x axis;
[0044]
[0045]
[0046] In the formula, r, g, and b represent the RGB channels of the image respectively.
[0047] Furthermore, the first domain adaptation framework further includes a first continuous learning module and a second continuous learning module, wherein:
[0048] The first continuous learning module includes a first deep learning model and a second deep learning model, the second deep learning model is initialized with parameter weights of the first deep learning model, and the weights are updated during training;
[0049] In the first continuous learning module, the input of the second deep learning model is an image The input of the first deep learning model is the target domain image Calculate the fitting loss according to the outputs of the second deep learning model and the first deep learning model, and constrain the outputs of the second deep learning model to be consistent with the outputs of the first deep learning model;
[0050] The second continuous learning module includes a second deep learning model and a first deep learning model, wherein the first deep learning model keeps parameter weights unchanged during training;
[0051] In the second continuous learning module, the inputs of the second deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the second deep learning model and the first deep learning model, so as to constrain the outputs of the second deep learning model and the first deep learning model to remain consistent.
[0052] Furthermore, the fitting loss is any fitting function.
[0053] Furthermore, the step of processing the unlabeled target domain image using the first image generator and the second deep learning model to obtain a target domain image with a pseudo label includes:
[0054] The first image generator is used to convert the target domain image into a target domain image with the characteristics of the source domain image, and the second deep learning model is used to process the target domain image with the characteristics of the source domain image to obtain a target domain image with a pseudo label.
[0055] Furthermore, the second domain adaptation framework includes a second multi-scale image reconstruction module, a second style feature alignment module, a target domain image supervision learning module and a third continuous learning module, wherein:
[0056] The second multi-scale image reconstruction module includes a third generator and a fourth generator, the second multi-scale image reconstruction module has the same structure as the first multi-scale image reconstruction module, and the second style feature alignment module has the same structure as the first style feature alignment module;
[0057] The third continuous learning module includes a third deep learning model and a first deep learning model, wherein the parameter weights of the first deep learning model remain unchanged during training;
[0058] In the third continuous learning module, the inputs of the third deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the third deep learning model and the first deep learning model, so as to constrain the outputs of the third deep learning model and the first deep learning model to be consistent;
[0059] The target domain image supervised learning module includes a third deep learning model, which is initialized with the parameter weights of the first deep learning model and updates the weights during training. After inputting into the third deep learning model, the output of the third deep learning model is obtained, and the output of the third deep learning model is constrained Keep the image annotations consistent with those in the supervised target domain image dataset.
[0060] A second aspect of the present invention provides a method for progressive adaptation of a passive domain medical image deep learning model, characterized in that when the collected target domain image is annotated, the following steps are performed:
[0061] Using the annotated target domain images, construct a supervised target domain image dataset;
[0062] Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model;
[0063] After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
[0064] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0065] (1) Compared with the prior art which can only solve the problem in a single scenario of "Scenario 1" or "Scenario 2", the solution proposed by the present invention systematically solves the problems in both scenarios of "Scenario 1" and "Scenario 2";
[0066] (2) Compared with the previous focus on improving the business performance of the deep learning model in the target domain, the method proposed in the present invention can not only improve the business performance of the deep learning model on the target domain image, but also maintain the business performance of the model on the source domain image;
[0067] (3) The framework proposed in this paper adopts an end-to-end training strategy. This pipeline approach reduces error propagation and optimizes the global learning goal more focused and effectively, ultimately producing excellent results in a more resource-efficient manner.
[0068] (4) Compared with previous inventions that can only be applied to a certain computer vision task such as image segmentation and image classification, the framework proposed by the present invention can be combined with any computer vision task (image classification, image segmentation, object detection), so the framework has stronger task versatility;
[0069] (5) In view of the problem that the performance of the current unsupervised domain adaptation algorithm is still insufficient to meet clinical needs after adapting to the new center, the present invention proposes a deployment solution that gradually optimizes from unsupervised to semi-supervised. First, the unsupervised algorithm is used to adapt the new center image and generate labels to reduce the data labeling workload of clinicians. The labeled data is then used to train the semi-supervised algorithm to further improve the model performance to meet clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1A flowchart of a progressive adaptive method for a passive domain medical image deep learning model under “Scenario 1” provided in an embodiment of the present invention.
[0071] Figure 2 A schematic diagram of the structure of a first domain adaptation framework provided by an embodiment of the present invention;
[0072] Figure 3 A schematic diagram of the structure of a second domain adaptation framework provided by an embodiment of the present invention;
[0073] Figure 4 A flowchart of a progressive adaptive method for a passive domain medical image deep learning model under “Scenario 2” provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0074] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0075] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;
[0076] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0077] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0078] Example 1
[0079] This embodiment provides a method for progressively adapting a deep learning model for passive domain medical images. Figure 1 As shown in Figure 1, when the collected target domain images are unlabeled, the following steps are included:
[0080] Obtaining the first deep learning model that has been applied to medical image analysis tasks for source domain images;
[0081] According to the first deep learning model, a first domain adaptation framework (USL-CLMS framework) and a second domain adaptation framework (SSL-CLMS framework) are respectively constructed, wherein the first domain adaptation framework and the second domain adaptation framework respectively convert the target domain image style into the source domain image style, and then adjust the parameters of the first deep learning module;
[0082] Use unlabeled target domain images to build an unsupervised target domain image dataset;
[0083] Collecting a public dataset of the same type as the unlabeled target domain image, and merging the public dataset with the unsupervised target domain image dataset into an enhanced dataset;
[0084] Using the unsupervised target domain image dataset and the enhanced dataset to train the first domain adaptation framework, obtain a trained first domain adaptation framework, and obtain a first image generator and a second deep learning model;
[0085] Processing the unlabeled target domain image using a first image generator and a second deep learning model to obtain a target domain image with a pseudo label;
[0086] Using the target domain images with pseudo labels, and checking and correcting the pseudo labels, to construct a supervised target domain image dataset;
[0087] Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model;
[0088] After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
[0089] In a further embodiment, Figure 2 As shown, the first domain adaptation frame includes a first multi-scale image reconstruction module, which is used to convert the target domain image style into the source domain image style. The main purpose of this module is to ensure that the image visual appearance can be converted and that the image does not lose key information during the conversion process. The first multi-scale image reconstruction module includes a first image generator and a second image generator, wherein:
[0090] The input of the first image generator is a target domain image, and the output of the first image generator is a target domain image having the style of the source domain image;
[0091] The input of the second image generator is the target domain image having the style of the source domain image, and the output of the second image generator is the reconstructed target domain image having the style of the target domain image.
[0092] In this embodiment, the reconstruction of the image includes two parts: the reconstruction of the whole image and the reconstruction of the patch image. Among them, the patch image is obtained by randomly cropping the whole image. The reconstruction of the whole image is to ensure that the global information of the image is not lost during the style conversion process, while the reconstruction of the patch image is to ensure that the local information of the image is not lost during the conversion process.
[0093] In a further embodiment, the first domain adaptation block further includes a first style feature alignment module, and the first style feature alignment module is based on the first deep learning model, wherein:
[0094] Obtaining source domain image style information from the Batch Normalization layer in the first deep learning model;
[0095] Inputting the output image of the first image generator into the first deep learning model to obtain image style feature information of the output image of the first image generator;
[0096] An image style loss is calculated according to the source domain image style information and the source domain image style information, and the image style loss is used to constrain the first image generator.
[0097] In a further embodiment, using the unsupervised target domain image dataset and the enhanced dataset to train the first domain adaptation framework includes using the unsupervised target domain image dataset to train the first image generator and the second image generator, including:
[0098] Whole-image reconstruction: In order to change the visual appearance of the target domain image, the target domain image is Input to an encoder-decoder type image generator G t→n In the target domain image Input to the image first image generator G t→n In the image first image generator G t→n The output is In order to reduce the difficulty of image conversion and ensure that the generated image has the same semantic information as the target domain image, Do the following processing to get the image
[0099]
[0100] To improve the generated graph The quality will Input to the first image generator G t→n A second image generator G of the same type n→r In the example, the second image generator G n→r The output is an image Calculate the reconstruction loss L rebuild_f For images Constrain the image The visual appearance of the target domain image is less than the preset threshold, the reconstruction loss L rebuild_f is the L1 loss function, calculated as:
[0101]
[0102] Patch image reconstruction: The target domain image is randomly cropped to obtain a patch image. Input to the first image generator G t→n After getting the image Then the image Input to the second image generator G n→r Get the image Using reconstruction loss L rebuild_p For images Constraints are calculated as follows:
[0103]
[0104] Loss function L for multi-scale image reconstruction rebuild for:
[0105] L rebuild =L rebuild_f +L rebuild_p
[0106] In order to achieve the conversion of the target domain image style to the source domain image style, some papers have proved that the Batch Normalization layer in the trained deep learning model stores running average statistics: the running mean and variance of the batch of images, which implicitly captures the feature distribution of the training data. In addition, shallow features contain appearance information, while deep features focus on semantic information. Based on this, this embodiment uses the first P shallow Batch Normalization statistics stored in the deep learning model trained with source domain data as a representative of the source domain image style information. Then calculate the loss L wassertauin , used to optimize the generated image to follow the source domain data distribution, Input into the first deep learning model to calculate the image style loss L wassertauin :
[0107]
[0108] In the formula, and Respectively represent the image After input into the first deep learning model, the mean and standard deviation of the feature map before the mth batch normalization layer are calculated; t represents the number of iterations of the current training, t-1 represents the number of iterations of the previous training, and α is used to balance the statistics when the number of iterations is t and The statistic when the number of iterations is t-1 and Hyperparameters between ; and are the mean and standard deviation stored in the mth batch normalization layer in the first deep learning model trained with source domain data; ||·||2 represents the L2 loss function; L wassertauin The summation is performed on the first H layers that are most likely to capture low-level style information;
[0109] Using the loss function L rebuild and L wassertauin The first image generator and the second image generator are trained.
[0110] In a further embodiment, in order to further constrain the quality of the generated image, based on the existing total variation loss function (TVLoss), the TVLoss is improved to constrain the generated image The quality is as follows:
[0111] Loss tv =Loss channel +Loss spatial
[0112] In the formula, Loss spatial is the loss function in image space, Loss channel is the loss function of the image channel, where:
[0113]
[0114]
[0115] In the formula, h and w represent the coordinates of the image on the y-axis and x-axis respectively, sum represents the summation function, and || represents the absolute value. Indicated in the image The matrix area taken from 0 to h-1 on the y axis and from 0 to w on the x axis; Indicated in the image The matrix area taken from 1 to h on the y axis and 0 to w on the x axis; Indicated in the image The matrix area taken from 0 to h on the y axis and from 0 to w-1 on the x axis; Indicated in the image The matrix area taken from 0 to h-1 on the y axis and 1 to w on the x axis;
[0116]
[0117] In the formula, r, g, and b represent the RGB channels of the image respectively.
[0118] In a further embodiment, in order to ensure that the image does not lose semantic information critical to the downstream task during the conversion, the downstream task model is used in the art to constrain the image conversion. The specific method is:
[0119] The first domain adaptation framework further includes a first continuous learning module and a second continuous learning module, wherein:
[0120] The first continuous learning module includes a first deep learning model and a second deep learning model, the second deep learning model is initialized with parameter weights of the first deep learning model, and the weights are updated during training;
[0121] In the first continuous learning module, the input of the second deep learning model is an image The input of the first deep learning model is the target domain image Calculate the fitting loss according to the outputs of the second deep learning model and the first deep learning model, and constrain the outputs of the second deep learning model to be consistent with the outputs of the first deep learning model;
[0122] The second continuous learning module includes a second deep learning model and a first deep learning model, wherein the first deep learning model keeps parameter weights unchanged during training;
[0123] In the second continuous learning module, the inputs of the second deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the second deep learning model and the first deep learning model, so as to constrain the outputs of the second deep learning model and the first deep learning model to remain consistent.
[0124] In a further embodiment, the fitting loss is any fitting function.
[0125] like Figure 2 As shown in the figure, based on the deep learning model of the medical image analysis task extracted from the analysis of the medical image analysis task, the USL-CLMS framework in the unsupervised scenario is constructed. The two core ideas contained in this framework are: first, while ensuring the quality of image conversion, the target domain image style is converted into a standard domain morphological style, and the converted style is closer to the source domain image style; second, the response of the downstream task model to the image is maintained, thereby ensuring that the image does not lose the semantic feature information that is key to the downstream task during the conversion process.
[0126] When using this framework to train a deep learning model, we first use multi-scale image reconstruction technology ( Figure 2 A) and style feature alignment technology ( Figure 2B) to convert the target domain image into a source-domain image. Specifically: an Encoder-Decoder type generator is used to transform the visual appearance of the target domain image, a style feature alignment module is used to guide the style features of the converted image to be close to the style of the source domain image, and multi-scale image reconstruction technology and improved TV Loss are used to ensure the quality of the conversion graph. In order to ensure that the semantic feature information related to the image analysis task is not lost during the conversion process, prediction consistency is performed on the target domain image and the reconstructed image. This helps to retain key semantic information related to downstream tasks during the image conversion process ( Figure 2 C). Finally, an augmented dataset is constructed by merging the target domain images with publicly available medical images. By performing prediction consistency on this augmented dataset, the performance of the model on the source domain images is maintained ( Figure 2 D) This comprehensive approach ensures the robustness and adaptability of the model to different medical settings.
[0127] In this embodiment, compared with the self-training method of generating pseudo labels, the framework proposed in the embodiment of the present invention avoids the use of pseudo labels, effectively alleviating the problems caused by potential wrong labels. This not only contributes to the stability of the model, but also ensures that the learning process will not be damaged by inaccurate information introduced by pseudo labels.
[0128] In this embodiment, compared with the solution based on converting the style of the target domain image to the source domain, the semantic information of the image is not preserved in the image conversion stage, so it is easy to pass the erroneous information generated in the conversion to the downstream task stage in the cross-domain image conversion stage. The embodiment of the present invention constrains the image distribution conversion model through downstream analysis tasks, avoids the problem of traditional image style conversion easily losing key information, and ensures that the converted image not only conforms to the expected distribution, but also retains the key information for downstream analysis tasks.
[0129] In a further embodiment, the step of processing the unlabeled target domain image using the first image generator and the second deep learning model to obtain the target domain image with a pseudo label includes:
[0130] The first image generator is used to convert the target domain image into a target domain image with the characteristics of the source domain image, and the second deep learning model is used to process the target domain image with the characteristics of the source domain image to obtain a target domain image with a pseudo label.
[0131] In a specific embodiment, the pseudo-labels are corrected by hiring professional clinicians to check and correct the pseudo-labels.
[0132] In a further embodiment, the second domain adaptation framework is as follows Figure 3As shown, it includes a second multi-scale image reconstruction module, a second style feature alignment module, a target domain image supervision learning module and a third continuous learning module, wherein:
[0133] The second multi-scale image reconstruction module includes a third generator and a fourth generator, the second multi-scale image reconstruction module has the same structure as the first multi-scale image reconstruction module, and the second style feature alignment module has the same structure as the first style feature alignment module;
[0134] The third continuous learning module includes a third deep learning model and a first deep learning model, wherein the parameter weights of the first deep learning model remain unchanged during training;
[0135] In the third continuous learning module, the inputs of the third deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the third deep learning model and the first deep learning model, so as to constrain the outputs of the third deep learning model and the first deep learning model to be consistent;
[0136] The target domain image supervised learning module includes a third deep learning model, which is initialized with the parameter weights of the first deep learning model and updates the weights during training. After inputting into the third deep learning model, the output of the third deep learning model is obtained, and the output of the third deep learning model is constrained Keep the image annotations consistent with those in the supervised target domain image dataset.
[0137] In another embodiment of the present invention, when the collected target domain images are annotated, such as Figure 4 As shown, there are the following steps:
[0138] Using the annotated target domain images, construct a supervised target domain image dataset;
[0139] Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model;
[0140] After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
[0141] In this embodiment, compared with transfer learning, more attention is paid to improving the business performance of the deep learning model in the target domain, but it is difficult to maintain the performance in the source domain; the method proposed in the embodiment of the present invention can not only improve the business performance of the deep learning model on the target domain image, but also maintain the business performance of the model on the source domain image.
[0142] The same or similar reference numerals correspond to the same or similar components;
[0143] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;
[0144] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A progressive adaptive method for deep learning model of passive domain medical images, characterized in that: When the collected target domain images are unlabeled, the following steps are included: Obtaining the first deep learning model that has been applied to medical image analysis tasks for source domain images; According to the first deep learning model, a first domain adaptation framework and a second domain adaptation framework are respectively constructed, wherein the first domain adaptation framework and the second domain adaptation framework respectively convert the target domain image style into the source domain image style, and then adjust the parameters of the first deep learning module; Use unlabeled target domain images to build an unsupervised target domain image dataset; Collecting a public dataset of the same type as the unlabeled target domain image, and merging the public dataset with the unsupervised target domain image dataset into an enhanced dataset; Using the unsupervised target domain image dataset and the enhanced dataset to train the first domain adaptation framework, obtain a trained first domain adaptation framework, and obtain a first image generator and a second deep learning model; Processing the unlabeled target domain image using a first image generator and a second deep learning model to obtain a target domain image with a pseudo label; Using the target domain images with pseudo labels, and checking and correcting the pseudo labels, to construct a supervised target domain image dataset; Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model; After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
2. The method for progressively adapting the deep learning model of passive domain medical images according to claim 1 is characterized in that: The first domain adaptation framework includes a first multi-scale image reconstruction module, the first multi-scale image reconstruction module is used to transform the target domain image style into the source domain image style, the first multi-scale image reconstruction module includes a first image generator and a second image generator, wherein: The input of the first image generator is a target domain image, and the output of the first image generator is a target domain image having the style of the source domain image; The input of the second image generator is the target domain image having the style of the source domain image, and the output of the second image generator is the reconstructed target domain image having the style of the target domain image.
3. The method for progressively adapting the deep learning model of passive domain medical images according to claim 2 is characterized in that: The first domain adaptation block further includes a first style feature alignment module, and the first style feature alignment module is based on the first deep learning model, wherein: Obtaining source domain image style information from the Batch Normalization layer in the first deep learning model; Inputting the output image of the first image generator into the first deep learning model to obtain image style feature information of the output image of the first image generator; An image style loss is calculated according to the source domain image style information and the source domain image style feature information, and the image style loss is used to constrain the first image generator.
4. The method for progressively adapting the passive domain medical image deep learning model according to claim 3 is characterized in that: Training the first domain adaptation framework using the unsupervised target domain image dataset and the augmented dataset includes training the first image generator and the second image generator using the unsupervised target domain image dataset, including: Whole image reconstruction: The target domain image Input to the image first image generator G t→n In the image first image generator G t→n The output is And Do the following processing to get the image Will Input to the first image generator G t→n A second image generator G of the same type n→r In the example, the second image generator G n→r The output is an image Calculate the reconstruction loss L rebuild_f For images Constrain the image The visual appearance of the target domain image The difference between is less than the preset threshold, and the reconstruction loss L rebuild_f is the L1 loss function, calculated as: Patch image reconstruction: The target domain image is randomly cropped to obtain a patch image. Input to the first image generator G t→n After getting the image Then the image Input to the second image generator G n→r Get the image Using reconstruction loss L rebuild_p For images Constraints are calculated as follows: Loss function L for multi-scale image reconstruction rebuild for: L rebuild =L rebuild_f +L rebuild_p Image Input into the first deep learning model to calculate the image style loss L wassertauin : In the formula, and Respectively represent the image After input into the first deep learning model, the mean and standard deviation of the feature map before the mth batch normalization layer are calculated; t represents the number of iterations of the current training, t-1 represents the number of iterations of the previous training, and α is used to balance the statistics when the number of iterations is t and The statistic when the number of iterations is t-1 and Hyperparameters between ; and are the mean and standard deviation stored in the mth batch normalization layer in the first deep learning model trained on source domain data; ||·||2 represents the L2 loss function; Using the loss function L rebuild and L wassertauin The first image generator and the second image generator are trained.
5. The method for progressively adapting the deep learning model of passive domain medical images according to claim 4 is characterized in that: Training the first image generator and the second image generator also includes calculating an improved total variation loss function Loss tv ,as follows: Loss tv =Loss channel +Loss spatial In the formula, Loss spatial is the loss function in the image space, Loss channel is the loss function of the image channel, where: In the formula, h and w represent the coordinates of the image on the y-axis and x-axis respectively, sum represents the summation function, and || represents the absolute value. Indicated in the image The matrix area taken from 0 to h-1 on the y axis and from 0 to w on the x axis; Indicated in the image The matrix area taken from 1 to h on the y axis and 0 to w on the x axis; Indicated in the image The matrix area taken from 0 to h on the y axis and from 0 to w-1 on the x axis; Indicated in the image The matrix area taken from 0 to h-1 on the y axis and 1 to w on the x axis; In the formula, r, g, and b represent the RGB channels of the image respectively.
6. The method for progressively adapting the passive domain medical image deep learning model according to claim 1, characterized in that: The first domain adaptation framework further includes a first continuous learning module and a second continuous learning module, wherein: The first continuous learning module includes a first deep learning model and a second deep learning model, the second deep learning model is initialized with parameter weights of the first deep learning model, and the weights are updated during training; In the first continuous learning module, the input of the second deep learning model is an image The input of the first deep learning model is the target domain image Calculate the fitting loss according to the outputs of the second deep learning model and the first deep learning model, and constrain the outputs of the second deep learning model to be consistent with the outputs of the first deep learning model; The second continuous learning module includes a second deep learning model and a first deep learning model, wherein the first deep learning model keeps parameter weights unchanged during training; In the second continuous learning module, the inputs of the second deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the second deep learning model and the first deep learning model, so as to constrain the outputs of the second deep learning model and the first deep learning model to remain consistent.
7. The method for progressively adapting the passive domain medical image deep learning model according to claim 6, characterized in that: The fitting loss is any fitting function.
8. The passive domain medical image deep learning model progressive adaptive method according to claim 6, characterized in that: The method of processing the unlabeled target domain image by using the first image generator and the second deep learning model to obtain the target domain image with pseudo labels includes: The first image generator is used to convert the target domain image into a target domain image with the characteristics of the source domain image, and the second deep learning model is used to process the target domain image with the characteristics of the source domain image to obtain a target domain image with a pseudo label.
9. The method for progressively adapting the passive domain medical image deep learning model according to claim 8, characterized in that: The second domain adaptation framework includes a second multi-scale image reconstruction module, a second style feature alignment module, a target domain image supervision learning module and a third continuous learning module, wherein: The second multi-scale image reconstruction module includes a third generator and a fourth generator, the second multi-scale image reconstruction module has the same structure as the first multi-scale image reconstruction module, and the second style feature alignment module has the same structure as the first style feature alignment module; The third continuous learning module includes a third deep learning model and a first deep learning model, wherein the parameter weights of the first deep learning model remain unchanged during training; In the third continuous learning module, the inputs of the third deep learning model and the first deep learning model are both images in the enhanced data set, and the fitting loss is calculated according to the outputs of the third deep learning model and the first deep learning model, so as to constrain the outputs of the third deep learning model and the first deep learning model to be consistent; The target domain image supervised learning module includes a third deep learning model, which is initialized with the parameter weights of the first deep learning model and updates the weights during training. After inputting into the third deep learning model, the output of the third deep learning model is obtained, and the output of the third deep learning model is constrained Keep the image annotations consistent with those in the supervised target domain image dataset.
10. A progressive adaptive method for a deep learning model of passive domain medical images, characterized in that: When the collected target domain images are annotated, there are the following steps: Using the annotated target domain images, construct a supervised target domain image dataset; Using the supervised target domain image dataset and the enhanced dataset to train the second domain adaptation framework, obtain a trained second domain adaptation framework, and obtain a third image generator and a third deep learning model; After the style of the target domain image to be analyzed is converted into the style of the source domain image by using the third image generator, the converted target domain image to be analyzed is input into the third deep learning model to obtain an analysis result.
