Medical image automatic analysis method and system based on domain feature alignment transfer learning

Abstract

The application discloses a kind of medical image automatic analysis method and system based on domain feature alignment transfer learning, on target medical image dataset, first proportionally divided into training set, verification set and test set;First, a group of pre-training neural network models is obtained using different pre-training methods, then a generation of fine-tuning training is carried out on the target domain training dataset, the highest pre-training model of transferability is filtered according to the proposed transferability index, and the whole transfer learning step is completed based on the domain feature alignment module;The transferred neural network model is tested using the test set.By the application, the transferability of a group of pre-training model set for target task can be quantitatively evaluated in the transfer learning process;The features in the pre-training and target neural network model can be aligned along the channel during the transfer process, which fully improves the efficiency of transfer learning, and can be widely used in the transfer learning between various medical image datasets.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision technology, in particular to a medical image automatic analysis method and system based on domain feature alignment transfer learning, and also provides a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] Image automatic analysis algorithms based on deep learning require large-scale labeled samples for training, while medical images lack sufficient annotations due to data acquisition and ethical norms. Therefore, learning from a small number of annotations is a long-standing and challenging task for medical image automatic analysis based on deep learning. Researchers have found that the transfer learning process of "pre-training-then fine-tuning" can significantly improve the performance of various medical image automatic analysis tasks, such as organ segmentation, lesion classification, and lesion detection.

[0003] However, in the field of medical images, most current work focuses on improving the representation learning method in the pre-training stage, and few works focus on the transfer learning method in the fine-tuning process. In the field of natural images, a few works have proposed transfer learning methods to improve the fine-tuning process, mainly divided into two categories: the first category is based on regularization, which applies regularization to the target model using the weights or features of the pre-trained model during fine-tuning; the second category is based on policy network, which introduces multiple network modules to generate strategies to filter the transferable convolutional layers or filters layer by layer. However, the first method is only suitable for cases where the source domain and target domain have small differences, and the second method requires a large amount of computational cost. Due to different imaging methods, anatomical regions, and image acquisition factors, the distribution of medical image datasets is very different. In addition, three-dimensional medical tasks already require a lot of computational cost. Obviously, the above two transfer methods are not suitable for medical image tasks.

[0004] In the field of cross-domain transfer, a series of works have found that the batch normalization layer in the convolutional neural network stores domain-related information of the training dataset. Based on this assumption, researchers have focused on converting the image style between the source domain and target domain according to the statistical characteristics of the two domains, thereby improving the efficiency of transfer learning. In the field of cross-domain transfer, the semantic information between the source domain data and the target domain data is consistent, only the image style is different, and the source domain data is available during the transfer process. In the "pre-training-then fine-tuning" transfer learning of medical images, on the one hand, the distance between the source domain and the target domain may be large, and on the other hand, the source domain data is not available during the fine-tuning stage, so the transfer learning method based on statistical characteristics to convert the image style is not applicable in this case.

[0005] After searching, it is found that:

[0006] The Chinese invention patent application with the authorization announcement number CN 113269734B, "tumor image detection method and device based on meta-learning feature fusion strategy", pre-trains multiple network models on an open source dataset, inputs a tumor image into the network model, and extracts a multi-level source domain feature vector; the meta-learning convolution kernel is used to align the scales of the source domain feature vectors and the target domain feature vectors of different scales, and the source domain feature vectors after the scale alignment are fused into a single-level source domain feature map; the meta-learning convolution kernel is used to migrate the multi-scale single-level source domain feature map into the target domain feature vector of the input tumor image according to the structure of the target network; the target domain feature vector fused with the source domain feature vector is input into a detection network, and a cascaded multi-target detection network is used to complete the regression of the candidate frame and the classification of the detection target. The method has the following technical problems:

[0007] When the distribution difference between the source domain dataset and the target domain dataset is too large, directly fusing the source domain features into the target domain features may cause negative transfer and cannot improve the performance of the target task;

[0008] Different channels in the convolutional neural network learn different patterns of features, and whether the channels between the source domain features and the target domain features are aligned is not considered, which hinders the effectiveness of transfer learning;

[0009] The introduction of multi-level source domain features and meta-learning convolution kernels may cause a dramatic increase in model parameters, which requires high computing resources and is not suitable for 3D medical image analysis tasks that already require a large amount of computation. SUMMARY

[0010] The present application provides a medical image (medical CT image) automatic analysis method and system based on domain feature alignment transfer learning, and a corresponding computer terminal and computer readable storage medium.

[0011] According to one aspect of the present application, a medical image automatic analysis method based on domain feature alignment transfer learning is provided, comprising:

[0012] For a given source domain image training set, a source domain convolutional neural network model is built, the source domain convolutional neural network model is pre-trained, and a group of pre-trained models is obtained;

[0013] For a given number of target domain medical CT images, the images are divided into a training set, a validation set and a test set according to a set proportion, and a target domain dataset is generated;

[0014] Based on the backbone network framework of the pre-trained model, a corresponding target domain convolutional neural network model is built according to the target medical automatic analysis task, and a target domain model is obtained;

[0015] For the target domain model, a set of pre-training models are used to initialize the backbone network parameters in turn, and a generation of fine-tuning training is performed on the training set in the target domain dataset to obtain a generation of fine-tuned target domain model; the consistency of the source domain features generated by the pre-training model and the target domain features generated by the generation of fine-tuned target domain model along the channel distribution is calculated to calculate the transferability, and the transferability is used as a measure to screen the pre-training model with the highest transferability, that is, the pre-training model with the best performance for the target medical automatic analysis task;

[0016] The parameters of the best pre-training model are used to initialize the backbone network parameters of the target domain model, and then the training set in the target domain dataset is used to complete multiple generations of fine-tuning training, so as to realize the transfer learning of the best pre-training model on the target domain, thereby obtaining a medical image automatic analysis model;

[0017] The medical image automatic analysis model is used to complete the automatic analysis of medical images on the test set of the target domain dataset.

[0018] Preferably, the target domain convolutional neural network model has a cross-domain feature alignment and an attention mechanism batch normalization layer.

[0019] Preferably, the backbone network framework based on the pre-training model is used to build a corresponding target domain convolutional neural network model according to the target medical automatic analysis task, which comprises:

[0020] The input medical CT image is preprocessed;

[0021] A backbone network framework consistent with the pre-training model is used to build a top-level classifier on the backbone network according to the specific target medical task to obtain a target domain model backbone network.

[0022] In the batch normalization layer of the target domain model backbone network, the similarity of the source domain features and the target domain features along the channel distribution is calculated to generate a channel correlation matrix of the source domain features and the target domain features, and an attention mechanism is introduced for channel calibration, thereby realizing cross-domain feature alignment and obtaining a target domain convolutional neural network model.

[0023] Preferably, the preprocessing of the input medical CT image comprises resampling, HU value transformation, normalization and cropping.

[0024] Preferably, in the batch normalization layer of the target domain model backbone network, the channel correlation matrix of the source domain features and the target domain features is generated, and an attention mechanism is introduced for channel calibration, thereby realizing cross-domain feature alignment, which comprises:

[0025] The input feature of the batch normalization layer of the target domain model backbone network is x∈R B×K×H×W, where K is the number of feature map channels, B is the number of samples selected for one training, H is the image length, and W is the image width; first, calculate the mean value μ∈R 1×K×1×1 and variance σ 2 ∈R 1×K×1×1 of each channel of x, and then normalize x as follows:

[0026]

[0027] where x is the normalized feature, and ∈ is a constant to ensure that the denominator is not zero; Each channel of x obeys the standard normal distribution N(0, 1), and then a set of affine transformation parameters is used to transform the normalized feature x as follows:

[0028]

[0029] where f i is the output feature of the i-th channel of the batch normalization layer, γ i is the scaling affine transformation parameter of the i-th channel, and β i is the offset affine transformation parameter of the i-th channel.

[0030] Let the source domain affine transformation parameters in the pre-trained model be and the target domain affine transformation parameters be The cross-channel similarity C p,q between the source domain affine transformation parameters and the target domain affine transformation parameters is calculated as follows:

[0031]

[0032] where C is used to describe the distribution of the domain feature in each channel, t is a temperature coefficient for adjusting the steepness of the migration importance distribution, s represents the source domain, q indexes the channel corresponding to the source domain feature, t represents the target domain, and p indexes the channel corresponding to the target domain feature; C∈R K×K reflects the cross-channel correlation between the source domain and the target domain, and is used as a measure of the migration of the pre-trained model; in order to prevent negative migration, C p,q is sparsified according to the values of the diagonal line as follows:

[0033]

[0034] where C p,p is the correlation between the channels corresponding to the source domain feature and the target domain feature.

[0035] The cross-channel correlation C p,qAlign the source domain features and the target domain features as attention matrices, i.e. transform the target domain affine transformation parameters as follows:

[0036] γ c = C[γ s i ,...,γ s i ] T ,β c = C[β s i ,...,β s i ] T

[0037] wherein γ c is the calibrated scaling affine transformation parameter, and β c is the calibrated offset affine transformation parameter;

[0038] Finally, the aligned target domain output feature y i is:

[0039]

[0040] Realize cross-domain feature alignment.

[0041] Preferably, for the target domain model, a set of pre-training models are used to initialize the parameters of the backbone network thereof in sequence, and a training set in the target domain dataset is used for one generation of fine-tuning training to obtain one generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated based on the pre-training model and the target domain features generated by the one generation of fine-tuned target domain model is calculated to calculate the transferability, and the transferability is used as a measure to screen one pre-training model with the highest transferability, comprising:

[0042] A set of pre-training models are used to initialize the parameters of the backbone network of the target domain model respectively, and then a training set is used to perform one generation of fine-tuning training on a set of target domain models initialized with different pre-training model parameters to generate corresponding one generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated based on the pre-training model and the target domain features generated by the one generation of fine-tuned target domain model is calculated to calculate the transferability as a measure, and the transferability is used to further screen out a pre-training model with the highest transferability for transfer learning of the current target medical automatic analysis task; wherein:

[0043] In the fine-tuning training of a group of target domain models initialized with different pre-training model parameters, a loss function is constructed by using the artificially labeled true values and the prediction results of the target domain model, the target domain model is trained by using the training set, and the hyperparameters of the target domain model are optimized by using the verification set to generate a corresponding generation of fine-tuned target domain model.

[0044] Preferably, the consistency of the source domain features generated based on the pre-training model and the target domain features generated by the generation of fine-tuned target domain model along the channel distribution is used to calculate the transferability, comprising:

[0045]

[0046] Wherein, M is the transferability measure, l=1,...,L is the first L layers of the pre-training model and the target domain model, that is, the feature extractor, K is the number of feature map channels, q is the index of the source domain feature channel, and p is the index of the target domain feature channel. is the correlation between the features of the qth channel of the source domain and the features of the pth channel of the target domain.

[0047] According to another aspect of the present application, a medical image automatic analysis system based on domain feature alignment transfer learning is provided, characterized by comprising:

[0048] A pre-training model set generation module, which, for a given source domain image training set, builds a source domain convolutional neural network model, pre-trains the source domain convolutional neural network model, and obtains a group of pre-training models;

[0049] A target domain data set construction module, which is used for dividing a plurality of given target domain medical CT images into a target domain training set, a verification set and a test set according to a set proportion, and generating a target domain data set;

[0050] A target domain model establishment module, which, based on the backbone network framework of the pre-training model, builds a corresponding target domain convolutional neural network model according to the target medical automatic analysis task, and obtains a target domain model;

[0051] A pre-training model screening module, which initializes the target domain model backbone network parameters by using a group of pre-training models in turn, and performs a generation of fine-tuning training by using the training set in the target domain data set to obtain a generation of fine-tuned target domain model; the consistency of the source domain features generated based on the pre-training model and the target domain features generated by the generation of fine-tuned target domain model along the channel distribution is used to calculate the transferability, and the transferability is used as a measure to screen the pre-training model with the highest transferability, that is, the pre-training model with the best performance for the target medical automatic analysis task;

[0052] a model training module, which initializes parameters of a target domain model backbone network by using parameters of the best pre-training model obtained through screening, and performs multi-generation fine-tuning training by using a training set in the target domain data set, so as to realize transfer learning of the best pre-training model on the target domain, thereby obtaining the medical image automatic analysis model;

[0053] a medical image automatic analysis module, which performs automatic analysis of medical images on a test set of the target domain data set by using the medical image automatic analysis model.

[0054] According to a third aspect of the present application, a computer terminal is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, is configured to perform the method according to any one of the above embodiments, or to run the system according to the above embodiments.

[0055] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor, wherein the processor, when executing the program, is configured to perform the method according to any one of the above embodiments, or to run the system according to the above embodiments.

[0056] Due to the adoption of the above technical solutions, the present application has at least one of the following beneficial effects compared with the prior art:

[0057] The medical image automatic analysis method and system based on domain feature alignment transfer learning provided by the present application can quantitatively evaluate the transferability of a group of pre-training models for a target task in the transfer learning process.

[0058] The medical image automatic analysis method and system based on domain feature alignment transfer learning provided by the present application can align features along channels in the pre-training model and the target domain convolutional neural network model in the transfer process, thereby fully improving the efficiency of transfer learning and being widely applicable to transfer learning between various medical image data sets with large distribution differences.

[0059] The medical image automatic analysis method and system based on domain feature alignment transfer learning provided by the present application adopts a domain feature alignment batch normalization layer that can be plugged in and used in the fine-tuning process without introducing additional parameters, thereby relieving the demand for computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0060] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0061] Figure 1 A workflow diagram of the medical image automatic analysis method based on domain feature alignment transfer learning according to an embodiment of the present application.

[0062] Figure 2 A flow chart of a medical image automatic analysis method based on domain feature alignment transfer learning in a preferred embodiment of the present application;

[0063] Figure 3 A working schematic diagram of transfer learning in a preferred embodiment of the present application;

[0064] Figure 4 A convolutional neural network structure schematic diagram as a medical image automatic analysis model in a preferred embodiment of the present application;

[0065] Figure 5 A working schematic diagram of cross-domain feature alignment in a preferred embodiment of the present application;

[0066] Figure 6 A liver tumor segmentation result comparison chart of the method of the present application and the prior art method;

[0067] Figure 7 A lung nodule segmentation result comparison chart of the method of the present application and the prior art method;

[0068] Figure 8 A performance chart of the transferability measurement index proposed in a preferred embodiment of the present application on a plurality of pre-training models.

[0069] Figure 9 A component module schematic diagram of a medical image automatic analysis system based on domain feature alignment transfer learning in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application will be described in detail below: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0071] An embodiment of the present application provides a medical image automatic analysis method based on domain feature alignment transfer learning, which can quantitatively evaluate the transferability of a series of pre-training model sets for a target task in the transfer learning process; the features in the pre-training and target neural network models can be aligned along the channel in the transfer process, fully improving the efficiency of transfer learning, and can be widely used in transfer learning between various medical image data sets.

[0072] As shown in Figure 1 The medical image automatic analysis method based on domain feature alignment transfer learning provided by the embodiment can include:

[0073] S1. For a given source domain image training set, build a source domain convolutional neural network model. Pre-train the built source domain convolutional neural network model according to various self-supervised or supervised learning pre-training methods to obtain a set of pre-trained models.

[0074] S2, for a given number of target domain medical CT images, divide the images into target domain training set, validation set and test set according to a set ratio to generate target domain dataset;

[0075] S3, based on the backbone network framework of the pre-trained model, builds the corresponding target domain convolutional neural network model according to the target medical automatic analysis task to obtain the target domain model;

[0076] S4. For the target domain model, the parameters of its backbone network (i.e., the feature extractor in the neural network) are initialized sequentially using a set of pre-trained models. Then, a generation of fine-tuning training is performed using the training set in the target domain dataset to obtain a generation of fine-tuned target domain model. The transferability is calculated based on the consistency of the source domain features generated by the pre-trained model and the target domain features generated by the generation of fine-tuned target domain model along the channel distribution. The transferability is used as a metric to select the pre-trained model with the highest transferability, which is the pre-trained model with the best performance for the target medical automatic analysis task.

[0077] S5. The parameters of the backbone network of the target domain model are initialized using the parameters of the best pre-trained model obtained by screening. Then, the training set in the target domain dataset is used to complete the full fine-tuning training for multiple generations (about 200 generations) to achieve transfer learning of the best pre-trained model in the target domain and obtain the automatic medical image analysis model.

[0078] S6 utilizes an automated medical image analysis model to perform automated analysis of medical images on the test set of the target domain dataset.

[0079] The technical solutions provided by the above embodiments of the present invention will be further described below.

[0080] like Figure 2 As shown, a preferred embodiment of the present invention provides an automatic medical image analysis method based on domain feature alignment transfer learning, the method comprising the following steps:

[0081] S1: Obtain the given source domain image training set;

[0082] Build a source domain convolutional neural network model and generate a set of pre-trained models by adopting various supervised or unsupervised learning methods;

[0083] S2: Construct and partition the dataset;

[0084] For a given number of target domain medical CT images, they are respectively divided into target domain training set, validation set and test set according to a certain proportion, and a target domain data set is generated;

[0085] S3: Based on the backbone network framework of the pre-trained model, a corresponding target domain convolutional neural network model is built according to the target medical automatic analysis task, and a target domain model is obtained;

[0086] The structure diagram of the target domain convolutional neural network model (i.e. the target domain model) established in S3 is shown in Figure 4 , which can be used for a wide range of medical image segmentation tasks. Specifically, the target domain convolutional neural network model presents a U-shaped structure, which is composed of an encoder, a decoder and a skip connection. The skip connection can better integrate features at different levels to achieve accurate segmentation;

[0087] S4: In a set of pre-trained models, the pre-trained model with the highest transferability is selected based on the transferability metric;

[0088] The set of pre-trained models obtained by S1 is used to initialize the encoder part parameters of the target domain model built by S3 in turn, and then the target domain training set obtained by S2 is used for one generation of fine-tuning training. Finally, the transferability is calculated based on the consistency of the cross-domain features along the channel distribution;

[0089] S5: Complete transfer learning on the target data set;

[0090] The pre-trained model parameters selected by S4 are used to initialize the backbone network of the target domain model built by S3, and the target domain training set in S2 is trained to realize the transfer learning of the best pre-trained model on the target domain, and obtain the medical image automatic analysis model; as shown in Figure 3 ;

[0091] S6: Medical image automatic analysis

[0092] Using the obtained medical image automatic analysis model, the automatic analysis of medical images on the test set divided by S2 is completed.

[0093] In a preferred embodiment of S3, the target domain convolutional neural network model established is a neural network model with cross-domain feature alignment and attention mechanism characteristics for the batch normalization layer of the backbone network.

[0094] In a preferred embodiment of S3, based on the source domain convolutional neural network model, a corresponding convolutional neural network model is built according to the target medical automatic analysis task, which includes:

[0095] S31: Preprocess the input medical CT image, and in a preferred embodiment, the preprocessing can include resampling, HU value transformation, normalization and cropping;

[0096] S32: In the batch normalization layer of the target domain model backbone network, the similarity of the source domain feature and the target domain feature along the channel distribution is calculated to generate a source domain feature and a target domain feature generated channel correlation matrix, an attention mechanism is introduced for channel calibration, and then cross-domain feature alignment is realized to obtain a target domain convolutional neural network model;

[0097] In a preferred embodiment of S32, cross-domain feature alignment is defined as calculating cross-channel correlation of source domain features and target domain features in the batch normalization layer in the network, and then performing channel calibration to realize perceptible transfer learning of transferability. The specific operation process is as follows:

[0098] The input feature of the batch normalization layer of the target domain model backbone network is x ∈ R B×K×H×W , wherein K is the number of feature map channels, B is the number of selected samples for one training, H is the image length, and W is the image width; first, calculate the mean μ ∈ R 1×K×1×1 and variance σ 2 ∈ R 1×K×1×1 of each channel in x, then standardize x:

[0099]

[0100] wherein x is the standardized feature, ∈ is a constant to ensure that the denominator is not zero; Each channel of x obeys the standard normal distribution N(0, 1), and then a set of affine transformation parameters is used to transform the standardized feature x as follows:

[0101]

[0102] wherein f i is the output feature of the i-th channel of the batch normalization layer, γ i is the scaling affine transformation parameter of the i-th channel, and β i is the offset affine transformation parameter of the i-th channel;

[0103] Let the affine transformation parameter in the pre-trained network model be called the source domain affine transformation parameter; the target domain affine transformation parameter is called the target domain affine transformation parameter. The cross-channel similarity C p,q between the source domain affine transformation parameter and the target domain affine transformation parameter is calculated as:

[0104]

[0105] wherein The distribution of the domain-specific features in each channel is described, T is a temperature coefficient for adjusting the steepness of the migration importance distribution. s represents the source domain, q indexes the channel corresponding to the source domain feature, t represents the target domain, and p indexes the channel corresponding to the target domain feature. C ∈ R K×K Reflects the cross-channel correlation between the source domain and the target domain, which can be further used as a migration measure of the pre-trained model; in order to prevent negative migration, C is sparsified according to the value of the diagonal p,q is:

[0106]

[0107] Where C k,p is the correlation between the source domain features and the target domain features corresponding to the channel;

[0108] The cross-channel correlation C p,q is used as an attention matrix to align the source domain features and the target domain features, that is, the target domain affine parameters are transformed as follows:

[0109] γ c = C[γ s i ,...,γ s i ] T ,β c = C[β s i ,...,β s i ] T

[0110] Where γ c is the calibrated scaling affine transformation parameter, and β c is the calibrated offset affine transformation parameter;

[0111] Finally, the aligned target domain output feature y i is:

[0112]

[0113] The cross-domain feature alignment is realized.

[0114] In a preferred embodiment of S3, for the target domain model, a set of pre-trained models are used to initialize the parameters of the backbone network in turn, and a generation of fine-tuning training is performed on the training set in the target domain dataset to obtain a generation of fine-tuned target domain model; the consistency of the source domain features generated based on the pre-trained model and the target domain features generated based on the generation of fine-tuned target domain model along the channel distribution is used to calculate the transferability, and the transferability is used as a measure to screen the pre-trained model with the highest transferability, comprising:

[0115] The parameters of the target domain convolutional neural network model backbone network are initialized by using a set of pre-training models obtained by S1 respectively, and then a generation of fine-tuning training is performed. The similarity of the source domain features generated based on the pre-training model and the target domain features generated by the target domain model after one generation of fine-tuning is calculated to calculate the transferability measurement value, and the pre-training model with the highest transferability is further selected for transfer learning of the current target task; wherein:

[0116] In the fine-tuning training of a set of target domain convolutional neural network models initialized by different pre-training model parameters using the target domain training set, a loss function is constructed by using the artificial labeled true value and the prediction result of the target domain convolutional neural network model, the target domain model is trained using the target domain training set, and the hyperparameters of the target domain model are optimized using the target domain validation set.

[0117] The attention value generated by the cross-domain feature alignment is used to measure the reuse degree of the source domain features in the pre-training model in the target domain model after one generation of fine-tuning, and the average transferability of each layer and each channel is used as the transferability measurement of the entire pre-training model.

[0118] Further preferably, the attention value generated by the feature alignment mechanism is used to measure the reuse degree of the pre-training features (source domain features) in the target domain model, and the average transferability of each layer and each channel is used as the transferability measurement of the entire pre-training model:

[0119]

[0120] Wherein, M is the transferability measurement, l=1,...,L is the first L layers of the convolutional neural network model, that is, the feature extractor, K is the number of feature map channels, q indexes the source domain feature channel, and p indexes the target domain feature channel. is the correlation between the qth channel feature of the source domain and the pth channel feature of the target domain.

[0121] In some embodiments of the present application:

[0122] The division in S2 includes the target domain training set, the validation set and the test set, specifically, the target domain training set is used for the training of the target domain model in S3 in S4 and S5. After one generation of fine-tuning training in S4, the best pre-training model is selected, and a plurality of groups of target domain model hyperparameters are selected for training in S5. After the training is completed, the target domain validation set obtained by S2 is used to verify each group of target domain models, and the target task evaluation index is obtained. The target domain model with the optimal index on the target domain validation set is selected as the final medical image automatic analysis model; the target domain test set is used for model testing in S6.

[0123] The target domain convolutional neural network model established in S3 is a neural network model with a cross-channel domain feature matching module.

[0124] The S3-established cross-channel domain feature matching module is as shown in Figure 5 The module can, Indicate feature addition. S3 specifically includes:

[0125] S31: pre-process the input medical CT image, including resampling, HU value conversion, normalization and cropping;

[0126] S32: design a cross-domain feature alignment module; adopt a backbone network framework consistent with the pre-training model, build a top classifier on the backbone network according to the specific target medical task, obtain the target domain model backbone network; in the batch normalization layer of the target domain model backbone network, generate a channel correlation matrix of the source domain feature and the target domain feature, introduce an attention mechanism for channel calibration, and then realize cross-domain feature alignment.

[0127] A series of target domain models initialized by different pre-training models are trained quickly by one generation using the neural network with the domain feature alignment module, and the corresponding transferability measurement values are obtained, and the pre-training models with high transferability are further selected for transfer learning of the current target task.

[0128] The loss function is constructed using the manually labeled true value and the model prediction result, the training set of S2 is used to train the target domain neural network model, and the hyperparameters are optimized through the validation set of S2.

[0129] The structure of the target domain model with the domain feature alignment module in S32 is as shown in Figure 4 It is composed of three down-sampling layers and three up-sampling layers, and each basic convolution module is composed of a convolution layer using batch normalization (Batch Normalization, BN) and a RELU activation function. The skip connection fuses the bottom layer features and the high layer features to achieve more accurate segmentation results. In the target domain model, the batch normalization layers of the backbone network are replaced by the domain feature alignment module in S32, thereby achieving better transfer performance.

[0130] The structure of the domain feature alignment module in S32 is as shown in Figure 5 In the batch normalization layer, the transferability between the cross-channels of the source domain feature and the target domain feature is calculated according to the affine parameters of the pre-training model and the affine parameters of the target domain model.

[0131] Both the pre-training model screening and the complete fine-tuning training use the training set of S2 to train the target domain convolutional neural network model constructed by S3. The definition of the cross-domain feature alignment module in S22 is that in the batch normalization layer in the network, the cross-channel correlation of the source domain feature and the target domain feature is calculated, and the channel is calibrated, thereby realizing the transfer learning with perceptible transferability.

[0132] The technical solutions provided by the above embodiments of the application are described in detail below in combination with a specific application example. The specific application example is implemented on the premise of the technical solutions of the application, and detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following examples.

[0133] The implementation effects of the technical solutions provided by the above embodiments of the application are verified in the specific application example.

[0134] 1. Experimental conditions

[0135] In the specific application example, two CT image segmentation data sets are used as target tasks: a lung nodule segmentation data set on a lung CT image and a liver tumor segmentation data set on an abdominal CT image. The lung nodule segmentation data set contains a total of 1318 lung nodule 3D images with a size of 64x64x32, and the task target is to segment the lung nodule in the CT image. According to the 510:400:408 division, the training set, the validation set and the test set are divided. The liver segmentation data set contains 130 3D CT images, and the task target is to segment the liver and tumor in the CT image respectively. According to the 100:15:15 division, the training set, the validation set and the test set are divided. In the training stage, a 192x192x64 image block is randomly cropped from each 3D CT image as the input of the neural network; while in the validation and test stages, a sliding window strategy with a window size of 192x192x64 and a step size of 96x96x32 is used to sequentially divide the entire CT image into small image blocks to input the network to complete the whole image prediction.

[0136] The model is implemented by using Pytorch, and the computing hardware is an NVIDIA RTX 3090 GPU.

[0137] The parameters in the training process are as follows: the optimizer uses Adam, the training process is verified on the validation set every 5 generations, and the Dice similarity coefficient is selected as the segmentation performance evaluation index. When the Dice index on the validation set no longer rises for more than 20 generations, the training is stopped, and the model with the highest Dice coefficient on the validation set is selected as the final model; for the lung nodule segmentation task, 1e-3 is used as the initial learning rate, the batch size during training is set to 16, and the temperature coefficient t in the proposed feature matching module is set to 1; for the liver tumor segmentation task, 1e-2 is used as the initial learning rate, the learning rate is halved when training for 100 generations, the batch size during training is set to 4, and the temperature coefficient t in the proposed feature matching module is set to 0.5.

[0138] 2. Experimental content

[0139] The transfer learning method proposed by the application is verified from both qualitative and quantitative angles.

[0140] 2.1 Qualitative experimental results

[0141] As shown in Figure 6 , the images in the liver tumor test set are segmented using the present application, and the results obtained by other transfer learning methods are compared. It can be seen that the transfer learning method proposed in the present application can obtain better segmentation performance, especially for the recognition of small tumors.

[0142] As shown in Figure 7 , the images in the lung nodule test set are segmented using the present application, and the results obtained by other transfer learning methods are compared. It can be seen that the transfer learning method proposed in the present application can obtain better segmentation performance, especially for the performance improvement of difficult samples.

[0143] As shown in Figure 8 , the correlation between the transferability quantification index proposed in the present application and the real transfer performance obtained after complete standard fine-tuning on multiple pre-trained models is shown. Kendall's tau is used to measure the ranking correlation between the two, and the greater it is, the stronger the positive correlation. It can be seen that the transferability quantification index proposed in the present application and the real transfer performance show a positive correlation, verifying that the present application has good measurement transferability effect in a wide range of medical image pre-trained models.

[0144] 2.2 Quantitative analysis

[0145] The above-mentioned embodiments of the present application provide a method for fine-tuning training of five pre-trained models including self-supervised learning and supervised learning on lung nodule segmentation data set and liver tumor segmentation data set. The performance improvement of the segmentation performance of the fine-tuned model on the target task compared with the segmentation performance of the model trained from scratch on the target task can measure the efficiency of transfer learning. Two classical transfer learning methods, Stoch Norm and BSS, are selected for comparison. The segmentation evaluation index is the Dice similarity coefficient (DSC):

[0146]

[0147] Where p is the true label of the segmentation target, is the segmentation prediction of the model. The calculation method of the average DSC on a certain category is:

[0148]

[0149] N in the formula represents the number of images containing the category object, p j and respectively represent the real value and the predicted value on the jth image. For the multi-class segmentation task of liver and tumor, the average DSC coefficient of all categories is calculated as an index:

[0150]

[0151] M in the formula represents the number of segmentation categories.

[0152] The value range of DSC and ADSC is 0-1, and the closer to 1, the better the algorithm effect.

[0153] After testing the images in the test set by different methods and taking the average, the experimental results are shown in Table 1:

[0154] Table 1. Comparison of segmentation performance of different transfer learning methods on Model Genesis pre-trained model (the best result is bolded)

[0155]

[0156] From the comparison experiment results in Table 1, it can be seen that the method proposed in the present application realizes better transfer performance on multiple medical image data sets than the standard fine-tuning and the rest of the transfer learning methods. In the multi-class segmentation problem of liver and tumor, due to the large difference in area of the two categories, the model often faces the challenge of class imbalance learning. The transfer learning method based on domain feature alignment proposed in the present application has more obvious improvement in tumor segmentation performance, which shows that the method can improve the class imbalance learning.

[0157] An embodiment of the present application provides a medical image automatic analysis system based on domain feature alignment transfer learning.

[0158] As shown in Figure 9 , the system comprises:

[0159] A pre-trained model set generation module, which, for a given source domain image training set, builds a source domain convolutional neural network model, pre-trains the source domain convolutional neural network model, and obtains a group of pre-trained models;

[0160] A target domain data set construction module, which is used for dividing a plurality of target domain medical CT images into a target domain training set, a validation set and a test set according to a set proportion, and generating a target domain data set;

[0161] A target domain model establishment module, which, based on the backbone network framework of the pre-trained model, builds a corresponding target domain convolutional neural network model according to a target medical automatic analysis task, and obtains a target domain model;

[0162] The pre-training model screening module initializes the target domain model backbone network parameters by using a set of pre-training models in turn, and performs one generation of fine-tuning training by using the training set in the target domain data set to obtain one generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated based on the pre-training model and the target domain features generated by the one generation of fine-tuned target domain model is calculated to obtain the transferability, and the transferability is used as a measure to screen the pre-training model with the highest transferability, that is, the pre-training model with the best performance for the target medical automatic analysis task;

[0163] The model training module initializes the target domain model backbone network parameters by using the parameters of the best pre-training model screened, and performs multiple generations of fine-tuning training by using the training set in the target domain data set, so as to realize the transfer learning of the best pre-training model on the target domain, thereby obtaining the medical image automatic analysis model.

[0164] The medical image automatic analysis module uses the medical image automatic analysis model to complete the automatic analysis of the medical image on the test set of the target domain data set.

[0165] It should be noted that the steps in the method provided by the present application can be realized by using the corresponding modules, devices, units and the like in the system, and those skilled in the art can realize the composition of the system by referring to the technical scheme of the method, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be described here.

[0166] An embodiment of the present application provides a computer terminal, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor can be used to execute the medical image automatic analysis method based on domain feature alignment transfer learning of any one of the above-mentioned embodiments of the present application, or run the medical image automatic analysis system based on domain feature alignment transfer learning of any one of the above-mentioned embodiments of the present application.

[0167] Optionally, the memory is configured to store a program; the memory can include volatile memory (e.g., random-access memory (RAM) such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.), and / or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, etc. for implementing the above-described methods), computer instructions, etc. The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Moreover, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0168] The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Moreover, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0169] The processor is configured to execute the computer programs stored in the memory to implement the various steps in the methods or the various modules of the systems described above in the embodiments. Details can be found in the descriptions of the methods and systems above.

[0170] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled by a bus.

[0171] An embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute the medical image automatic analysis method based on domain feature alignment transfer learning according to any one of the above-mentioned embodiments of the present application, or run the medical image automatic analysis system based on domain feature alignment transfer learning according to any one of the above-mentioned embodiments of the present application.

[0172] The medical image automatic analysis method and system based on domain feature alignment transfer learning provided by the above embodiments of the present application first generate a group of pre-training neural network models on a given source domain data training set by using different pre-training methods; on a target medical image data set, a training set, a validation set and a test set are proportionally divided; the proposed method is used to perform one generation of fine-tuning training on the target training data set, the pre-training model with the highest transferability is selected according to the proposed transferability index, and the whole transfer learning step is completed based on the domain feature alignment module; the test set is used to test the transferred neural network model. Through the present application, the transferability of a group of pre-training models to the target task can be quantitatively evaluated in the transfer learning process; the features in the pre-training model and the target domain model can be aligned along the channel in the transfer process, thereby fully improving the efficiency of transfer learning, and the present application can be widely used in transfer learning between various medical image data sets.

[0173] The preferred embodiments disclosed herein are only a part of the present application, and the description of these embodiments is selected and described in order to better explain the principles and practical applications of the present application, and is not a limitation of the present application. Any modifications and changes made by those skilled in the art within the scope of the specification shall fall within the scope of the present application.

[0174] The above embodiments of the present application are not exhaustive and are well known in the art.

[0175] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or changes within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A medical image automatic analysis method based on domain feature alignment transfer learning, characterized in that, The application relates to a medical image automatic analysis method based on cross-domain feature alignment and attention mechanism. For a given source domain image training set, a source domain convolutional neural network model is built, the source domain convolutional neural network model is pre-trained, and a group of pre-training models are obtained; For a given target domain medical CT image, the image is divided into a training set, a validation set and a test set according to a set proportion, and a target domain data set is generated; Based on the backbone network framework of the pre-training model, a corresponding target domain convolutional neural network model is built according to a target medical automatic analysis task, and a target domain model is obtained; For the target domain model, the backbone network parameters are initialized by using a group of pre-training models in sequence, and a generation of fine-tuning training is carried out by using the training set in the target domain data set, so that a generation of fine-tuning target domain model is obtained; the consistency of the source domain features generated based on the pre-training model and the target domain features generated based on the generation of fine-tuning target domain model along the channel distribution is calculated to obtain the transferability, and the transferability is used as a measurement to screen a pre-training model with the highest transferability, that is, a pre-training model with the best performance for the target medical automatic analysis task; The parameters of the best pre-training model are used to initialize the backbone network parameters of the target domain model, and then the training set in the target domain data set is used to complete multiple generations of fine-tuning training, so that the transfer learning of the best pre-training model on the target domain is realized, and a medical image automatic analysis model is obtained; The medical image automatic analysis model is used to complete the automatic analysis of medical images on the test set of the target domain data set. The target domain convolutional neural network model has a cross-domain feature alignment and an attention mechanism batch normalization layer. The backbone network framework based on the pre-training model is used to build a corresponding target domain convolutional neural network model according to a target medical automatic analysis task, which comprises the following steps: The input medical CT image is preprocessed; A top classifier is built on the backbone network according to a specific target medical task by using a backbone network framework consistent with the pre-training model, and a target domain model backbone network is obtained; In the batch normalization layer of the target domain model backbone network, the similarity of the source domain features and the target domain features along the channel distribution is calculated, a channel correlation matrix of the source domain features and the target domain features is generated, an attention mechanism is introduced for channel calibration, and then cross-domain feature alignment is realized, and a target domain convolutional neural network model is obtained. 2.The medical image automatic analysis method based on domain feature alignment transfer learning according to claim 1, characterized in that, The preprocessing of the input medical CT image comprises resampling, HU value transformation, normalization and cropping. 3.The medical image automatic analysis method based on domain feature alignment transfer learning according to claim 1, characterized in that, In the batch normalization layer of the target domain model backbone network, the channel correlation matrix of the source domain features and the target domain features is generated, the attention mechanism is introduced for channel calibration, and then cross-domain feature alignment is realized, which comprises the following steps: An input feature of a batch normalization layer of the target domain model backbone network is x e R B×K×H×W , wherein K is a number of feature map channels, B is a number of samples selected for one training, H is an image length, and W is an image width; first, calculate a mean value mu e R 1×K×1×1 and a variance sigma 2 e R 1×K×1×1 in each channel of x, and then normalize x: where, is the standardized feature, and ∈ is a constant to ensure the denominator is not zero; Each channel of follows a standard normal distribution N(0, 1), and then a set of affine transformation parameters is applied to the standardized feature as follows: wherein f i is the output feature of the i-th channel of the batch normalization layer, γ i is the scaling affine transformation parameter of the i-th channel, β i is the shifting affine transformation parameter of the i-th channel; Let the source domain affine transformation parameters in the pre-training model be The target domain affine transformation parameters are Calculate the cross-channel similarity C between the source domain affine transformation parameters and the target domain affine transformation parameters p,q is: where, is used to describe the distribution of the domain-specific features in each channel, T is a temperature coefficient used to adjust the steepness of the migration importance distribution; s represents the source domain, q indexes the channel corresponding to the source domain feature, t represents the target domain, and p indexes the channel corresponding to the target domain feature; C ∈ R K×K reflects the cross-channel correlation between the source domain and the target domain, and serves as a measure of the migration of the pre-trained model; in order to prevent negative migration, C is sparsified according to the value of the diagonal line p,q is: wherein C p,p is the correlation between the corresponding channels of the source domain feature and the target domain feature; Utilizing cross-channel correlation C p,q Aligning source domain features and target domain features as attention matrices, i.e. transforming the target domain affine transformation parameters as follows: gamma c = C [gamma s i ,..., gamma s i ] T , beta c = C [beta s i ,..., beta s i ] T wherein γ c is the scaled affine transformation parameter after calibration, and β c is the offset affine transformation parameter after calibration. Finally, the aligned target domain output features y i is: Cross-domain feature alignment is realized. 4.The medical image automatic analysis method based on domain feature alignment transfer learning according to claim 1, characterized in that, The target domain model is initialized by a set of pre-training models in sequence, and a generation of fine-tuning training is performed on the training set in the target domain data set to obtain a generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated by the pre-training model and the target domain features generated by the generation of fine-tuned target domain model is calculated to calculate the transferability, and the transferability is used as a measure to screen the pre-training model with the highest transferability, comprising: The parameters of the backbone network of the target domain model are initialized by a set of pre-training models, and a generation of fine-tuning training is performed on the training set to generate a corresponding generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated by the pre-training model and the target domain features generated by the generation of fine-tuned target domain model is calculated to calculate the transferability as a measure, and the pre-training model with the highest transferability is further screened out for transfer learning of the current target medical automatic analysis task; wherein: In the fine-tuning training of a set of target domain models initialized by different pre-training model parameters using the training set, a loss function is constructed using the manually labeled true value and the prediction result of the target domain model, the target domain model is trained using the training set, and the hyperparameters of the target domain model are optimized using the validation set to generate a corresponding generation of fine-tuned target domain model. 5.The medical image automatic analysis method based on domain feature alignment transfer learning according to claim 4, characterized in that, The consistency of the channel distribution of the source domain features generated by the pre-training model and the target domain features generated by the generation of fine-tuned target domain model is calculated to calculate the transferability, comprising: where M is the transferability measure, l = 1,...,L are the first L layers of the pre-trained model and the target domain model, i.e., the feature extractor, K is the number of feature map channels, q is the index of the source domain feature channel, and p is the index of the target domain feature channel, is the correlation between the feature of the qth channel of the source domain and the feature of the pth channel of the target domain.

6. A medical image automatic analysis system based on domain feature alignment transfer learning for implementing the method of any one of claims 1-5, characterized in that, Comprising: A pre-training model set generation module, which, for a given source domain image training set, builds a source domain convolutional neural network model, pre-trains the source domain convolutional neural network model, and obtains a set of pre-training models; A target domain data set construction module, which is used to divide a given number of target domain medical CT images into a target domain training set, a validation set and a test set according to a set proportion, and generate a target domain data set; A target domain model establishment module, which, based on the backbone network framework of the pre-training model, builds a corresponding target domain convolutional neural network model according to the target medical automatic analysis task, and obtains a target domain model; A pre-training model screening module, which initializes the parameters of the backbone network of the target domain model by a set of pre-training models in sequence, and performs a generation of fine-tuning training on the training set in the target domain data set to obtain a generation of fine-tuned target domain model; the consistency of the channel distribution of the source domain features generated by the pre-training model and the target domain features generated by the generation of fine-tuned target domain model is calculated to calculate the transferability, and the pre-training model with the highest transferability is screened out as a measure, i.e. the pre-training model with the best performance for the target medical automatic analysis task. A model training module, which initializes parameters of a target domain model backbone network with parameters of the best pre-training model screened, and performs multi-generation fine-tuning training with a training set in the target domain data set, so as to realize transfer learning of the best pre-training model on the target domain, thereby obtaining a medical image automatic analysis model; A medical image automatic analysis module, which performs automatic analysis of medical images on a test set of the target domain data set by using the medical image automatic analysis model.

7. A terminal comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor can be used to execute the method in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the method in any one of claims 1-5.

Images