Auxiliary diagnosis method and device based on semi-supervised learning and multi-scale feature fusion

Abstract

The present invention provides an auxiliary diagnosis method and device based on semi-supervised learning and multi-scale feature fusion, the method comprising: inputting a CT image to be diagnosed into a multi-scale feature fusion module of a classification model to obtain a multi-scale fusion feature of the CT image to be diagnosed; inputting the multi-scale fusion feature into a classification module of the classification model to obtain an auxiliary medical diagnosis result of the CT image to be diagnosed; wherein the classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a first sample CT image with a label and a second sample CT image without a label. The present invention realizes that when the amount of effective annotated data is small, a classification model with higher generalization and accuracy can be obtained, thereby quickly and accurately obtaining auxiliary medical diagnosis results.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary medical diagnosis technology, and in particular to an auxiliary diagnosis method and device based on semi-supervised learning and multi-scale feature fusion. Background Art

[0002] Infectious diseases spread rapidly around the world, seriously threatening human life, health and safety. In order to fight infectious diseases, it is crucial to effectively screen patients suspected of infection. Compared with reverse transcription polymerase chain reaction, CT (Computed Tomography) images have become an effective means of diagnosing whether patients are infected with infectious diseases due to their high accuracy and wide popularity.

[0003] In the prior art, doctors need to examine a large number of CT images to determine whether the patient corresponding to the CT image is infected with an infectious disease, which requires doctors to spend a lot of energy to repeatedly examine and diagnose. Excessive diagnostic pressure will affect the accuracy and efficiency of doctors' CT image examinations, making it difficult to make a quick and accurate diagnosis, and it is also difficult for patients to be diagnosed and isolated in time.

[0004] In order to solve this problem, existing technologies use deep learning to analyze CT images, which can efficiently and accurately diagnose and predict the lesion area in CT, and assist doctors in providing accurate diagnosis. However, a large number of labeled training data sets are needed to train a deep learning model that can accurately diagnose. On the one hand, the organization and annotation of medical images requires a lot of energy from professionals, so large-scale annotation work is not supported; on the other hand, because medical images involve patient privacy, medical images cannot be used publicly. Therefore, in this scenario, it is difficult to obtain high-quality labeled medical image data, and it is also difficult to train a high-performance deep learning model, resulting in poor auxiliary medical diagnosis effect. Summary of the invention

[0005] The present invention provides an auxiliary diagnosis method and device based on semi-supervised learning and multi-scale feature fusion, which are used to solve the defects in the prior art that there is a shortage of effectively labeled CT image samples, it is difficult to train a high-performance deep learning model, and the auxiliary medical diagnosis effect is poor. In the case of a shortage of effectively labeled CT image samples, the auxiliary medical diagnosis results can be obtained quickly and accurately.

[0006] The present invention provides an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion, comprising:

[0007] Inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed;

[0008] Inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed;

[0009] The classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a labeled first sample CT image and an unlabeled second sample CT image.

[0010] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the classification model is trained based on the following steps:

[0011] Based on the first training data set, performing semi-supervised learning training on the initial model of the classification model to obtain a pre-trained model of the classification model;

[0012] Inputting a second sample CT image in the first training data set into the pre-training model to obtain a simulation label corresponding to an auxiliary medical diagnosis result of the second sample CT image;

[0013] adding the simulated label of the second sample CT image to the first training data set;

[0014] Performing image mixing on the sample CT images in the first training data set to which the simulation labels are added, to obtain a second training data set;

[0015] Based on the second training data set, the pre-training model is trained to obtain the classification model.

[0016] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the sample CT images in the first training data set with the simulation label added are mixed to obtain a second training data set, including:

[0017] Performing data enhancement on the sample CT images in the first training data set to which the simulation labels are added, to obtain an enhanced first training data set;

[0018] Processing the enhanced first training data set to obtain multiple groups of sample CT image pairs;

[0019] For each sample CT image pair in the plurality of groups of sample CT image pairs, the following operations are performed:

[0020] Calculate the class activation map of any sample CT image in the current sample CT image pair;

[0021] Determining, according to the class activation map, a target region in any sample CT image that has the highest contribution to an auxiliary medical diagnosis result;

[0022] Pasting the target area to a corresponding position of another sample CT image in the current sample CT image pair to obtain a third sample CT image corresponding to the current sample CT image pair;

[0023] Obtaining a first label of the third sample CT image according to the size of the target area, the label of any one of the sample CT images, and the label of the other sample CT image;

[0024] The second training sample set is constructed according to the third sample CT images corresponding to all sample CT image pairs and the first labels of the third sample CT images.

[0025] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the data enhancement includes one or more combinations of standardization, geometric transformation, random adjustment of brightness and random adjustment of contrast;

[0026] The geometric transformation includes one or more combinations of translation, flipping and rotation.

[0027] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the loss function of the classification model is generated by jointly constructing a first loss function and a second loss function;

[0028] Wherein, the first loss function aims to minimize the similarity of the multi-scale fusion features of the third sample CT image and the other sample CT image;

[0029] The second loss function aims to minimize the difference between the first label and the second label of the third sample CT image; the second label is obtained by predicting the auxiliary medical diagnosis result of the third sample CT image based on the classification model.

[0030] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the second sample CT image in the first training data set is input into the pre-training model to obtain a simulation label corresponding to the auxiliary medical diagnosis result of the second sample CT image, including:

[0031] Inputting the second sample CT image into the pre-trained model to obtain the probability that the second sample CT image belongs to each preset medical diagnosis category;

[0032] Performing entropy minimization processing on the probability that the second sample CT image belongs to each preset medical diagnosis category to obtain a result of entropy minimization processing of the second sample CT image;

[0033] According to the minimization entropy processing result, a simulation label of the second sample CT image is obtained.

[0034] According to an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention, the multi-scale feature fusion module includes an identity mapping module;

[0035] Wherein, the identity mapping module includes a first feature extraction unit, a feature fusion unit and a second feature extraction unit;

[0036] The first feature extraction unit is used to perform multi-scale feature extraction on the CT image to be diagnosed, so as to obtain feature maps of multiple different scales of the CT image to be diagnosed;

[0037] The feature fusion unit is used to fuse the multiple feature maps of different scales to obtain a fused feature map of the CT image to be diagnosed;

[0038] The second feature extraction unit is generated based on a non-local neural network and is used to perform a non-local operation on the fused feature map to obtain a multi-scale fused feature of the CT image to be diagnosed.

[0039] The present invention also provides an auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion, comprising:

[0040] A feature extraction module, used for inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed;

[0041] An auxiliary diagnosis module, used for inputting the multi-scale fusion features into the classification module of the classification model to obtain an auxiliary medical diagnosis result of the CT image to be diagnosed;

[0042] The classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a labeled first sample CT image and an unlabeled second sample CT image.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion as described above is implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the auxiliary diagnosis methods based on semi-supervised learning and multi-scale feature fusion as described above.

[0045] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described auxiliary diagnosis methods based on semi-supervised learning and multi-scale feature fusion.

[0046] The auxiliary diagnosis method and device based on semi-supervised learning and multi-scale feature fusion provided by the present invention use a small amount of labeled first sample CT images and a large amount of unlabeled second sample CT images as training data sets, so that the training data sets contain a large number of rich samples, and use the training data sets containing rich samples to perform semi-supervised training on the classification model, so that the classification model that can accurately output auxiliary medical diagnosis results can be quickly trained to obtain the classification model, which can effectively reduce the demand for the amount of labeled data and improve the performance of the classification model, so that the auxiliary medical diagnosis results are more accurate; in addition, by performing multi-scale extraction and fusion on the CT images to be diagnosed, the features of each scale in the CT images to be diagnosed that are related to the auxiliary medical diagnosis results can be retained, thereby further improving the auxiliary medical diagnosis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is one of the flow charts of the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0049] Figure 2 It is a structural schematic diagram of image mixing in the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0050] Figure 3 It is a structural schematic diagram of a multi-scale feature fusion module in the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0051] Figure 4 This is the second flow chart of the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0052] Figure 5 It is a structural schematic diagram of a confusion matrix in the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0053] Figure 6It is a structural schematic diagram of simulation results in the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0054] Figure 7 It is a structural schematic diagram of an auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion provided by the present invention;

[0055] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In recent years, thanks to the rapid development of computer performance, deep learning has made breakthrough progress in many fields, such as computer vision, natural language processing, etc. Computer diagnosis technology based on deep learning has also been continuously improved and widely used. In the diagnosis of certain diseases, the accuracy of computer diagnosis has reached a very high level.

[0058] Usually, training deep learning models requires a large amount of labeled data for training, but the organization and labeling of medical images requires a lot of effort from professionals. Moreover, due to the privacy of patients, medical images cannot be used publicly. Therefore, it is difficult to obtain abundant high-quality labeled medical image data. The lack of high-quality labeled medical image data makes it difficult to train high-performance deep learning models, resulting in poor auxiliary medical diagnosis effects, which in turn limits the further development of deep learning in the medical field.

[0059] To address this technical problem, this embodiment provides an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion. By introducing a training data set containing a large number of unlabeled samples and a small number of labeled samples, and using a semi-supervised learning method, the hidden distribution information learned from the training data set is used to promote the classifier to move in the correct decision direction, thereby obtaining a classification model with higher generalization and accuracy, and then quickly and accurately obtaining auxiliary medical diagnosis results.

[0060] It should be noted that the execution subject in this embodiment may be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, a super mobile personal computer, a netbook, or a personal digital assistant, etc., and the non-mobile electronic device may be a server, a network attached storage, a personal computer, a smart TV, a teller machine, or a self-service machine, etc., which is not specifically limited in the present invention.

[0061] Combine the following Figure 1 The auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion of the present invention is described. The application scenarios of this method include, but are not limited to, auxiliary diagnosis of infectious diseases and auxiliary diagnosis of non-infectious diseases, etc. This embodiment does not specifically limit this.

[0062] The method comprises: step 101, inputting a CT image to be diagnosed into a multi-scale feature fusion module of a classification model to obtain a multi-scale fusion feature of the CT image to be diagnosed;

[0063] Among them, the CT image to be diagnosed is a CT image to be used for auxiliary medical diagnosis; the CT image to be diagnosed may include various types of images related to auxiliary medical diagnosis, such as a patient's lung CT image, laryngeal CT image, etc., which is not specifically limited in this embodiment.

[0064] The source of the CT image to be diagnosed may be a CT image of the patient acquired in real time, or may be a CT image of the patient acquired in advance. This embodiment does not specifically limit the source of the CT image to be diagnosed.

[0065] The classification model is constructed and generated based on the multi-scale feature fusion module and the classification module; wherein the multi-scale feature fusion module includes but is not limited to other units such as feature extraction units and fusion units; the number of feature extraction units and the number of fusion units can be set according to actual needs, and this embodiment does not specifically limit this. The feature extraction unit includes multiple sub-feature extraction units, and each sub-feature extraction unit corresponds to feature extraction of one scale.

[0066] Before executing step 101, the classification model needs to be trained. The specific training steps include:

[0067] Acquire a first training data set, wherein the first training data set includes a small number of labeled first sample CT images and a large number of unlabeled second sample CT images;

[0068] Then, directly based on the training data set, the classification model is trained by semi-supervised learning to update the parameter information of the classification model;

[0069] Alternatively, after preprocessing the first training data set, such as data enhancement and data mixing, the classification model is trained through semi-supervised learning based on the preprocessed first training data set to obtain a classification model that can quickly and accurately output auxiliary medical diagnosis results. This embodiment does not specifically limit the specific training method of the classification model.

[0070] Since semi-supervised learning is a learning method that combines supervised learning with unsupervised learning, it can use a small amount of labeled data and a large amount of unlabeled data to alleviate the problem of insufficient data. Specifically, semi-supervised learning uses a large amount of unlabeled data and a small amount of labeled data at the same time to quickly train a classification model that can accurately output auxiliary medical diagnosis results. While effectively reducing the need for labeled data, it can improve the performance of the classification model and make the classification results more accurate.

[0071] After obtaining the trained classification model, the CT image to be diagnosed can be input into the multi-scale feature fusion module of the classification model, and at least one set of multi-scale feature extraction and multi-scale feature fusion operations can be performed on the CT image to be diagnosed to obtain the multi-scale fusion features of the CT image to be diagnosed.

[0072] Step 102, inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed;

[0073] The classification model is obtained by semi-supervised learning based on a training data set; the training data set includes a first sample CT image with a label and a second sample CT image without a label.

[0074] Among them, the auxiliary medical diagnosis result is a medical diagnosis result that assists the doctor's diagnosis. The auxiliary medical diagnosis result can be a category of whether a certain disease is present. For example, the specific form of the diagnosis result can be negative or positive, or normal or abnormal, etc.; it can also be a category of disease, such as lung swelling, lung inflammation, etc. This embodiment does not specifically limit this.

[0075] Optionally, after obtaining the multi-scale fusion features of the CT image to be diagnosed, the multi-scale fusion features can be learned based on the classification module to predict the auxiliary medical diagnosis results of the CT image to be diagnosed.

[0076] This embodiment uses a small amount of labeled first sample CT images and a large amount of unlabeled second sample CT images as a training data set, so that the training data set contains a large number of rich samples, and uses the training data set containing rich samples to perform semi-supervised training on the classification model, which can quickly train to obtain a classification model that can accurately output auxiliary medical diagnosis results. While effectively reducing the demand for labeled data volume, the performance of the classification model can be improved, making the auxiliary medical diagnosis results more accurate; in addition, by performing multi-scale extraction and fusion on the CT image to be diagnosed, the features of each scale in the CT image to be diagnosed that are related to the auxiliary medical diagnosis results can be retained, further improving the auxiliary medical diagnosis results.

[0077] On the basis of the above-mentioned embodiments, the classification model in this embodiment is obtained by training based on the following steps: based on the first training data set, the initial model of the classification model is trained by semi-supervised learning to obtain a pre-trained model of the classification model; the second sample CT image in the first training data set is input into the pre-trained model to obtain a simulated label corresponding to the auxiliary medical diagnosis result of the second sample CT image; the simulated label of the second sample CT image is added to the first training data set; the sample CT images in the first training data set with the simulated label are mixed to obtain a second training data set; based on the second training data set, the pre-trained model is trained to obtain the classification model.

[0078] The sample CT image may be a first sample CT image and / or a second sample CT image.

[0079] Optionally, specific training steps of the classification model include:

[0080] First, a first training data set is obtained; wherein the first training data set includes a first sub-training data set constructed by a labeled first sample CT image, and a second sub-training data set constructed by an unlabeled second sample CT image.

[0081] Table 1 Division of training data set

[0082] category Training set Validation set Test Set Illness 191 60 98 normal 234 58 105 total 425 118 203

[0083] After obtaining the first training data set, all training data can be used for training, or the training data set can be divided into a training set, a validation set and a test set. The training set is used for training, and the validation set and the test set are used for validation testing. The specific division ratio can be set according to actual needs, as shown in Table 1.

[0084] Among them, the labeled CT dataset is obtained from the public dataset, and the specific number and acquisition source can be set according to actual needs.

[0085] For example, the first sub-training dataset contains 349 first sample CT images of positive diagnosis categories and 397 first sample CT images of negative diagnosis categories. Among them, the first sample CT images corresponding to the positive diagnosis category can be 760 COVID-19 preprints from the preprint servers medRxiv and bioRxiv, and the first sample CT images corresponding to the positive diagnosis category can be normal or CT scans with other types of diseases (diseases unrelated to the disease to be diagnosed). The second sub-training dataset comes from the LUNA (Lung Nodule Analysis) dataset, which consists of low-dose lung CT images and is designed specifically for the detection and segmentation of lung nodules in patients.

[0086] Then, based on the first training data set, the initial model of the classification model is pre-trained by semi-supervised learning to obtain a pre-trained model of the classification model.

[0087] In order to obtain the label of the second sample CT image, after obtaining the pre-trained model of the classification model, the second sample CT image may be input into the pre-trained model to obtain the simulated label of the second sample CT image.

[0088] Among them, the specific method of obtaining the simulation label of the second sample CT image includes: directly obtaining the simulation label of the second sample CT image according to the auxiliary medical diagnosis result output by the pre-training model; or after data processing the auxiliary medical diagnosis result output by the pre-training model, obtaining the simulation label of the second sample CT image according to the data processing result. This embodiment does not specifically limit this.

[0089] After obtaining the simulated label of the second sample CT image, the simulated label of the second sample CT image can be added to the first training data set so that each sample in the first training data set contains a complete label, and the first training data set after the addition contains the real label of the first sample CT image and the simulated label of the second sample CT image.

[0090] Then, the sample CT images in the complete first training data set with the simulation labels added are directly mixed to obtain the second training data set; or the sample CT images in the complete first training data set with the simulation labels added are data enhanced and then mixed to obtain the second training data set, etc. This embodiment does not specifically limit this.

[0091] By performing image mixing on the sample CT images of the first training data set, not only can the first training data set be enhanced, but the second training data set obtained after image mixing can also contain more abundant and diverse sample CT images, thereby effectively preventing the classification model from overfitting, and making the classification model trained according to the second training data set more accurate and have better robustness and generalization.

[0092] Then, based on the second training data set after image mixing, the pre-trained model is trained until the preset termination condition of the classification model is met. The preset termination condition includes the convergence of the classification model and / or the reaching of the maximum number of iterations, etc., which is not specifically limited in this embodiment.

[0093] In this embodiment, the classification model is first pre-trained with an initial first training data set, and based on the pre-trained model, a simulated label of a second sample CT image is obtained and added to the first training data set, so that each sample CT image of the first training data set has a complete label; then, image mixing is performed on the sample CT images in the first training data set with complete labels to perform data enhancement on the first training data set, so that the second training data set after image mixing contains more abundant and diverse sample CT images; then, the pre-trained model is trained with the second training data set containing more abundant and diverse sample CT images to quickly obtain a classification model with good robustness, generalization and accuracy.

[0094] On the basis of the above-mentioned embodiment, the image mixing of the sample CT images in the first training data set to which the simulation label is added is performed in this embodiment to obtain the second training data set, including: performing data enhancement on the sample CT images in the first training data set to which the simulation label is added to obtain the enhanced first training data set; processing the enhanced first training data set to obtain multiple groups of sample CT image pairs; performing the following operations for each sample CT image pair in the multiple groups of sample CT image pairs: calculating the class activation map of any sample CT image in the current sample CT image pair; determining the target area with the highest contribution to the auxiliary medical diagnosis result in any sample CT image according to the class activation map; pasting the target area to the corresponding position of another sample CT image in the current sample CT image pair to obtain the third sample CT image corresponding to the current sample CT image pair; obtaining the first label of the third sample CT image according to the size of the target area, the label of any sample CT image, and the label of the other sample CT image; constructing the second training sample set according to the third sample CT images corresponding to all sample CT image pairs and the first label of the third sample CT image.

[0095] Optionally, in order to further increase the types and quantity of sample CT images, before image mixing is performed on the sample CT images in the first training data set, data enhancement may be performed on the sample CT images in the first training data set to which simulation labels are added; specific data enhancement methods include but are not limited to geometric transformation and random adjustment of image attribute values, etc.; wherein the attributes include but are not limited to brightness and contrast, etc.

[0096] The number of data enhancements can also be set according to actual needs, such as performing two different data enhancements on each unlabeled second sample CT image; or performing one data enhancement on each labeled first sample CT image.

[0097] Then, the sample CT images in the first training data set after data enhancement are mixed to further increase the types and quantity of the sample CT images, so that the samples in the first training sample set are more abundant and diverse.

[0098] Optionally, any two sample CT images in the enhanced first training data set are divided into a group of sample CT image pairs to obtain multiple groups of sample CT image pairs.

[0099] For each set of sample CT image pairs, the following image blending steps are performed:

[0100] Firstly, any sample CT image in each group of sample CT image pairs is selected, and the class activation map of any sample CT image is calculated.

[0101] Among them, the class activation map is a response-based visual interpretation method; it generates an attention heat map by weighted summing the weights of the fully connected layer and the feature map, highlighting important areas that are closely related to the auxiliary medical diagnosis results.

[0102] The formula for calculating the class activation map of any sample CT image is:

[0103]

[0104] Among them, CAM (I i ) is the class activation map of the i-th sample CT image; It is an upsampling operation, which is used to upsample the size of the feature map to the same size as the input image (sample CT image); i is the i-th sample CT image, F l (I i ) is the lth feature map of the i-th sample CT image, w is the weight of the classification channel, y i is the number of categories of the i-th sample CT image, and d is the number of feature maps of the i-th sample CT image.

[0105] like Figure 2 As shown, after obtaining the class activation map of any sample CT image, the contribution of each pixel point in any sample CT image to the auxiliary medical diagnosis category result can be determined according to the class activation map; and according to the contribution of each pixel point, the target area with the highest contribution to the auxiliary medical diagnosis result is selected from any sample CT image as the region of interest.

[0106] For example, according to the contribution of each pixel, a target area with a scale of 56×56 and a highest contribution is selected from a class activation map with a scale of 224×224 as the region of interest. The size of the target area can be determined according to actual needs, and this embodiment does not specifically limit this.

[0107] Then, the target region is pasted to the corresponding position of another sample CT image to obtain a third CT image, namely a new sample CT image, wherein the pasting position of the target region is consistent with the position of the target region in any sample CT image, namely, they are mapped to each other.

[0108] According to the size of the target area, the label of any sample CT image, and the label of another sample CT image, the label of the third sample CT image is updated. The specific formula is:

[0109] x′=B⊙x 1 +(1-B)⊙x 2 ;

[0110] y′=λy 1 +(1-λ)y 2 ;

[0111] Where B is a binary mask indicating which pixels belong to which of any two sample CT images, and 1 is a binary mask filled with 1s. λ is the ratio of the size of the target region to the total size of the other sample CT image; x 1 and x 2 are any two sample CT images; y 1 and 2 are the labels of any two sample CT images; x′ and y′ are the third sample CT image and the first label of the third sample CT image; ⊙ is element-wise multiplication.

[0112] The above process is repeated to continue to perform the above image mixing step on all sample CT image pairs in the enhanced training data set to construct and generate a second training sample set.

[0113] In this embodiment, by performing data enhancement on sample image data and performing class activation image combination on the unlabeled training samples after data enhancement with virtual labels and the labeled training samples after data enhancement, the acquired second training samples contain richer and more diverse sample CT images, thereby making the trained classification model have good robustness, generalization and accuracy.

[0114] Based on the above embodiments, the data enhancement in this embodiment includes one or more combinations of standardization, geometric transformation, random adjustment of brightness and random adjustment of contrast; the geometric transformation includes one or more combinations of translation, flipping and rotation.

[0115] The standardization includes maximum and minimum standardization and log function conversion standardization, etc., which are not specifically limited in this embodiment.

[0116] It should be noted that the data enhancement in this embodiment is not limited to the above-mentioned methods, and other methods for data enhancement of the sample CT image may also be used, such as adding random noise or random cropping to the sample CT image.

[0117] This embodiment can effectively expand the training data set by performing data enhancement on the sample CT images, improve the robustness of the samples in different environments, and enable the trained classification model to adapt to background interference with greater robustness, generalization and compatibility, thereby improving the accuracy of the diagnosis results.

[0118] On the basis of the above-mentioned embodiment, the loss function of the classification model in this embodiment is jointly constructed and generated based on a first loss function and a second loss function; wherein, the first loss function aims to minimize the similarity between the multi-scale fusion features of the third sample CT image and the other sample CT image; the second loss function aims to minimize the difference between the first label and the second label of the third sample CT image; the second label is obtained by predicting the auxiliary medical diagnosis result of the third sample CT image based on the classification model.

[0119] Among them, the first loss function aims to obtain the same multi-scale fusion features of the original sample CT image (another sample CT image without image fusion) and the new sample CT image (the third sample CT image) through the multi-feature fusion module. The cross entropy loss of the two images is applied to the output of the features to construct the first loss function, so that the multi-scale fusion features of the sample CT images before and after data enhancement are consistent, thereby improving the accuracy of the classification model. The specific calculation formula is as follows:

[0120] L=L BCE (F, F′);

[0121] Among them, L is the first loss function, L BCE (·) represents the binary cross entropy loss function, F and F′ represent the original sample CT image and the new sample CT image respectively, which are subjected to the multi-scale fusion features obtained by the multi-feature fusion module.

[0122] The second loss function is constructed and generated with the goal of minimizing the difference between the first label and the second label of the third sample CT image.

[0123] Optionally, the first loss function and the second loss function may be fused to obtain a loss function of the classification model.

[0124] The fusion method includes weighted addition, direct addition, etc., which is not specifically limited in this embodiment.

[0125] This embodiment combines a first loss function that aims to make the multi-scale fusion features of the third sample CT image and another sample CT image the same, and a second loss function that aims to minimize the difference between the first label and the second label of the third sample CT image. The present invention proposes a new loss function to train the classification model, which not only makes the third sample CT image and another sample CT image produce spatially consistent feature representations, but also ensures that the difference between the predicted label and the true label is minimized, so that the performance of the classification model is better, and the auxiliary medical diagnosis results are more accurate.

[0126] On the basis of the above embodiments, the method of inputting the second sample CT image in the first training data set into the pre-training model to obtain a simulated label corresponding to the auxiliary medical diagnosis result of the second sample CT image in this embodiment includes: inputting the second sample CT image into the pre-training model to obtain the probability that the second sample CT image belongs to each preset medical diagnosis category; performing minimization entropy processing on the probability that the second sample CT image belongs to each preset medical diagnosis category to obtain a minimized entropy processing result of the second sample CT image; and obtaining a simulated label of the second sample CT image according to the minimized entropy processing result.

[0127] Optionally, since there are fewer labeled sample data in semi-supervised learning, in order to make the trained classification model more reliable, the auxiliary medical diagnosis results of the unlabeled second sample CT image can be processed by minimizing entropy, forcing the classification model to make a low entropy prediction for the unlabeled second sample CT image. The specific steps are as follows:

[0128] First, the second sample CT image is input into the pre-trained model to obtain the probability that the second sample CT image belongs to each preset medical diagnosis category;

[0129] Then, the probability that the second sample CT image belongs to each preset medical diagnosis category is processed by minimizing entropy to obtain the minimizing entropy processing result of the second sample CT image;

[0130] Among them, the method of minimizing entropy processing can be to use a sharpening function to minimize entropy. The specific formula is:

[0131]

[0132] Among them, Sharpen(p, T) i is the minimization entropy processing result of the second sample CT image i; p j is the probability that the second sample CT image i belongs to the preset medical diagnosis category j, T is an adjustment parameter used to adjust the classification entropy; L is the number of preset medical diagnosis categories; p i is the final prediction result of the second sample CT image i.

[0133] This embodiment obtains corresponding simulated labels by performing entropy minimization processing on the auxiliary medical diagnosis results of the second sample CT image, so that the classification model can make low entropy predictions for the unlabeled second sample CT image, thereby making the trained classification model more reliable.

[0134] On the basis of the above embodiments, the multi-scale feature fusion module in this embodiment includes an identity mapping module; wherein the identity mapping module includes a first feature extraction unit, a feature fusion unit and a second feature extraction unit; the first feature extraction unit is used to perform multi-scale feature extraction on the CT image to be diagnosed to obtain feature maps of multiple different scales of the CT image to be diagnosed; the feature fusion unit is used to fuse the feature maps of multiple different scales to obtain a fused feature map of the CT image to be diagnosed; the second feature extraction unit is generated based on a non-local neural network, and is used to perform non-local operations on the fused feature map to obtain multi-scale fusion features of the CT image to be diagnosed.

[0135] Among them, the classification model adopts the residual neural network model in the classic image classification model, and adds a multi-scale feature fusion module to it to enhance the feature extraction ability of the classification model.

[0136] The multi-scale feature fusion module includes at least one set of identity mapping modules generated based on the residual neural network.

[0137] The identity mapping module includes a first feature extraction unit, which includes multiple feature extraction branches for extracting features of different scales, and extracts multiple feature maps of different scales by performing parallel multi-convolution operations on the diagnostic CT image. The number of feature extraction branches can be set according to actual needs, such as two.

[0138] The identity mapping module also includes a feature fusion unit, which is used to splice and fuse multiple feature maps of different scales extracted by the feature extraction branch. Specifically, multiple feature maps of different scales are spliced ​​and scaled through 1×1 convolution, so that the fused feature map is consistent with the size of the CT image to be diagnosed.

[0139] The identity mapping module also includes a second feature extraction unit, which is specifically generated based on a non-local neural network and is used to perform non-local operations on the fused feature map to capture the long-range dependent features in the fused feature map, so that the acquired features can more effectively characterize the auxiliary medical diagnosis results, thereby obtaining more accurate features of the image and effectively improving the accuracy of the classification model.

[0140] like Figure 3 As shown in FIG. 1 , an exemplary structure of a multi-scale feature fusion module is shown. The first feature extraction unit includes two feature extraction branches, one of which is constructed by two convolution layers with convolution kernels of 1x1 and 3x3, and the other is constructed by three convolution layers with convolution kernels of 1x1, 3x3 and 3x3.

[0141] This embodiment extracts multi-scale features from the CT image to be diagnosed and performs non-local operations, so that the acquired features can more effectively characterize the auxiliary medical diagnosis results and effectively improve the accuracy of the classification model.

[0142] like Figure 4 As shown, it is a complete flow chart of an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided in this embodiment, and the specific steps include:

[0143] Step 401, obtaining a first training data set; wherein the first training data set includes a first sample CT image with a label and a second sample CT image without a label;

[0144] Step 402, performing data enhancement on the sample CT images in the first training data set, specifically including rotation and random cropping;

[0145] Step 403, obtaining a simulated label of the second sample CT image; specifically, inputting the second sample CT image into a pre-trained model of a classification model obtained by semi-supervised learning based on the first training data set, obtaining an auxiliary medical diagnosis result of the second sample CT image, and performing entropy minimization processing on the auxiliary medical diagnosis result to obtain a simulated label of the second sample CT image.

[0146] Step 404, performing class activation image mixing on the sample CT images in the first training data set with the simulation labels added thereto, to generate a second training data set;

[0147] Step 405, adding the cross entropy loss function between the multi-scale fusion features of the original sample CT image (another sample CT image) and the new sample CT image (the third sample CT image) to the loss function of the classification model;

[0148] Step 406, training the classification model based on the second training data set and the loss function;

[0149] Step 407: input the CT image to be diagnosed into the trained classification model to obtain an auxiliary medical diagnosis result.

[0150] In summary, in order to automatically diagnose the CT images to be diagnosed and solve the problem of insufficient samples available for training models, this embodiment proposes an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion. Specifically, a convolutional neural network with a multi-scale feature fusion module and a residual network is proposed to achieve accurate diagnosis of the CT images to be diagnosed, and good performance can be achieved even when the number of labeled samples is small.

[0151] In order to verify the effectiveness of the auxiliary diagnosis method in this embodiment, this embodiment selects 425 labeled first sample CT images and 500 unlabeled second sample CT images as training data sets to perform semi-supervised learning on the classification model. The offline area index AUC, the accuracy index Auccracy, and the score index F1-Score are used to evaluate the performance of the classification model.

[0152] As shown in Table 2, various indicators of the trained classification model on the test set are shown when only labeled data is used for supervised learning and when both labeled data and unlabeled data are used for semi-supervised learning (the auxiliary diagnosis method of this embodiment).

[0153] Table 2 Comparison results of supervised learning and semi-supervised learning classification indicators

[0154] Labeled samples Unlabeled samples AUC Accuracy F1-score 425 0 0.827 0.786 0.768 425 500 0.911 0.891 0.883

[0155] It can be seen from Table 2 that after using semi-supervised learning, each indicator has been improved to a certain extent. Therefore, the auxiliary diagnosis method in this embodiment has better accuracy and effectiveness.

[0156] Figure 5 is the confusion matrix of the results of supervised learning using labeled data and semi-supervised learning using both labeled data and unlabeled data on the test set; Figure 5 It can be seen that this embodiment can effectively improve the feature extraction ability of the classification model by introducing unlabeled samples, reduce the number of misjudgments of the classification model, and increase the reliability of the classification model.

[0157] In order to further verify the effect of each part of the present invention on the experimental results, the following ablation study was conducted, which is divided into four cases: (1) using the residual neural network alone; (2) using the residual neural network plus a multi-scale feature module; (3) using the residual neural network plus semi-supervised learning; (4) using the residual neural network plus a multi-feature extraction module plus semi-supervised learning (the auxiliary diagnosis method in this embodiment).

[0158] like Figure 6 As shown in the figure, the residual neural network with the multi-feature extraction module is better than the original residual network, indicating that the multi-scale feature extraction module can enable the network to extract multi-scale image features of different depths and improve the performance of the model. The use of residual neural network, semi-supervised learning and multi-scale feature extraction module at the same time achieves the best effect, so the auxiliary diagnosis method in this embodiment has better accuracy and effectiveness.

[0159] The auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion provided by the present invention is described below. The auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion described below and the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion described above can be referenced to each other.

[0160] like Figure 7 As shown, this embodiment provides an auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion, the device comprising: a feature extraction module 701 and an auxiliary diagnosis module 702, wherein:

[0161] The feature extraction module 701 is used to input the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed;

[0162] Optionally, the classification model is trained first, and the specific training steps include:

[0163] Acquire a first training data set, wherein the first training data set includes a small number of labeled first sample CT images and a large number of unlabeled second sample CT images;

[0164] Then, directly based on the first training data set, semi-supervised learning training is performed on the classification model to update parameter information of the classification model;

[0165] Alternatively, after preprocessing the first training data set, such as data enhancement and data mixing, the classification model is trained through semi-supervised learning based on the preprocessed first training data set to obtain a classification model that can quickly and accurately output auxiliary medical diagnosis results. This embodiment does not specifically limit the specific training method of the classification model.

[0166] Since semi-supervised learning is a learning method that combines supervised learning with unsupervised learning, it can use a small amount of labeled data and a large amount of unlabeled data to alleviate the problem of insufficient data. Specifically, semi-supervised learning uses a large amount of unlabeled data and a small amount of labeled data at the same time to quickly train a classification model that can accurately output auxiliary medical diagnosis results. While effectively reducing the need for labeled data, it can improve the performance of the classification model and make the classification results more accurate.

[0167] After obtaining the trained classification model, the CT image to be diagnosed can be input into the multi-scale feature fusion module of the classification model, and at least one set of multi-scale feature extraction and multi-scale feature fusion operations can be performed on the CT image to be diagnosed to obtain the multi-scale fusion features of the CT image to be diagnosed.

[0168] The auxiliary diagnosis module 702 is used to input the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed; wherein, the classification model is obtained by semi-supervised learning training based on a training data set; the training data set includes a first sample CT image with a label and a second sample CT image without a label.

[0169] Optionally, after obtaining the multi-scale fusion features of the CT image to be diagnosed, the multi-scale fusion features can be learned based on the classification module to predict the auxiliary medical diagnosis results of the CT image to be diagnosed.

[0170] This embodiment uses a small amount of labeled first sample CT images and a large amount of unlabeled second sample CT images as a training data set, so that the training data set contains a large number of rich samples, and uses the training data set containing rich samples to perform semi-supervised training on the classification model, which can quickly train to obtain a classification model that can accurately output auxiliary medical diagnosis results. While effectively reducing the demand for labeled data volume, the performance of the classification model can be improved, making the auxiliary medical diagnosis results more accurate; in addition, by performing multi-scale extraction and fusion on the CT image to be diagnosed, the features of each scale in the CT image to be diagnosed that are related to the auxiliary medical diagnosis results can be retained, further improving the auxiliary medical diagnosis results.

[0171] On the basis of the above embodiment, the present embodiment also includes a training module, which is used to: based on the first training data set, perform semi-supervised learning training on the initial model of the classification model to obtain a pre-trained model of the classification model; input the second sample CT image in the first training data set into the pre-trained model to obtain a simulation label corresponding to the auxiliary medical diagnosis result of the second sample CT image; add the simulation label of the second sample CT image to the first training data set; perform image mixing on the sample CT images in the first training data set with the simulation label added to obtain a second training data set; and train the pre-trained model based on the second training data set to obtain the classification model.

[0172] On the basis of the above-mentioned embodiment, the training module in this embodiment includes a mixing module, which is used to: perform data enhancement on the sample CT images in the first training data set to which the simulation label is added to obtain the enhanced first training data set; process the enhanced first training data set to obtain multiple groups of sample CT image pairs; perform the following operations for each sample CT image pair in the multiple groups of sample CT image pairs: calculate the class activation map of any sample CT image in the current sample CT image pair; determine the target area with the highest contribution to the auxiliary medical diagnosis result in any sample CT image according to the class activation map; paste the target area to the corresponding position of another sample CT image in the current sample CT image pair to obtain the third sample CT image corresponding to the current sample CT image pair; obtain the first label of the third sample CT image according to the size of the target area, the label of any sample CT image, and the label of the other sample CT image; construct the second training sample set according to the third sample CT images corresponding to all sample CT image pairs and the first label of the third sample CT image.

[0173] Based on the above embodiments, the data enhancement in this embodiment includes one or more combinations of standardization, geometric transformation, random adjustment of brightness and random adjustment of contrast; the geometric transformation includes one or more combinations of translation, flipping and rotation.

[0174] On the basis of the above-mentioned embodiment, the loss function of the classification model in this embodiment is jointly constructed and generated based on a first loss function and a second loss function; wherein, the first loss function aims to minimize the similarity between the multi-scale fusion features of the third sample CT image and the other sample CT image; the second loss function aims to minimize the difference between the first label and the second label of the third sample CT image; the second label is obtained by predicting the auxiliary medical diagnosis result of the third sample CT image based on the classification model.

[0175] On the basis of the above embodiments, the training module in this embodiment is further used to: input the second sample CT image into the pre-training model to obtain the probability that the second sample CT image belongs to each preset medical diagnosis category; perform entropy minimization processing on the probability that the second sample CT image belongs to each preset medical diagnosis category to obtain the minimized entropy processing result of the second sample CT image; and obtain the simulated label of the second sample CT image according to the minimized entropy processing result.

[0176] On the basis of the above embodiments, the multi-scale feature fusion module in this embodiment includes an identity mapping module; wherein the identity mapping module includes a first feature extraction unit, a feature fusion unit and a second feature extraction unit; the first feature extraction unit is used to perform multi-scale feature extraction on the CT image to be diagnosed to obtain feature maps of multiple different scales of the CT image to be diagnosed; the feature fusion unit is used to fuse the feature maps of multiple different scales to obtain a fused feature map of the CT image to be diagnosed; the second feature extraction unit is generated based on a non-local neural network, and is used to perform non-local operations on the fused feature map to obtain multi-scale fusion features of the CT image to be diagnosed.

[0177] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 communicate with each other through the communication bus 804. The processor 801 may call the logic instructions in the memory 803 to execute an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion, the method comprising: inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed; inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed; wherein the classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a first sample CT image with a label and a second sample CT image without a label.

[0178] In addition, the logic instructions in the above-mentioned memory 803 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0179] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the above methods, and the method includes: inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed; inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed; wherein the classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a first sample CT image with a label and a second sample CT image without a label.

[0180] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion provided by the above-mentioned methods, the method comprising: inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed; inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed; wherein the classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set comprises a first sample CT image with a label and a second sample CT image without a label.

[0181] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion, characterized in that: include: Inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed; Inputting the multi-scale fusion features into the classification module of the classification model to obtain the auxiliary medical diagnosis result of the CT image to be diagnosed; The classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a labeled first sample CT image and an unlabeled second sample CT image; The loss function of the classification model is generated based on the joint construction of the first loss function and the second loss function; Wherein, the first loss function aims to minimize the similarity of the multi-scale fusion features of the third sample CT image and another sample CT image; The second loss function aims to minimize the difference between the first label and the second label of the third sample CT image; the second label is obtained by predicting the auxiliary medical diagnosis result of the third sample CT image based on the classification model; The third sample CT image and the another sample CT image are acquired based on the following steps: Performing data enhancement on the sample CT images in the first training data set to which the simulation labels are added, to obtain an enhanced first training data set; Processing the enhanced first training data set to obtain multiple groups of sample CT image pairs; For each sample CT image pair in the plurality of groups of sample CT image pairs, the following operations are performed: Calculate the class activation map of any sample CT image in the current sample CT image pair; Determining, according to the class activation map, a target region in any sample CT image that has the highest contribution to an auxiliary medical diagnosis result; Paste the target area to a corresponding position of another sample CT image in the current sample CT image pair to obtain the third sample CT image corresponding to the current sample CT image pair; The classification model is trained based on the following steps: Based on the first training data set, performing semi-supervised learning training on the initial model of the classification model to obtain a pre-trained model of the classification model; Inputting a second sample CT image in the first training data set into the pre-training model to obtain a simulation label corresponding to an auxiliary medical diagnosis result of the second sample CT image; adding the simulated label of the second sample CT image to the first training data set; Performing image mixing on the sample CT images in the first training data set to which the simulation labels are added, to obtain a second training data set; Based on the second training data set, the pre-training model is trained to obtain the classification model.

2. The auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion according to claim 1, characterized in that: The step of performing image mixing on the sample CT images in the first training data set to which the simulation labels are added to obtain a second training data set comprises: Obtaining a first label of the third sample CT image according to the size of the target area, the label of any one of the sample CT images, and the label of the other sample CT image; A second training sample set is constructed according to the third sample CT images corresponding to all the sample CT image pairs and the first labels of the third sample CT images.

3. The auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion according to claim 2, characterized in that: The data enhancement includes one or more combinations of standardization, geometric transformation, random brightness adjustment, and random contrast adjustment; The geometric transformation includes one or more combinations of translation, flipping and rotation.

4. The auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: The step of inputting the second sample CT image in the first training data set into the pre-training model to obtain a simulation label corresponding to the auxiliary medical diagnosis result of the second sample CT image includes: Inputting the second sample CT image into the pre-trained model to obtain the probability that the second sample CT image belongs to each preset medical diagnosis category; Performing entropy minimization processing on the probability that the second sample CT image belongs to each preset medical diagnosis category to obtain a result of entropy minimization processing of the second sample CT image; According to the minimization entropy processing result, a simulation label of the second sample CT image is obtained.

5. The auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion according to any one of claims 1 to 3, characterized in that: The multi-scale feature fusion module includes an identity mapping module; Wherein, the identity mapping module includes a first feature extraction unit, a feature fusion unit and a second feature extraction unit; The first feature extraction unit is used to perform multi-scale feature extraction on the CT image to be diagnosed, so as to obtain feature maps of multiple different scales of the CT image to be diagnosed; The feature fusion unit is used to fuse the multiple feature maps of different scales to obtain a fused feature map of the CT image to be diagnosed; The second feature extraction unit is generated based on a non-local neural network and is used to perform a non-local operation on the fused feature map to obtain a multi-scale fused feature of the CT image to be diagnosed.

6. An auxiliary diagnosis device based on semi-supervised learning and multi-scale feature fusion, characterized in that: include: A feature extraction module, used for inputting the CT image to be diagnosed into the multi-scale feature fusion module of the classification model to obtain the multi-scale fusion features of the CT image to be diagnosed; An auxiliary diagnosis module, used for inputting the multi-scale fusion features into the classification module of the classification model to obtain an auxiliary medical diagnosis result of the CT image to be diagnosed; The classification model is obtained by semi-supervised learning training based on a first training data set; the first training data set includes a labeled first sample CT image and an unlabeled second sample CT image; The loss function of the classification model is generated based on the joint construction of the first loss function and the second loss function; Wherein, the first loss function aims to minimize the similarity of the multi-scale fusion features of the third sample CT image and another sample CT image; The second loss function aims to minimize the difference between the first label and the second label of the third sample CT image; the second label is obtained by predicting the auxiliary medical diagnosis result of the third sample CT image based on the classification model; The third sample CT image and the another sample CT image are acquired based on the following steps: Performing data enhancement on the sample CT images in the first training data set to which the simulation labels are added, to obtain an enhanced first training data set; Processing the enhanced first training data set to obtain multiple groups of sample CT image pairs; For each sample CT image pair in the plurality of groups of sample CT image pairs, the following operations are performed: Calculate the class activation map of any sample CT image in the current sample CT image pair; Determining, according to the class activation map, a target region in any sample CT image that has the highest contribution to an auxiliary medical diagnosis result; Paste the target area to a corresponding position of another sample CT image in the current sample CT image pair to obtain the third sample CT image corresponding to the current sample CT image pair; The classification model is trained based on the following steps: Based on the first training data set, performing semi-supervised learning training on the initial model of the classification model to obtain a pre-trained model of the classification model; Inputting a second sample CT image in the first training data set into the pre-training model to obtain a simulation label corresponding to an auxiliary medical diagnosis result of the second sample CT image; adding the simulated label of the second sample CT image to the first training data set; Performing image mixing on the sample CT images in the first training data set to which the simulation labels are added, to obtain a second training data set; Based on the second training data set, the pre-training model is trained to obtain the classification model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, an auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the auxiliary diagnosis method based on semi-supervised learning and multi-scale feature fusion as described in any one of claims 1 to 5 is implemented.

Images