Mixed self-supervised learning method for malignant degree of single pulmonary nodule based on CT image

By adopting a hybrid self-supervised learning method in medical images, combining encoder and momentum encoder for feature extraction and pixel reconstruction, the difficulty of feature extraction caused by differences in natural images and medical imaging is solved, and the pulmonary granulomatous nodules and Classification accuracy of solid lung adenocarcinoma.

CN119027369AActive Publication Date: 2024-11-26NORTHEASTERN UNIV CHINA

Patent Information

Application Number
CN202410935205.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-26
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

Existing self-supervised learning methods are difficult to effectively distinguish between pulmonary granulomatous nodules and solid lung adenocarcinoma in medical image classification, especially due to differences in natural images and medical imaging, which lead to potential loss of feature extraction and pathological features.

Method used

Using a hybrid self-supervised learning method based on CT images, the horizontal plane and coronal plane images of lung lesions were obtained, and feature extraction and pixel reconstruction were performed in combination with encoder, momentum encoder and decoder, loss values ​​were calculated and jointly iterative training was carried out to construct Classification model of pulmonary granulomatous nodules and solid lung adenocarcinoma.

Benefits of technology

The classification model identification ability of lung granulomatous nodules and solid lung adenocarcinoma is enhanced, and the comprehensive context information in medical imaging is fully utilized, ensuring the thoroughness of data representation and the maximum sample information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027369A_ABST
    Figure CN119027369A_ABST
Patent Text Reader

Abstract

The invention discloses a hybrid self-supervised learning method for the malignancy degree of a single pulmonary nodule based on a CT image. The method comprises the following steps: inputting a horizontal plane image of a lung focus into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung focus; inputting the coronal plane image of the lung focus into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung focus, and storing the coronal plane feature vector into a dynamic dictionary; based on the horizontal plane feature vector and a decoder, carrying out pixel reconstruction on the covered part in the horizontal plane image to obtain a target image; calculating a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image; and according to the loss value, performing common iterative training on the encoder, the momentum encoder and the decoder until a preset condition is met, and constructing a preset lung granuloma nodule and solid lung adenocarcinoma classification model. The application can distinguish single atypical pulmonary granuloma nodules from solid lung adenocarcinoma.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of self-supervised learning, and in particular to a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images. Background Art

[0002] Lung cancer is the most lethal disease among all cancers in the world, posing a serious threat to human life and health. Pulmonary granulomatous nodules (PGN) with spiculated or lobed appearance often resemble solid lung adenocarcinoma (SLA) in CT morphology, making it difficult to distinguish.

[0003] Computer-aided diagnosis (CAD), especially enhanced by deep learning techniques, provides a novel approach to diagnosing lung diseases, such as differentiating solitary granulomas from adenocarcinomas. However, a significant limitation when using deep learning in diagnosis is that training neural networks requires a large amount of accurately annotated medical data. Acquiring this data requires a lot of resources, including time and money.

[0004] Currently, the emergence of self-supervised learning is considered a potential solution to the challenge of finely annotated medical data. Unlike traditional supervised learning, self-supervised learning utilizes unlabeled data and derives proxy tasks from the data itself to train models without manual annotation. However, despite the key progress made in self-supervised learning, the differences between natural images and medical imaging modalities still make the application of self-supervised learning in the medical field face significant challenges. Medical images mainly come from radiological, functional, magnetic resonance, and ultrasound imaging modalities, while natural images are mainly captured by ambient light. This difference highlights the significant differences in application and algorithm design. One notable difference lies in the dimensionality and nature of the images. Many medical images are 3D single-channel grayscale representations, while natural images are mainly 2D color images. This difference may lead to the potential loss of important pathological features when deep learning architectures originally designed for natural images are applied to medical data. In addition, the high similarity between medical images of the same anatomical region or similar physiological state stems from the intrinsic similarity of human tissue. In natural images, subtle structural changes may be considered insignificant, while these subtle changes in medical images may be indicative of pathological changes. Given the unique challenges posed by the differences between natural images and medical imaging, traditional neural networks often struggle to classify pulmonary granulomatous nodules and solid lung adenocarcinomas. Summary of the invention

[0005] In view of this, the present application provides a hybrid self-supervised learning method for the malignancy of a single lung nodule based on CT images. The main purpose is to distinguish between a single atypical pulmonary granulomatous nodule and solid lung adenocarcinoma before surgery through CT images and self-supervised learning methods.

[0006] According to the first aspect of the present application, a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images is provided, the method comprising:

[0007] Acquire horizontal and coronal images of lung lesions;

[0008] Inputting the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion;

[0009] Inputting the coronal plane image into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung lesion, and storing the coronal plane feature vector in a dynamic dictionary;

[0010] Based on the horizontal plane feature vector and the decoder, reconstruct pixels of the masked part in the horizontal plane image to obtain a target image;

[0011] Calculating a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image;

[0012] According to the loss value, the encoder, the momentum encoder and the decoder are jointly iteratively trained until preset conditions are met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0013] According to a second aspect of the present application, a hybrid self-supervised learning device for malignancy of a single pulmonary nodule based on CT images is provided, the device comprising:

[0014] An acquisition unit, used for acquiring horizontal plane images and coronal plane images of lung lesions;

[0015] A first extraction unit, configured to input the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion;

[0016] A second extraction unit is used to input the coronal plane image into a momentum encoder for feature extraction, obtain a coronal plane feature vector of the lung lesion, and store the coronal plane feature vector in a dynamic dictionary;

[0017] A reconstruction unit, configured to reconstruct pixels of a masked portion of the horizontal plane image based on the horizontal plane feature vector and a decoder to obtain a target image;

[0018] a calculation unit, configured to calculate a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary, and the target image;

[0019] A training unit is used to jointly iteratively train the encoder, the momentum encoder and the decoder according to the loss value until a preset condition is met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0020] According to a third aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the hybrid self-supervised learning method for the malignancy degree of a single pulmonary nodule based on CT images is implemented.

[0021] According to the fourth aspect of the present application, an electronic device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the program, the hybrid self-supervised learning method for the malignancy degree of a single pulmonary nodule based on CT images is implemented.

[0022] By means of the above technical solution, the present application provides a hybrid self-supervised learning method for the malignancy of a single lung nodule based on CT images. By training the encoder, momentum encoder and decoder, the method successfully integrates the technical advantages of contrastive self-supervised learning and generative self-supervised learning. This synthesis enables the model to not only extract features from medical images from a global perspective, but also to carefully examine unique image areas, thereby enhancing the discrimination ability of the preset pulmonary granulomatous nodules and solid lung adenocarcinoma classification models. The present application innovatively proposes a breakthrough self-supervised learning method that makes full use of the comprehensive contextual information present in medical imaging. Compared with traditional 2D medical imaging methods, the present application promotes multi-faceted lesion feature extraction, ensures the thoroughness of data representation and maximizes the information content of the sample.

[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A flow chart of a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images provided in an embodiment of the present application is shown;

[0026] Figure 2 Shown are samples of lesion images (CT and pathology) of PGN and SLA in Dataset 1 and Dataset 2 provided in the embodiments of the present application, as well as schematic diagrams of excluded lesion images;

[0027] Figure 3 A schematic diagram of the MCMAE-NET network architecture provided in an embodiment of the present application is shown;

[0028] Figure 4 A flow chart of another hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images provided in an embodiment of the present application is shown;

[0029] Figure 5 The MCMAE-NET network structure and detailed parameter diagram provided in the embodiment of the present application are shown;

[0030] Figure 6 A structural schematic diagram of a hybrid self-supervised learning device for the malignancy of a single pulmonary nodule based on CT images provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0032] Given the unique challenges posed by the differences between natural images and medical imaging, traditional neural networks often struggle to classify pulmonary granulomatous nodules and solid lung adenocarcinomas.

[0033] To solve the above problems, an embodiment of the present invention proposes an innovative self-supervised learning network MCMAE-NET, which is designed for medical image classification, especially focusing on distinguishing pulmonary granulomatous nodules and solid lung adenocarcinomas in CT scans of lung cancer patients. The architecture of MCMAE-NET integrates several key components to achieve its excellent results. First and foremost, the embodiment of the present invention curates sample pairs by extracting regions of interest (ROIs) from the horizontal and coronal planes of each case, thereby ensuring a comprehensive and diverse dataset. Subsequently, the MCMAE-NET model, as a hybrid of self-supervised learning strategies, effectively combines the advantages of contrastive and generative self-supervised learning. This integration enables MCMAE-NET to exploit the ability to distinguish feature differences between instances, while also emphasizing feature changes within instances.

[0034] Based on the above inventive concept, the embodiment of the present invention provides a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images, such as Figure 1As shown, the method includes:

[0035] Step 101: Acquire a horizontal plane image and a coronal plane image of a lung lesion.

[0036] The horizontal plane image is formed by stitching the horizontal plane slices, and the coronal plane image is formed by stitching the coronal plane slices. The slices in the horizontal plane image and the coronal plane image are randomly covered.

[0037] For the embodiment of the present invention, when constructing the sample training set, it is necessary to pre-set the inclusion criteria and exclusion criteria of the sample data. Among them, the inclusion criteria include: 1. Patients with pathological diagnosis of pulmonary granulomatous lesions (tuberculosis or fungal granuloma) and adenocarcinoma confirmed by surgical resection or image-guided biopsy; 2. All patients underwent conventional and enhanced CT scans using the same CT machine and standardized reconstruction parameters within two weeks after surgery; 3. Isolated solid SPNs, ranging in size from 7 to 30 mm, without calcification or fat components, showing spinous processes, lobes or pleural indentations, and no associated atelectasis or lymphadenopathy; 4. Laboratory analysis of conventional tumor markers (CEA, CA125, CA153) performed within one week before surgery, with positive thresholds set to >5ng / ml, >35ng / ml and >25ng / ml, respectively, according to the reference range of the embodiment of the present invention. Exclusion criteria include: 1. Nodules with features highly suggestive of benign lesions, such as caseous necrosis with cavitation of tuberculoma or characteristic halo sign of fungal granuloma; 2. Individuals with a history of other malignancies or concurrent malignancies; 3. Cases imaged using different algorithms, different slice thicknesses, or different CT machines. Some typical examples (including CT and pathological images) are shown in Figure 2 .

[0038] Among these cases, 333 cases, including 105 PGN and 228 SLA, were collected from a certain hospital. This specific dataset was designated as Dataset 1 and served as the basis for model training, testing, and validation. In addition, an additional 161 cases, including 67 PGN and 94 SLA, were collected from other hospitals. This external dataset was referred to as Dataset 2 and was used for external model validation to evaluate the generalization ability of the model. CT images of all patients were acquired using a multi-detector CT system (AS+128-Slice; Siemens Healthineers, Germany). For Dataset 1, the CT scanning parameters were set as follows: tube voltage was 120 kVp, tube current averaged 299.88 mAs with a standard deviation of ±134.62 mAs, and average slice thickness was 2.31 mm with a standard deviation of ±0.71 mm. For dataset 2, the parameters included a tube voltage of 120 kVp, a tube current of an average of 194.31 mAs with a standard deviation of ±116.13 mAs, and a mean slice thickness of 4.03 mm with a standard deviation of ±1.57 mm. All images of these datasets were exported in DICOM format and subsequently used for image feature extraction of the MCMAE-NET model.

[0039] The lesion size in the embodiment of the present invention is defined as the maximum diameter of the tumor in the axial image. The spinous process is defined as a linear or pointed extension emanating from the edge of a nodule or mass that extends into the lung parenchyma without contacting the pleural surface. The lobular feature is characterized by a wavy, lobed structure of a portion of the lesion surface, excluding areas adjacent to the pleura. Pleural indentations are defined as linear structures extending from the tumor to the pleural surface. These morphological assessments provide important information for the subsequent analysis and classification of PGN and SLA. The clinical features of Dataset 1 are summarized in Table 1 in the embodiment of the present invention.

[0040] Table 1

[0041]

[0042] Among them, * indicates that the result is significant. a and b represent two-sample t test and chi-square test respectively.

[0043] Furthermore, in order to ensure the quality and consistency of the dataset during training and evaluation, the embodiments of the present invention use a comprehensive series of data processing and enhancement techniques. The preprocessing of CT images involves key steps such as standardizing resolution, reducing noise, and enhancing contrast. These measures are essential to ensure the consistency of the dataset and improve the quality of the input data, which is crucial for the subsequent analysis stage.

[0044] First, given the differences in slice thickness between different scans, the preliminary steps included resampling all CT images to a uniform slice thickness of 1 mm using trilinear interpolation. This meticulous process ensured that all images conformed to a consistent format, facilitating subsequent seamless analysis. In addition, a key aspect in the preprocessing process is intensity normalization. CT images were intensity normalized to mitigate image intensity variations caused by differences in acquisition settings. This calibration involves standardizing Hounsfield units (HU) to establish a uniform intensity scale in all images. Therefore, all CT data were uniformly set to a window width of 1400HU and a window level of -500HU. In addition, in order to reduce interference from surrounding information and focus on the lesion, the embodiment of the present invention selected the center point of each lesion as an anchor point. The embodiment of the present invention then extracted a square ROI area with a side length of 50 mm, which exceeded the maximum diameter of the tumor. Multiple slices containing tumor lesions were obtained from the horizontal and coronal planes, respectively. Subsequently, the slices from the horizontal plane are arranged in order and spliced ​​to obtain a horizontal plane spliced ​​image. Similarly, the slices from the coronal plane are arranged in order and spliced ​​to obtain a coronal plane spliced ​​image. Finally, based on the horizontal plane spliced ​​images and coronal plane spliced ​​images of the same lesion, sample pairs are generated as the training data set of the model, and the training data set includes multiple sample pairs of different lesions. When training the model, the horizontal plane spliced ​​images and coronal plane spliced ​​images in the training data set are randomly masked to obtain the horizontal plane image and coronal plane image of the lesion.

[0045] Based on this, step 101 specifically includes: obtaining horizontal plane slices and coronal plane slices of the lung lesion; stitching the horizontal plane slices and the coronal plane slices respectively to obtain a horizontal plane stitched image and a coronal plane stitched image; randomly masking the slices in the horizontal plane stitched image and the coronal plane stitched image respectively to obtain the horizontal plane image and the coronal plane image.

[0046] For example, 25 slices containing tumor lesions are obtained from the horizontal plane and the coronal plane, respectively. Subsequently, these slices from the horizontal plane and the coronal plane are arranged in order to generate two groups of 5x5 image slices, and then each group of image slices is spliced ​​and randomly masked to obtain the horizontal plane image X q and coronal image X k .

[0047] In order to mitigate the risk of overfitting during model training, the embodiment of the present invention uses a variety of data enhancement techniques to expand the data set. This method successfully improves the diversity of the data set, thereby improving the generalization ability of the model. In the embodiment of the present invention, specific enhancement techniques such as rotation, horizontal flipping, and four-way translation centered on the central anchor point are implemented to generate additional valuable data. The combination of these techniques significantly increases the size of the data set, and a total of 13,320 pairs of samples are generated in data set 1. Subsequently, data set 1 was used for self-supervised pre-training of the MCMAE-NET model. In the fine-tuning stage of downstream tasks, the unexpanded original data set is used for training, and the data is divided into training and test sets in a ratio of 8:2, and a five-fold cross validation is performed. Dataset 2 is used as an external validation set to independently evaluate the performance of the model.

[0048] Figure 3 The overall architecture of the self-supervised learning network MCMAE-NET is innovatively combined with generative and contrastive self-supervised learning. It mainly includes three components, namely, data processing module, generative self-supervised learning task module and contrastive self-supervised learning task module. Among them, the data processing module is used to splice the horizontal slices and coronal slices respectively to obtain horizontal plane splicing images and coronal plane splicing images. The horizontal plane splicing images and coronal plane splicing images can be used as input images of the MCMAE-NET network after random masking; the generative self-supervised learning task module includes encoder Encoder and decoder Decoder; the contrastive self-supervised learning task module includes momentum encoder MomentumEncoder. The above modules work together to greatly improve the accuracy and efficiency of the classification of pulmonary granulomatous nodules PGN and solid lung adenocarcinoma SLA in CT images.

[0049] Step 102: Input the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion.

[0050] like Figure 3 As shown, after initializing the encoder, the horizontal image X q is input into the encoder for feature extraction to obtain the horizontal plane feature vector q of the lung lesion.

[0051] Step 103: input the coronal plane image into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung lesion, and store the coronal plane feature vector in a dynamic dictionary.

[0052] The dynamic dictionary stores coronal plane feature vectors of different lesions.

[0053] like Figure 3As shown, after initializing the momentum encoder, the coronal image X k The data is input into the momentum encoder for feature extraction to obtain the coronal plane feature vector k of the lung lesion, which is then stored in a dynamic dictionary. The coronal plane feature vectors of different lesions are all stored in the dynamic dictionary.

[0054] Step 104: Based on the horizontal plane feature vector and the decoder, reconstruct pixels of the masked part in the horizontal plane image to obtain a target image.

[0055] According to the embodiment of the present invention, the mask token of the masked slice in the horizontal plane image is combined with the horizontal plane feature vector and then input into the decoder, and the masked slice is pixel-reconstructed to obtain the target image Target.

[0056] Step 105: Calculate a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image.

[0057] For the embodiment of the present invention, in order to calculate the loss value, step 105 specifically includes: calculating the similarity loss based on the horizontal plane feature vector and each coronal plane feature vector in the dynamic dictionary; calculating the pixel loss based on the target image and the horizontal plane stitched image; based on a preset temperature coefficient, adding the similarity loss and the pixel loss to obtain the loss value.

[0058] Among them, the preset temperature coefficient is used to control the concentration level of the distribution and can be set according to actual business needs.

[0059] Specifically, the similarity between the horizontal plane feature vector and the coronal plane feature vector from the same lesion is high, while the similarity between the horizontal plane feature vector and the coronal plane feature vector from different lesions is low. Based on this, a similarity loss function can be constructed to calculate the similarity loss. At the same time, the pixel difference between the reconstructed target image and the horizontal plane stitching image is calculated to obtain the pixel loss. Then, based on the preset temperature coefficient, the similarity loss and the pixel loss are added to obtain the current loss value.

[0060] Step 106: According to the loss value, the encoder, the momentum encoder and the decoder are jointly iteratively trained until preset conditions are met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0061] For the embodiment of the present invention, when the MCMAE-NET network is iteratively trained, step 106 specifically includes: according to the loss value, updating the parameters of the encoder and the decoder to obtain an updated encoder and an updated decoder; based on the parameters in the updated encoder, updating the parameters in the momentum encoder to obtain an updated momentum encoder; repeating the iterative update process of the encoder, the momentum encoder and the decoder until a preset number of iterations is reached, and outputting the final trained encoder; based on the final trained encoder and the fully connected layer, constructing the preset pulmonary granulomatous nodule and solid lung adenocarcinoma classification model.

[0062] Specifically, the momentum encoder performs a momentum update operation during the update process, that is, the parameters in the momentum encoder change with the changes in the encoder. After updating the encoder, the momentum encoder is modified using part of its updated parameters, such as using 1% of the encoder update parameters to modify the momentum encoder. This ensures a slow and controlled update of the momentum encoder. When the loss value is minimized or reaches the preset number of iterations, the iterative training is stopped, and the final trained encoder is output. After that, a fully connected layer is connected behind the trained encoder for task classification. Finally, the trained encoder and the fully connected layer are fine-tuned to obtain the preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0063] The embodiment of the present invention provides a hybrid self-supervised learning method for the malignancy of a single lung nodule based on CT images. By training the encoder, momentum encoder and decoder, the method successfully integrates the technical advantages of contrastive self-supervised learning and generative self-supervised learning. This synthesis enables the model to not only extract features from medical images from a global perspective, but also to carefully examine unique image areas, thereby enhancing the discrimination ability of the preset classification models for granulomatous lung nodules and solid lung adenocarcinomas. The embodiment of the present invention innovatively proposes a breakthrough self-supervised learning method that makes full use of the comprehensive contextual information present in medical imaging. Compared with traditional 2D medical imaging methods, the embodiment of the present invention promotes the extraction of multi-faceted lesion features, ensures the thoroughness of data representation and maximizes the information content of the sample.

[0064] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the implementation of this embodiment, this embodiment also provides another hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images, such as Figure 4 As shown, the method includes:

[0065] Step 201: Acquire a horizontal plane image and a coronal plane image of a lung lesion.

[0066] In the field of self-supervised learning, embodiments of the present invention study the synergy of contrastive self-supervised learning and generative self-supervised learning. Contrastive self-supervised learning, as the name implies, reveals unique image features by contrasting positive and negative examples in a high-dimensional space. On the other hand, generative self-supervised learning exploits the power of image reconstruction to obtain valuable image information. It can be seen that the masked autoencoder (MAE) tends to take a global perspective when considering images, while the momentum contrast (MoCo) method tends to review unique image regions and strategically locate positive and negative examples.

[0067] MoCo is a model based on contrastive self-supervised learning that performs well in various visual tasks. It adopts a unique strategy involving dynamic dictionaries and momentum-based updates. This approach enables MoCo to effectively learn robust and unique features by contrasting positive and negative data samples. MAE is a model based on generative self-supervised learning that performs well in various visual tasks. It utilizes an asymmetric encoder-decoder architecture, and its success is mainly attributed to the use of Vanilla Vision Transformer, which is able to extract global features from the input data. This architecture enables MAE to effectively reconstruct the missing parts of the input and thus learn comprehensive representations.

[0068] In order to optimize the training of the model and enhance the capabilities of the Transformer, the embodiment of the present invention introduces hybrid self-supervised learning as a key intermediate task in MCMAE-NET. This strategic addition enables the Transformer to explore the unique characteristics of features and the overall image context.

[0069] The detailed network architecture of the MCMAE-NET model is shown in Figure 5 As shown, the encoder and momentum encoder adopt the ViT-large model, which features 24 stacked encoder blocks, 1024 token vector lengths, and 16 heads in the multi-head attention mechanism. These modules specialize in processing unmasked image slices. The following is an overall description of the image data processing flow of this scheme. First, the image data is flattened into a one-dimensional array. Subsequently, these arrays are linearly transformed and then merged with the position embedding of the original image and augmented by adding a class tag at the beginning. Due to the asymmetric architecture of the encoder-decoder, in which the encoder only processes unmasked slices to save computing resources, the decoder is configured with 8 stacked decoder blocks and a token vector length of 512. The task of the decoder is to process not only the unmasked tokens encoded by the encoder, but also the masked tokens. It should be noted that these masked tokens are not derived from the embedding transformation of the previous masked slices; instead, they are learnable and shared among all masked slices.

[0070] In traditional contrastive learning, the size of the dictionary is usually equal to the size of the mini-batch. However, this approach is often limited by the limitations of GPU memory and computing power, making it impossible to use large batch sizes. To address this limitation, the MoCo framework uses a queue-based mechanism to store dictionaries, which contain feature vectors of images, allowing the use of larger dictionary sizes. Nevertheless, in the case where the dataset size is not abnormally large, choosing a dictionary that is too large may hinder the convergence of the model. Therefore, an embodiment of the present invention chooses to set the dictionary size to 700, with a dimension of 128 for each vector. Maintenance of the queue involves enqueuing the feature vectors of the latest batch of images and dequeuing the feature vectors of the earliest batch.

[0071] In order to ensure that the keys in the queue remain consistent, it is necessary to gradually update the momentum encoder associated with the dictionary. This is achieved by implementing a momentum-based approach. Specifically, after updating the encoder, the momentum encoder is modified using only 1% of its updated parameters. This approach ensures a slow and controlled update of the momentum encoder, maintaining stability and consistency of the keys in the queue.

[0072] The above is the overall processing flow of image data. When performing model training, the process of obtaining horizontal plane images and coronal plane images is exactly the same as step 101, and will not be repeated here.

[0073] Step 202: Input the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion.

[0074] For the embodiment of the present invention, with respect to the process of extracting the horizontal plane feature vector by the encoder, step 202 specifically includes: calculating the position embedding value of the unmasked slice in the horizontal plane image; extracting the embedding vector of the horizontal plane image using the convolutional layer in the encoder; superimposing the position embedding value of the unmasked slice in the horizontal plane image with the embedding vector of the horizontal plane image to obtain the feature vector of the horizontal plane image; masking the feature vector of the horizontal plane image to obtain the masked feature vector of the horizontal plane image; adding the target embedding vector of the horizontal plane image, and concatenating the target embedding vector of the horizontal plane image with the masked feature vector of the horizontal plane image to obtain the target feature vector of the horizontal plane image; processing the target feature vector of the horizontal plane image using the converter model in the encoder to obtain the horizontal plane feature vector of the lung lesion.

[0075] Specifically, Figure 5As shown, the horizontal plane image is input into a two-dimensional convolution layer for feature extraction and array conversion to obtain an embedding vector of 25×1024. The embedding vector is then superimposed with the position embedding value of the unmasked slice to obtain a feature vector of 25×1024 for the horizontal plane image. The specific calculation formula for the position embedding value is as follows:

[0076]

[0077] Among them, PE(pos,2i) and PE(pos,2i+1) represent the position embedding values ​​of even (2i) and odd (2i+1) dimensions respectively, pos represents a specific position in the sequence, and d model is the embedding dimension of the model. Embodiments of the present invention enhance tag embedding by integrating position data, thereby achieving a more detailed and comprehensive representation of image spatial relationships.

[0078] Then, the feature vector 25×1024 of the horizontal plane image is masked to obtain the masked feature vector 12×1024, which is then concatenated with the target embedding vector Class Token to obtain the target feature vector 13×1024. Finally, the target feature vector 13×1024×L(24) is input into the transformer model Transformer Blocks for processing, and finally normalized to obtain the horizontal plane feature vector 13×1024.

[0079] Step 203: input the coronal plane image into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung lesion, and store the coronal plane feature vector in a dynamic dictionary.

[0080] For the embodiment of the present invention, with respect to the process of extracting the coronal plane feature vector by the momentum encoder, step 203 specifically includes: calculating the position embedding value of the unmasked slice in the coronal plane image; extracting the embedding vector of the coronal plane image by using the convolutional layer in the momentum encoder; superimposing the position embedding value of the unmasked slice in the coronal plane image with the embedding vector of the coronal plane image to obtain the feature vector of the coronal plane image; masking the feature vector of the coronal plane image to obtain the masked feature vector of the coronal plane image; adding the target embedding vector of the coronal plane image, concatenating the target embedding vector of the coronal plane image with the masked feature vector of the coronal plane image to obtain the target feature vector of the coronal plane image; processing the target feature vector of the coronal plane image by using the converter model in the momentum encoder to obtain the coronal plane feature vector of the lung lesion.

[0081] Specifically, Figure 5As shown, the coronal image is input into a two-dimensional convolution layer for feature extraction and array conversion to obtain an embedding vector of 25×1024, and then the embedding vector is superimposed with the position embedding value of the unmasked slice to obtain a feature vector of the coronal image of 25×1024. The specific calculation formula of the position embedding value is the same as the calculation formula in step 202.

[0082] Then, the feature vector 25×1024 of the coronal plane image is masked to obtain the masked feature vector 12×1024, which is then concatenated with the target embedding vector Class Token to obtain the target feature vector 13×1024. Finally, the target feature vector 13×1024×L(24) is input into the transformer model Transformer Blocks for processing and finally normalized to obtain the coronal plane feature vector 13×1024.

[0083] Step 204: Input the horizontal plane feature vector and the coronal plane feature vector into their respective one-dimensional convolutional layers for dimensionality reduction to obtain a reduced horizontal plane feature vector and a reduced coronal plane feature vector.

[0084] The embodiment of the present invention can reduce the dimension of the extracted lesion features by introducing a feature flattening module, namely a one-dimensional convolutional layer. This integration makes the combination of contrast and generative self-supervised learning technology more effective.

[0085] Step 205: Based on the reduced horizontal plane feature vector and the decoder, reconstruct pixels of the masked part in the horizontal plane image to obtain a target image.

[0086] For the embodiment of the present invention, with respect to the pixel reconstruction process, step 205 specifically includes: calculating the position embedding value of the masked slice in the horizontal plane image; performing dimensionality reduction processing on the horizontal plane feature vector using the fully connected layer of the decoder to obtain a reduced-dimensional horizontal plane feature vector; combining the mask token of the masked slice in the horizontal plane image with the reduced-dimensional horizontal plane feature vector to obtain a combined horizontal plane feature vector; superimposing the combined horizontal plane feature vector with the position embedding values ​​of the masked slice and the uncovered slice in the horizontal plane image to obtain a superimposed horizontal plane feature vector; and processing the superimposed horizontal plane feature vector using the converter model in the decoder to obtain the target image.

[0087] Specifically, Figure 5As shown in the figure, the horizontal plane feature vector is input into the fully connected layer for dimensionality reduction to obtain the reduced horizontal plane feature vector 13×512, which is combined with the mask token of the masked slice to obtain the combined horizontal plane feature vector 26×512, and then the combined horizontal plane feature vector is superimposed with the position embedding value to obtain the superimposed horizontal plane feature vector 26×512, and then the superimposed horizontal plane feature vector is input into the transformer model TransformerBlocks for processing, and finally after normalization, fully connected layer, elimination of target embedding vector and other processes, the target image 250×250×3 is obtained.

[0088] Step 206: Calculate the loss value based on the reduced horizontal plane feature vector, each reduced coronal plane feature vector in the dynamic dictionary and the target image.

[0089] In the embodiment of the present invention, the horizontal plane feature vector q and the coronal plane feature vector k are dimensionally reduced through their respective one-dimensional convolutional layers to generate reduced horizontal plane feature vectors q + and the reduced water coronal eigenvector k + It should be noted that k + It is found in a queue-like memory bank, which evolves with the introduction of new training batches, seamlessly replacing the oldest batches. Therefore, the embodiment of the present invention uses the InfoNCE loss function to measure q + and k + The similarity between them is used as the cornerstone of the comparison task. The formula of InfoNCE loss function is as follows:

[0090]

[0091] Among them, q + is the reduced horizontal plane feature vector, used as the query key, and is the reduced coronal plane feature vector in the dynamic dictionary, serving as the key in the dictionary. Among these, q + and are considered as positive pairs, and other keys are regarded as q + Negative sample pairs, With q + come from the same lesion. In addition, τ represents a temperature parameter that controls the concentration level of the distribution. By minimizing the infoNCE loss, the model is trained to effectively distinguish positive and negative sample pairs, thereby enhancing the ability to learn discriminative features from the data.

[0092] Furthermore, the decoder's task is to predict and reconstruct the pixel values ​​of the masked slice. Subsequently, the mean square error loss function MSE is used to quantify the difference between the predicted pixel value and the original pixel value. The formula of the MSE loss function is as follows:

[0093]

[0094] Where N is the total number of missing pixels in the original image. represents the true pixel value of the i-th pixel, Represents the pixel value predicted by the model. By minimizing the MSE loss, the model is trained to accurately reconstruct each pixel, thereby promoting the learning of global features of the image.

[0095] Finally, the InfoNCE loss function is combined with the MSE loss function using a specific weight coefficient λ as the loss function of the optimization model, through which the loss value can be calculated. The formula of the loss function is as follows:

[0096] L = λ × L MSE +(1-λ)×L InfoNCE

[0097] Among them, λ is the weight coefficient, which can be set according to actual business needs.

[0098] Step 207: According to the loss value, the encoder, the momentum encoder and the decoder are jointly iteratively trained until a preset condition is met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0099] In an embodiment of the present invention, the encoder and the momentum encoder have the same dimension. During the training process, the momentum encoder will perform a momentum update operation based on the update of the encoder, that is, the parameters of the momentum encoder change as the parameters of the encoder change. After the training is completed, a fully connected layer is connected behind the trained encoder for task classification, and finally the trained encoder and the fully connected layer are fine-tuned to obtain a preset classification model of pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0100] During specific training, the embodiment of the present invention used an NVIDIA GeForce RTX 4070 with 12GB of memory, and implemented the training and testing of the MCMAE-NET model in PyTorch (version 1.7), including all variants and comparison models. All graphics are created in Python using matplotlib. The network of the embodiment of the present invention is an evolution of MAE and MoCo, and the pre-trained model parameters cannot be directly loaded into MCMAE-NET, so training from scratch is required. Fortunately, after several iterations, the network parameters are finally optimally converged.

[0101] In the pre-training stage, the image size is 250×250 and the batch size is set to 64. The embodiment of the present invention uses the AdamW optimizer for optimization, sets betas=(0.9,0.95), sets the training cycle to 100, and the initial learning rate is 1×10-2. The learning rate is reduced by 0.1 at the 120th and 160th cycles. In the fine-tuning stage of the downstream task, the embodiment of the present invention utilizes the pre-trained encoder module, adds a two-class fully connected layer, keeps the same image size and batch size, trains for 100 cycles, and the initial learning rate is 1×10-3. The learning rate is reduced by 0.1 at the 40th and 70th cycles.

[0102] Furthermore, the embodiment of the present invention selects commonly used evaluation indicators to evaluate the performance of the model. Specifically, it includes the area under the curve (AUC) with 95% confidence interval (95% CI), accuracy (ACC), sensitivity (SEN) and specificity (SPE). AUC quantifies the ability of the model to distinguish categories.

[0103]

[0104] Among them, the true positive rate (TPR) and the false positive rate (FPR) vary with different thresholds t. The AUC is presented together with its 95% confidence interval, providing a statistical range indicating the interval where the true AUC value may be at a 95% confidence level. This metric enhances the interpretability and reliability of the AUC indicator.

[0105] ACC reflects the overall effectiveness of the model in correctly classifying positive and negative samples. SEN plays a key role in determining the model's ability to identify true positive samples, which is critical to ensure that no real cases are missed. In contrast, SPE evaluates the model's ability to accurately identify negative samples, which is critical to minimizing the occurrence of false positives.

[0106]

[0107] Among them, TP represents true positive examples, TN represents true negative examples, FP represents false positive examples, and FN represents false negative examples.

[0108] During actual classification, the horizontal plane image of the lesion to be predicted is directly obtained, and the horizontal plane image of the lesion to be predicted is input into a preset classification model of pulmonary granulomatous nodules and solid lung adenocarcinoma to distinguish whether the lesion belongs to a pulmonary granulomatous nodule or a solid lung adenocarcinoma.

[0109] Another hybrid self-supervised learning method for the malignancy of a single lung nodule based on CT images provided by an embodiment of the present invention successfully integrates the technical advantages of contrastive self-supervised learning and generative self-supervised learning by training the encoder, momentum encoder and decoder. This synthesis enables the model to not only extract features from medical images from a global perspective, but also to carefully examine unique image areas, thereby enhancing the discrimination ability of the preset pulmonary granulomatous nodules and solid lung adenocarcinoma classification models. The embodiment of the present invention innovatively proposes a breakthrough self-supervised learning method that makes full use of the comprehensive contextual information present in medical imaging. Compared with traditional 2D medical imaging methods, the embodiment of the present invention promotes multi-faceted lesion feature extraction, ensures the thoroughness of data representation and maximizes the information content of the sample.

[0110] Further, as Figure 1 and Figure 4 The specific implementation of the method shown in the embodiment provides a hybrid self-supervised learning device for the malignancy of a single pulmonary nodule based on CT images, such as Figure 6 As shown, the device includes: an acquisition unit 31, a first extraction unit 32, a second extraction unit 33, a reconstruction unit 34, a calculation unit 35 and a training unit 36.

[0111] The acquisition unit 31 can be used to acquire horizontal plane images and coronal plane images of lung lesions.

[0112] The first extraction unit 32 may be used to input the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion.

[0113] The second extraction unit 33 can be used to input the coronal plane image into the momentum encoder for feature extraction, obtain the coronal plane feature vector of the lung lesion, and store the coronal plane feature vector in a dynamic dictionary.

[0114] The reconstruction unit 34 may be configured to perform pixel reconstruction on the masked portion of the horizontal plane image based on the horizontal plane feature vector and the decoder to obtain a target image.

[0115] The calculation unit 35 may be configured to calculate a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary, and the target image.

[0116] The training unit 36 ​​can be used to jointly iteratively train the encoder, the momentum encoder and the decoder according to the loss value until a preset condition is met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

[0117] In some embodiments, the acquisition unit 31 can be specifically used to acquire horizontal plane slices and coronal plane slices of lung lesions; respectively splice the horizontal plane slices and the coronal plane slices to obtain a horizontal plane spliced ​​image and a coronal plane spliced ​​image; and respectively randomly mask the slices in the horizontal plane spliced ​​image and the coronal plane spliced ​​image to obtain the horizontal plane image and the coronal plane image.

[0118] In some embodiments, the first extraction unit 32 can be specifically used to calculate the position embedding value of the unmasked slice in the horizontal plane image; use the convolution layer in the encoder to extract the embedding vector of the horizontal plane image; superimpose the position embedding value of the unmasked slice in the horizontal plane image with the embedding vector of the horizontal plane image to obtain the feature vector of the horizontal plane image; mask the feature vector of the horizontal plane image to obtain the masked feature vector of the horizontal plane image; add the target embedding vector of the horizontal plane image, and concatenate the target embedding vector of the horizontal plane image with the masked feature vector of the horizontal plane image to obtain the target feature vector of the horizontal plane image; use the converter model in the encoder to process the target feature vector of the horizontal plane image to obtain the horizontal plane feature vector of the lung lesion.

[0119] In some embodiments, the second extraction unit 33 can be specifically used to calculate the position embedding value of the unmasked slice in the coronal image; use the convolution layer in the momentum encoder to extract the embedding vector of the coronal image; superimpose the position embedding value of the unmasked slice in the coronal image with the embedding vector of the coronal image to obtain the feature vector of the coronal image; mask the feature vector of the coronal image to obtain the masked feature vector of the coronal image; add the target embedding vector of the coronal image, and concatenate the target embedding vector of the coronal image with the masked feature vector of the coronal image to obtain the target feature vector of the coronal image; use the converter model in the momentum encoder to process the target feature vector of the coronal image to obtain the coronal feature vector of the lung lesion.

[0120] In some embodiments, the device further comprises: a dimensionality reduction unit.

[0121] The dimension reduction unit can be used to input the horizontal plane feature vector and the coronal plane feature vector into respective one-dimensional convolutional layers for dimensionality reduction, so as to obtain a reduced horizontal plane feature vector and a reduced coronal plane feature vector.

[0122] The reconstruction unit 34 may be specifically configured to perform pixel reconstruction on the masked portion of the horizontal plane image based on the reduced horizontal plane feature vector and the decoder to obtain a target image.

[0123] The calculation unit 35 may be specifically configured to calculate the loss value based on the reduced horizontal plane feature vector, each reduced coronal plane feature vector in the dynamic dictionary and the target image.

[0124] In some embodiments, the reconstruction unit 34 can also be specifically used to calculate the position embedding value of the masked slice in the horizontal plane image; use the fully connected layer of the decoder to reduce the dimension of the horizontal plane feature vector to obtain the reduced-dimensional horizontal plane feature vector; combine the mask token of the masked slice in the horizontal plane image with the reduced-dimensional horizontal plane feature vector to obtain a combined horizontal plane feature vector; superimpose the combined horizontal plane feature vector with the position embedding values ​​of the masked slice and the uncovered slice in the horizontal plane image to obtain a superimposed horizontal plane feature vector; use the converter model in the decoder to process the superimposed horizontal plane feature vector to obtain the target image.

[0125] In some embodiments, the calculation unit 35 can be specifically used to calculate the similarity loss based on the horizontal plane feature vector and each coronal plane feature vector in the dynamic dictionary; calculate the pixel loss based on the target image and the horizontal plane stitched image; based on a preset temperature coefficient, add the similarity loss and the pixel loss to obtain the loss value.

[0126] In some embodiments, the training unit 36 ​​can be specifically used to update the parameters of the encoder and the decoder according to the loss value to obtain an updated encoder and an updated decoder; based on the parameters in the updated encoder, update the parameters in the momentum encoder to obtain an updated momentum encoder; repeat the iterative update process of the encoder, the momentum encoder and the decoder until a preset number of iterations is reached, and output the final trained encoder; based on the final trained encoder and the fully connected layer, construct the preset pulmonary granulomatous nodule and solid lung adenocarcinoma classification model.

[0127] It should be noted that for other corresponding descriptions of the functional units involved in the hybrid self-supervised learning device for the malignancy of a single pulmonary nodule based on CT images provided in the embodiment of the present invention, reference can be made to Figure 1 and Figure 4 The corresponding description in will not be repeated here.

[0128] Based on the above Figure 1 and Figure 4The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 and Figure 5 The figure shows a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images.

[0129] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0130] Based on the above Figure 1 and Figure 4 The method shown, and Figure 6 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be a personal computer, a tablet computer, a server, or other network equipment, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 4 The figure shows a hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images.

[0131] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0132] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0133] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0135] The embodiment of the present invention successfully integrates the technical advantages of contrastive self-supervised learning and generative self-supervised learning by training the encoder, momentum encoder and decoder. This synthesis enables the model to not only extract features from medical images from a global perspective, but also to carefully examine unique image regions, thereby enhancing the discrimination ability of the preset pulmonary granulomatous nodules and solid lung adenocarcinoma classification models. The embodiment of the present invention innovatively proposes a breakthrough self-supervised learning method that fully utilizes the comprehensive contextual information present in medical imaging. Compared with traditional 2D medical imaging methods, the embodiment of the present invention promotes multi-faceted lesion feature extraction, ensures the thoroughness of data representation and maximizes the information content of the sample.

[0136] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0137] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A hybrid self-supervised learning method for the malignancy of a single pulmonary nodule based on CT images, characterized in that: include: Acquire horizontal and coronal images of lung lesions; Inputting the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion; Inputting the coronal plane image into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung lesion, and storing the coronal plane feature vector in a dynamic dictionary; Based on the horizontal plane feature vector and the decoder, reconstruct pixels of the masked part in the horizontal plane image to obtain a target image; Calculating a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image; According to the loss value, the encoder, the momentum encoder and the decoder are jointly iteratively trained until preset conditions are met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

2. The method according to claim 1, characterized in that Acquire transverse and coronal images of lung lesions, including: Obtain horizontal and coronal sections of lung lesions; Separately stitching the horizontal slices and the coronal slices to obtain a horizontal stitching image and a coronal stitching image; The slices in the horizontal plane stitched image and the coronal plane stitched image are respectively randomly masked to obtain the horizontal plane image and the coronal plane image.

3. The method according to claim 1, characterized in that The horizontal plane image is input into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion, including: Calculating the position embedding value of the unmasked slice in the horizontal plane image; extracting an embedding vector of the horizontal plane image using a convolutional layer in the encoder; Superimposing the position embedding value of the unmasked slice in the horizontal plane image with the embedding vector of the horizontal plane image to obtain a feature vector of the horizontal plane image; Masking the feature vector of the horizontal plane image to obtain the masked feature vector of the horizontal plane image; Adding a target embedding vector of the horizontal plane image, concatenating the target embedding vector of the horizontal plane image with the masked feature vector of the horizontal plane image, to obtain a target feature vector of the horizontal plane image; Processing the target feature vector of the horizontal plane image using the converter model in the encoder to obtain the horizontal plane feature vector of the lung lesion; and / or The coronal plane image is input into a momentum encoder for feature extraction to obtain a coronal plane feature vector of the lung lesion, including: Calculating the position embedding value of the unmasked slice in the coronal image; extracting an embedding vector of the coronal image using a convolutional layer in the momentum encoder; Superimposing the position embedding value of the unmasked slice in the coronal plane image with the embedding vector of the coronal plane image to obtain a feature vector of the coronal plane image; Masking the feature vector of the coronal plane image to obtain the masked feature vector of the coronal plane image; Adding the target embedding vector of the coronal plane image, concatenating the target embedding vector of the coronal plane image with the masked feature vector of the coronal plane image, to obtain the target feature vector of the coronal plane image; The target feature vector of the coronal plane image is processed using the converter model in the momentum encoder to obtain the coronal plane feature vector of the lung lesion.

4. The method according to claim 1, characterized in that The method further comprises: Inputting the horizontal plane feature vector and the coronal plane feature vector into respective one-dimensional convolutional layers for dimensionality reduction, thereby obtaining a reduced horizontal plane feature vector and a reduced coronal plane feature vector; Based on the horizontal plane feature vector and the decoder, pixel reconstruction is performed on the masked part in the horizontal plane image to obtain a target image, including: Based on the reduced horizontal plane feature vector and the decoder, reconstruct pixels of the masked part in the horizontal plane image to obtain a target image; Calculating a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image, including: The loss value is calculated based on the reduced horizontal plane feature vector, each reduced coronal plane feature vector in the dynamic dictionary and the target image.

5. The method according to claim 3, characterized in that: Based on the horizontal plane feature vector and the decoder, pixel reconstruction is performed on the masked part in the horizontal plane image to obtain a target image, including: Calculating the position embedding value of the masked slice in the horizontal plane image; Using the fully connected layer of the decoder to perform dimensionality reduction processing on the horizontal plane feature vector to obtain a horizontal plane feature vector after dimensionality reduction; Combining the mask token of the masked slice in the horizontal plane image with the reduced-dimensional horizontal plane feature vector to obtain a combined horizontal plane feature vector; Superimposing the combined horizontal plane feature vector with the position embedding values ​​of the masked slice and the uncovered slice in the horizontal plane image to obtain a superimposed horizontal plane feature vector; The superimposed horizontal plane feature vector is processed using a transformer model in the decoder to obtain the target image.

6. The method according to claim 2, characterized in that Calculating a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary and the target image, including: Calculating similarity loss based on the horizontal plane feature vector and each coronal plane feature vector in the dynamic dictionary; Calculating pixel loss based on the target image and the horizontal plane stitching image; Based on a preset temperature coefficient, the similarity loss and the pixel loss are added to obtain the loss value.

7. The method according to claim 1, characterized in that According to the loss value, the encoder, the momentum encoder and the decoder are jointly iteratively trained until a preset condition is met, and a preset pulmonary granulomatous nodule and solid pulmonary adenocarcinoma classification model is constructed, including: According to the loss value, updating parameters of the encoder and the decoder to obtain an updated encoder and an updated decoder; Based on the parameters in the updated encoder, updating the parameters in the momentum encoder to obtain an updated momentum encoder; Repeating the iterative updating process of the encoder, the momentum encoder and the decoder until a preset number of iterations is reached, and outputting a finally trained encoder; Based on the final trained encoder and fully connected layers, the preset pulmonary granulomatous nodules and solid lung adenocarcinoma classification model is constructed.

8. A hybrid self-supervised learning device for the malignancy of a single pulmonary nodule based on CT images, characterized in that: include: An acquisition unit, used for acquiring horizontal plane images and coronal plane images of lung lesions; A first extraction unit, configured to input the horizontal plane image into an encoder for feature extraction to obtain a horizontal plane feature vector of the lung lesion; A second extraction unit is used to input the coronal plane image into a momentum encoder for feature extraction, obtain a coronal plane feature vector of the lung lesion, and store the coronal plane feature vector in a dynamic dictionary; A reconstruction unit, configured to reconstruct pixels of a masked portion of the horizontal plane image based on the horizontal plane feature vector and a decoder to obtain a target image; a calculation unit, configured to calculate a loss value based on the horizontal plane feature vector, each coronal plane feature vector in the dynamic dictionary, and the target image; A training unit is used to jointly iteratively train the encoder, the momentum encoder and the decoder according to the loss value until a preset condition is met, thereby constructing a preset classification model for pulmonary granulomatous nodules and solid lung adenocarcinoma.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Lightweight CNN (Convolutional Neural Network)-based pulmonary nodule benign and malignant prediction method

    CN108038844A

  • Weak supervision voice-video positioning method and system based on semantic interaction

    CN116524915A

  • Lung CT anomaly detection method based on multi-scale cutting and self-supervised reconstruction

    CN117237269A

Cited By

  • Multi-modal pulmonary nodule recognition system and construction method, device, equipment, program product and medium thereof

    CN122597410A