A method, system, device and medium for detecting multiple lesions of lung cancer

Abstract

The present invention discloses a method, system, device and medium for detecting multiple lung cancer lesions, which relates to the detection of lung cancer lesions in the field of artificial intelligence. The purpose is to solve the technical problems in the prior art, such as the small number of multi-lesion samples and the lack of a network model for detecting multiple lesions. First, sample data is obtained, and a multi-lesion data generation model is constructed to expand a small number of multi-lesion samples. The multi-lesion data generation model includes a probability diffusion model and a prior information learning network model. The probability diffusion model includes a diffusion encoder and a diffusion decoder, and the prior information learning network model includes a Transformer encoder and a prior fully connected layer. Then, the sample data and the expanded sample data are input into the constructed multiple lesion detection model for training. Finally, the trained multiple lesion detection model is used to perform real-time detection of multiple lung cancer lesions.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and relates to the detection of lung cancer lesions, in particular to a method, system, device and medium for detecting multiple lung cancer lesions based on a small number of samples. Background Art

[0002] With the popularization and application of high-resolution chest imaging systems and intelligent lung cancer screening programs, the number of patients with multiple lung cancer lesions in clinical manifestations is gradually increasing. These patients with multiple lung cancer lesions can be divided into multiple primary lung cancer (MPLC) and intrapulmonary metastasis (IM). MPLC refers to the simultaneous or sequential discovery of two or more primary lung cancer lesions in the lungs, and these lesions are not only anatomically separated but also independent in origin. IM refers to the situation where other primary cancer lesions (including lung cancer, breast cancer, colorectal cancer, etc.) form metastatic lesions in the lungs of patients. The treatment strategy for IM patients mainly depends on the type of primary cancer and the overall health status of the patients. However, there are no generally recognized guidelines and standards for the diagnosis criteria of MPLC and its differentiation from IM, especially in cases with similar histology. The existing diagnostic criteria are mainly based on clinicopathological features, but far from meeting the clinical needs. Therefore, an efficient and accurate detection method based on artificial intelligence technology is expected to provide new ideas and possibilities for the precise treatment of patients with multiple lung cancer lesions.

[0003] In the prior art, detection methods for single lung cancer lesions have been widely studied. Most of these studies focus on the detection and differentiation of single lesions, while there are few studies focusing on multiple lung cancer lesions.

[0004] In the method for diagnosing multiple lung cancer lesions, the invention patent with the application number of 202010147633.0 discloses a system for diagnosing multiple lung cancer lesions. The system includes: a sample information processing module, a first diagnosis module, a second diagnosis module, and an information output module. The first diagnosis module and the second diagnosis module judge whether the lesions are from multiple primaries or metastases according to the driver genes and hot spot mutations of lung cancer carcinogenesis, thereby realizing the molecular determination of multiple lung cancer lesions. This system can accurately and quickly diagnose patients with multiple lung cancer lesions reasonably, and there is no subjective factor involved in the diagnosis process, which has important clinical application value.

[0005] If the method in the above invention patent is used to judge whether the lesions are from multiple primaries or metastases according to the driver genes and hot spot mutations of lung cancer carcinogenesis, this method requires continuous gene detection, and its detection and diagnosis are still very complex, time-consuming and laborious. Patients also need to pay a high cost, and it is quite difficult to promote.

[0006] Common single-lesion detection of lung cancer is based on convolutional neural networks to extract features from images and classify image patches through preset anchor boxes. As is well known, if neural networks are used for feature extraction, a large amount of sample data is required to train the neural networks. Moreover, in the existing technology, a large amount of single-lesion sample data exists and can be obtained without much effort. However, the sample data for multiple lesions is very scarce. Therefore, there are basically no related technologies that use neural networks to detect multiple lesions. Thus, it is necessary to solve problems such as the small number of multiple-lesion samples and the neural network model for multiple lesions. Summary of the Invention

[0007] The purpose of the present invention is to solve the technical problems in the existing technology, namely, the small number of multiple-lesion samples and the lack of a network model for detecting multiple lesions, and to provide a method, system, device, and medium for detecting multiple lesions of lung cancer.

[0008] To achieve the above purpose, the present invention specifically adopts the following technical solutions:

[0009] A method for detecting multiple lesions of lung cancer includes the following steps:

[0010] Step S1, obtaining sample data;

[0011] Obtaining CT sample images and label data of multiple lesions of lung cancer;

[0012] Step S2, expanding sample data;

[0013] Constructing a multiple-lesion data generation model, which includes a probabilistic diffusion model and a prior information learning network model. The probabilistic diffusion model includes a diffusion encoder and a diffusion decoder, and the prior information learning network model includes a Transformer encoder and a prior fully connected layer;

[0014] The label data in the CT sample images of multiple lesions of lung cancer is used as the input of the probabilistic diffusion model after adding random noise. The CT sample images of multiple lesions of lung cancer are output as prior information features after passing through the Transformer encoder. The prior information features and the output of the diffusion encoder are jointly used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is concatenated with the output of the probabilistic diffusion model to generate simulated synthetic data of intrapulmonary metastases of multiple primary lung cancers as supplementary sample data for the CT sample images of multiple lesions of lung cancer;

[0015] Step S3, constructing a multiple-lesion detection model;

[0016] Constructing a multiple-lesion detection model based on the state space;

[0017] Step S4, training the multiple-lesion detection model;

[0018] Input the lung cancer multi-lesion CT sample images and label data obtained in step S1 and the synthetic data of multiple primary lung cancers and intrapulmonary metastases obtained in step S2 into the multiple-lesion detection model to train the multiple-lesion detection model;

[0019] Step S5, multiple-lesion detection;

[0020] Obtain the CT image to be detected and input it into the multiple-lesion detection model, and the multiple-lesion detection model outputs the detection result.

[0021] Furthermore, in step S1, preprocess the obtained lung cancer multi-lesion CT sample images. The specific method is: perform normalization processing on the lung cancer multi-lesion CT sample images, and after cropping the lung cancer multi-lesion CT sample images, change them into CT images of 300*472*472; then normalize the CT images to the interval [0,1].

[0022] Furthermore, in step S2, the diffusion encoder includes a first group of regularization layers, a first sigmoid linear unit layer, a first convolutional layer, a second group of regularization layers, a second sigmoid linear unit layer, a second convolutional layer, a first attention layer, and a first downsampling layer arranged in sequence. And after the output of the second convolutional layer is concatenated with the input residual of the first group of regularization layers, it is used as the input of the first attention layer;

[0023] The diffusion decoder includes a fourth spatial normalization layer, a fourth sigmoid linear unit layer, a fourth convolutional layer, a fifth spatial normalization layer, a fifth sigmoid linear unit layer, a fifth convolutional layer, a second attention layer, and a sixth downsampling layer arranged in sequence. And after the output of the fifth convolutional layer is concatenated with the input residual of the fourth spatial normalization layer, it is used as the input of the second attention layer.

[0024] Furthermore, in step S3, the multiple-lesion detection model includes a mask feature extraction module, a state space feature aggregation module, a radiomics analysis module, a clinical analysis module, and a multi-modal feature fusion module. The input of the mask feature extraction module is used as the input of the state space feature aggregation module, and the outputs of the state space feature aggregation module, the radiomics analysis module, and the clinical analysis module are all used as the input of the multi-modal feature fusion module.

[0025] Furthermore, the mask feature extraction module includes a teacher network and a student network;

[0026] The state space feature aggregation module includes a first SSM layer, a second SSM layer, and a pooling layer arranged in sequence;

[0027] The radiomics analysis module includes a radiomics feature extractor sub-module, a first-order feature sub-module, a singular value decomposition sub-module, a texture module, and a LASSO feature selection module;

[0028] The clinical analysis module includes a radiology record sub-module, a radiology sign sub-module, and a patient basic information sub-module;

[0029] The multi-modal feature fusion module includes a Transformer layer and an MLP layer.

[0030] Furthermore, in step S4, when training the multiple-lesion detection model, in the mask feature extraction module, the CT sample images of multiple lung cancer lesions and the synthetic data of intrapulmonary metastases of multiple primary lung cancers are input into the teacher network, and the teacher network outputs an instance mask; the CT sample images of multiple lung cancer lesions and the synthetic data of intrapulmonary metastases of multiple primary lung cancers are input into the student network, and the student network learns and updates the parameters by combining the instance mask output by the teacher network, and the parameters of the student network are used to update the parameters of the teacher network through EMA.

[0031] Furthermore, the way of giving EMA update is:

[0032] θ t ← λθ t +(1 - λ)θ s

[0033] where θ t is the weight parameter of the teacher network, and θ s is the weight parameter of the student network; λ is a hyperparameter, and during training, λ increases from 0.996 to 1 according to a cosine schedule.

[0034] A multiple-lesion detection system for lung cancer, comprising:

[0035] A sample data acquisition module, used to acquire CT sample images of multiple lung cancer lesions and label data;

[0036] A sample data augmentation module, used to construct a multiple-lesion data generation model. The multiple-lesion data generation model includes a probabilistic diffusion model and a prior information learning network model. The probabilistic diffusion model includes a diffusion encoder and a diffusion decoder, and the prior information learning network model includes a Transformer encoder and a prior fully-connected layer;

[0037] The labeled data in the CT sample images of multiple lung cancer lesions is used as the input of the probability diffusion model after adding random noise. The CT sample images of multiple lung cancer lesions pass through the Transformer encoder and then output prior information features. The prior information features and the output of the diffusion encoder are jointly used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is concatenated with the output of the probability diffusion model to generate simulated synthetic data of intrapulmonary metastases of multiple primary lung cancers, which is used as supplementary sample data for the CT sample images of multiple lung cancer lesions.

[0038] The multiple lesion detection model construction module is used to construct a multiple lesion detection model based on the state space.

[0039] The multiple lesion detection model training module is used to input the CT sample images of multiple lung cancer lesions and the labeled data obtained by the sample data acquisition module, as well as the synthetic data of intrapulmonary metastases of multiple primary lung cancers obtained by the sample data augmentation module, into the multiple lesion detection model to train the multiple lesion detection model.

[0040] The multiple lesion detection module is used to obtain the CT image to be detected and input it into the multiple lesion detection model, and the multiple lesion detection model outputs the detection result.

[0041] A computer device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the above method.

[0042] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above method.

[0043] The beneficial effects of the present invention are as follows:

[0044] 1. In the present invention, it can be applied to chest CT images. By constructing a multiple lesion data generation model to augment a small amount of sample data, it can use a small amount of data to accurately complete the task of detecting multiple lung cancer lesions, effectively solving the problem of the small number of multiple lesion samples. The generation and detection models used in this method only require two stages. Through the generation model, the amount of multiple lesion sample data can be increased, and the detection model can be fully trained, making the detection of the detection model more accurate and efficient.

[0045] 2. In the present invention, the prior knowledge related to lung cancer is used for constraint, and the probability diffusion network model is used for image synthesis to achieve the expansion of the training dataset. Based on the feature aggregator of the structured state space model, the internal relationship of the instance sequence is efficiently modeled, and an effective and accurate multiple primary lung cancer detection model is constructed. Description of the Drawings

[0046] Figure 1 It is a schematic flow diagram of the present invention;

[0047] Figure 2 It is a schematic structural diagram of the multi-lesion data generation model in the present invention;

[0048] Figure 3 It is a schematic structural diagram of the multi-lesion detection model in the present invention. Detailed implementation manners

[0049] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention.

[0050] Therefore, based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] Embodiment 1

[0052] This embodiment provides a method for detecting multiple lung cancer lesions, which can expand the samples with a small amount of multi-lesion sample data and be used for the training of subsequent detection models, thus effectively solving the problem of insufficient multi-lesion sample data. The specific steps are as follows:

[0053] Step S1, obtain sample data;

[0054] Obtain CT sample images and label data of multiple lung cancer lesions.

[0055] The CT sample images of multiple lung cancer lesions all come from West China Hospital of Sichuan University, and the amount of sample data is limited. The accurate label data of the CT sample images of multiple lung cancer lesions also all come from West China Hospital of Sichuan University.

[0056] In this embodiment, complex preprocessing processes are not required for the sample images. Only simple normalization processing is performed on the original CT sample images of multiple lung cancer lesions, that is, the CT images are cropped and then changed into CT images with a size of 300*472*472, and then the CT images are normalized to the interval [0,1].

[0057] Since the amount of data of the original CT sample images is small, it is necessary to add noise to the existing small amount of CT image data to expand the data set.

[0058] Step S2, expand sample data;

[0059] Construct a multi-lesion data generation model. The multi-lesion data generation model includes a probabilistic diffusion model and a prior information learning network model. The probabilistic diffusion model includes a diffusion encoder and a diffusion decoder. The prior information learning network model includes a Transformer encoder and a prior fully connected layer;

[0060] The labeled data in the multi-lesion CT sample images of lung cancer is used as the input of the probabilistic diffusion model after adding random noise. The multi-lesion CT sample images of lung cancer are output with prior information features after passing through the Transformer encoder. The prior information features and the output of the diffusion encoder are jointly used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is concatenated with the output of the probabilistic diffusion model to generate simulated synthetic data of intrapulmonary metastases of multiple primary lung cancers, which is used as supplementary sample data for the multi-lesion CT sample images of lung cancer.

[0061] In this embodiment, prior information features such as the morphology and size of lung cancer lesions extracted by the Transformer encoder are used to prepare for embedding the prior information into the feature map of the generated data in the lungs by subsequent spatial adaptive normalization and introducing a conditional control module to regulate the number of generated images. The Transformer encoder is composed of multiple transformer sub-modules. Each sub-module includes a multi-head attention sub-layer, a fully connected sub-layer, and a residual connection.

[0062] The probabilistic diffusion model completes the training of the neural network by performing a forward diffusion process to gradually add Gaussian noise to the initially input image and then performing a reverse diffusion process to remove the noise from the image. In synthesizing images constrained by prior conditions, in this embodiment, an existing accurately labeled dataset is used as the training input, and classifier-free guidance is selected to control the sampling process to reduce the training cost.

[0063] This embodiment introduces the prior information extracted by the transformer, including the location, size, and shape of lung cancer lesions, etc. The Spatially-Adaptive Normalization (SPADE) module is used to align the feature space and fuse the feature information under different feature dimensions to generate reasonable simulated data. Specifically, the diffusion encoder in this probability diffusion model includes a first set of regularization layers, a first sigmoid linear unit layer, a first convolutional layer, a second set of regularization layers, a second sigmoid linear unit layer, a second convolutional layer, a first attention layer, and a first downsampling layer arranged in sequence. After the output of the second convolutional layer is concatenated with the input residual of the first set of regularization layers, it serves as the input of the first attention layer. The diffusion decoder in this probability diffusion model includes a fourth spatial normalization layer, a fourth sigmoid linear unit layer, a fourth convolutional layer, a fifth spatial normalization layer, a fifth sigmoid linear unit layer, a fifth convolutional layer, a second attention layer, and a sixth downsampling layer arranged in sequence. After the output of the fifth convolutional layer is concatenated with the input residual of the fourth spatial normalization layer, it serves as the input of the second attention layer.

[0064] The output of the probability diffusion model is concatenated with the output of the prior information learning network to generate a simulated image (i.e., synthetic data for intrapulmonary metastases of multiple primary lung cancers), and the Adam algorithm is used to update the parameters of the backup model.

[0065] Step S3, construct a multiple-lesion detection model;

[0066] Construct a multiple-lesion detection model based on the state space.

[0067] The multiple-lesion detection model includes a mask feature extraction module, a state space feature aggregation module, a radiomics analysis module, a clinical analysis module, and a multi-modal feature fusion module. The input of the mask feature extraction module serves as the input of the state space feature aggregation module. The outputs of the state space feature aggregation module, the radiomics analysis module, and the clinical analysis module all serve as the inputs of the multi-modal feature fusion module.

[0068] The mask feature extraction module includes a teacher network and a student network;

[0069] The state space feature aggregation module includes a first SSM layer, a second SSM layer, and a pooling layer arranged in sequence;

[0070] The radiomics analysis module includes a radiomics feature extraction sub-module, a first-order feature sub-module, a singular value decomposition sub-module, a texture module, and a LASSO feature selection module;

[0071] The clinical analysis module includes a radiology record sub-module, a radiology sign sub-module, and a patient basic information sub-module;

[0072] The multi-modal feature fusion module includes a Transformer layer and an MLP layer.

[0073] In this embodiment, a state space feature aggregation module is also constructed. Multiple instance features are regarded as sequence inputs to capture the implicit context relationships and dynamic changes between sequences, so as to identify the connections and differences between lesions. This module consists of an SSM layer and a pooling layer. The sequence features are aggregated through the pooling layer to obtain global patient-level features, which are further used as deep learning features and become an input to the multi-modal feature fusion module. s Utilize the radiomics features of the CT image mask and use the singular value decomposition algorithm and LASSO feature selection to extract clinical features. R Utilize the clinical text corresponding to the CT image, fuse the basic patient information for word embedding operation, and obtain the output. c Finally, a multi-modal feature fusion module is constructed, including a Transformer layer and an MLP layer. The inputs are s , R and c , and its output is the final detection information.

[0074] In this embodiment, a state space-based multiple lesion detection model for multi-instance learning is constructed. Through feature extraction and feature interaction of lesions and their microenvironments, effective global information can be obtained, and further combined with multi-modal features such as clinical and radiomics to achieve accurate discrimination.

[0075] Step S4, train the multiple lesion detection model;

[0076] Input the lung cancer multiple lesion CT sample images and label data obtained in step S1, and the synthetic data of multiple primary lung cancers with intrapulmonary metastases obtained in step S2 into the multiple lesion detection model to train the multiple lesion detection model.

[0077] When training the multiple lesion detection model, in the mask feature extraction module, the lung cancer multiple lesion CT sample images and the synthetic data of multiple primary lung cancers with intrapulmonary metastases are input into the teacher network, and the teacher network outputs instance masks; the lung cancer multiple lesion CT sample images and the synthetic data of multiple primary lung cancers with intrapulmonary metastases are input into the student network, and the student network learns and updates the parameters by combining the instance masks output by the teacher network. The parameters of the student network are used to update the parameters of the teacher network through EMA.

[0078] In order to improve the representation ability of the feature extractor in the absence of instance-level labels, the present invention designs a knowledge distillation model including a teacher network and a student network. The output instance mask of the teacher network is used as the instance feature available to the student network, and the update of the teacher network parameters is guided by the exponential moving average output by the student network. The way of giving EMA update is as follows:

[0079] θ t ←λθ t +(1 - λ)θ s

[0080] Among them, θ t is the weight parameter of the teacher network, and θ s is the weight parameter of the student network; λ is a hyperparameter, and during training, λ increases from 0.996 to 1 according to a cosine schedule.

[0081] Regarding the training of the multiple-lesion detection model, except for the above content, the specific training method and loss function can adopt the existing technology.

[0082] During the training process, different data augmentation strategies are applied to the same lesion to obtain different perspectives and encode them, and the feature expression ability of the encoder is alternately trained by constraining the feature similarity under different perspectives. To enhance the difference between lesion features, the difference between different instances is amplified by minimizing the mutual information between instances in different clusters, and the mutual information in the same cluster is maximized to constrain lesions of the same type.

[0083] Step S5, multiple-lesion detection;

[0084] Obtain the CT image to be detected and input it into the multiple-lesion detection model, and the multiple-lesion detection model outputs the detection result.

[0085] Embodiment 2

[0086] This embodiment provides a multiple-lesion detection system for lung cancer, which specifically includes:

[0087] A sample data acquisition module for acquiring CT sample images and label data of multiple lung cancer lesions.

[0088] The CT sample images of multiple lung cancer lesions are all from West China Hospital of Sichuan University, and the amount of sample data is limited. The accurate label data of the CT sample images of multiple lung cancer lesions are also all from West China Hospital of Sichuan University.

[0089] In this embodiment, no complex preprocessing process is required for the sample images. It only performs simple normalization on the original CT sample images of multiple lung cancer lesions, that is, the CT image is cropped and then becomes a CT image with a size of 300*472*472, and then the CT image is normalized to the [0, 1] interval.

[0090] Since the amount of data of the original CT sample images is small, it is necessary to add noise to the existing small amount of CT image data to expand the dataset.

[0091] The sample data augmentation module is used to construct a multi-lesion data generation model. The multi-lesion data generation model includes a probabilistic diffusion model and a prior information learning network model. The probabilistic diffusion model includes a diffusion encoder and a diffusion decoder. The prior information learning network model includes a Transformer encoder and a prior fully connected layer;

[0092] The labeled data in the multi-lesion CT sample images of lung cancer is used as the input of the probabilistic diffusion model after adding random noise. The multi-lesion CT sample images of lung cancer are output as prior information features after passing through the Transformer encoder. The prior information features and the output of the diffusion encoder are jointly used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is concatenated with the output of the probabilistic diffusion model to generate simulated synthetic data of intrapulmonary metastases of multiple primary lung cancers, which serves as supplementary sample data for the multi-lesion CT sample images of lung cancer.

[0093] In this embodiment, prior information features such as the morphology and size of lung cancer lesions extracted by the Transformer encoder are used to prepare for embedding prior information into the feature map of the generated data in the lungs by subsequent spatial adaptive normalization and introducing a conditional control module to regulate the number of generated images. The Transformer encoder is composed of multiple transformer sub-modules. Each sub-module includes a multi-head attention sub-layer, a fully connected sub-layer, and a residual connection.

[0094] The probabilistic diffusion model completes the training of the neural network by performing a forward diffusion process to gradually add Gaussian noise to the initially input image and then a reverse diffusion process to remove the noise from the image. In synthesizing images constrained by prior conditions, in this embodiment, an existing accurately labeled dataset is used as the training input, and classifier-free guidance is selected to control the sampling process to reduce the training cost.

[0095] This embodiment introduces the prior information extracted by the transformer, including the location, size, and shape of lung cancer lesions, etc. The Spatially-Adaptive Normalization (SPADE) module is used to align the feature space and fuse the feature information under different feature dimensions to generate reasonable simulated data. Specifically, the diffusion encoder in this probability diffusion model includes a first set of regularization layers, a first sigmoid linear unit layer, a first convolutional layer, a second set of regularization layers, a second sigmoid linear unit layer, a second convolutional layer, a first attention layer, and a first downsampling layer arranged in sequence. After the output of the second convolutional layer is concatenated with the input residual of the first set of regularization layers, it serves as the input of the first attention layer. The diffusion decoder in this probability diffusion model includes a fourth spatial normalization layer, a fourth sigmoid linear unit layer, a fourth convolutional layer, a fifth spatial normalization layer, a fifth sigmoid linear unit layer, a fifth convolutional layer, a second attention layer, and a sixth downsampling layer arranged in sequence. After the output of the fifth convolutional layer is concatenated with the input residual of the fourth spatial normalization layer, it serves as the input of the second attention layer.

[0096] The output of the probability diffusion model is concatenated with the output of the prior information learning network to generate a simulated image (i.e., the synthetic data of intrapulmonary metastases of multiple primary lung cancers), and the Adam algorithm is used to update the parameters of the backup model.

[0097] The multiple-lesion detection model construction module is used to construct a multiple-lesion detection model based on the state space.

[0098] The multiple-lesion detection model includes a mask feature extraction module, a state space feature aggregation module, a radiomics analysis module, a clinical analysis module, and a multi-modal feature fusion module. The input of the mask feature extraction module serves as the input of the state space feature aggregation module, and the outputs of the state space feature aggregation module, the radiomics analysis module, and the clinical analysis module all serve as the inputs of the multi-modal feature fusion module.

[0099] The mask feature extraction module includes a teacher network and a student network;

[0100] The state space feature aggregation module includes a first SSM layer, a second SSM layer, and a pooling layer arranged in sequence;

[0101] The radiomics analysis module includes a radiomics feature extraction sub-module, a first-order feature sub-module, a singular value decomposition sub-module, a texture module, and a LASSO feature selection module;

[0102] The clinical analysis module includes a radiology record sub-module, a radiology sign sub-module, and a patient basic information sub-module;

[0103] The multi-modal feature fusion module includes a Transformer layer and an MLP layer.

[0104] In this embodiment, a state space feature aggregation module is also constructed. Multiple instance features are regarded as sequence inputs to capture the implicit context relationships and dynamic changes between sequences, so as to identify the connections and differences between lesions. This module consists of an SSM layer and a pooling layer. The sequence features are aggregated through the pooling layer to obtain global patient-level features, which are further used as deep learning features and become an input to the multi-modal feature fusion module. s Utilize the radiomics features of the CT image mask and use the singular value decomposition algorithm and LASSO feature selection to extract clinical features. R Utilize the clinical text corresponding to the CT image, fuse the basic patient information for word embedding operation, and obtain the output. c Finally, a multi-modal feature fusion module is constructed, including a transformer layer and an MLP layer. The inputs are s , R and c , and its output is the final detection information.

[0105] In this embodiment, a state space-based multiple lesion detection model for multi-instance learning is constructed. Through feature extraction and feature interaction of lesions and their microenvironments, effective global information can be obtained, and further combined with multi-modal features such as clinical and radiomics to achieve accurate discrimination.

[0106] The multiple lesion detection model training module is used to input the lung cancer multiple lesion CT sample images and label data obtained by the sample data acquisition module, as well as the synthetic data of multiple primary lung cancers with intrapulmonary metastases obtained by the sample data augmentation module, into the multiple lesion detection model to train the multiple lesion detection model.

[0107] When training the multiple lesion detection model, in the mask feature extraction module, the lung cancer multiple lesion CT sample images and the synthetic data of multiple primary lung cancers with intrapulmonary metastases are input into the teacher network, and the teacher network outputs instance masks; the lung cancer multiple lesion CT sample images and the synthetic data of multiple primary lung cancers with intrapulmonary metastases are input into the student network, and the student network learns and updates the parameters by combining the instance masks output by the teacher network. The parameters of the student network are used to update the parameters of the teacher network through EMA.

[0108] In order to improve the representation ability of the feature extractor in the absence of instance-level labels, the present invention designs a knowledge distillation model including a teacher network and a student network. The instance masks output by the teacher network are used as instance features that can be utilized by the student network, and the update of the teacher network parameters is guided by the exponential moving average output by the student network. The way of giving the EMA update is:

[0109] θ t←λθ t +(1 - λ)θ s where θ t is the weight parameter of the teacher network, and θ s is the weight parameter of the student network; λ is a hyperparameter, and during training, λ increases from 0.996 to 1 according to a cosine schedule.

[0110] Regarding the training of the multiple lesion detection model, except for the above content, the specific training method and loss function can adopt existing technologies.

[0111] During the training process, different data augmentation strategies are applied to the same lesion to obtain different perspectives and encode them. By constraining the feature similarity under different perspectives, the feature expression ability of the encoder is alternately trained. To enhance the difference between lesion features, the difference between different instances is amplified by minimizing the mutual information between instances in different clusters, and the mutual information in the same cluster is maximized to constrain lesions of the same type.

[0112] The multiple lesion detection module is used to obtain the CT image to be detected and input it into the multiple lesion detection model, and the multiple lesion detection model outputs the detection result.

[0113] Embodiment 3

[0114] A computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the steps of the lung cancer multiple lesion detection method.

[0115] where the computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.

[0116] The memory at least includes one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Of course, the memory may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory is commonly used to store the operating system installed on the computer device and various application software, such as the program code of the lung cancer multiple lesion detection method. In addition, the memory may also be used to temporarily store various data that have been output or will be output.

[0117] In some embodiments, the processor may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the lung cancer multiple lesion detection method.

[0118] Embodiment 4

[0119] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the lung cancer multiple lesion detection method.

[0120] Wherein, the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to cause the at least one processor to execute the steps of the lung cancer multiple lesion detection method as described above.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the lung cancer multiple lesion detection method described in the embodiments of the present application.

Claims

1. A method for detecting multiple lesions of lung cancer, characterized in that: The steps include: Step S1, obtaining sample data; Obtain multi-lesion CT sample images and label data for lung cancer; Step S2, expanding sample data; A multi-lesion data generation model is constructed, and the multi-lesion data generation model includes a probability diffusion model and a prior information learning network model. The probability diffusion model includes a diffusion encoder and a diffusion decoder. The prior information learning network model includes a Transformer encoder and a prior fully connected layer. The label data in the multi-lesion CT sample image of lung cancer is used as the input of the probability diffusion model after adding random noise. The multi-lesion CT sample image of lung cancer outputs prior information features after passing through the Transformer encoder. The prior information features and the output of the diffusion encoder are used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is spliced ​​with the output of the probability diffusion model to generate simulated multi-primary lung cancer intrapulmonary metastasis synthetic data as supplementary sample data of the multi-lesion CT sample image of lung cancer. Step S3, constructing a multiple lesion detection model; Construct a state-space-based multiple lesion detection model; Step S4, training a multiple lesion detection model; Inputting the lung cancer multi-lesion CT sample image and label data obtained in step S1 and the synthetic data of multiple primary lung cancer intrapulmonary metastasis obtained in step S2 into the multiple lesion detection model to train the multiple lesion detection model; Step S5, multiple lesion detection; Obtaining a CT image to be detected and inputting it into a multiple lesion detection model, which then outputs a detection result; In step S3, the multiple lesion detection model includes a mask feature extraction module, a state-space feature aggregation module, a radiomics analysis module, a clinical analysis module and a multimodal feature fusion module, the input of the mask feature extraction module is used as the input of the state-space feature aggregation module, and the outputs of the state-space feature aggregation module, the radiomics analysis module and the clinical analysis module are all used as the input of the multimodal feature fusion module; The mask feature extraction module includes a teacher network and a student network; The state space feature aggregation module includes a first SSM layer, a second SSM layer and a pooling layer which are arranged in sequence; The radiomics analysis module includes a radiomics feature extraction submodule, a first-order feature submodule, a singular value decomposition submodule, a texture module, and a LASSO feature selection module; The clinical analysis module includes the radiology record submodule, the radiology sign submodule, and the patient basic information submodule; The lower multimodal feature fusion module includes the Transformer layer and the MLP layer.

2. A method for detecting multiple lesions of lung cancer as claimed in claim 1, characterized in that: In step S1, the acquired lung cancer multi-lesion CT sample image is preprocessed, specifically, the lung cancer multi-lesion CT sample image is normalized, and the lung cancer multi-lesion CT sample image is cropped into a 300*472*472 CT image; and the CT image is normalized to the [0,1] interval.

3. A method for detecting multiple lesions of lung cancer as claimed in claim 1, characterized in that: In step S2, the diffusion encoder includes a first group of regularization layers, a first sigmoid linear unit layer, a first convolution layer, a second group of regularization layers, a second sigmoid linear unit layer, a second convolution layer, a first attention layer and a first downsampling layer, and the output of the second convolution layer is concatenated with the input residual of the first group of regularization layers as the input of the first attention layer; The diffusion decoder includes a fourth spatial normalization layer, a fourth sigmoid linear unit layer, a fourth convolutional layer, a fifth spatial normalization layer, a fifth sigmoid linear unit layer, a fifth convolutional layer, a second attention layer and a sixth downsampling layer, which are arranged in sequence, and the output of the fifth convolutional layer is concatenated with the input residual of the fourth spatial normalization layer as the input of the second attention layer.

4. A method for detecting multiple lesions of lung cancer as claimed in claim 1, characterized in that: In step S4, when training the multiple lesion detection model, in the mask feature extraction module, the multi-lesion CT sample images of lung cancer and the synthetic data of multiple primary lung cancer intrapulmonary metastasis are input into the teacher network, and the teacher network outputs the instance mask; Multi-lesion CT sample images of lung cancer and synthetic data of multiple primary lung cancer metastases are input into the student network. The student network learns and updates parameters based on the instance masks output by the teacher network. The parameters of the student network update the parameters of the teacher network through EMA.

5. A method for detecting multiple lesions of lung cancer as claimed in claim 4, characterized in that: EMA updates are given as: in, is the weight parameter of the teacher network, is the weight parameter of the student network; is a hyperparameter, and during training, λ increases from 0.996 to 1 according to a cosine schedule.

6. A system for detecting multiple lesions of lung cancer, characterized in that: include: The sample data acquisition module is used to obtain the sample images and label data of multi-lesion CT of lung cancer; The sample data expansion module is used to build a multi-lesion data generation model. The multi-lesion data generation model includes a probability diffusion model and a prior information learning network model. The probability diffusion model includes a diffusion encoder and a diffusion decoder. The prior information learning network model includes a Transformer encoder and a prior fully connected layer. The label data in the multi-lesion CT sample image of lung cancer is used as the input of the probability diffusion model after random noise is added. The multi-lesion CT sample image of lung cancer outputs the prior information features after passing through the Transformer encoder. The prior information features and the output of the diffusion encoder are used as the input of the diffusion decoder. The prior information features are also input into the prior fully connected layer. The output of the prior fully connected layer is spliced ​​with the output of the probability diffusion model to generate simulated multi-primary lung cancer intrapulmonary metastasis synthetic data as supplementary sample data of the multi-lesion CT sample image of lung cancer. Multiple lesion detection model building module, used to build a multiple lesion detection model based on state space; A multiple lesion detection model training module is used to input the multiple lesion CT sample images and label data of lung cancer obtained by the sample data acquisition module and the synthetic data of multiple primary lung cancer intrapulmonary metastasis obtained by the sample data expansion module into the multiple lesion detection model to train the multiple lesion detection model; The multiple lesion detection module is used to obtain the CT image to be detected and input it into the multiple lesion detection model, and the multiple lesion detection model outputs the detection result; In the multiple lesion detection model construction module, the multiple lesion detection model includes a mask feature extraction module, a state space feature aggregation module, a radiomics analysis module, a clinical analysis module and a multimodal feature fusion module. The input of the mask feature extraction module is used as the input of the state space feature aggregation module, and the outputs of the state space feature aggregation module, the radiomics analysis module and the clinical analysis module are all used as the input of the multimodal feature fusion module. The mask feature extraction module includes a teacher network and a student network; The state space feature aggregation module includes a first SSM layer, a second SSM layer and a pooling layer which are arranged in sequence; The radiomics analysis module includes a radiomics feature extraction submodule, a first-order feature submodule, a singular value decomposition submodule, a texture module, and a LASSO feature selection module; The clinical analysis module includes the radiology record submodule, the radiology sign submodule, and the patient basic information submodule; The lower multimodal feature fusion module includes the Transformer layer and the MLP layer.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images