Multimodal lung nodule recognition system, method, device, equipment, program product and medium for constructing same

Abstract

The application belongs to the technical field of lung nodule detection, and discloses a multi-modal lung nodule recognition system and a lung segmentation model, a construction method, device, equipment, program product and medium of a lung nodule detection and classification model, to solve the problem of limited lung nodule detection and recognition performance. The system comprises an acquisition unit for acquiring multiple chest CT images and marker detection results of a subject, a processing unit for recognizing lung tissue based on the multiple chest CT images, reconstructing a lung voxel model using the lung tissue, and preprocessing the multiple marker detection results to obtain a marker feature vector, an analysis unit for fusing the lung voxel model and the marker feature vector, extracting lung nodule features, and obtaining position information and classification information of the lung nodule, and an output unit for outputting the lung nodule classification information obtained by the analysis unit. Through the multi-modal lung nodule recognition system, marker-assisted lung nodule image diagnosis can be introduced, and accurate lung nodule recognition results can be automatically given.

Description

Technical Field

[0001] This application belongs to the field of lung nodule detection technology, specifically relating to a multimodal lung nodule identification system and its construction method, device, equipment, program product and medium. Background Technology

[0002] Lung cancer is one of the malignant tumors with the highest incidence and mortality rates. Pulmonary nodules are a typical sign of lung cancer, and early diagnosis is crucial for improving patient survival. While current imaging diagnostic methods have made significant progress in early screening, such as the method for identifying medical signs of pulmonary nodules in Chinese patent document CN107092918A, they have limitations in differentiating between benign and malignant nodules, especially in judging small nodules, which relies heavily on physician experience. Furthermore, internationally used biochemical tumor marker tests, such as CEA and CYFRA21-1 in conjunction with gene markers as described in Chinese patent CN116426636A, have limited accuracy when used alone. Therefore, how to integrate imaging data with biomarkers using advanced artificial intelligence technology to construct an accurate diagnostic model for pulmonary nodules has become a key technical issue for improving the early diagnosis of lung cancer.

[0003] Therefore, an automated lung nodule identification system based on multimodal data is needed to address the problem of high subjectivity in lung nodule identification, especially the high false positive rate in clinical practice. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this application is to provide a multimodal lung nodule recognition system. This system can automatically generate a three-dimensional model of the lungs based on CT image information provided by the subject. It uses a trained target recognition network and lung nodule classification network to extract effective lung nodule information from the model and automatically determine whether the subject has malignant nodules. This avoids diagnostic bias caused by the subject's subjectivity, thereby improving the detection rate of malignant nodules and reducing false positive diagnoses. This allows for accurate and timely warnings for potential lung cancer patients and also avoids misdiagnosis of more benign patients.

[0005] Specifically, this application relates to the following aspects: According to one aspect of this application, a multimodal lung nodule identification system is provided, comprising: an acquisition unit for acquiring multiple chest CT images and multiple biomarker detection results of a subject; a processing unit for identifying multiple lung tissues of the subject based on the multiple chest CT images using a second neural network, reconstructing a lung voxel model using the multiple lung tissues, and preprocessing the multiple biomarker detection results to obtain biomarker feature vectors; an analysis unit for fusing the lung voxel model and biomarker feature vectors, extracting lung nodule features through a first neural network to obtain lung nodule location information and classification information; and an output unit for outputting the lung nodule location information and classification information obtained by the analysis unit; wherein the multiple biomarker detection results include detection results of nucleic acid biomarker piR-hsa-8393202, nucleic acid biomarker piR-hsa-8429916, tumor biomarker CEA, and tumor biomarker CYFRA21-1.

[0006] According to some embodiments of this application, the analysis unit is configured to fuse a lung voxel model and a marker feature vector before extracting lung nodule features. The fusion of the lung voxel model and the marker feature vector, and the extraction of lung nodule features through a first neural network, includes: upsampling the marker feature vector into a marker feature tensor, aligning it with the voxel tensor of the lung voxel model and concatenating them into an input feature map; using the first neural network to extract lung nodule features from the input feature map to determine the lung nodule region in the lung voxel model; and using the first neural network to further classify the lung nodule region to determine whether the lung nodule region is benign or malignant.

[0007] According to some embodiments of this application, the analysis unit is configured to fuse a lung voxel model and a marker feature vector during the extraction of lung nodule features. The fusion of the lung voxel model and the marker feature vector, and the extraction of lung nodule features through a first neural network, includes: calculating the attention weight between the voxel tensor and the marker feature vector to weight the marker feature vector; summing the voxel tensor and the weighted marker feature vector to obtain an input feature map; using the first neural network to extract lung nodule features from the input feature map to determine the lung nodule region in the lung voxel model; and using the first neural network to further classify the lung nodule region to determine whether the lung nodule region is benign or malignant.

[0008] According to some embodiments of this application, the first neural network includes a lung nodule detection unit and a lung nodule classification unit; the lung nodule detection unit receives an input feature map and extracts lung nodule features using multiple first convolutional blocks that have residual connections to obtain a detection region of the lung nodule region; the lung nodule classification unit extracts pathological features of the lung nodule region based on the detection region using multiple second convolutional blocks to perform binary classification of the lung nodule region as benign or malignant.

[0009] According to some embodiments of this application, the lung nodule detection unit further includes multiple third convolutional layers for further determining the scores of lung nodule regions and non-lung nodule regions in the detection region through sliding window convolution to obtain the lung nodule region; the lung nodule detection unit further includes multiple fourth convolutional layers for obtaining the offset value between the lung nodule region and the corresponding candidate region based on the lung nodule region determined by the multiple third convolutional layers to correct the position of the lung nodule region.

[0010] According to some embodiments of this application, when the first neural network classifies lung nodule regions, it modulates the contribution of voxel tensors to classification by gating conditions based on the weighting of marker feature tensors, so as to reduce the dimensionality of voxel tensors and extract multi-scale information.

[0011] According to some embodiments of this application, the convolution kernel used in the first neural network is a 3D convolution kernel.

[0012] According to some embodiments of this application, identifying multiple lung tissues of a subject based on multiple chest CT images using a second neural network includes: performing multiple convolutions and downsampling operations on each chest CT image in the multiple chest CT images using the second neural network to obtain a first feature map; performing multiple convolutions and upsampling operations on the first feature map to obtain a second feature map; and inputting the second feature map into a convolutional layer and a pooling layer to obtain a segmentation mask for multiple lung tissues; wherein, the second neural network copies and clips the feature map before each downsampling and connects it to the feature map after each upsampling; the second neural network outputs segmentation masks for the trachea, arteries / veins, lobes, and segments of the lungs, as well as a mask for high-density areas of suspected nodules; the mask for high-density areas of suspected nodules is configured to be output to the lung nodule detection unit of the first neural network to limit the candidate detection region.

[0013] According to some embodiments of this application, each convolution in the multiple convolutions performed by the second neural network on each chest CT image is implemented through multiple third convolution blocks; for each of the multiple third convolution blocks: the feature map output by each convolutional layer is concatenated with the feature maps input to all convolutional layers after that convolutional layer; the feature map input to the first convolutional layer is concatenated with the feature maps output by each convolutional layer to form the feature map output by the third convolution block.

[0014] According to some embodiments of this application, the processing unit reconstructs a lung voxel model using multiple lung tissues, including: stitching the segmentation masks of each tissue into three-dimensional voxel data using a volume rendering method to generate a lung voxel model; and performing data cleaning, outlier removal, unit standardization, reference interval normalization, marker feature vector construction, and modal dimension alignment on the detection results of multiple markers to obtain marker feature vectors.

[0015] According to another aspect of this application, a method for constructing a lung segmentation model, a lung nodule detection and classification model is provided, comprising: inputting multiple chest CT images of a first population and a second population into a lung segmentation model; setting initial hyperparameters of an optimizer; calculating a first loss function value and iterating the lung segmentation model using gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value drops to a preset level; using the iterated lung segmentation model to output lung tissue masks from multiple chest CT images; reconstructing a lung voxel model; preprocessing and upsampling the marker detection results of the first and second populations and aligning them with the lung voxel model to obtain an input feature map; inputting the input feature map into a lung nodule detection and classification model; setting initial hyperparameters of an optimizer; calculating a second loss function value and iterating the lung nodule detection and classification model using gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value drops to a preset level; wherein, the lung segmentation model is a second neural network as described in any one of claims 1-10, and the lung nodule detection and classification model is a first neural network as described in any one of claims 1-10; the first population is patients with malignant lung nodules, and the second population is patients with benign lung nodules.

[0016] According to another aspect of this application, an apparatus for constructing a lung segmentation model, a lung nodule detection and classification model is provided, comprising: a first training unit, used to input multiple chest CT images of a first population and a second population into the lung segmentation model, set initial hyperparameters of the optimizer, calculate a first loss function value and iterate the lung segmentation model through gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value drops to a preset level; a reconstruction and alignment unit, used to output lung tissue masks of multiple chest CT images using the iterated lung segmentation model, reconstruct a lung voxel model, preprocess and upsample the marker detection results of the first population and the second population and align them with the lung voxel model to obtain an input feature map; and a second training unit, used to input the input feature map into the lung nodule detection and classification model, set initial hyperparameters of the optimizer, calculate a second loss function value and iterate the lung nodule detection and classification model through gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value drops to a preset level.

[0017] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions, which, when executed by the processor, cause the processor to perform the aforementioned methods for constructing the lung segmentation model, lung nodule detection, and classification model.

[0018] According to another aspect of this application, a computer program product is provided, which includes computer program instructions; when the computer program instructions are run by a processor, the processor causes the processor to execute the aforementioned methods for constructing the lung segmentation model, lung nodule detection and classification model.

[0019] According to another aspect of this application, a computer-readable storage medium is provided having computer program instructions stored thereon; when executed by a processor, the computer program instructions cause the processor to perform the aforementioned methods for constructing the lung segmentation model, lung nodule detection and classification model. Attached Figure Description

[0020] Figure 1 The figure shows a structural block diagram of a multimodal lung nodule identification system according to an embodiment of this application.

[0021] Figure 2 The figure shows a schematic diagram of the structure of the convolutional layer of a second neural network according to an embodiment of this application.

[0022] Figure 3 The illustration shows a schematic diagram of a first neural network identifying a target lung nodule region according to an embodiment of this application.

[0023] Figure 4 Analysis graphs of melting curves for piR-hsa-8429916, piR-hsa-8393202, and internal reference U6.

[0024] Figure 5 The amplification curves of representative real-time PCR for piR-hsa-8429916, piR-hsa-8393202, and internal control U6 are shown.

[0025] Figure 6 The illustration shows a flowchart of a method for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application.

[0026] Figure 7 The diagram illustrates a block diagram of an apparatus for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application.

[0027] Figure 8 A block diagram of an electronic device according to an embodiment of this application is illustrated. Detailed Implementation

[0028] The present application is further illustrated below with reference to embodiments. It should be understood that the embodiments are only used to further illustrate and explain the present application and are not intended to limit the present application.

[0029] Unless otherwise defined, technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art. While similar or identical methods and materials may be applied in experimental or practical applications, materials and methods are described herein. In case of conflict, the definitions included herein shall prevail. Furthermore, materials, methods, and examples are for illustrative purposes only and are not intended to be limiting. The present application is further described below with reference to specific embodiments, but is not intended to limit the scope of the application.

[0030] Application Overview Non-coding RNAs (piRNAs) such as piR-hsa-interacting RNAs (PIWIs) have shown potential in the field of liquid biopsy for tumors. Previous studies, using the Illumina small RNA sequencing platform and double-validation with tumor tissue and serum, have identified piR-hsa-8393202 and piR-hsa-8429916 as characteristic upregulated biomarkers for lung cancer. piRNAs epigenetically silence transposons such as LINE-1 through the PIWI / piRNA pathway. This regulatory mechanism is a common biological pathway in the malignant transformation of major pathological types of lung cancer. Members of the PIWI protein family are abnormally highly expressed in various pathological types of lung cancer and are associated with tumor proliferation, invasion, and prognosis. Therefore, piR-hsa-8393202 and piR-hsa-8429916, as functional effector molecules of the aforementioned pathway, also have potential applications in the differential diagnosis of malignant pulmonary nodules other than lung adenocarcinoma.

[0031] This application employs a four-level screening strategy for piRNA biomarkers in lung nodules: high-throughput discovery, differential expression validation, independent cohort validation, and machine learning ranking. First, piRNA omics analysis was performed on lung cancer tissues and paired adjacent normal tissues using the Illumina small RNA sequencing platform. Differential expression analysis was conducted using the DESeq2 algorithm, retaining candidate piRNAs with high fold-over-fold differential expression and corrected p-values ​​below significance. Second, the expression stability of candidate piRNAs was validated using qRT-PCR in an independent serum cohort, eliminating candidates with large batch-to-batch coefficients of variation, inconsistent serum-tissue expression trends, or non-single-peak melting curves. Third, the validated candidate piRNAs were incorporated into a random forest classifier, and the marginal contribution of each biomarker to lung cancer / non-lung cancer classification was assessed using the SHAP value interpretation framework. piRNAs with high SHAP values ​​and high levels of AUC, sensitivity, and specificity were retained as the desired biomarkers.

[0032] Furthermore, this application also utilizes two tumor-associated antigen markers at the protein levels, CEA and CYFRA21-1, as complementary multidimensional biomarkers for piRNA based on the following biological mechanisms: CEA mainly reflects the secretory activity of tumor cells and tumor burden, while CYFRA21-1 mainly reflects the degradation status of cytokeratin 19. Both have high sensitivity in the middle and late stages of lung tumor progression, but insufficient sensitivity in early lung cancer due to low tumor burden; while piRNA regulates tumor stem cell proliferation and transposon silencing through the PIWI / piRNA pathway, and abnormal expression can be detected in plasma in the very early stages of tumor development. Therefore, piRNA and protein markers complement each other on the biological timeline: piRNA compensates for the sensitivity blind spot of protein markers in early lung cancer, while protein markers compensate for the false positives that may occur in piRNA in specific benign lesions, such as active pneumonia and granulomatous inflammation.

[0033] Furthermore, since CT images provide spatial morphological information such as the size, margins, density, and calcification type of lung nodules, which are completely independent of the two types of biomarkers in terms of anatomical and molecular spatial dimensions, we consider using their image information as a third type of biomarker. The three modalities complement each other in the temporal and spatial dimensions of different stages of lung tumor development, and the combined application of these modalities provides a much greater information gain than simple superposition.

[0034] After introducing the basic principles and advantages of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0035] Exemplary System Figure 1 The figure shows a structural block diagram of a multimodal lung nodule identification system according to an embodiment of this application.

[0036] like Figure 1 As shown, the multimodal lung nodule identification system according to an embodiment of this application includes the following parts.

[0037] The acquisition unit obtains multiple chest CT images and multiple biomarker detection results from the subject.

[0038] The acquisition unit utilizes software with a communication interface that uses a common protocol or a communication protocol provided by the existing CT equipment manufacturer to acquire multiple chest CT images of the subject from the CT equipment. For example, it can directly transmit the chest scan results generated by the CT equipment to a computer using the DICOM protocol interface provided by a common PACS system, or obtain chest scan results from the CT equipment through GE's AW Server SDK. Alternatively, chest scan results can be transferred from the CT equipment to the acquisition unit via readable electronic storage devices, such as encrypted hard drives, encrypted USB flash drives, or DICOM encrypted network APIs.

[0039] It is understood that the acquisition unit here, as well as the processing unit, analysis unit and output unit described later, are all electronic devices, computer devices or cloud-based computer program products with microprocessors and readable storage media. They can all be used for data input, storage, processing and output to achieve the functions described above and below.

[0040] After obtaining the chest scan results of the subject, the results usually need to be preprocessed to make them easier for deep learning models to process. Preprocessing includes data format conversion, such as DICOM to NIfTI, which can be achieved using Python packages like nibabel. Preprocessing also includes pixel value calibration. Manufacturers of various CT equipment usually provide compensation parameters based on the age and aging of the equipment to adjust the pixel value of each pixel in the chest scan results, which can be calculated by introducing these compensation parameters. In addition, preprocessing includes isotropic resampling of the result data after format conversion to improve the accuracy of 3D reconstruction, which can be achieved using integrated algorithm packages such as Spacing, as well as noise filtering, which can be achieved using noise reduction algorithms. Specific algorithms will not be listed here.

[0041] In this way, through data acquisition and preprocessing, multiple chest CT images of the required subjects can be obtained. For the other modality of biomarker detection results required by the system described in this application, the detection results of tumor markers CEA and CYFRA21-1 can be directly obtained from the hospital's HIS or LIS system.

[0042] The detection results of nucleic acid biomarkers piR-hsa-8393202 and piR-hsa-8429916 were obtained from the hospital system, research sample bank, or database of cooperating testing institutions. The cooperating testing institutions possessed clinical gene amplification testing laboratory qualifications and passed the interlaboratory quality assessment organized by the National Health Commission's Clinical Laboratory Center. All data related to the above modal biomarkers were obtained with the consent of the subjects and the hospitals and institutions.

[0043] In one example, the detection of piRNA and biomarker screening followed a standardized procedure. Specifically, 5 mL of peripheral venous blood was collected from the subject in an EDTA anticoagulant tube. Plasma was separated by centrifugation at 1600 g for 10 min within 2 hours, aliquoted, and stored at -80°C, avoiding repeated freeze-thaw cycles. Samples subjected to more than two freeze-thaw cycles required re-collection. A commercially available plasma small RNA extraction kit was used, preferably one capable of efficiently recovering piRNA. Total RNA was extracted according to the kit's instructions. After extraction, RNA concentration and purity were determined using NanoDrop One, requiring an A260 / A280 ratio of 1.8-2.1 and a total RNA extraction yield ≥10 ng. Then, a poly(A) tail was added to the 3' end of the piRNA using the tailing method, followed by reverse transcription using oligo(dT) primers and a reverse transcription kit. The reverse transcription products were detected using SYBR Green qPCR, with U6 snRNA as an internal control gene. Optionally, a synthetic spike-in control, cel-miR-39, was added as an exogenous quality control. Preferably, a dual internal control strategy was used, with U6 snRNA correcting for differences in extraction efficiency between samples, and cel-miR-39 correcting for losses during the extraction process. In one example, the quality control criteria included a blank control, a negative control (healthy mixed plasma pool), a positive control (lung cancer patient mixed plasma pool), and a spike-in control (cel-miR-39) for each batch of tests. The negative control was required to have a Ct value ≥35 or no obvious amplification curve, the positive control to have a Ct value ≤30, and the spike-in control to have a Ct value within a preset range to verify extraction efficiency. The coefficient of variation (CV) of the Ct value for the tested samples in replicates was <5%.

[0044] 2 can be used -ΔΔCt The relative expression level was calculated as follows: ΔCt = Ct (target piRNA) – Ct (U6 snRNA), ΔΔCt = ΔCt (test sample) – ΔCt (calibration sample). The calibration sample was a healthy mixed plasma pool standard prepared using the same extraction and detection procedures as the test sample. This standard was prepared, aliquoted, and frozen during the establishment of the reference interval and used for calibration of each batch of tests to ensure the comparability of results between batches. Since the absolute values ​​of expression levels in qRT-PCR relative quantification are highly dependent on the calibration sample batch, reaction system, and instrument baseline threshold and are not directly comparable, a self-constructed reference interval was used in this example instead of directly applying values ​​from external literature.

[0045] Specifically, a healthy mixed plasma pool standard was prepared using the exact same extraction and detection procedures as the test samples, and this standard was used as the calibration benchmark. Based on the relative expression distribution of healthy control samples tested simultaneously in this batch, an internal reference range was initially established. Simultaneously, a relative expression fold increase of ≥2.0 between the malignant group and the healthy group, with a p-value <0.05 between groups, was used as the criterion for positive upregulation of piRNA markers. In this example, the calculated malignant / healthy ratios of piR-hsa-8429916 and piR-hsa-8393202 were 2.57 and 2.23, respectively, meeting the above marker criteria.

[0046] To verify the expression characteristics of the aforementioned piRNA biomarkers in the plasma of patients with malignant pulmonary nodules, this example was further validated by qRT-PCR in an independent cohort. Specifically, a poly(A) tail was added to the 3' end of the piRNA using the tailing method, followed by reverse transcription using oligo(dT) universal primers. Then, using U6 snRNA as an internal reference gene, the candidate piRNAs were detected using SYBR Green qPCR. The specific primer pairs used are shown in Table 1 below: Table 1 Primers for the selected markers

[0047] In this study, piRNA detection employed a tailing method using oligo (dT) universal primers for reverse transcription. Therefore, the reverse primer for piRNA used a universal sequence, not a marker-specific sequence. U6 was an internal reference gene, and a specific reverse primer was used. In this application, all subjects agreed to participate in sample collection and signed informed consent forms. 5 mL of peripheral venous blood was collected from each subject in an EDTA anticoagulant tube. Plasma was separated within 2 hours by centrifugation at 1600 g for 10 min, aliquoted, and stored at -80℃, avoiding repeated freeze-thaw cycles. Total RNA was extracted according to the aforementioned standardized procedure. RNA concentration and purity were determined using NanoDrop One, requiring an A260 / A280 ratio of 1.8-2.1 and a total RNA extraction amount ≥10 ng. Thermo Fisher SuperScript IV reverse transcription kit was used for tailing reverse transcription. The reverse transcription products were then detected by SYBR Green qPCR using Takara TB Green Premix Ex Taq II. The reaction volume was 20 μL, and the reaction conditions were: 95℃ pre-denaturation for 30 s; 95℃ denaturation for 5 s; 60℃ annealing / extension for 34 s, for a total of 40 cycles. For each batch of tests, template-free control, negative control, positive control, and spike-in control were included as described above. The negative control required a Ct value ≥35 or no obvious amplification curve, the positive control required a Ct value ≤30, and the coefficient of variation (CV) of the Ct value in the replicates of the tested samples should be <5%.

[0048] like Figure 4 Melting curve analysis showed that the amplification products of piR-hsa-8429916, piR-hsa-8393202, and the internal control U6 all had a single melting peak. The Tm values ​​of each product were approximately 82.0℃, 83.5℃, and 82.5℃, respectively, with a difference of no more than 2.5℃, indicating good primer specificity and no primer dimers or non-specific amplification were observed. Figure 5 As shown in the representative amplification curves, the amplification curves of the markers exhibit a standard S-shape, and the amplification efficiency is normal. In addition, the standard deviation of the Ct values ​​of each replicate well is <0.25, and the coefficient of variation meets the interlaboratory quality assessment standards.

[0049] Furthermore, adopting 2 -ΔΔCt Relative expression levels were calculated using a method with U6 as an internal reference and calibrated samples from a healthy control group as a baseline. Plasma samples from 7 patients with pathologically confirmed malignant pulmonary nodules and 7 healthy controls were analyzed. The results are shown in Table 2 below: Table 2. Marker Difference Test

[0050] In this example, an independent pool of healthy mixed plasma was used as the calibration benchmark. The absolute values ​​of relative expression levels were affected by batch-to-batch variations in the calibration samples; therefore, the absolute mean value of the malignant group is not directly comparable to the reference range for healthy individuals reported in the literature. The significant trend of fold change between the malignant and healthy groups supports the auxiliary diagnostic value of piRNA biomarkers in the identification of malignant pulmonary nodules. The results of this biomarker detection example show that both piR-hsa-8429916 and piR-hsa-8393202 were significantly upregulated in the plasma of patients with malignant pulmonary nodules, and the difference between the two groups was statistically significant.

[0051] In particular, considering that benign lesions such as inflammatory nodules, tuberculomas, and hamartomas, which resemble malignant nodules morphologically, are prone to high false-positive rates when diagnosing solely based on CT images, the aforementioned piRNA markers are characteristically upregulated in patients with malignant pulmonary nodules, while their expression levels in the plasma of healthy controls are within the reference range. Therefore, when CT images suggest a suspicious nodule but piRNA markers do not reach the positive threshold, it may indicate a benign lesion, thus potentially filtering out false-positive signals from CT morphology. Conversely, when CT images show atypical features but piRNA markers are significantly upregulated, it can compensate for the sensitivity blind spot of imaging in early malignant nodules, potentially improving the early detection rate.

[0052] The analysis unit integrates this data with lung voxel information as an additional data dimension for lung nodule region classification, or utilizes a multi-head attention mechanism to establish voxel-biomarker associations. In the decision layer of lung nodule region classification, biomarker information is used as a gating signal to modulate the activation intensity of voxel information. This measures the importance of each pathological feature in the deeply associated voxels for the benign / malignant classification of lung nodules from the perspective of biomarkers. This is expected to improve the performance of the analysis unit's deep lung nodule classification model, such as helping to reduce the generation of false positive classification results and improving the accuracy of lung nodule identification.

[0053] The processing unit identifies multiple lung tissues of the subject based on multiple chest CT images, reconstructs a lung voxel model using multiple lung tissues, and preprocesses the detection results of multiple biomarkers to obtain biomarker feature vectors.

[0054] The reason for using a lung voxel model instead of lung images as input information for one modality of the analysis unit is that the lung voxel model can better carry the three-dimensional spatial feature information of the lung target region. The inter-slice correlation information between layer-by-layer CT images is difficult to fully utilize if only two-dimensional convolution is used. This is because constructing a lung voxel model can better preserve the original pathological information of lung nodules. For example, a common clinical problem is that layer-by-layer analysis of CT images inevitably leads to the fragmentation of information between slices. Features such as spiculation of lung nodules, which are radially distributed in three-dimensional space, are difficult to obtain in the feature map processing of multiple CT images regardless of the two-dimensional convolution method used. This is because the kernel parameters of the two-dimensional convolution kernel are set independently on different channels of the image, and the channels themselves lack information reflecting the radially distributed pathological features. However, when analyzing a three-dimensional voxel model, the weights along the depth are shared when the convolution kernel in the three-dimensional convolution mode is convolved on the feature map with depth of the voxel model. In this way, each convolution can learn the feature information between depths. The analysis unit can then use its first neural network to better learn the pathological feature information of lung nodules that are widely distributed in the third dimension.

[0055] Specifically, the processing unit first uses a second neural network, such as a segmentation model, to identify and extract voxel-level segmentation results of multiple lung tissues from multiple chest CT images of the subject. Then, it uses these lung tissue segmentation results to reconstruct the subject's lung voxel model. Available segmentation models can include UNet or ResNet, which utilize their convolutional neural networks to extract segmentation mask features of the lung tissues. For example, the U-shaped network structure provided by UNet can be used, where the data encoding and decoding parts each have multiple corresponding layers. Each layer can be understood as a convolutional block containing multiple convolutional layers. After each convolutional block extracts features and obtains a new feature map, it uses downsampling techniques, such as pooling layers, to reduce the feature map size processed by the next level of convolutional blocks to learn more complex features. Similarly, the decoding part uses upsampling techniques, such as deconvolution, after the convolutional block outputs a new feature map to increase the feature map size processed by the next level of convolutional blocks, ensuring that the feature map size is restored at the last level to output the mask. Between each layer, the encoding part copies the feature map output by the convolutional block, performs cropping (usually center cropping), and then performs skip connections, that is, splices it onto the feature map of the corresponding convolutional block to be input in the decoding part, in order to supplement the global features missing in the feature map under the U-shaped structure.

[0056] However, such neural network structures, especially in the encoding part, are prone to problems due to severe information attenuation in feature transfer between multiple layers. There is a significant disconnect between the shallow texture features captured by multiple convolutional layers in lower-level convolutional blocks and the deep semantic features captured by multiple convolutional layers in higher-level convolutional blocks, a problem that worsens with increasing the number of layers. As a result, some lung nodules, which are not difficult to detect but easily overlooked, are ignored by the segmentation model and completely lost during subsequent lung voxel reconstruction. Examples include microcalcifications or spiculated features distributed in multiple adjacent tissues.

[0057] To address this issue and improve the accuracy of voxel modeling, this application does not construct the second neural network in the processing unit entirely based on the UNet network. Instead, it optimizes the data interaction of multiple conventional convolutional layers in each level's convolutional block to reduce the fragmentation between features at different levels. Specifically, as... Figure 2As shown, for each level of convolutional block, multiple convolutional layers are configured to: concatenate the output feature map of the preceding layer to the input feature map of all subsequent layers; and concatenate the output feature map of the last layer with all feature maps of the preceding layers (including the inputs). This ensures that subsequent feature maps retain information from preceding feature maps. By increasing the connection paths between convolutional layers, the feature reuse rate is significantly improved, effectively avoiding the problem of intermediate layer feature information disappearing after multiple cross-layer transitions, thus improving the reconstruction accuracy of the lung voxel model. Furthermore, although this approach increases the information content of the feature maps, it does not increase the number of parameters in the entire network. Moreover, by intentionally connecting the outputs and inputs of preceding and following convolutional layers, redundant and invalid information included in the segmented mask is avoided, further improving the reconstruction efficiency of the lung voxel model.

[0058] In this way, the segmentation model utilizes an improved second neural network based on UNet to receive multiple chest CT images and process them to obtain multiple lung tissue masks. These tissue masks include: trachea, arteries / veins, lobes and segments, as well as masks of suspected nodular high-density areas. A voxel model of the subject's lungs is reconstructed using these tissue masks. The semantic segmentation masks of lung tissues in DICOM format or preprocessed NIfTI format are concatenated into a three-dimensional voxel grid. In Python, the VTK or SimpleITK packages are used to process the segmentation mask sequence to weightedly fuse the segmentation masks of different tissues, generating three-dimensional voxel data aligned with the original CT spatial resolution. The voxel size is preferably consistent with the spacing after isotropic resampling, for example, 1 mm × 1 mm × 1 mm. This lung voxel model is output in the form of a three-dimensional tensor, which can be directly processed by subsequent analysis units or concatenated with the feature tensors of markers.

[0059] In an alternative example, a generative reconstruction method based on a 3D generative adversarial network can be used. Each tissue segmentation mask is input as a channel into the network, and the Dice loss and WGAN-GP gradient penalty are iteratively optimized to generate 3D voxel data. This approach also outputs an a×b×c 3D voxel array and can serve as an alternative to the volume rendering method described above.

[0060] The detection results of multiple biomarkers were preprocessed to obtain biomarker feature vectors. This preprocessing included data cleaning and outlier removal. Based on the hospital's internal quality control rules, biomarker detection results exceeding the instrument's linear range or from batches with out-of-control batches were removed. For missing values, multiple imputation or mean filling based on similar patient groups (e.g., age group, sex, smoking history) was used. For piRNA detection results, samples with RNA concentrations <10 ng / μL or A260 / A280 <1.6 were removed. Unit standardization was also performed to convert tumor biomarker concentrations from different detection platforms to standard units, such as ng / mL for CEA and ng / mL for CYFRA21-1. The conversion formulas were based on calibration coefficients provided by each detection platform or interlaboratory quality assessment reports. piRNA detection results are relative expression levels and do not require unit conversion.

[0061] The preprocessing further includes normalization of the reference interval, converting the detection results of each biomarker into a Z-score to eliminate differences in the dimensions and distributions of different biomarkers. Additionally, the preprocessing includes biomarker feature vector construction, concatenating the normalized detection results of multiple biomarkers into a multidimensional biomarker feature vector. Optionally, the preprocessing also includes temporal feature extension; when there are multiple biomarker detection records for the same subject at different time periods, such as t1, t2, etc., the dynamic change rate of the biomarker is calculated: Δz. i = (z i (t2)-z i (t1)) / (t2- t1), the rate of change Δz i The feature vector is appended to form an extended marker feature vector, increasing its dimensionality. Furthermore, preprocessing includes modal dimension alignment. Since the marker feature vector is a one-dimensional vector, its dimension needs to be expanded to align with the voxel tensors of the three-dimensional lung voxel model. Therefore, the marker feature vector... The feature maps are mapped to the channel dimension of the convolutional layer through the fully connected layer: (Formula 1) in, For the mapping matrix, The bias vector is used, and ReLU is the activation function. The vector is upsampled through deconvolution to a feature tensor matching the dimensions of the voxel tensor space. In a specific processing example, the configuration of the deconvolution layer is shown in Table 3 below: Table 3. Deconvolution parameters of feature vectors of markers

[0062] Thus, after multiple deconvolutions, the spatial dimension expands from 1×1×1 to D×H×W, thereby satisfying the alignment requirements of features between modalities. The number of deconvolution layers can preferably be log 2 (min (D, H, W)). For example, when the voxel tensor spatial dimension is 64×64×64, 6 deconvolution layers are required. In one example, the number of channels C is set to 32 or 64. After multiple deconvolutions, the output is a marker feature tensor aligned with the voxel tensor spatial dimension. The total number of parameters is controllable, meeting the computational resource constraints of the clinical deployment environment.

[0063] Specifically, the system employs two methods for fusing voxel and biochemical marker features across different modalities. In the first method, the analysis unit stitches together features from different modalities before processing the data, directly using the fused 3D feature map as input to determine the location and benign or malignant nature of lung nodules. Specifically, the voxel tensor value of the lung voxel model reconstructed by the processing unit is... F v = [a, b, c, c, c], expanding the dimension of the marker feature vector [a', b', 1, 1] to [a', b', 1, 1, 1], and then upsampling it through deconvolution to obtain the marker feature tensor. F b = [a',b',c,c,c], according to F b The weighting is determined by the gating condition modulating the contribution of the voxel tensor to the classification. Specifically, a gating signal G is set, which is generated by global average pooling of the marker feature tensor followed by Sigmoid activation: (Formula 2) in GAP For global average pooling, For the mapping matrix, The bias matrix, The output G is a sigmoid function, with each channel element taking values ​​in the range [0,1], resulting in the modulated voxel features becoming: (Formula 3) in, For element-wise multiplication, for The baseline feature map is transformed by a 1×1×1 convolutional kernel or a 3D average pooling layer. This transformation is used to reduce the dimensionality of the original voxel features or extract multi-scale information, enabling the gating signal G to be applied to the original features. Features after transformation Selective weighting is applied between the two channels; when the feature tensor of the marker indicates that the marker is significantly abnormal, the gate value of the corresponding channel approaches 1, which enhances the contribution of the voxel features of that channel to the classification, and vice versa.

[0064] The weighted features are then used again as voxel tensor values ​​for the lung voxel model. The tensor for aligning the two modes by channel is: F fuse = α F v (1 - α) F b (Formula 4) Here, ⊕ represents the channel concatenation operation, and α is the weight value of the feature tensor of the two modalities. The initial value can be set to α = 0.5, which can be set as a learnable model parameter during the training phase of the first neural network in the analysis unit. This parameter can be dynamically adjusted during the iteration of the first neural network to optimize the network's classification performance. F fuse It can be used as the input feature map of the analysis unit. It has three dimensions and includes not only the spatial distribution information and time series information of multiple lung images of the subject, but also the expression information of four biomarkers piR-hsa-8393202, piR-hsa-8429916, CEA and CYFRA21-1 in the subject. The modal information is complementary and improves the richness of features.

[0065] In the second approach, a cross-modal attention mechanism can be used to fuse the voxel tensors and feature vectors of the lung voxel model. After the voxel tensors and feature vectors are input into the first neural network of the analysis unit, a tiny version of the 3D-Swin Transformer (patch size = 4×4×4, embeddim = 48) based on window multi-head self-attention is introduced to extract multi-scale feature maps of the voxel tensors, for example, those extracted from the voxel tensors. F v = [a, b, c, c, c], [a, 2b, c / 2, c / 2, c / 2] and [a, 4b, c / 4, c / 4, c / 4], while the feature vectors of the markers still retain their original form. F b Then, cross-modal attention is used to compute multi-level attention between voxel tensors and feature vectors, including: (Formula 5) for and Channel attention between For integration and The channel weight matrix is ​​used in the system described in this application. Attention is applied to different voxel tensors to assign weights to the channels of each voxel tensor for classifying lung nodules as benign or malignant; multi-level attention also includes: (Formula 6) for and Spatial attention between , They are respectively Query and The key, The scaling factor in attention calculation is also utilized in the system described in this application. The weights of different voxel tensor locations for classifying lung nodules as benign or malignant are assigned based on the attention level of different voxel tensors.

[0066] In this way, by using the aforementioned cross-modal attention to calculate the mutual cross-attention between the two modalities, the features of the marker modality can be used as gating signals to adjust the weighting of the features of the voxel modality. This allows the first neural network, with the help of the features of the marker modality, to focus more on the important information in the voxel tensor and reduce the adverse effects of redundant features or features with low classification contribution on lung nodule classification. Preferably, this channel attention and spatial attention network are set in the fusion layer of the first neural network of the analysis unit (i.e., before the classification head), which can intuitively adjust the weighting of voxel features, while the marker feature vector, after upsampling and expansion, is only aligned with the voxel tensor as the input feature map.

[0067] The analysis unit fuses the lung voxel model with the feature vectors of the markers to extract lung nodule features, thereby obtaining the location and classification information of the lung nodules. The first neural network includes a lung nodule detection unit and a lung nodule classification unit. Specifically, after acquiring the input feature maps to be fused / to be fused, the first neural network first uses its lung nodule detection unit to detect the target lung nodule region to obtain location information, and then uses the lung nodule classification unit to determine whether the target lung nodule region is benign or malignant. In one example, the lung nodule detection unit is a region generation network. After inputting the feature map through a sliding window convolution, it passes through two fully connected layers to generate two parts: a score and an offset value for the target lung nodule region. The offset value is the coordinate difference between the center point of the candidate box and the center point of the actual lung nodule, i.e., the coordinate difference between the blue point and the green point. It is understandable that, since the input feature map is a three-dimensional feature map, the sliding window used by the region generation network is set to a three-dimensional sliding window. When it slides in the depth dimension of the input feature map, it shares the same convolution parameters, which can better capture the spatial relationships in the input feature map that is mainly based on the lung voxel model. This is especially important for the identification of lung nodules.

[0068] The classification features of pulmonary nodules extracted by the pulmonary nodule classification unit mainly come from the main clinical signs of pulmonary nodules reconstructed from voxel models. These clinical signs are the characteristics of pulmonary nodules as seen on imaging, including their high-density nodule body, abnormal margins that distinguish them from normal lung tissue, and induced lesions. The pulmonary nodule classification unit extracts these pathological features as effective information for classification. Specifically, according to the Fleischner Society consensus proposed in 2017, nodules are classified into three categories based on imaging density: solid nodules, which are high-density and can completely or almost completely obscure pulmonary vascular markings; pure ground-glass nodules, which are semi-transparent high-density shadows, with pulmonary vessels and bronchial markings visible; and partially solid nodules, which contain both ground-glass-like low-density areas and solid high-density areas within the lesion. For the above nodules, their margin morphology includes smooth, lobulated, spiculated, and indistinct borders; while the interior of the margin is mainly composed of homogeneous solid, localized calcification (central, homogeneous, patchy, or punctate), cystic degeneration, or necrosis.

[0069] The purpose of using chest CT images for 3D modeling and nodule detection is to better consider the pathological features of ground-glass nodules and some solid nodules, such as irregular edges and localized calcifications, which is currently lacking in the field of intelligent lung nodule interpretation. The characteristics of 3D convolution kernels allow them to instantiate these pathological features by incorporating depth information into existing cross-sectional lung images, while introducing biomarker detection information can help the network further determine the severity of these features.

[0070] During the training phase, the loss value of the first neural network is determined as follows: Figure 3As shown, in each iteration, the candidate box offset value is used to continuously correct the position of each point in the target lung nodule region displayed by the candidate box score, so as to optimize the target detection capability of the lung nodule detection unit. For the lung nodule identification stage of the analysis unit, after detecting the correct target lung nodule region, or at least as close as possible to the correct position of the lung nodule in the lung voxel model, the lung nodule classification unit uses the target lung nodule region as input and performs binary classification on the region by extracting classification features. In one example, the lung nodule classification unit is set as a three-dimensional convolutional neural network, including multiple convolutional layers with three-dimensional convolutional kernels. In a further example, residual connections are set for each convolutional layer to avoid the classification performance degradation caused by gradient vanishing. In another example, at least some convolutional layers are equipped with cross-modal attention heads, which use the information of the marker modality to gate the contribution of each voxel tensor to the classification of lung nodules as benign or malignant, so as to complete the classification of the benign or malignant lung nodules of the subject.

[0071] Specifically, before using the lung nodule detection unit to detect the target lung nodule region, a segmentation network can be used to preliminarily segment the region where the lung nodule is located in the input feature map. The preliminarily segmented region is then used as the input data for the lung nodule detection unit to improve its detection accuracy. The classification network is a ResNet 18 with three-dimensional convolutional kernels.

[0072] The output unit outputs the classification information of the lung nodules obtained by the analysis unit. The output unit receives the feature map finally output by the lung nodule classification unit and converts it into classification probabilities for the lung nodule region; the result with the highest probability is the classification result. This output unit can be a pooling layer, preferably a combination of a pyramid pooling layer and one or two fully connected layers, which can better cooperate with the aforementioned cross-modal attention mechanism to generate effective information.

[0073] As described above, the effectiveness of the system is demonstrated from the perspective of fundamental principles of information theory, medical imaging, and molecular biology. The CT images used in the system provide morphological features of malignant pulmonary nodules, piRNA markers reflect epigenetic changes in lung tumor cells, and CEA / CYFRA21-1 reflects the expression of lung tumor-associated antigens at the protein level. These three information sources are biologically independent, but their combined use increases information entropy. From another perspective, according to the mutual information principle in information theory, the fusion of independent information sources can reduce classification uncertainty, and its gain effect can be qualitatively described as: I(F fuse ;Y) ≥ max {I (F v ; Y), I (F b ; Y)}, where I is mutual information, F fuse As a characteristic of fusion, F v As a voxel characteristic, F bAs a biomarker feature, Y represents the benign or malignant label of lung nodules. The information content of the fused feature is no less than that of any single modality feature, which can supplement independent discriminative information for CT images at the molecular level. In conventional CT diagnosis of lung nodules, benign lesions such as inflammatory nodules, tuberculomas, and hamartomas can lead to a high false positive rate because their morphological characteristics are similar to those of malignant nodules. After introducing piRNA and protein biomarkers, piR-hsa-8393202 and piR-hsa-8429916 are characteristically upregulated in malignant lesions, while their expression levels in the plasma of patients with benign lesions are usually within the healthy reference range. By supplementing the information with protein biomarkers, benign and malignant lung nodules can be distinguished at the molecular level, effectively filtering false positive signals from CT morphology.

[0074] Furthermore, the sensitivity of tumor marker detection alone is limited for early-stage lung cancer because the tumor burden is low in the early stages, and the markers have not yet risen significantly. By fusing the three-dimensional morphological features of CT images, when images suggest suspicious nodules but markers have not reached the positive threshold, the spiculation, lobulation, and pleural traction features extracted by three-dimensional convolution can assist in the judgment, potentially compensating for the sensitivity blind spot of marker detection. Compared to two-dimensional convolutional analysis, the three-dimensional voxel model reconstructs lung tissue using the network structure proposed in this application. The three-dimensional convolution kernels share weights in the depth direction, capturing spatial features lost in two-dimensional methods, such as the three-dimensional radial distribution of spiculation and the three-dimensional concave-convex morphology of lobulation. This advantage is an inherent characteristic of three-dimensional convolutional neural networks and has been widely validated in the field of medical image segmentation.

[0075] Furthermore, the system applies cross-modal gated attention in the model, automatically enhancing the activation intensity of malignant feature channels when biomarkers are abnormal and suppressing redundant features when biomarkers are normal, significantly outperforming single CT imaging models or single serum biomarker models. Building upon this, the system introduces piRNA epigenetic biomarkers, forming a trimodal fusion of imaging, protein biomarkers, and nucleic acid biomarkers, resulting in improvements in lung nodule detection in terms of specificity, sensitivity, and other aspects.

[0076] Exemplary methods This application also provides a method for constructing a lung segmentation model, a lung nodule detection and classification model, for constructing a model for segmenting lung tissue and a model for identifying lung nodule regions in a three-dimensional lung model.

[0077] Figure 6 The illustration shows a flowchart of a method for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application.

[0078] like Figure 6 As shown, the method for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application includes the following steps.

[0079] Step S110: Input multiple chest CT images of the first and second populations into the lung segmentation model, set the initial hyperparameters of the optimizer, calculate the first loss function value, and iterate the lung segmentation model through gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value drops to a preset level. Step S120: Using the iterated lung segmentation model, output the lung tissue mask of the multiple chest CT images, reconstruct the lung voxel model, preprocess and upsample the marker detection results of the first and second populations and align them with the lung voxel model to obtain the input feature map; Step S130: Input the input feature map into the lung nodule detection and classification model, set the initial hyperparameters of the optimizer, calculate the second loss function value, and iterate the lung nodule detection and classification model through gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value drops to a preset level.

[0080] Specifically, one of the first and second populations consists of individuals diagnosed with malignant pulmonary nodules, while the other consists of individuals diagnosed with benign or no pulmonary nodules. The lung tissue mask is the segmentation mask for the trachea, arteries / veins, lesions, lobes, and segments of the lungs used to reconstruct the voxel model of the lungs. The biomarker detection results include the detection results of nucleic acid biomarkers piR-hsa-8393202 and piR-hsa-8429916, the detection results of tumor biomarkers CEA and CYFRA21-1. The first loss function value is the weighted sum of the Dice loss function value and the cross-entropy loss function value, with each of the Dice loss function value and the cross-entropy loss function value having a weight of 0.5. The second loss function value includes detection loss and classification loss. The detection loss includes a classification loss term and a regression loss term, where the classification loss term uses the cross-entropy loss function to distinguish between foreground and background, and the regression loss term can use the Smooth L1 loss function to correct the candidate box position offset. The classification loss uses the cross-entropy loss function to distinguish between benign and malignant pulmonary nodules. The initial learning rates of the detection and classification parts are the same. The backpropagation gradient is initially exclusively used by the detection part, and after a predetermined number of iterations, it is shared by the detection and classification parts with a weighted average. The initial weights are 0.5 for each part. The Adam optimizer is preferred.

[0081] As can be seen from the above, the lung segmentation model, lung nodule detection and classification model construction method provided in this application can be used to sequentially construct the segmentation model of the second neural network and the lung nodule detection unit and lung nodule classification unit contained in the first neural network according to the foregoing embodiments of this application. Therefore, further data processing, such as the reconstruction of the lung voxel model, the preprocessing of marker detection results and upsampling, will not be repeated here.

[0082] Exemplary device Figure 7 The diagram illustrates a block diagram of an apparatus for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application.

[0083] like Figure 7 As shown, the apparatus for constructing a lung segmentation model, a lung nodule detection and classification model according to an embodiment of this application includes the following units.

[0084] The first training unit 210 inputs multiple chest CT images of the first and second populations into the lung segmentation model, sets the initial hyperparameters of the optimizer, calculates the first loss function value, and iterates the lung segmentation model through gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value drops to a preset level. The reconstruction and alignment unit 220 uses the iterated lung segmentation model to output the lung tissue mask of the multiple chest CT images, reconstructs the lung voxel model, preprocesses and upsamples the marker detection results of the first and second populations and aligns them with the lung voxel model to obtain the input feature map. The second training unit 230 inputs the feature map into the lung nodule detection and classification model, sets the initial hyperparameters of the optimizer, calculates the second loss function value, and iterates the lung nodule detection and classification model through gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value drops to a preset level.

[0085] Here, those skilled in the art will understand that the specific functions and operations of each unit and module in the above-described lung segmentation model, lung nodule detection, and classification model construction device 200 have been referenced above. Figure 6 The methods for constructing lung segmentation models, lung nodule detection and classification models are described in detail in the previous section, and therefore, their repeated descriptions will be omitted.

[0086] As described above, the lung segmentation model, lung nodule detection, and classification model construction apparatus 200 according to the embodiments of this application can be implemented in various terminal devices, such as servers for storing multiple CT images and lung voxel models. In some examples, the lung segmentation model, lung nodule detection, and classification model construction apparatus 200 according to the embodiments of this application can be integrated into the terminal device as a software module and / or a hardware module. For example, the lung segmentation model, lung nodule detection, and classification model construction apparatus 200 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the lung segmentation model, lung nodule detection, and classification model construction apparatus 200 can also be one of many hardware modules of the terminal device.

[0087] Alternatively, in other examples, the lung segmentation model, lung nodule detection and classification model construction device 200 and the terminal device may also be separate devices, and the lung segmentation model, lung nodule detection and classification model construction device 200 may be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0088] Exemplary electronic devices Figure 8 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0089] like Figure 8 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0090] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0091] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the lung segmentation model, lung nodule detection and classification model construction method, and / or other desired functions described in the various embodiments of this application above. The computer-readable storage medium may also store various contents such as multiple CT images, first loss function values, second loss function values, etc.

[0092] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0093] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0094] The output device 14 can output various information to the outside, including location information and classification information of lung nodule regions. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0095] Of course, for the sake of simplicity, Figure 8Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0096] Exemplary computer program products and computer-readable storage media In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for constructing lung segmentation models, lung nodule detection and classification models according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0097] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C, Python, or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for constructing lung segmentation models, lung nodule detection and classification models according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0099] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0100] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0101] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0102] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0103] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0104] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A multimodal lung nodule identification system, characterized in that, include: The acquisition unit obtains multiple chest CT images and multiple biomarker detection results from the subject; The processing unit identifies multiple lung tissues of the subject through a second neural network based on the multiple chest CT images, reconstructs a lung voxel model using the multiple lung tissues, and preprocesses the detection results of the multiple biomarkers to obtain biomarker feature vectors. The analysis unit integrates the lung voxel model with the marker feature vector and extracts lung nodule features through a first neural network to obtain the location and classification information of the lung nodules. The output unit outputs the location information and classification information of the lung nodules obtained by the analysis unit. The results of the multiple biomarkers include the results of nucleic acid biomarker piR-hsa-8393202, nucleic acid biomarker piR-hsa-8429916, tumor biomarker CEA, and tumor biomarker CYFRA21-1.

2. The multimodal lung nodule identification system according to claim 1, characterized in that, The analysis unit is configured to fuse the lung voxel model and the biomarker feature vector before extracting the lung nodule features. The fusion of the lung voxel model and the biomarker feature vector, followed by the extraction of lung nodule features through a first neural network, includes: The upsampled feature vector of the marker is a feature tensor, which is aligned with the voxel tensor of the lung voxel model and concatenated to form an input feature map; The lung nodule features in the input feature map are extracted using a first neural network to determine the lung nodule region in the lung voxel model; The first neural network is used to further classify the lung nodule region to determine whether the lung nodule region is benign or malignant.

3. The multimodal lung nodule identification system according to claim 1, characterized in that, The analysis unit is configured to fuse the lung voxel model and the biomarker feature vector during the extraction of lung nodule features. The fusion of the lung voxel model and the biomarker feature vector, and the extraction of lung nodule features through a first neural network, includes: Calculate the attention weights between the voxel tensor and the marker feature vectors to weight the marker feature vectors, and sum the voxel tensor and the weighted marker feature vectors to obtain the input feature map; The lung nodule features in the input feature map are extracted using a first neural network to determine the lung nodule region in the lung voxel model; The first neural network is used to further classify the lung nodule region to determine whether the lung nodule region is benign or malignant.

4. The multimodal lung nodule identification system according to claim 2 or 3, characterized in that, The first neural network includes a lung nodule detection unit and a lung nodule classification unit; The lung nodule detection unit receives the input feature map and extracts the lung nodule features using multiple first convolutional blocks, each with residual connections, to obtain the detection region of the lung nodule region. The lung nodule classification unit extracts pathological features of the lung nodule region based on the detection region using multiple second convolutional blocks, so as to perform binary classification of the lung nodule region as benign or malignant.

5. The multimodal lung nodule identification system according to claim 4, characterized in that, The lung nodule detection unit also includes multiple third convolutional layers, which are used to further determine the fractions of the lung nodule region and the non-lung nodule region in the detection region by sliding window convolution, so as to obtain the lung nodule region; The lung nodule detection unit further includes multiple fourth convolutional layers, which are used to obtain the offset value between the lung nodule region and the corresponding candidate region based on the lung nodule region determined by the multiple third convolutional layers, so as to correct the position of the lung nodule region.

6. The multimodal lung nodule identification system according to claim 2, characterized in that, When the first neural network classifies the lung nodule region, it modulates the contribution of the voxel tensor to the classification based on the weighting of the marker feature tensor, thereby reducing the dimensionality of the voxel tensor and extracting multi-scale information.

7. The multimodal lung nodule identification system according to claim 2 or 3, characterized in that, The first neural network uses a 3D convolution kernel.

8. The multimodal lung nodule identification system according to claim 1, characterized in that, Based on the multiple chest CT images, a second neural network identifies multiple lung tissues of the subject, including: The first feature map is obtained by performing multiple convolutions and downsampling on each of the multiple chest CT images using a second neural network. The second feature map is then obtained by performing multiple convolutions and upsampling operations on the first feature map. The second feature map is then input into a convolutional layer and a pooling layer to obtain a segmentation mask for the multiple lung tissues. The second neural network copies and prunes the feature map before each downsampling and then connects it to the feature map after each upsampling. The second neural network outputs segmentation masks for the trachea, arteries / veins, lobes, and segments of the lungs, as well as high-density area masks for suspected nodules; the high-density area masks for suspected nodules are configured to be output to the lung nodule detection unit of the first neural network to limit the candidate detection area.

9. The multimodal lung nodule identification system according to claim 8, characterized in that, Each convolution in the second neural network's multiple convolutions of each chest CT image is implemented through multiple third convolutional blocks; For each of the plurality of third convolutional blocks: wherein the feature map output by each convolutional layer is concatenated with the feature maps input by all convolutional layers following that convolutional layer; The feature map of the first convolutional layer is concatenated with the feature maps output by each convolutional layer to form the feature map output by the third convolutional block.

10. The multimodal lung nodule identification system according to claim 1, characterized in that, The processing unit reconstructs a lung voxel model using the multiple lung tissues, including: The lung voxel model is generated by stitching together the segmentation masks of each tissue into three-dimensional voxel data using a volume rendering method. Data cleaning, outlier removal, unit standardization, reference interval normalization, marker feature vector construction, and modal dimension alignment are performed on the detection results of the multiple markers to obtain the marker feature vectors.

11. A method for constructing lung segmentation models, lung nodule detection and classification models, characterized in that, include: Multiple chest CT images of the first and second populations are input into the lung segmentation model. The initial hyperparameters of the optimizer are set, the first loss function value is calculated, and the lung segmentation model is iterated by gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value is reduced to a preset level. The lung tissue mask of the multiple chest CT images is output using the iterated lung segmentation model. The lung voxel model is reconstructed, and the marker detection results of the first and second populations are preprocessed and upsampled and aligned with the lung voxel model to obtain the input feature map. The input feature map is input into the lung nodule detection and classification model. The initial hyperparameters of the optimizer are set, the second loss function value is calculated, and the lung nodule detection and classification model is iterated by gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value is reduced to a preset level. Wherein, the lung segmentation model is the second neural network as described in any one of claims 1-10, and the lung nodule detection and classification model is the first neural network as described in any one of claims 1-10; The first group consists of patients with malignant pulmonary nodules, and the second group consists of patients with benign pulmonary nodules.

12. A device for constructing lung segmentation models, lung nodule detection and classification models, characterized in that, include: The first training unit is used to input multiple chest CT images of the first and second populations into the lung segmentation model, set the initial hyperparameters of the optimizer, calculate the first loss function value, and iterate the lung segmentation model through gradient descent of the optimizer until the maximum number of iterations is reached or the first loss function value drops to a preset level. The reconstruction and alignment unit is used to output the lung tissue mask of the multiple chest CT images using the iterated lung segmentation model, reconstruct the lung voxel model, preprocess and upsample the marker detection results of the first and second populations and align them with the lung voxel model to obtain the input feature map. The second training unit is used to input the input feature map into the lung nodule detection and classification model, set the initial hyperparameters of the optimizer, calculate the second loss function value, and iterate the lung nodule detection and classification model through gradient descent of the optimizer until the maximum number of iterations is reached or the second loss function value drops to a preset level.

13. An electronic device, characterized in that, include: processor; as well as A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the method for constructing the lung segmentation model, lung nodule detection, and classification model as described in claim 11.

14. A computer program product, characterized in that, Includes computer program instructions; When the computer program instructions are executed by the processor, the processor performs the method for constructing the lung segmentation model, lung nodule detection and classification model as described in claim 11.

15. A computer-readable storage medium, characterized in that, It stores computer program instructions; When the computer program instructions are executed by the processor, the processor performs the method for constructing the lung segmentation model, lung nodule detection and classification model as described in claim 11.

Images