Pulmonary nodule malignancy probability prediction method based on radiomics and clinical features

Abstract

The invention discloses a pulmonary nodule malignancy probability prediction method based on radiomics and clinical features, and the method comprises the following steps: collecting and processing original lung image data, and extracting pulmonary nodule image blocks; collecting and processing clinical data of a target patient, and constructing a structured clinical feature vector; inputting the pulmonary nodule image block and the structured feature vector into a dual-channel adaptive fusion network, and extracting an image feature vector and a structured embedded feature vector; a modal sensing module judges that a modal is missing, a modal compensation module is started when the modal is missing, a pseudo-dual-channel output feature pair is generated, and otherwise, a cross attention fusion feature vector is generated; a dynamic fusion module carries out dynamic fusion on the cross attention fusion feature vector or the pseudo dual-channel feature pair, and outputs a dynamic fusion feature vector; and inputting the dynamic fusion feature vector into a classifier, and outputting a malignant probability value of the pulmonary nodule. According to the invention, a dual-channel adaptive fusion network is adopted to realize the intelligent prediction of the malignant probability of pulmonary nodules.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical artificial intelligence, and in particular to a lung nodule malignancy probability prediction method based on imageomics and clinical features. BACKGROUND

[0002] With the continuous rise in the incidence of lung cancer, computer tomography (CT) based lung nodule early detection technology has become an important means of clinical routine screening. Existing auxiliary diagnosis systems usually rely on artificial rules or single modality deep learning models to classify nodules as benign or malignant, mainly through pattern recognition by extracting image density, shape, and edge dominant features. On this basis, some studies introduce imageomics methods to enhance the representation ability of nodules by using high-throughput image features, thereby improving the recognition accuracy of the model. However, due to the reliance on image features alone, it is often difficult to fully depict the individual risk factors of patients.

[0003] In recent years, multi-modal learning methods have been gradually applied to lung nodule diagnosis tasks, by jointly modeling image features and structured clinical data, the stability of the prediction has been improved to some extent. However, existing multi-modal fusion methods generally use static splicing strategies, which fail to dynamically adjust the model path according to the modality integrity of the sample, resulting in decreased inference ability in actual scenarios where modalities are missing. In addition, most fusion strategies only perform shallow splicing, lacking deep semantic interaction mechanisms, and the complementary information between images and structured features is not fully utilized.

[0004] Therefore, how to provide a lung nodule malignancy probability prediction method based on imageomics and clinical features is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide a lung nodule malignancy probability prediction method based on imageomics and clinical features. The present application constructs a dual-path adaptive fusion network, jointly uses a three-dimensional convolutional neural network to extract spatial semantic features of nodule images and a multilayer perceptron to extract structured clinical features, and through a modality perception control mechanism, a cross-attention module, and a modality compensation strategy, realizes deep semantic interaction and dynamic fusion of multi-modal features, and has the advantages of adapting to modality missing, strong prediction result stability, and high clinical generalizability.

[0006] The lung nodule malignancy probability prediction method based on imageomics and clinical features according to the embodiment of the present application comprises the following steps:

[0007] Performing chest computer tomography on a target patient, collecting lung original image data, and performing image preprocessing operations to extract lung nodule image blocks;

[0008] Collecting structured clinical data of a target patient and performing standardized coding processing to construct a structured clinical feature vector;

[0009] Inputting the lung nodule image block and the structured feature vector into an image channel and a structured channel of a dual-channel adaptive fusion network respectively to extract an image feature vector and a structured embedding feature vector respectively;

[0010] Judging a modality loss by a modality perception module of the dual-channel adaptive fusion network, enabling a modality compensation module when the modality is lost to generate a pseudo dual-channel output feature pair, or generating a cross-attention fusion feature vector;

[0011] Performing dynamic fusion on the cross-attention fusion feature vector or the pseudo dual-channel feature pair by a dynamic fusion module of the dual-channel adaptive fusion network to output a dynamic fusion feature vector;

[0012] Inputting the dynamic fusion feature vector into a classifier to perform nonlinear mapping and Sigmoid activation operation and output a malignancy probability value of the lung nodule.

[0013] Optionally, the image preprocessing operation includes window width and window level adjustment, three-dimensional median filter denoising, lung parenchyma region extraction, lung nodule candidate region positioning, and image block cropping.

[0014] Optionally, the collecting structured clinical data of a target patient and performing standardized coding processing to construct a structured clinical feature vector specifically includes:

[0015] Collecting structured clinical data of a target patient, the structured clinical data including age, gender, smoking history, family tumor history, basic comorbidity identification, and nodule follow-up time to construct a clinical original feature vector;

[0016] Performing missing value processing on the clinical original feature vector to generate a complete clinical feature vector;

[0017] Using Z-score standardization for numerical features in the complete clinical feature vector and using one-hot encoding for categorical features to discretize each categorical value into an independent binary vector;

[0018] Concatenating the normalized numerical features and the discretized categorical features according to the original dimension order to obtain a structured clinical feature vector.

[0019] Optionally, the inputting the lung nodule image block and the structured feature vector into an image channel and a structured channel of a dual-channel adaptive fusion network respectively to extract an image feature vector and a structured embedding feature vector specifically includes:

[0020] A dual-path adaptive fusion network is constructed, which includes an image path, a structured path, a modality perception module, a cross-attention module, a modality compensation module, and a dynamic fusion module.

[0021] The image path is a multi-layer convolutional neural network structure, which includes a first convolutional layer, a first batch normalization layer, a first activation layer, a first max pooling layer, a second convolutional layer, a second batch normalization layer, a second activation layer, and a second max pooling layer connected in sequence.

[0022] The image pathway uses three-dimensional convolution to extract features from lung nodule image blocks, obtaining image feature vectors;

[0023] The structured clinical feature vector is input into the structured pathway and processed sequentially through the input layer, two hidden layers, and the output layer to obtain the structured embedded feature vector.

[0024] Optionally, the modality perception module of the dual-path adaptive fusion network determines modality loss. If a modality is missing, the modality compensation module is activated to generate a pseudo-dual-path output feature pair; otherwise, the cross-attention fusion feature vector is generated. Specifically, this includes:

[0025] The image feature vector output from the image path and the structured embedding feature vector output from the structured path are input into the modality perception module, and the validity flag is used to determine whether the image modality or structured modality is missing.

[0026] When the modality perception module determines that the dual-modality is available, the cross-attention module is activated to perform bidirectional feature interaction operations.

[0027] The attention output vector guided by the image direction and the attention output vector guided by the structured image direction are concatenated to construct a cross-attention fusion feature vector.

[0028] When the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair.

[0029] Optionally, the step of inputting the image feature vector output from the image path and the structured embedding feature vector output from the structured path into the modality perception module, and determining whether the image modality or structured modality is missing based on the validity flag, specifically includes:

[0030] The image feature vector output from the image path and the structured embedded feature vector output from the structured path are input into the modality perception module;

[0031] The validity of image feature vectors is judged, and the L1 norm is used as the criterion to obtain the validity indicator of image pathways;

[0032] The effectiveness of structured embedding feature vectors is judged using the L1 norm as the criterion, and the effectiveness indicator of structured pathways is obtained.

[0033] Based on the combination of the values ​​of the image pathway validity flag and the structured pathway validity flag, the modality status flag is determined. The modality status flag includes a bimodal availability status, a single modality missing status, and a complete lack of modalities status.

[0034] Optionally, the step of enabling the cross-attention module to perform bidirectional feature interaction operations when the modality perception module determines that the modality is available specifically includes:

[0035] When the modality perception module determines that the dual-modality is available, the cross-attention module is activated to perform bidirectional attention calculation on the image feature vector and the structured embedding feature vector;

[0036] Perform a linear mapping operation on the image feature vector to generate an image query matrix. At the same time, perform a linear mapping operation on the structured embedding feature vector to generate a structured key matrix and a structured value matrix. Calculate the attention output vector that guides the structured direction of the image.

[0037] A linear mapping operation is performed on the structured embedding feature vector to generate a structured query matrix. Simultaneously, a linear mapping operation is performed on the image feature vector to generate an image key matrix and an image value matrix. The attention output vector for structured guided image orientation is then calculated.

[0038] Optionally, when the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair, specifically including:

[0039] If the modal perception module determines that a single modality is missing, the modal compensation module is activated.

[0040] When the image modality is missing but the structured modality is valid, the modality compensation module inputs the structured embedded feature vector into the structured guided image compensation function to generate the estimated image feature vector.

[0041] When the structured path is missing but the image path is valid, the modal compensation module inputs the image feature vector into the image-guided structured compensation function to generate an estimated structured embedding feature vector.

[0042] The estimated feature vector generated by the modality compensation module is combined with the effective feature vectors existing in the current sample to form a pseudo dual-path feature pair that simultaneously contains image information and structured information.

[0043] Optionally, the dynamic fusion module of the dual-path adaptive fusion network dynamically fuses the cross-attention fusion features or pseudo-dual-path feature pairs, and outputs a dynamic fusion feature vector, specifically including:

[0044] The cross-attention fusion feature vector output by the cross-attention module or the pseudo-dual-path feature pair output by the modality compensation module is input into the dynamic fusion module;

[0045] When the input is a cross-attention fusion feature vector, the dynamic fusion module performs a non-linear transformation on the cross-attention fusion feature vector to obtain a dynamic fusion feature vector.

[0046] When the input is a pseudo-dual-path feature pair, the dynamic fusion module performs a linear mapping on each feature vector to obtain image features and structured features with a unified dimension of feature representation;

[0047] Calculate the fusion weight coefficients corresponding to image features and the fusion weight coefficients corresponding to structured features respectively;

[0048] Based on the fusion weight coefficient, image features and structured features of the same dimension are weighted and fused to obtain a dynamic fused feature vector.

[0049] Optionally, the step of inputting the fused feature vector into the classifier, performing nonlinear mapping and sigmoid activation operations, and outputting the malignancy probability value of the lung nodule specifically includes:

[0050] The dynamic fusion feature vector output by the dynamic fusion module is input into the classifier, which consists of at least one fully connected layer, a batch normalization layer, a nonlinear activation function layer, and an output layer.

[0051] In the first fully connected layer of the classifier, the dynamically fused feature vector is linearly mapped to obtain the intermediate feature representation;

[0052] Batch normalization and nonlinear activation are performed on the intermediate feature representations, with the modified linear unit selected as the activation function, to obtain the activated feature representations.

[0053] The activated feature representation is input into the second fully connected layer, the linear mapping result of the output layer is calculated, and the Sigmoid activation function is applied to the linear mapping result of the output layer to obtain the predicted value of the malignancy probability of the lung nodule.

[0054] The beneficial effects of this invention are:

[0055] This invention constructs a dual-pathway adaptive fusion network, structurally enabling parallel modeling and deep interaction of image features and structured clinical features. It extracts spatial texture and density distribution information of lung nodules through the image pathway and captures individual patient risk factors through the structured pathway. Combined with a modality perception mechanism, it can dynamically identify modality integrity based on sample input, thereby controlling subsequent processing paths. This solves the practical problem of traditional multimodal models failing to operate stably in the presence of missing modalities, improving the model's robustness in clinical applications.

[0056] This invention introduces a cross-attention mechanism under the condition of simultaneous dual-modal data. Through bidirectional attention calculation, it establishes semantic associations between images and structured features, enabling the model to fully exploit complementary information between different modalities at the feature level. Compared to static feature stitching, the cross-attention mechanism achieves deep coupling between information pathways, helping to enhance the model's ability to identify complex lesion features, thereby improving the accuracy and stability of malignancy probability prediction.

[0057] Furthermore, this invention incorporates a modality compensation module and a dynamic fusion mechanism. In cases where some modalities are missing, the missing pathway output is estimated using existing modalities, and adaptive fusion is achieved through a learnable weighting strategy, effectively reducing the impact of incomplete input information on the prediction results. Through these mechanisms, this invention improves clinical adaptability while maintaining model predictive performance, providing greater feasibility and flexibility for practical deployment in intelligent diagnosis of pulmonary nodules. Attached Figure Description

[0058] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0059] Fig. 1 This is a flowchart of the method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics proposed in this invention.

[0060] Fig. 2 This is a schematic diagram of the dual-pathway adaptive fusion network in the lung nodule malignancy probability prediction method based on radiomics and clinical features proposed in this invention.

[0061] Fig. 3 This is a schematic diagram of the feature interaction mechanism of the cross-attention module in the lung nodule malignancy probability prediction method based on radiomics and clinical features proposed in this invention. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0063] refer to Figs. 1-3 A method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics includes the following steps:

[0064] The target patient underwent a chest computed tomography scan to acquire raw lung image data, and image preprocessing was performed to extract lung nodule image blocks.

[0065] Collect structured clinical data of target patients and perform standardized coding to construct structured clinical feature vectors;

[0066] Lung nodule image patches and structured feature vectors are input into the image path and structured path of a dual-path adaptive fusion network, respectively, to extract image feature vectors and structured embedding feature vectors.

[0067] The modality perception module of the dual-path adaptive fusion network determines modality missing. If a modality is missing, the modality compensation module is activated to generate pseudo-dual-path output feature pairs; otherwise, a cross-attention fusion feature vector is generated.

[0068] The dynamic fusion module of the dual-path adaptive fusion network dynamically fuses the cross-attention fusion feature vector or pseudo-dual-path feature pairs and outputs the dynamic fusion feature vector.

[0069] The dynamically fused feature vector is input into the classifier, and nonlinear mapping and Sigmoid activation operations are performed to output the malignancy probability value of the lung nodule.

[0070] In this embodiment, the step of performing a chest computed tomography scan on the target patient, acquiring raw lung image data, and performing image preprocessing to extract lung nodule image blocks specifically includes:

[0071] Acquire a sequence of chest computed tomography images of the target patient in a resting state, wherein the image sequence is a series of two-dimensional grayscale slices;

[0072] The image sequence is processed by adjusting the window width and window level. The window level and window width values ​​in the lung window parameters are used to linearly stretch the original pixel grayscale. Specifically, the upper and lower limits of the grayscale mapping interval are determined according to the set window level and window width. The original pixel grayscale values ​​located in the interval are mapped proportionally to the [0,1] interval. Pixel values ​​exceeding the upper limit of the interval are set to 1, and pixel values ​​below the lower limit of the interval are set to 0, thus obtaining the normalized image data.

[0073] The normalized image sequence is subjected to three-dimensional median filtering. A cube-shaped filter kernel slides in the image voxel space, and the gray value of each voxel is replaced with the median value of all voxel gray values ​​in its neighborhood.

[0074] The lung parenchyma region segmentation operation is performed on the filtered image sequence. First, non-lung regions are screened out based on the gray-scale thresholding method. Then, morphological closing operation is used to fill lung cavities and remove small-area artifacts. Furthermore, the largest connected regions of the left and right lungs are retained through connected component analysis. Finally, a three-dimensional lung mask image covering the lung parenchyma structure is generated.

[0075] Within the extracted lung parenchyma region, based on image grayscale distribution features and anatomical priors, a region growing and candidate detection algorithm based on nodule diameter thresholds are used to identify suspicious lung nodule regions with clear boundaries, high density, and approximately spherical shape. The spatial center coordinates of each candidate region are recorded, including the lateral position, longitudinal position, and slice thickness direction position, as well as the maximum radial length of the nodule as a size parameter.

[0076] Using the center coordinates of the lung nodule as the center, a cube image block is cropped to a fixed size and output as the lung nodule image block.

[0077] In this embodiment, the process of collecting structured clinical data of the target patient and performing standardized coding to construct a structured clinical feature vector specifically includes:

[0078] Structured clinical data of the target patients were collected, including age, gender, smoking history, family history of cancer, basic comorbidity markers and nodule follow-up time, and a clinical original feature vector was constructed.

[0079] The original clinical feature vector is processed to handle missing values. For missing values, the mean imputation, mode imputation or independent encoding is used to generate a complete clinical feature vector.

[0080] Normalization is performed on the complete clinical feature vector. For numerical features, Z-score standardization is used, which involves subtracting the mean of all samples from each numerical feature and then dividing by the standard deviation to convert it into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0081] Categorical features are discretized using one-hot encoding, mapping each category value to an independent binary vector;

[0082] The normalized numerical features and the discretized categorical features are concatenated in the original dimensional order to obtain the structured clinical feature vector.

[0083] In this embodiment, the step of inputting the lung nodule image patch and the structured feature vector into the image path and structured path of the dual-path adaptive fusion network, respectively, and extracting the image feature vector and the structured embedded feature vector, specifically includes:

[0084] A dual-path adaptive fusion network is constructed, which includes an image path, a structured path, a modality perception module, a cross-attention module, a modality compensation module, and a dynamic fusion module.

[0085] The image path is a multi-layer convolutional neural network structure, which includes a first convolutional layer, a first batch normalization layer, a first activation layer, a first max pooling layer, a second convolutional layer, a second batch normalization layer, a second activation layer, and a second max pooling layer connected in sequence.

[0086] The convolutional layers preferably use a 3×3×3 convolutional kernel, and the pooling layers preferably use a 2×2×2 max pooling kernel. The specific parameters can be adjusted according to the voxel resolution of the input lung nodule image block.

[0087] In the image pathway, lung nodule image blocks are used as input, and local spatial structural features are extracted through three-dimensional convolution operations to generate image feature vectors.

[0088] The structured pathway is a multilayer perceptron structure, preferably including an input layer, two hidden layers and an output layer, which respectively perform the embedding mapping of structured features and the dimensionality unification operation;

[0089] The input structure is a structured clinical feature vector, and the output is a structured embedded feature vector. The dimension of the structured embedded feature vector is consistent with or can be aligned with the image feature vector.

[0090] In this embodiment, the modality perception module of the dual-path adaptive fusion network determines modality loss. If a modality loss is detected, the modality compensation module is activated to generate a pseudo-dual-path output feature pair; otherwise, the cross-attention fusion feature vector is generated. Specifically, this includes:

[0091] The image feature vector output from the image path and the structured embedding feature vector output from the structured path are input into the modality perception module, and the validity flag is used to determine whether the image modality or structured modality is missing.

[0092] When the modality perception module determines that the dual-modality is available, the cross-attention module is activated to perform bidirectional feature interaction operations.

[0093] The attention output vector guided by the image direction and the attention output vector guided by the structured image direction are concatenated to construct a cross-attention fusion feature vector.

[0094] When the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair.

[0095] In this embodiment, the step of inputting the image feature vector output from the image path and the structured embedding feature vector output from the structured path into the modality perception module, and determining whether the image modality or structured modality is missing based on the validity flag, specifically includes:

[0096] The image feature vector output from the image path and the structured embedded feature vector output from the structured path are input into the modality perception module;

[0097] The validity of image feature vectors is judged using the L1 norm as the criterion. When the L1 norm of the image feature vector is greater than zero, the image modality is considered valid and the validity flag of the image path is assigned a value of 1. If the L1 norm of the image feature vector is equal to zero, the image modality is considered missing and the validity flag of the image path is assigned a value of 0.

[0098] The validity of structured embedding feature vectors is judged using the L1 norm as the criterion. When the L1 norm of the structured embedding feature vector is greater than zero, the structured mode is considered valid and the validity flag of the structured path is assigned a value of 1. If the L1 norm of the structured embedding feature vector is equal to zero, the structured mode is considered missing and the validity flag of the structured path is assigned a value of 0.

[0099] Based on the combination of the values ​​of the image path validity flag and the structured path validity flag, the modal status flag is determined. If both the image path validity flag and the structured path validity flag are 1, it is determined to be a dual-modal available state, and the modal status flag is assigned a value of 2.

[0100] If only the image path validity flag is 1 or only the structured path validity flag is 1, it is determined to be a single modality missing state, and the modality status flag is assigned a value of 1.

[0101] If both the image pathway validity flag and the structured pathway validity flag are 0, it is determined to be a state where all modalities are missing, and the modality status flag is assigned a value of 0.

[0102] In this embodiment, the step of enabling the cross-attention module to perform bidirectional feature interaction operation when the modality perception module determines that the modality is available specifically includes:

[0103] When the modality perception module determines that the modality is available (i.e., the modality status flag is 2), the cross-attention module is activated to perform bidirectional attention calculation on the image feature vector and the structured embedding feature vector.

[0104] Perform a linear mapping operation on the image feature vectors to generate an image query matrix. Simultaneously, perform a linear mapping operation on the structured feature vectors to generate a structured key matrix and a structured value matrix. Calculate the attention output vector that guides the image's structured direction.

[0105]

[0106] Among them, F img→clin Q is the attention output vector that guides the structured direction of the image. img K is the image query matrix obtained by linearly mapping the image feature vectors output from the image path. clin V is the structured key matrix obtained by linear mapping of the structured embedded feature vectors output from the structured pathway. clin The structured value matrix is ​​obtained by linear mapping from the structured embedded feature vectors output by the structured pathway, and softmax(·) is the activation function that normalizes the attention scoring results. This is the scaling factor;

[0107] The formula for calculating the attention output vector guiding the structured direction in this patent originates from the "ScaledDot-ProductAttention" mechanism in computer science. This mechanism scales the result of the dot product between the query vector and the key vector by dividing it by the square root of the dimension, normalizes it using softmax, and then multiplies it by the value vector to generate a weighted output. This application adapts this mechanism to cross-modal interaction scenarios between lung nodule imaging features and structured clinical features, retains the original algorithm structure, and uniformly maps the input vector dimension to a fixed-length tensor to ensure the dot product operation is valid. In this formula, the dot product term Q... img , As a scalar, the softmax output is the weight coefficients, which are ultimately combined with the value vector V. clin When multiplied, the dimensions on both sides of the formula are consistent, satisfying the requirement of consistency between mathematical form and physical semantics;

[0108] Perform a linear mapping operation on the structured feature vectors to generate a structured query matrix. Simultaneously, perform a linear mapping operation on the image feature vectors to generate an image key matrix and an image value matrix. Calculate the attention output vector that guides the image orientation in a structured manner.

[0109]

[0110] Among them, F clin→img Q is the attention output vector that guides the structured image orientation. clin K is the structured query matrix obtained by linear mapping from the structured embedded feature vectors output by the structured pathway. img V is the image key matrix obtained by linearly mapping the image feature vectors output from the image path. img The image value matrix is ​​obtained by linearly mapping the image feature vectors output from the image path;

[0111] The formula for calculating the attention output vector of the structured guided image direction in this patent originates from the "ScaledDot-ProductAttention" mechanism in computer science. This mechanism scales the result of the dot product between the query vector and the key vector by dividing it by the square root of the dimension, normalizes it using softmax, and then multiplies it by the value vector to generate a weighted output. This application adapts this mechanism to cross-modal interaction scenarios between lung nodule imaging features and structured clinical features, retains the original algorithm structure, and uniformly maps the input vector dimension to a fixed-length tensor to ensure the dot product operation is valid. In this formula, the dot product term Q... clin , As a scalar, the softmax output is the weight coefficients, which are ultimately combined with the value vector V. img When multiplied, the dimensions on both sides of the formula are consistent, satisfying the requirement of consistency between mathematical form and physical semantics.

[0112] In this embodiment, when the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair, specifically including:

[0113] When the modal perception module determines that a single modality is missing, i.e., the value of the modal state flag is 1, the modal compensation module is activated.

[0114] When the image modality is missing but the structured modality is valid, that is, the validity flag of the image path is 0 and the validity flag of the structured path is 1, the modality compensation module inputs the structured embedded feature vector into the structured guided image compensation function to generate the estimated image feature vector.

[0115] The structured guided image compensation function trains a multilayer perceptron model that maps from the structured embedded feature space to the image feature space. During the training phase, it uses paired structured embedded features and corresponding image features as input and output samples to learn the nonlinear mapping relationship between the two.

[0116] During the inference phase, when the image modality is missing but the structured modality is valid, the structured embedding features are used as input and passed sequentially through the input layer, multiple hidden layers and the output layer to generate an estimated image feature vector.

[0117] When the structured modality is missing but the image modality is valid, that is, the validity flag of the image path is 1 and the validity flag of the structured path is 0, the modality compensation module inputs the image feature vector into the image-guided structured compensation function to generate an estimated structured feature vector.

[0118] The image-guided structured compensation function constructs a multilayer perceptron model that maps from the image feature space to the structured feature space. During the training phase, image feature vectors and corresponding structured embedded features are used as sample pairs. The nonlinear mapping relationship between image features and structured embedded features is learned by minimizing the reconstruction error. During the inference phase, when the structured mode is missing but the image mode is valid, the image feature vector is used as input and passed through a fully connected layer and a nonlinear activation function in sequence to output an estimated structured feature vector.

[0119] The estimated feature vector generated by the modality compensation module is combined with the valid feature vectors existing in the current sample. When the image modality is valid, the image feature vector is used directly; when the image modality is missing, the estimated image feature vector generated by the structured embedded features is used.

[0120] When the structured modality is valid, the structured embedded feature vector is used directly; when the structured modality is missing, the estimated structured feature vector generated from image features is used, thus forming a pseudo dual-path feature pair that simultaneously contains image information and structured information.

[0121] In this embodiment, the dynamic fusion module of the dual-path adaptive fusion network dynamically fuses the cross-attention fusion feature vector or pseudo-dual-path feature pair, and outputs the dynamic fusion feature vector, specifically including:

[0122] The cross-attention fusion feature vector output by the cross-attention module or the pseudo-dual-path feature pair output by the modality compensation module is input into the dynamic fusion module;

[0123] When the input is a cross-attention fusion feature vector, the dynamic fusion module performs a non-linear transformation on the cross-attention fusion feature vector to obtain a dynamic fusion feature vector.

[0124] When the input is a pseudo-dual-path feature pair, the dynamic fusion module performs a linear mapping on each feature to obtain image features and structured features with a unified dimension of feature representation;

[0125] The inner product of the image features and structured features with the trainable weight vector with the feature representation of the same dimension is calculated separately, and then normalized by the exponential function to obtain the fusion weight coefficients corresponding to the image features and the fusion weight coefficients corresponding to the structured features.

[0126] Based on the fusion weight coefficients corresponding to image features and fusion weight coefficients corresponding to structured features, image features and structured features of the same dimension are weighted and fused to obtain a dynamic fused feature vector.

[0127] In this embodiment, the step of inputting the fused feature vector into the classifier, performing nonlinear mapping and Sigmoid activation operations, and outputting the malignancy probability value of the lung nodule specifically includes:

[0128] The dynamic fusion feature vector output by the dynamic fusion module is input into the classifier, which consists of at least one fully connected layer, a batch normalization layer, a nonlinear activation function layer, and an output layer.

[0129] In the first fully connected layer of the classifier, the fused feature vector is linearly mapped to obtain the intermediate feature representation;

[0130] Batch normalization and nonlinear activation are performed on the intermediate feature representations, with the modified linear unit selected as the activation function, to obtain the activated feature representations.

[0131] The activated feature representation is input into the second fully connected layer, the linear mapping result of the output layer is calculated, and the Sigmoid activation function is applied to the linear mapping result of the output layer to obtain the predicted value of the malignancy probability of the lung nodule.

[0132] Example 1:

[0133] To verify the feasibility of this invention in practice, it was applied to a lung CT image-assisted diagnostic scenario in the radiology department of a tertiary hospital. By collecting lung CT image data from real patients and corresponding structured clinical data, a lung nodule malignancy probability prediction system based on radiomics and clinical features was constructed. This system, with a dual-pathway adaptive fusion network at its core, can automatically identify modality integrity and adjust feature processing paths under real-world clinical conditions where image and structured information sources are incomplete, thereby improving the robustness and practicality of the prediction system.

[0134] In the experiment, the research team collected a total of 1,824 pathologically validated pulmonary nodule samples, including 956 benign nodules and 868 malignant nodules. Each sample contained a complete chest CT image sequence and structured clinical information, including fields for age, gender, smoking history, family history, and comorbidities. The image data were uniformly preprocessed into nodule image blocks of 64×64×64 pixels, and the clinical features were standardized before being input into the structured pathway.

[0135] To test the performance of this invention in real-world modality loss scenarios, three input scenarios were simulated: 1) both images and structured data are available; 2) images are missing, and only structured data is used; 3) structured data is missing, and only images are used. The comparison models are the traditional static-fusion model, a single-path CNN model using only images (Image-Only), and an MLP model using only structured data (Clinical-Only).

[0136] The prediction results on the test set show that, under the complete modality condition where both image and structured information are present, the model proposed in this invention outperforms the comparison model in all four metrics: AUC, accuracy, recall, and F1 score, demonstrating stronger classification performance. Under the simulated modality missing condition, this invention exhibits the smallest decrease in AUC, demonstrating good adaptability. Regarding inference time, this invention controls the average prediction time per instance to within 0.42 seconds while maintaining accuracy, meeting the real-time response requirements of clinical auxiliary diagnosis.

[0137] Table 1 shows a comparison of the performance metrics of each model on the test set under different input modalities. The specific values ​​are shown below:

[0138] Table 1 Comparison of prediction performance of each model in different modal scenarios

[0139]

[0140] As can be seen from the table above, this invention exhibits higher classification performance compared to traditional static splicing models when multimodal input is complete, and maintains stable predictive ability even with missing modalities, demonstrating its good adaptability and practicality in complex real-world scenarios. The system as a whole, while maintaining clinical interpretability, can achieve rapid risk assessment of pulmonary nodules and possesses the technical feasibility for expansion into large-scale assisted diagnostic systems.

[0141] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics, characterized in that, Includes the following steps: The target patient underwent a chest computed tomography scan to acquire raw lung image data, and image preprocessing was performed to extract lung nodule image blocks. Collect structured clinical data of target patients and perform standardized coding to construct structured clinical feature vectors; Lung nodule image patches and structured feature vectors are input into the image path and structured path of a dual-path adaptive fusion network, respectively, to extract image feature vectors and structured embedding feature vectors. The modality perception module of the dual-path adaptive fusion network determines modality missing. If a modality is missing, the modality compensation module is activated to generate pseudo-dual-path output feature pairs; otherwise, a cross-attention fusion feature vector is generated. The dynamic fusion module of the dual-path adaptive fusion network dynamically fuses the cross-attention fusion feature vector or pseudo-dual-path feature pairs and outputs the dynamic fusion feature vector. The dynamically fused feature vector is input into the classifier, and nonlinear mapping and Sigmoid activation operations are performed to output the malignancy probability value of the lung nodule.

2. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The image preprocessing operations include window width and level adjustment, three-dimensional median filtering for noise reduction, lung parenchyma region extraction, lung nodule candidate region localization, and image block cropping.

3. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The process of collecting structured clinical data from target patients and performing standardized coding to construct structured clinical feature vectors specifically includes: Structured clinical data of the target patients were collected, including age, gender, smoking history, family history of cancer, basic comorbidity markers and nodule follow-up time, and a clinical original feature vector was constructed. Missing values ​​are processed from the original clinical feature vector to generate a complete clinical feature vector; The numerical features in the complete clinical feature vector are standardized using Z-score, and the categorical features are discretized using one-hot encoding, mapping each categorical value to an independent binary vector. The normalized numerical features and the discretized categorical features are concatenated in the original dimensional order to obtain the structured clinical feature vector.

4. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The step of inputting lung nodule image patches and structured feature vectors into the image path and structured path of a dual-path adaptive fusion network, respectively, and extracting image feature vectors and structured embedded feature vectors, specifically includes: A dual-path adaptive fusion network is constructed, which includes an image path, a structured path, a modality perception module, a cross-attention module, a modality compensation module, and a dynamic fusion module. The image path is a multi-layer convolutional neural network structure, which includes a first convolutional layer, a first batch normalization layer, a first activation layer, a first max pooling layer, a second convolutional layer, a second batch normalization layer, a second activation layer, and a second max pooling layer connected in sequence. The image pathway uses three-dimensional convolution to extract features from lung nodule image blocks, obtaining image feature vectors; The structured clinical feature vector is input into the structured pathway and processed sequentially through the input layer, two hidden layers, and the output layer to obtain the structured embedded feature vector.

5. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The modality perception module of the dual-path adaptive fusion network determines modality loss. If a modality is missing, the modality compensation module is activated to generate a pseudo-dual-path output feature pair; otherwise, the cross-attention fusion feature vector is generated. Specifically, this includes: The image feature vector output from the image path and the structured embedding feature vector output from the structured path are input into the modality perception module, and the validity flag is used to determine whether the image modality or structured modality is missing. When the modality perception module determines that the dual-modality is available, the cross-attention module is activated to perform bidirectional feature interaction operations. The attention output vector guided by the image direction and the attention output vector guided by the structured image direction are concatenated to construct a cross-attention fusion feature vector. When the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair.

6. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 5, characterized in that, The step of inputting the image feature vector output from the image path and the structured embedding feature vector output from the structured path into the modality perception module, and determining whether the image modality or structured modality is missing based on the validity flag, specifically includes: The image feature vector output from the image path and the structured embedded feature vector output from the structured path are input into the modality perception module; The validity of image feature vectors is judged, and the L1 norm is used as the criterion to obtain the validity indicator of image pathways; The effectiveness of structured embedding feature vectors is judged using the L1 norm as the criterion, and the effectiveness indicator of structured pathways is obtained. Based on the combination of the values ​​of the image pathway validity flag and the structured pathway validity flag, the modality status flag is determined. The modality status flag includes a bimodal availability status, a single modality missing status, and a complete lack of modalities status.

7. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 5, characterized in that, The step of enabling the cross-attention module to perform bidirectional feature interaction operations when the modality perception module determines that the modality is available includes: When the modality perception module determines that the dual-modality is available, the cross-attention module is activated to perform bidirectional attention calculation on the image feature vector and the structured embedding feature vector; Perform a linear mapping operation on the image feature vector to generate an image query matrix. At the same time, perform a linear mapping operation on the structured embedding feature vector to generate a structured key matrix and a structured value matrix. Calculate the attention output vector that guides the structured direction of the image. A linear mapping operation is performed on the structured embedding feature vector to generate a structured query matrix. Simultaneously, a linear mapping operation is performed on the image feature vector to generate an image key matrix and an image value matrix. The attention output vector for structured guided image orientation is then calculated.

8. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 5, characterized in that, When the modality perception module determines that a single modality is missing, the modality compensation module estimates the output of the missing path and constructs a pseudo-dual-path output feature pair, specifically including: If the modal perception module determines that a single modality is missing, the modal compensation module is activated. When the image modality is missing but the structured modality is valid, the modality compensation module inputs the structured embedded feature vector into the structured guided image compensation function to generate the estimated image feature vector. When the structured path is missing but the image path is valid, the modal compensation module inputs the image feature vector into the image-guided structured compensation function to generate an estimated structured embedding feature vector. The estimated feature vector generated by the modality compensation module is combined with the effective feature vectors existing in the current sample to form a pseudo dual-path feature pair that simultaneously contains image information and structured information.

9. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The dynamic fusion module of the dual-path adaptive fusion network dynamically fuses cross-attention fusion features or pseudo-dual-path feature pairs, and outputs a dynamic fusion feature vector, specifically including: The cross-attention fusion feature vector output by the cross-attention module or the pseudo-dual-path feature pair output by the modality compensation module is input into the dynamic fusion module; When the input is a cross-attention fusion feature vector, the dynamic fusion module performs a non-linear transformation on the cross-attention fusion feature vector to obtain a dynamic fusion feature vector. When the input is a pseudo-dual-path feature pair, the dynamic fusion module performs a linear mapping on each feature vector to obtain image features and structured features with a unified dimension of feature representation; Calculate the fusion weight coefficients corresponding to image features and the fusion weight coefficients corresponding to structured features respectively; Based on the fusion weight coefficient, image features and structured features of the same dimension are weighted and fused to obtain a dynamic fused feature vector.

10. The method for predicting the malignancy probability of pulmonary nodules based on radiomics and clinical characteristics according to claim 1, characterized in that, The process of inputting the fused feature vector into the classifier, performing nonlinear mapping and sigmoid activation operations, and outputting the malignancy probability value of the lung nodule specifically includes: The dynamic fusion feature vector output by the dynamic fusion module is input into the classifier, which consists of at least one fully connected layer, a batch normalization layer, a nonlinear activation function layer, and an output layer. In the first fully connected layer of the classifier, the dynamically fused feature vector is linearly mapped to obtain the intermediate feature representation; Batch normalization and nonlinear activation are performed on the intermediate feature representations, with the modified linear unit selected as the activation function, to obtain the activated feature representations. The activated feature representation is input into the second fully connected layer, the linear mapping result of the output layer is calculated, and the Sigmoid activation function is applied to the linear mapping result of the output layer to obtain the predicted value of the malignancy probability of the lung nodule.

Images