Lung cancer occult lymph node metastasis prediction method and device based on multi-modal fusion

Abstract

The present application relates to the medical health technical field, disclose a kind of lung cancer occult lymph node metastasis prediction method and device based on multimodal fusion, comprising: obtaining medical image data and clinical text data as original multimodal data;The original multimodal data is pre-trained with multimodal consistency, and robust multiscale fusion feature is obtained;Cross-modal prediction fine-tuning is carried out to the robust multiscale fusion feature, and cross-modal semantic alignment feature is obtained;The statistical morphological features and spatial geometric features of lung nodule are extracted by double-branch parallel, the statistical morphological features and spatial geometric features are converted into high-dimensional abstract features by depth feature mapping, and multidimensional supplementary features are constructed;The cross-modal semantic alignment feature is combined with the multidimensional supplementary feature, and the lung cancer occult lymph node metastasis prediction result is obtained by multiscale prediction decoding processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical health, in particular to a lung cancer occult lymph node metastasis prediction method and device based on multi-modal fusion. BACKGROUND

[0002] Occult lymph node metastasis is a key link in the progression of lung cancer, and its early accurate prediction is of decisive significance for treatment plan formulation, prognosis evaluation and improvement of survival rate. Even the most advanced nuclear medicine examination such as PET has a diagnostic rate of less than 70% for occult lymph node metastasis of lung cancer. Especially when the lymph node diameter is less than 4-5mm, and in the case of some pathological types such as adherent adenocarcinoma, the diagnostic efficiency is lower. Therefore, this prediction task faces multiple clinical challenges: first, occult metastasis has no typical imaging features, and conventional imaging examinations (CT, MRI, etc.) are prone to missed diagnosis; second, prediction relies on multi-dimensional information (images, clinical, pathological, etc.), and single modal data is difficult to fully capture metastasis-related features; third, traditional prediction methods (such as pathological biopsy) are invasive operations and cannot achieve preoperative non-invasive prediction; fourth, the existing model has low fusion efficiency for multi-modal data, and it is difficult to mine the complementary correlation between cross-modal data, resulting in insufficient prediction accuracy and robustness.

[0003] Existing lung cancer occult lymph node metastasis prediction methods mostly rely on single modal data or simple modal splicing, and have limited generalization ability in complex clinical scenarios. At the same time, most models lack deep fusion of clinical semantic information and image features, making it difficult to effectively use key clues in pathological reports, clinical histories and other texts, resulting in low prediction accuracy of early occult metastasis of lung cancer. Although multi-modal fusion has become a research hotspot in the field of medical prediction, how to achieve efficient alignment and complementary fusion of image, text and other multi-modal data to improve the accuracy and non-invasiveness of occult lymph node metastasis prediction has become a key problem that needs to be solved in current clinical practice. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a lung cancer occult lymph node metastasis prediction method and device based on multi-modal fusion to solve the above technical problems.

[0005] In a first aspect, a lung cancer occult lymph node metastasis prediction method based on multi-modal fusion is provided, comprising: obtaining medical image data and clinical text data as original multi-modal data; performing multi-modal consistency pre-training on the original multi-modal data to obtain robust multi-scale fusion features; performing cross-modal prediction fine-tuning on the robust multi-scale fusion features to obtain cross-modal semantic alignment features; The statistical morphological features and spatial geometric features of lung nodules are extracted in parallel by two branches. The statistical morphological features and spatial geometric features are then transformed into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features. By combining the cross-modal semantic alignment features with the multi-dimensional supplementary features and performing multi-scale prediction decoding processing, the prediction results of occult lymph node metastasis in lung cancer are obtained.

[0006] Furthermore, multimodal consistency pre-training is performed on the original multimodal data to obtain robust multi-scale fusion features, including: The original medical image data is subjected to random adjustments of brightness, contrast, and saturation, Gaussian blurring, and random cropping to obtain perturbed image data. The clinical text data is subjected to synonym replacement, random insertion, and sentence rearrangement to obtain perturbed text data; The perturbed image data and the perturbed text data are combined to obtain the perturbed multimodal data; The original multimodal data and the perturbed multimodal data are respectively input into the multimodal consistency encoder to obtain two sets of transition prediction results; By using a hybrid loss to supervise the transfer prediction results and the actual transfer labels of each group, and by using consistency constraints to strengthen the consistency between the two groups of transfer prediction results, robust multi-scale fusion features are obtained.

[0007] Furthermore, the robust multi-scale fusion features are fine-tuned for cross-modal prediction to obtain cross-modal semantic alignment features, including: Robust multi-scale fusion features are input into the student model, and the feature dimensions are adjusted through a lightweight projection network so that the features output by the student model match the global semantic feature dimensions output by the frozen pre-trained teacher model, thus obtaining semantic similarity. Distillation loss is constructed based on semantic similarity to guide the student model to learn and transfer relevant semantic features, thereby obtaining semantically enhanced fusion features; Clinical text data was encoded using a medical-specific text encoder to obtain semantic embeddings of the transfer-related text. Using the semantically enhanced fusion features as queries and the text semantic embedding as keys and values, text semantic information is injected into image features through an attention mechanism to obtain cross-modal semantic alignment features.

[0008] Furthermore, statistical morphological features of the lung nodules were extracted, including: First-order statistical features were obtained by gray-level histogram analysis, and texture features were obtained by gray-level co-occurrence matrix calculation and gray-level run length matrix analysis. At the same time, morphological features describing the geometric shape of nodules were extracted. By combining first-order statistical features, texture features, and morphological features, statistical morphological features are obtained.

[0009] Furthermore, the spatial geometric features of the lung nodules are extracted, including: Set the horizontal and vertical reference axes; The shortest distance between the tumor margin and the reference axis and the farthest distance between the lung field quadrant and the reference axis were obtained by distance calculation. The spatial geometric features are obtained by calculating the proportion of relative distance in the horizontal direction, the proportion of relative distance in the vertical direction, and the sum of the two.

[0010] Furthermore, the cross-modal semantic alignment features are combined with the multi-dimensional complementary features, and multi-scale prediction decoding processing is performed, including: Multiple parallel decoding branches are constructed, each receiving a pair of cross-modal semantic alignment features at adjacent scales; Bilinear interpolation upsampling is performed on the low-resolution cross-modal semantic alignment features in each branch; The upsampled low-resolution features are concatenated with the corresponding high-resolution cross-modal semantic alignment features along the channel dimension to obtain the concatenated features.

[0011] Furthermore, the splicing features are refined, including: Multiple rounds of residual convolution are performed on the spliced ​​features, and multi-layer feature information is preserved through residual connections to obtain refined features.

[0012] Furthermore, the refined features are enhanced, including: The refined features are processed using an efficient channel attention module and a pyramid spatial attention module to highlight relevant salient features and suppress redundant information, resulting in enhanced multi-scale features.

[0013] Furthermore, the predictive results for occult lymph node metastasis in lung cancer were obtained, including: The enhanced multi-scale features output from all parallel decoding branches are uniformly adjusted to the same resolution and then stitched together to obtain fused features; The fused features are correlated and integrated with multidimensional supplementary features, and a segmentation prediction mask is generated through lightweight convolutional head processing to obtain the prediction results of occult lymph node metastasis in lung cancer.

[0014] Secondly, a multimodal fusion-based prediction device for occult lymph node metastasis in lung cancer is provided, based on any of the preceding methods for predicting occult lymph node metastasis in lung cancer, including: The acquisition module is configured to acquire medical image data and clinical text data as raw multimodal data; The pre-training module is configured to perform multimodal consistency pre-training on the original multimodal data to obtain robust multi-scale fusion features; The fine-tuning module is configured to perform cross-modal prediction fine-tuning on the robust multi-scale fusion features to obtain cross-modal semantic alignment features; The module is configured to extract statistical morphological features and spatial geometric features of lung nodules in parallel through two branches, and then transform the statistical morphological features and spatial geometric features into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features; The decoding processing module is configured to combine the cross-modal semantic alignment features with the multi-dimensional supplementary features, and obtain the prediction result of occult lymph node metastasis of lung cancer through multi-scale prediction decoding processing.

[0015] The invention employing the above technical solution has the following advantages: 1. This invention, by fusing multimodal data of images and text, can predict occult lymph node metastasis in lung cancer without invasive pathological biopsy, providing a reliable basis for preoperative treatment decisions and reducing patient suffering and medical costs.

[0016] 2. This invention achieves deep interaction between text and image features through a cross-modal semantic alignment module, accurately mining transfer-related cross-modal complementary information, and the fusion effect is better than the traditional simple stitching method.

[0017] 3. This invention leverages the global semantic advantages of the DINOv3 teacher model through a feature distillation module to enhance the extraction and discrimination of transfer-related features; the medical-specific text encoder accurately captures key transfer clues in clinical texts, further improving prediction accuracy.

[0018] 4. This invention uses a multi-scale prediction decoder to specifically model lymph node regions of different sizes through parallel branches, effectively solving the problem of missed diagnosis of small-sized occult lymph node metastasis and improving the comprehensiveness of prediction.

[0019] 5. This invention uses supplementary modules such as Branch 1 and Branch 2 to achieve comprehensive extraction of statistical morphology and spatial geometric features of lung nodules. Combined with depth mapping and multimodal fusion, it constructs a multi-dimensional and multi-level feature representation system to further improve the accuracy of prediction. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0021] Figure 1 This is a prediction framework diagram of the lung cancer occult lymph node metastasis prediction method and device based on multimodal fusion in this invention; Figure 2 This is a diagram illustrating the feature distillation enhancement process in the multimodal fusion-based method and apparatus for predicting occult lymph node metastasis in lung cancer, as described in this invention. Figure 3 This is a flowchart illustrating the internal structure and feature processing of the scale adaptive decoder in the multimodal fusion-based lung cancer occult lymph node metastasis prediction method and device of the present invention. Figure 4 This is a schematic diagram illustrating the definition of calculation parameters for the spatial geometric features of lung nodules in the lung cancer occult lymph node metastasis prediction method and device based on multimodal fusion of the present invention. Figure 5 This is a flowchart of the lung cancer occult lymph node metastasis prediction method based on multimodal fusion according to the present invention; Figure 6 This is a flowchart of the lung cancer occult lymph node metastasis prediction device based on multimodal fusion according to the present invention. Detailed Implementation

[0022] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0023] like Figures 1-6 As shown, the multimodal fusion-based method for predicting occult lymph node metastasis in lung cancer according to the present invention includes: Step S01: Acquire medical imaging data and clinical text data as raw multimodal data; Step S02: Perform multimodal consistency pre-training on the original multimodal data to obtain robust multi-scale fusion features; Step S03: Fine-tune the robust multi-scale fusion features for cross-modal prediction to obtain cross-modal semantic alignment features; Step S04: Extract the statistical morphological features and spatial geometric features of lung nodules in parallel through dual branches, and transform the statistical morphological features and spatial geometric features into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features; Step S05: Combine cross-modal semantic alignment features with multi-dimensional supplementary features, and obtain the prediction results of occult lymph node metastasis in lung cancer through multi-scale prediction and decoding processing.

[0024] Specifically, this invention provides a method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion. This method achieves deep alignment and fusion of multimodal data, including medical images (such as CT and MRI) and clinical texts (such as pathology reports and medical history records), through a modular process. The overall architecture is divided into three stages: multimodal consistency pre-training, cross-modal prediction fine-tuning, and multi-scale fusion prediction. The core includes four functional modules: a multimodal consistency encoder, a feature distillation enhancement module, a cross-modal semantic alignment module, and a multi-scale prediction decoder, to achieve robust multimodal feature extraction, semantic enhancement, cross-modal fusion, and accurate metastasis prediction.

[0025] like Figure 1 As shown, Phase 1 (Phase I): Multimodal consistency encoder, trained under strong data perturbation to learn robust multimodal feature representations; Phase 2 (Phase II): Cross-modal prediction fine-tuning, enhancing semantics through feature distillation and achieving image-text alignment through cross-modal attention; Phase 3 (Phase III): Fusing lung nodule images segmented from the first two phases with the metrics proposed in this invention, and outputting multi-scale prediction results for transfer prediction.

[0026] The multimodal fusion prediction framework proposed in this invention adopts a three-stage architecture of "pre-training-fine-tuning-prediction": Stage 1 is multimodal consistency pre-training, which trains the encoder to learn perturbation-robust multimodal joint feature representation by strongly perturbing image and text data, and outputs four fusion feature maps (f1, f2, f3, f4) at different scales; Stage 2 is cross-modal prediction fine-tuning, which first enhances the semantic discriminative ability of the fusion features through a feature distillation module (focusing on transfer-related features), and then uses a cross-modal attention mechanism to achieve deep alignment between clinical text semantics and image features; Stage 3 is that the multi-scale prediction decoder performs radiomics analysis on lymph node regions of different sizes and locations, and integrates the relevant indicators proposed in this invention to complete accurate transfer prediction. This architecture achieves the organic unity of multimodal robust coding, semantic enhancement and cross-modal fusion, and significantly improves the accuracy and generalization ability of non-invasive prediction.

[0027] In this embodiment, multimodal consistency pre-training is performed on the original multimodal data to obtain robust multi-scale fusion features, including: The original medical image data is subjected to random adjustments of brightness, contrast, and saturation, Gaussian blurring, and random cropping to obtain perturbed image data. Synonym replacement, random insertion, and sentence rearrangement were performed on the clinical text data to obtain perturbed text data. The perturbed image data and the perturbed text data are combined to obtain the perturbed multimodal data; The original multimodal data and the perturbed multimodal data are respectively input into the multimodal consistency encoder to obtain two sets of transition prediction results; By using a hybrid loss to supervise the transfer prediction results and the actual transfer labels of each group, and by using consistency constraints to strengthen the consistency between the two groups of transfer prediction results, robust multi-scale fusion features are obtained.

[0028] Specifically, this step is the core of improving the robustness of multimodal features. Based on the joint architecture of Unet and text encoder, a consistency-driven pre-training strategy is adopted to process medical image and clinical text data simultaneously, learn robust multimodal joint feature representations, and provide a reliable feature foundation for subsequent predictions.

[0029] The specific implementation process is as follows: Multimodal strong perturbation enhancement: Given an input image I and its corresponding clinical text T, perturbation versions I' and T' are generated using the image enhancement function T_img(·) and the text enhancement function T_text(·), respectively. Image enhancement techniques include random adjustments to brightness, contrast, and saturation, Gaussian blur, and random cropping to simulate the real degradation of clinical images; text enhancement techniques include synonym replacement, random insertion, and sentence rearrangement to preserve the core semantics of the text while increasing data diversity.

[0030] Multimodal parallel encoding: The original multimodal data (I, T) and the perturbed multimodal data (I', T') are input into the multimodal consistency encoder to obtain two sets of prediction results Pa (transition prediction probability of the original data) and Pb (transition prediction probability of the perturbed data). The encoder is then connected to a classification prediction head to supervise robust learning.

[0031] Hybrid Loss Supervision: Each prediction result is supervised by a hybrid loss consisting of cross-entropy loss (CE) and F1 loss, along with the true transfer label G (1 for transfer, 0 for no transfer), to ensure prediction accuracy. The loss formula is as follows: in, Consistency Constraint Enhancement: A symmetric consistency loss is introduced to encourage consistency between the prediction results of the original multimodal data and the perturbed multimodal data. The prediction probabilities are binarized using a threshold function B(P, θ) (1 for P≥θ, 0 otherwise). The consistency loss is defined as follows: Total pre-training loss: The final pre-training loss is the sum of the hybrid loss and the consistency loss, ensuring that the encoder learns perturbation-invariant and semantically stable features. In this embodiment, robust multi-scale fusion features are fine-tuned for cross-modal prediction to obtain cross-modal semantic alignment features, including: Robust multi-scale fusion features are input into the student model, and the feature dimensions are adjusted through a lightweight projection network so that the features output by the student model match the global semantic feature dimensions output by the frozen pre-trained teacher model, thus obtaining semantic similarity. Distillation loss is constructed based on semantic similarity to guide the student model to learn and transfer relevant semantic features, thereby obtaining semantically enhanced fusion features; Clinical text data was encoded using a medical-specific text encoder to obtain semantic embeddings of the transfer-related text. Using semantically enhanced fusion features as queries and textual semantic embeddings as keys and values, textual semantic information is injected into image features through an attention mechanism to obtain cross-modal semantic alignment features.

[0032] Specifically, the aim is to enhance the semantic discriminative power of multimodal fusion features, focusing on strengthening transfer-related features. Global semantic supervision is provided through a frozen DINOv3 teacher model to guide the student model (multimodal consistency encoder) to learn more discriminative multi-scale multimodal representations.

[0033] The specific implementation process is as follows: Multimodal feature extraction and dimensionality matching: The stage-one encoder outputs four hierarchical multimodal fusion feature maps f_i (i=1,2,3,4), and the DINOv3 teacher model outputs the normalized features of the final Transformer layer as teacher features t_i (focusing on tumor and lymph node related semantics). Each student feature f_i is projected to the same dimension as the teacher feature t_i using a lightweight MLP.

[0034] Distillation loss calculation: A distillation loss is constructed based on cosine similarity to promote semantic alignment between student features and teacher features. The loss formula is as follows: in, The total number of spatial tokens across all scales. Let be the number of patches for the i-th layer features. This process facilitates the encoder in learning richer, more discriminative multi-scale representations, outputting distilled features. This lays a solid foundation for subsequent text alignment.

[0035] This technology effectively injects semantic information related to occult lymph node metastasis (such as tumor invasion depth, lymph node size, and pathological type) from clinical text into image features, enabling fine-grained interaction between text semantics and multi-scale image features. The core technology employs a cross-modal attention mechanism to improve the accuracy of multimodal fusion.

[0036] The specific implementation process is as follows: Text semantic embedding extraction: The clinical text report was encoded using a frozen medical text encoder (CXR-BERT), focusing on extracting semantic information related to transfer, and generating a text semantic embedding T with dimension L×C (L is the text length and C is the number of feature channels).

[0037] Cross-modal attention computation: Using the semantically enhanced multimodal fusion feature f'_i (∈R^Hi×Wi×C) as the query, and the text semantic embedding T as the key and value, the transfer-related text semantics are accurately injected into the image features through an attention mechanism. The computation process is as follows: in, For learnable projection matrices, This is the scaling factor.

[0038] Output For the first Enhance the visual features of the text layer.

[0039] In this embodiment, the statistical morphological features of lung nodules are extracted, including: First-order statistical features were obtained by gray-level histogram analysis, and texture features were obtained by gray-level co-occurrence matrix calculation and gray-level run length matrix analysis. At the same time, morphological features describing the geometric shape of nodules were extracted. By combining first-order statistical features, texture features, and morphological features, statistical morphological features are obtained.

[0040] In this embodiment, the spatial geometric features of the lung nodules are extracted, including: Set the horizontal and vertical reference axes; The shortest distance between the tumor margin and the reference axis and the farthest distance between the lung field quadrant and the reference axis were obtained by distance calculation. The spatial geometric features are obtained by calculating the proportion of relative distance in the horizontal direction, the proportion of relative distance in the vertical direction, and the sum of the two.

[0041] In this embodiment, cross-modal semantic alignment features are combined with multi-dimensional complementary features, and multi-scale prediction decoding processing is performed, including: Construct multiple parallel decoding branches, each receiving a pair of cross-modal semantic alignment features at adjacent scales; Bilinear interpolation upsampling is performed on the low-resolution cross-modal semantic alignment features in each branch; The upsampled low-resolution features are concatenated with the corresponding high-resolution cross-modal semantic alignment features along the channel dimension to obtain the concatenated features.

[0042] In this embodiment, the splicing features are refined, including: Multiple rounds of residual convolution are performed on the spliced ​​features, and multi-layer feature information is preserved through residual connections to obtain refined features.

[0043] In this embodiment, the refinement features are enhanced, including: The refined features are processed using an efficient channel attention module and a pyramid spatial attention module, highlighting transfer-related salient features and suppressing redundant information, resulting in enhanced multi-scale features.

[0044] Specifically, such as Figure 3 As shown, to address anatomical structures at different scales, we designed a Scale Adaptive Decoder (SAD), which consists of three parallel decoding branches. These branches process the text-aligned multi-scale features and specifically model small, medium, and large anatomical structures. Each decoder branch receives a pair of feature maps from adjacent scales. These features are obtained by the encoder after cross-modal alignment. Among them, the shallow feature map It preserves fine-grained spatial information, while deep feature maps This contains broader contextual and semantic information, which helps in identifying large structures. To fuse these two feature maps, the low-resolution one is first processed... Perform bilinear interpolation upsampling, then... splicing along the channel dimension: in, express Upsampling operation. Features after concatenation. Refinement is achieved through a series of residual convolution operations: in, and for convolution, and for Convolution. Residual connections facilitate gradient propagation and preserve information from multiple layers of features.

[0045] Subsequently, an efficient channel attention module (ECA) and a pyramid spatial attention module (PSA) are introduced to highlight salient features and suppress redundant information. The above process is performed on three decoding branches: , and Receive feature pairs respectively , and The outputs of the three branches are upsampled to the same resolution and then concatenated. in, and They represent and The upsampling operation is then performed. The concatenated fused features are then passed through a lightweight convolutional head to generate the final segmentation prediction mask: in, and They are respectively and convolution, This represents the Sigmoid activation function.

[0046] Through this mechanism, SAD can flexibly model between global context and local details, enabling refined and sensitive segmentation of multi-scale structures.

[0047] In this embodiment, the prediction results of occult lymph node metastasis in lung cancer are obtained, including: The enhanced multi-scale features output from all parallel decoding branches are uniformly adjusted to the same resolution and then stitched together to obtain fused features; By associating and integrating fusion features with multidimensional supplementary features, and generating segmentation prediction masks through lightweight convolutional head processing, the prediction results of occult lymph node metastasis in lung cancer are obtained.

[0048] Specifically, as an important supplement to multimodal fusion features, the statistical morphological features and spatial geometric features of lung nodules are extracted in parallel through two branches, constructing a multidimensional feature foundation of "statistical attributes + spatial relationships", which provides comprehensive feature support for subsequent deep mapping and metastasis prediction.

[0049] Branch 1 (First Order: First-order statistical and morphological branch) Input: Lung nodule mask regions segmented after processing in Stage 1 (multimodal consistency pre-training) and Stage 2 (cross-modal prediction fine-tuning) (precisely locking the target nodule region and eliminating background interference).

[0050] Core operations: Extracting the first-order statistical and morphological features of the nodule region, specifically including: First-order statistical features are calculated based on gray-level histograms and their probability distributions. Commonly used features include mean, variance, standard deviation, skewness, kurtosis, and entropy. The mean measures the overall brightness level of a lung image, variance and standard deviation reflect the amplitude of gray-level fluctuations, while skewness and kurtosis describe the asymmetry of the gray-level distribution and the distribution of extreme values. Entropy, by calculating the information content of the gray-level probability distribution, quantitatively characterizes the complexity of the image; when there are many gray-level variations or fragmented structures in the image, the entropy value is usually high.

[0051] Because first-order statistical features do not depend on spatial structural information between pixels, they are insensitive to segmentation accuracy and local structural noise, making them particularly suitable for image analysis acquired under natural light conditions. In lung disease scenarios, these features can reflect nodule status, size variations, and brightness unevenness at a holistic level, providing quantitative indicators with physical meaning and good stability for subsequent models.

[0052] Texture features in radiomics are used to describe the spatial organization of pixel gray levels and are an important means of characterizing tissue heterogeneity. This study uses the pyradiomics toolkit to extract texture features based on the gray level co-occurrence matrix (GLCM) and gray level run length matrix (GLRLM) to characterize the spatial regularity of gray level distribution in lung images.

[0053] Gray-Level Co-occurrence Matrix (GLCM) Feature Extraction Principle: The Gray-Level Co-occurrence Matrix (GLCM) describes image texture characteristics by statistically analyzing the co-occurrence frequency of pixel pairs with different gray levels under a given spatial distance and orientation. Specifically, for pixels with gray levels i and j, the element P(i,j) in the GLCM represents the probability of that gray level pair occurring simultaneously under specific orientations and pixel spacing conditions.

[0054] During feature extraction, pyradiomics first discretizes the lung image using grayscale, and then constructs a grayscale co-occurrence matrix (GLCM) based on a preset pixel distance (usually one pixel) and multiple directions (such as 0°, 45°, 90°, and 135°). To enhance the rotation invariance of the features, pyradiomics statistically summarizes the GLCMs in different directions, thereby obtaining a texture description that is insensitive to changes in orientation.

[0055] Based on GLCM, various texture features can be calculated, among which contrast is used to measure the magnitude of local gray-level changes in an image: Homogeneity reflects the smoothness of grayscale distribution. Energy characterizes the regularity of texture structure: Correlation describes the degree of spatial correlation between grayscale values: in, These represent the mean and standard deviation of GLCM in the row and column directions, respectively.

[0056] Principle of Gray-Scale Run-Length Matrix (GLRLM) Feature Extraction: The gray-level run length matrix (GLLM) describes the length distribution of consecutive pixel sequences (i.e., "runs") with the same gray level in an image. The element R(i,l) in GLLM represents the number of times a run of gray level i and length l occurs in the image.

[0057] In the implementation of pyradiomics, GLRLM is also constructed based on the grayscale discretized image, and the run information of pixels with consecutive identical grayscale values ​​is statistically analyzed along multiple directions. By combining the results from different directions, the influence of image orientation changes on features can be reduced.

[0058] Features calculated based on GLRLM can characterize the texture structure of an image from different perspectives. Short run emphasis is used to measure the proportion of fine texture structures in an image. Long run emphasis reflects the distribution of large, smooth regions: Furthermore, gray level non-uniformity can be used to quantitatively describe the degree of non-uniformity in gray level distribution and texture structure: in This represents the total number of all trips.

[0059] In lung image analysis, the GLRLM feature is sensitive to local fine texture changes and can help reflect the microstructure of lung nodules and local brightness changes.

[0060] The size (volume, diameter), roundness (measures how close the nodule shape is to a circle), and surface area (reflects the roughness of the nodule surface) of the nodule describe its geometric morphological properties.

[0061] Branch 2 (Geometric Location Spatial Geometric Feature Branch) Input: Regions of interest (ROIs) containing lung nodules and their corresponding lung field quadrants in chest CT images, clearly defining the boundaries of the lung field quadrants where the nodules are located.

[0062] like Figure 4 As shown, the core definition (the distance measurement index proposed in this invention): Key parameter definitions: a: The shortest distance from the anterior (posterior) edge of the tumor to the X-axis (the X-axis is a horizontal reference axis, set along the left and right direction of the lung field); A: The farthest distance of the lung field in this quadrant from the X-axis (i.e., the maximum span of the lung field quadrant in the X-axis direction); b: The shortest distance from the inner edge of the tumor to the Y-axis (the Y-axis is the vertical reference axis, set along the vertical direction of the lung field); B: The farthest distance of the lung field on the side of the tumor from the Y-axis (i.e., the maximum span of the lung field in the Y-axis direction).

[0063] Feature calculation: The spatial distance relationship between the lung nodule and the lung edge is quantified using the following three indicators to reflect the location characteristics of the nodule within the lung field: Indicator 1: a / A: The ratio of the relative distance between the anterior (posterior) margin of the tumor and the X-axis boundary of the lung field, reflecting the degree of displacement of the nodule in the horizontal direction of the lung field; Indicator 2: b / B: The ratio of the relative distance between the inner edge of the tumor and the Y-axis boundary of the lung field, reflecting the degree of displacement of the nodule in the vertical direction of the lung field; Indicator 3: a / A+b / B: The sum of the relative distances in the horizontal and vertical directions, which comprehensively measures the overall proximity of the nodule to the lung edge (the smaller the value, the closer the nodule is to the lung edge; the larger the value, the closer the nodule is to the center of the lung field).

[0064] Module 4: ResNet Backbone (Deep Feature Mapping Module) Core operation: Taking the concatenated multi-dimensional feature vector as input, feature mapping is performed through the backbone network structure of ResNet (Residual Network). By leveraging residual blocks to solve the gradient vanishing problem in deep networks, shallow statistical and geometric features are transformed into higher-dimensional and more abstract deep features, enhancing the non-linear expressive power of features and adapting to complex transition prediction tasks.

[0065] Module 5: Prediction (Feature Adaptation Output Module) Core operation: The deep features output by ResNet are used as supplementary features and are associated and integrated with the preceding multimodal fusion features (features after multimodal consistency encoding, feature distillation, and cross-modal alignment) to provide a comprehensive input of "multimodal fusion features + multidimensional supplementary features" for the subsequent multi-scale prediction decoder, ultimately supporting the accurate prediction decision of "hidden lymph node metastasis".

[0066] Specifically, the multimodal fusion-based method for predicting occult lymph node metastasis in lung cancer in this embodiment is used to non-invasively predict the occult lymph node metastasis status of lung cancer patients. It is only provided as a reference and is not a method for doctors to diagnose and treat the disease.

[0067] In other embodiments, a multimodal fusion-based device for predicting occult lymph node metastasis in lung cancer is provided, and a multimodal fusion-based method for predicting occult lymph node metastasis in lung cancer based on any of the preceding embodiments is provided, including: The acquisition module is configured to acquire medical image data and clinical text data as raw multimodal data; The pre-training module is configured to perform multimodal consistency pre-training on the original multimodal data to obtain robust multi-scale fusion features; The fine-tuning module is configured to perform cross-modal prediction fine-tuning on robust multi-scale fusion features to obtain cross-modal semantic alignment features; The module is configured to extract the statistical morphological features and spatial geometric features of lung nodules in parallel through two branches, and then transform the statistical morphological features and spatial geometric features into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features; The decoding processing module is configured to combine cross-modal semantic alignment features with multi-dimensional supplementary features, and obtain the prediction results of occult lymph node metastasis in lung cancer through multi-scale prediction decoding processing.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion, characterized in that, include: Acquire medical imaging data and clinical text data as raw multimodal data; Multimodal consistency pre-training is performed on the original multimodal data to obtain robust multi-scale fusion features; Cross-modal prediction fine-tuning is performed on the robust multi-scale fusion features to obtain cross-modal semantic alignment features; The statistical morphological features and spatial geometric features of lung nodules are extracted in parallel by two branches. The statistical morphological features and spatial geometric features are then transformed into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features. By combining the cross-modal semantic alignment features with the multi-dimensional supplementary features and performing multi-scale prediction decoding processing, the prediction results of occult lymph node metastasis in lung cancer are obtained.

2. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 1, characterized in that, Multimodal consistency pre-training is performed on the original multimodal data to obtain robust multi-scale fusion features, including: The original medical image data is subjected to random adjustments of brightness, contrast, and saturation, Gaussian blurring, and random cropping to obtain perturbed image data. The clinical text data is subjected to synonym replacement, random insertion, and sentence rearrangement to obtain perturbed text data; The perturbed image data and the perturbed text data are combined to obtain the perturbed multimodal data; The original multimodal data and the perturbed multimodal data are respectively input into the multimodal consistency encoder to obtain two sets of transition prediction results; By using a hybrid loss to supervise the transfer prediction results and the actual transfer labels of each group, and by using consistency constraints to strengthen the consistency between the two groups of transfer prediction results, robust multi-scale fusion features are obtained.

3. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 1, characterized in that, Cross-modal prediction fine-tuning is performed on the robust multi-scale fusion features to obtain cross-modal semantic alignment features, including: Robust multi-scale fusion features are input into the student model, and the feature dimensions are adjusted through a lightweight projection network so that the features output by the student model match the global semantic feature dimensions output by the frozen pre-trained teacher model, thus obtaining semantic similarity. Distillation loss is constructed based on semantic similarity to guide the student model to learn and transfer relevant semantic features, thereby obtaining semantically enhanced fusion features; Clinical text data was encoded using a medical-specific text encoder to obtain semantic embeddings of the transfer-related text. Using the semantically enhanced fusion features as queries and the text semantic embedding as keys and values, text semantic information is injected into image features through an attention mechanism to obtain cross-modal semantic alignment features.

4. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 1, characterized in that, Statistical morphological features of lung nodules were extracted, including: First-order statistical features were obtained by gray-level histogram analysis, and texture features were obtained by gray-level co-occurrence matrix calculation and gray-level run length matrix analysis. At the same time, morphological features describing the geometric shape of nodules were extracted. By combining first-order statistical features, texture features, and morphological features, statistical morphological features are obtained.

5. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 1, characterized in that, Extracting the spatial geometric features of lung nodules, including: Set the horizontal and vertical reference axes; The shortest distance between the tumor margin and the reference axis and the farthest distance between the lung field quadrant and the reference axis were obtained by distance calculation. The spatial geometric features are obtained by calculating the proportion of relative distance in the horizontal direction, the proportion of relative distance in the vertical direction, and the sum of the two.

6. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 1, characterized in that, The cross-modal semantic alignment features are combined with the multi-dimensional complementary features, and multi-scale prediction decoding processing is performed, including: Construct multiple parallel decoding branches, each receiving a pair of cross-modal semantic alignment features at adjacent scales; Bilinear interpolation upsampling is performed on the low-resolution cross-modal semantic alignment features in each branch; The upsampled low-resolution features are concatenated with the corresponding high-resolution cross-modal semantic alignment features along the channel dimension to obtain the concatenated features.

7. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 6, characterized in that, The splicing features are refined, including: Multiple rounds of residual convolution are performed on the spliced ​​features, and multi-layer feature information is preserved through residual connections to obtain refined features.

8. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 7, characterized in that, The refinement features are enhanced by, including: The refined features are processed using an efficient channel attention module and a pyramid spatial attention module to highlight relevant salient features and suppress redundant information, resulting in enhanced multi-scale features.

9. The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to claim 8, characterized in that, The predicted results for occult lymph node metastasis in lung cancer include: The enhanced multi-scale features output from all parallel decoding branches are uniformly adjusted to the same resolution and then stitched together to obtain fused features; The fused features are correlated and integrated with multidimensional supplementary features, and a segmentation prediction mask is generated through lightweight convolutional head processing to obtain the prediction results of occult lymph node metastasis in lung cancer.

10. A multimodal fusion-based device for predicting occult lymph node metastasis in lung cancer, characterized in that, The method for predicting occult lymph node metastasis in lung cancer based on multimodal fusion according to any one of claims 1 to 9 includes: The acquisition module is configured to acquire medical image data and clinical text data as raw multimodal data; The pre-training module is configured to perform multimodal consistency pre-training on the original multimodal data to obtain robust multi-scale fusion features; The fine-tuning module is configured to perform cross-modal prediction fine-tuning on the robust multi-scale fusion features to obtain cross-modal semantic alignment features; The module is configured to extract statistical morphological features and spatial geometric features of lung nodules in parallel through two branches, and then transform the statistical morphological features and spatial geometric features into high-dimensional abstract features through deep feature mapping to construct multi-dimensional supplementary features; The decoding processing module is configured to combine the cross-modal semantic alignment features with the multi-dimensional supplementary features, and obtain the prediction result of occult lymph node metastasis of lung cancer through multi-scale prediction decoding processing.

Images