A lung lesion prediction method based on multi-modal data and a storage medium thereof

Abstract

The application provides a lung lesion prediction method based on multi-modal data and a storage medium thereof. The chest CXR image of a patient and the clinical physiological data of the patient are obtained. The chest CXR image and the clinical physiological data are input into a multi-modal model. The lung lesion prediction of the chest CXR image is performed based on the clinical physiological data by the multi-modal model. The prediction result of the lesion in the chest CXR image is obtained. The prediction result includes a lesion area and an influence degree of the lesion on the patient. The influence degree includes mild and severe. Furthermore, under the condition of the fusion of the multi-scale spatial features and the structured clinical physiological data of the chest CXR image, the limitation of a single image mode is broken. When the lesion image performance is atypical or fuzzy, the clinical information can be effectively supplemented and cross-verified, so that the accuracy and robustness of lesion identification and classification are greatly improved.

Description

Technical Field

[0001] This application relates to the field of medical image recognition, and in particular to a method for predicting lung lesions based on multimodal data and its storage medium. Background Technology

[0002] In recent years, while deep learning-based medical image analysis technology has made significant progress, it still faces a series of key bottlenecks and challenges in practical clinical deployment. First, the limitations of single-modal modeling are becoming increasingly apparent: most existing methods rely solely on chest X-ray images for lesion detection or classification, neglecting crucial clinical physiological information. When imaging manifestations are atypical, lesions have low contrast with normal tissue, or similar image artifacts exist, models relying solely on visual features are prone to failure, leading to missed diagnoses or misdiagnoses. Second, the complexity of real-world data constrains model robustness: tabular data collected in clinical settings often suffers from missing indicators, non-standard recording, multi-center distribution differences, and class imbalances. Existing models often lack robust processing mechanisms, resulting in significant performance degradation and insufficient generalization ability when applied across institutions. More critically, the "black box" nature of model decision-making hinders clinical adoption: most end-to-end predictive models lack verifiable interpretable outputs, failing to provide intuitive and credible evidence chains for diagnostic conclusions. Medical personnel struggle to understand and verify the reasoning process, which not only affects clinical trust but also fails to meet increasingly stringent medical device regulatory approval requirements. Summary of the Invention

[0003] The main objective of this application is to propose a lung lesion prediction method and its storage medium based on multimodal data. By fusion of imaging and clinical data through multi-task collaboration, the method significantly improves the accuracy and robustness of lesion prediction and provides verifiable decision-making basis.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for predicting lung lesions based on multimodal data, comprising: Acquire chest CXR images of the patient, as well as the patient's clinical physiological data; Chest CXR images and clinical physiological data are input into a multimodal model. The multimodal model predicts lung lesions in chest CXR images based on clinical physiological data, and obtains the prediction results of lesions in chest CXR images. The prediction results include the lesion area and the degree of impact of the lesions on the patient, including mild and severe impact. The multimodal model includes an image encoder, an image segmentation module, and a multimodal classification module. Predicting lung lesions includes the following steps: The chest CXR image is input into the image encoder, and the image encoder performs multi-scale spatial semantic feature extraction on the chest CXR image to obtain multiple spatial feature maps. Multiple spatial feature maps are input into the image segmentation module, which then segments the multiple spatial feature maps to obtain the lesion region. Clinical physiological data and multiple spatial feature maps are input into the multimodal classification module. The multimodal classification module classifies the lesion impact of the clinical physiological data and multiple spatial feature maps to obtain the degree of impact.

[0005] Furthermore, in some embodiments, the image encoder includes a SwinTransformer structure with multiple spatial levels. The image encoder performs multi-scale spatial feature extraction on the chest CXR image to obtain multiple spatial feature maps, including: The chest CXR image was divided into multiple image blocks by meshing and non-overlapping segments; By using linear projection, multiple image patches are embedded with low-dimensional vectors to form an initial feature sequence; The initial feature sequence is input into the Swin Transformer structure for feature extraction to obtain the spatial feature sequence corresponding to each spatial level. Each spatial feature sequence is restructured into an image spatial structure to obtain spatial feature maps corresponding to each spatial level. The spatial resolutions of the various spatial feature maps are different.

[0006] Furthermore, in some embodiments, the initial feature sequence is input into the Swin Transformer structure for feature extraction to obtain spatial feature sequences corresponding to each spatial level, including: The initial feature sequence is input into the bottom spatial level of the Swing Transformer structure, and the feature is encoded through a window-based self-attention mechanism to obtain the spatial feature sequence at the bottom spatial level. After spatial downsampling and channel expansion, the spatial feature sequence at the bottom spatial level is input to the next spatial level. Feature encoding is performed through a window-based self-attention mechanism to obtain the spatial feature sequence of the next spatial level. The spatial feature sequence of the upper spatial level is used as the spatial feature sequence of the lower spatial level. The process of spatial downsampling and channel expansion of the spatial feature sequence of the lower spatial level is repeated and then input into the upper spatial level. Feature encoding is performed through a window-based self-attention mechanism to obtain the spatial feature sequence of the upper spatial level. This process is repeated until the upper spatial level is the top spatial level, thus obtaining the spatial feature sequence corresponding to each spatial level.

[0007] Furthermore, in some embodiments, the image segmentation module includes an image decoder, which segments multiple spatial feature maps to obtain lesion regions, including: Multiple spatial feature maps are input into the image decoder. The image decoder performs multi-scale feature fusion and progressive upsampling of spatial resolution on the multiple spatial feature maps to reconstruct a pixel-level segmentation map that matches the size of the chest CXR image. Applying the first classification activation function to the pixel-level segmentation map yields a probability distribution map representing the lesion region for each pixel. The probability distribution map is binarized to obtain the lesion area.

[0008] Furthermore, in some embodiments, the multimodal classification module includes a multilayer perceptron and a classifier. The multimodal classification module classifies the impact of lesions on clinical physiological data and multiple spatial feature maps to obtain the degree of impact, including: Imputation of missing data in clinical physiological data; The imputed clinical physiological data is input into a multilayer perceptron, and the vector features of the imputed clinical physiological data are extracted by the multilayer perceptron to obtain the clinical physiological features. In multiple spatial feature maps, the spatial feature map at the top spatial level is subjected to global averaging to obtain the image semantic features. Clinical physiological features and image semantic features are concatenated to obtain fused features; The fused features are input into a classifier, which then predicts the impact of the fused features on lesions to obtain the degree of impact.

[0009] Furthermore, in some embodiments, missing data in clinical physiological data are filled in, including: Use preset placeholder values ​​to fill in the missing items in clinical physiological data to form initial imputation data; For each physiological indicator in the clinical physiological data, a binary mask vector is generated to indicate whether the data has been filled. The initial padding data is concatenated with the binary mask vector.

[0010] Furthermore, in some embodiments, a classifier is used to predict the lesion impact of the fused features to obtain the degree of impact, including: By using a classifier to perform nonlinear transformation and feature mapping on the fused features, the predicted value of the impact of lesions in chest CXR images on patients is output. The predicted value is compared with a preset decision threshold. If the predicted value is greater than or equal to the decision threshold, a value representing a mild impact is generated. If the predicted value is less than the decision threshold, a value representing a severe impact is generated.

[0011] Furthermore, in some embodiments, the multimodal model is obtained through the following steps: Obtain a first sample dataset and a second sample dataset; wherein, the first sample dataset includes multiple first chest CXR sample images with lesion areas marked, multiple second chest CXR sample images with no lesion areas marked, and the second sample dataset includes multiple third chest CXR sample images with lesion areas marked and the degree of influence marked, and clinical physiological data corresponding to the patients to which each third chest CXR sample image belongs. Initialize the original model containing the image encoder, image segmentation module, and multimodal classification module; The first sample dataset is input into the original model to iteratively train the image encoder and image segmentation module to obtain an intermediate model; The second sample dataset is input into the intermediate model to iteratively train the image encoder, image segmentation module, and multimodal classification module to obtain the multimodal model.

[0012] Furthermore, in some embodiments, the second sample dataset is input into an intermediate model to iteratively train the image encoder, image segmentation module, and multimodal classification module to obtain a multimodal model, including: A batch sample was obtained from the second sample dataset. The batch sample included a third chest CXR sample image labeled with the lesion area and the degree of influence, as well as the clinical physiological data corresponding to the third chest CXR sample image. The third chest CXR sample image from the batch samples is input to the image encoder, and the multiple sample feature maps extracted by the image encoder are input to the image segmentation module and the segmentation gradient loss value of the image segmentation module is calculated. The feature maps of multiple samples extracted by the image encoder and clinical physiological data are input into the multimodal classification module, and the classification gradient loss value of the multimodal classification module is calculated. The segmentation gradient loss value and the classification gradient loss value are weighted and summed to obtain the total gradient loss value of the intermediate model; Based on the total gradient loss value, the parameters of the image encoder, image segmentation module, and multimodal classification module are updated in reverse using the backpropagation algorithm; If the parameters of the image encoder, image segmentation module, and multimodal classification module all meet the training conditions, the training of the intermediate model is terminated, and the multimodal model is obtained.

[0013] To achieve the above objectives, a second aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the lung lesion prediction method based on multimodal data described in the first aspect of the present application.

[0014] The embodiments of the first aspect of this application have the following beneficial effects: By integrating multi-scale spatial features of chest CXR images with structured clinical physiological data, the model overcomes the limitations of a single image modality. When the lesion image is atypical or blurry, it can effectively supplement and cross-validate based on clinical information, thereby significantly improving the accuracy and robustness of lesion identification and classification. Secondly, the model adopts an integrated design, sharing the features extracted by the image encoder to simultaneously serve two tasks: lesion region segmentation and severity classification. This not only improves computational efficiency but also enables classification decisions to obtain spatial evidence support from pixel-level segmentation results, forming a traceable and verifiable interpretive decision chain, greatly enhancing doctors' trust in the model's output. Overall, it provides a solution for intelligent assisted diagnosis of lung diseases that simultaneously possesses high accuracy and high reliability. Attached Figure Description

[0015] Figure 1 This is an optional flowchart of a lung lesion prediction method based on multimodal data provided in the embodiments of this application; Figure 2 This is an optional architecture diagram of the multimodal model provided in the embodiments of this application; Figure 3 This is an optional flowchart of image feature extraction performed by the image encoder provided in this application embodiment; Figure 4 This is provided by the embodiments of this application. Figure 3 An optional flowchart for step S203; Figure 5 This is an optional schematic diagram illustrating the determination of pose data of a UAV relative to a defect point provided in an embodiment of this application; Figure 6 This is an optional flowchart of the multimodal classification module provided in this application for classifying the impact of lesions; Figure 7 This is an optional flowchart provided in the embodiments of this application for filling in missing items in clinical physiological data; Figure 8This is an optional flowchart of a classifier providing embodiments of this application predicting the impact of fused features on lesions; Figure 9 This is an optional flowchart of training a multimodal model provided in the embodiments of this application; Figure 10 This is another optional flowchart for training a multimodal model provided in the embodiments of this application; Figure 11 This is an optional flowchart for training the intermediate model provided in the embodiments of this application; Figure 12 This is a performance rating chart of a multimodal model trained on the same second sample dataset and other models, provided in an embodiment of this application. Figure 13 This is a model performance rating chart of the ablation experiment provided in the embodiments of this application; Figure 14 This is a comparison chart of the outputs of the multimodal model provided in this application embodiment and the existing baseline model; Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0018] It should also be noted that in the description of this application, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If the terms "first" and "second" are used, they are only for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0021] In recent years, while deep learning-based medical image analysis technology has made significant progress, it still faces a series of key bottlenecks and challenges in practical clinical deployment. First, the limitations of single-modal modeling are becoming increasingly apparent: most existing methods rely solely on chest X-ray images for lesion detection or classification, neglecting crucial clinical physiological information. When imaging manifestations are atypical, lesions have low contrast with normal tissue, or similar image artifacts exist, models relying solely on visual features are prone to failure, leading to missed diagnoses or misdiagnoses. Second, the complexity of real-world data constrains model robustness: tabular data collected in clinical settings often suffers from missing indicators, non-standard recording, multi-center distribution differences, and class imbalances. Existing models often lack robust processing mechanisms, resulting in significant performance degradation and insufficient generalization ability when applied across institutions. More critically, the "black box" nature of model decision-making hinders clinical adoption: most end-to-end predictive models lack verifiable interpretable outputs, failing to provide intuitive and credible evidence chains for diagnostic conclusions. Medical personnel struggle to understand and verify the reasoning process, which not only affects clinical trust but also fails to meet increasingly stringent medical device regulatory approval requirements.

[0022] To address these issues, this application proposes a lung lesion prediction method based on multimodal data and its storage medium. By collaboratively fusing imaging and clinical data through multiple tasks, it significantly improves the accuracy and robustness of lesion prediction and provides verifiable decision-making basis.

[0023] This application provides a method for predicting lung lesions based on multimodal data and its storage medium, which is specifically illustrated through the following embodiments.

[0024] Firstly, referring to Figure 1 As shown, Figure 1 This is an optional flowchart of a lung lesion prediction method based on multimodal data provided in the embodiments of this application. The method may include, but is not limited to, steps S101 to S102.

[0025] Step S101: Obtain the patient's chest CXR image and the patient's clinical physiological data.

[0026] Among them, the chest CXR image is the chest X-ray image. X-ray imaging is a technique that uses X-rays to image the internal structure of the human body. After being received by a detector or film, it forms a grayscale image. The chest CXR image is used to evaluate lesions of the lungs, heart, chest cavity, etc.

[0027] The aforementioned clinical physiological data, including key physiological indicators such as age, sex, blood pressure, heart rate, blood oxygen saturation, white blood cell count, and respiratory rate, are not specifically limited in this application.

[0028] Step S102: Input the chest CXR image and clinical physiological data into the multimodal model, and use the multimodal model to predict lung lesions in the chest CXR image based on the clinical physiological data to obtain the prediction results of lesions in the chest CXR image.

[0029] The prediction results include the lesion area and the degree of impact of the lesion on the patient, with the degree of impact including mild and severe.

[0030] It is worth noting that, referring to the reference Figure 2 As shown, Figure 2 This is an optional architecture diagram of the multimodal model provided in the embodiments of this application. The multimodal model includes an image encoder, an image segmentation module, and a multimodal classification module.

[0031] It should be noted that the above process of predicting lung lesions in chest CXR images includes the following steps: inputting the chest CXR image into an image encoder, extracting multi-scale spatial semantic features from the chest CXR image using the image encoder to obtain multiple spatial feature maps; then, inputting the multiple spatial feature maps into an image segmentation module, segmenting the multiple spatial feature maps into lesion regions using the image segmentation module; finally, inputting clinical physiological data and multiple spatial feature maps into a multimodal classification module, classifying the clinical physiological data and multiple spatial feature maps into lesion impact categories to obtain the degree of impact.

[0032] In one possible embodiment, the processor first acquires data by using a digital X-ray machine to obtain a chest CXR image of a patient suspected of having lung lesions. Simultaneously, it extracts and standardizes the corresponding patient's clinical and physiological data from the electronic medical record system. Next, it performs feature extraction by inputting the CXR image into an image encoder, which performs convolution and pooling operations to extract multi-scale spatial semantic features, outputting four spatial feature maps at different scales. Then, it performs lesion segmentation by inputting the four spatial feature maps into an image segmentation module, which uses a U-Net architecture for pixel-level segmentation to generate lesion region masks to accurately mark the location and extent of lesions. Subsequently, it performs impact classification by inputting the standardized clinical and physiological data and the four spatial feature maps into a multimodal classification module, which uses an attention mechanism to fuse bimodal information to classify the degree of lesion impact. Finally, it outputs the results by integrating the segmented lesion regions with the classified degree of impact (mild / severe) to generate a CXR image with visual annotations and a diagnostic report.

[0033] In steps S101 and S102, by integrating multi-scale spatial features of chest CXR images with structured clinical and physiological data, the model overcomes the limitations of a single image modality. When lesion images are atypical or blurry, it can effectively supplement and cross-validate based on clinical information, thereby significantly improving the accuracy and robustness of lesion identification and classification. Secondly, the model adopts an integrated design, sharing features extracted by the image encoder to simultaneously serve both lesion region segmentation and severity classification tasks. This not only improves computational efficiency but also enables classification decisions to obtain spatial evidence support from pixel-level segmentation results, forming a traceable and verifiable interpretive decision chain, greatly enhancing doctors' trust in the model's output. Overall, it provides a high-precision and high-reliability solution for intelligent assisted diagnosis of lung diseases.

[0034] Furthermore, refer to Figure 3 As shown, Figure 3 This is an optional flowchart of image feature extraction by an image encoder provided in the embodiments of this application. The method may include, but is not limited to, steps S201 to S204.

[0035] Step S201: The chest CXR image is meshed and non-overlapping to obtain multiple image blocks.

[0036] Step S202: By using linear projection, multiple image patches are embedded with low-dimensional vectors to form an initial feature sequence.

[0037] Step S203: Input the initial feature sequence into the Swing Transformer structure for feature extraction to obtain the spatial feature sequence corresponding to each spatial level.

[0038] Step S204: Reconstruct the image spatial structure of each spatial feature sequence to obtain the spatial feature map corresponding to each spatial level.

[0039] The spatial resolutions of the various spatial feature maps are different.

[0040] Specifically, in steps S201 to S204, the CXR image is first processed. After preprocessing, the image is divided into non-overlapping 16×16 pixel blocks. Each block is embedded into a low-dimensional feature vector through linear projection, forming an initial feature sequence. This sequence is then input into a hierarchical image encoder built around the Swin Transformer. The encoder consists of four sequentially connected stages, each employing a window-based self-attention mechanism for local feature relationship modeling. After the output of the previous stage, the feature sequence is spatially downsampled and channel-expanded through block merging, thereby progressively generating four spatial feature sequences with decreasing resolution and increasing semantic hierarchy. Reorganizing these sequences according to their spatial dimensions yields four spatial feature maps corresponding to different scales. .

[0041] Furthermore, the image encoder includes a Swin Transformer structure with multiple spatial levels, as shown in the reference. Figure 4 As shown, Figure 4 This is provided by the embodiments of this application. Figure 3 An optional flowchart for step S203, the method may include, but is not limited to, steps S301 to S303.

[0042] Step S301: Input the initial feature sequence into the bottom spatial level of the Swing Transformer structure, and encode the features through a window-based self-attention mechanism to obtain the spatial feature sequence at the bottom spatial level.

[0043] Step S302: After performing spatial downsampling and channel expansion on the spatial feature sequence at the bottom spatial level, input it to the next spatial level and encode the features using a window-based self-attention mechanism to obtain the spatial feature sequence of the next spatial level.

[0044] Step S303: Take the spatial feature sequence of the upper spatial level as the spatial feature sequence of the lower spatial level, repeat the process of spatial downsampling and channel expansion of the spatial feature sequence of the lower spatial level, input it into the upper spatial level, and perform feature encoding through a window-based self-attention mechanism to obtain the spatial feature sequence of the upper spatial level, until the upper spatial level is the top spatial level, and obtain the spatial feature sequence corresponding to each spatial level.

[0045] In one specific embodiment of steps S301 to S303, the initial feature sequence is input to the bottom spatial level (i.e., the first stage) of the SwinTransformer structure. At this level, the feature sequence is encoded using a window-based self-attention mechanism to fully model local context information, thereby obtaining the spatial feature sequence of this bottom layer. Subsequently, spatial downsampling and channel expansion operations (usually achieved through block merging) are performed on this sequence to reduce its spatial resolution and increase the feature dimension. The processed feature sequence is then fed into the next layer (i.e., the second stage) spatial level and undergoes the same window-based self-attention encoding to obtain the corresponding spatial feature sequence for that layer. This process is iteratively performed in a loop: the feature sequence output from the current layer is treated as a new "bottom layer" input, and spatial downsampling, channel expansion, and window self-attention encoding are repeatedly performed, progressively passing the process to higher levels. This hierarchical processing continues until the feature sequence is passed and encoded to the top spatial level (i.e., the fourth stage). Ultimately, through this bottom-up, hierarchical abstraction approach, the encoder synchronously outputs the spatial feature sequences corresponding to all four spatial levels, providing a multi-scale, multi-level feature representation foundation for subsequent segmentation and classification tasks.

[0046] Furthermore, the image segmentation module includes an image decoder, as shown in the reference... Figure 5 As shown, Figure 5 This is an optional flowchart of the image segmentation module provided in the embodiments of this application for lesion segmentation. The process of this method may include, but is not limited to, steps S401 to S403.

[0047] Step S401: Input multiple spatial feature maps into the image decoder, and perform multi-scale feature fusion and progressive upsampling of spatial resolution on the multiple spatial feature maps to reconstruct a pixel-level segmentation map that matches the size of the chest CXR image.

[0048] Step S402: Apply the first classification activation function to the pixel-level segmentation map to obtain a probability distribution map representing that each pixel belongs to the lesion region.

[0049] Step S403: Binarize the probability distribution map to obtain the lesion area.

[0050] In one specific embodiment of steps S401 to S403, the image segmentation module completes accurate segmentation of the lesion region through its internal image decoder. This decoder receives four feature maps with different spatial resolutions output from the image encoder (such as a SwingTransformer). The core of the decoding process lies in multi-scale feature fusion and spatial resolution reconstruction: First, starting with the feature map at the highest semantic level (lowest resolution), it is upsampled using bilinear interpolation or transposed convolution to double its spatial size. Subsequently, skip connections are used to concatenate and fuse the upsampled features with feature maps of the same spatial size from the corresponding level of the encoder along the channel dimension. This operation effectively combines high-level semantic information with low-level detail texture. This upsampling-fusion step is repeated step by step until the spatial size of the feature map is restored to match the original input chest CXR image. Finally, after further refinement by a series of convolutional layers, a pixel-level segmentation map is output. Next, a Sigmoid activation function is applied to each pixel position of this segmentation map, converting it into a probability distribution map where the value of each pixel represents its confidence level (between 0 and 1) in belonging to the lesion region. Finally, the probability map is binarized by setting a decision threshold (e.g., 0.5), and pixels with probability values ​​greater than or equal to the threshold are identified as lesion pixels, thereby generating a clear, binary lesion region mask, realizing pixel-level localization and delineation of lesions.

[0051] Furthermore, the multimodal classification module includes a multilayer perceptron and a classifier, as shown in the reference... Figure 6 As shown, Figure 6 This is an optional flowchart of a multimodal classification module for classifying lesion impact provided in the embodiments of this application. The method may include, but is not limited to, steps S501 to S505.

[0052] Step S501: Fill in missing items in the clinical physiological data.

[0053] Step S502: Input the imputed clinical physiological data into a multilayer perceptron, and extract the vector features of the imputed clinical physiological data through the multilayer perceptron to obtain the clinical physiological features.

[0054] Specifically, the imputed clinical physiological data is input into a multilayer perceptron (MLP), and the MLP extracts the vector features of the imputed clinical physiological data to obtain the clinical physiological features. R represents the spatial dimension. The formulaic expression for the extraction of clinical physiological features by a multilayer perceptron is as follows: .

[0055] Step S503: In multiple spatial feature maps, perform global averaging on the spatial feature map at the top spatial level to obtain image semantic features.

[0056] In one embodiment, in multiple spatial feature maps Select the spatial feature map at the top spatial level. This spatial feature map The highest semantic level (lowest resolution) feature map is used, followed by the spatial feature map. Global averaging is performed to obtain image-level semantic vector features. That is, image semantic features, which are formally expressed as: .

[0057] Step S504: Combine clinical physiological features with image semantic features to obtain fused features.

[0058] Specifically, the fusion feature is formally expressed as:

[0059] Step S505: Input the fused features into the classifier, and use the classifier to predict the impact of the fused features on the lesions to obtain the degree of impact.

[0060] The loss function of the classifier is the binary cross-entropy loss function. .

[0061] It should be noted that in steps S501 to S505, firstly, the system imputes the acquired clinical physiological data (such as age, laboratory indicators, etc.) that may contain missing items. Then, the imputed clinical physiological data is input into a multilayer perceptron, where its fully connected layers and nonlinear activation functions extract high-level abstract features, thereby obtaining a structured clinical physiological feature vector. On the image feature side, the module selects the top-level (i.e., lowest resolution, highest semantic) spatial feature map output by the image encoder (such as Swin Transformer), performs global average pooling on the features at all spatial locations, and aggregates them to obtain a compact image semantic feature vector that represents the overall semantics of the image. Next, the above clinical physiological feature vector and image semantic feature vector are concatenated along the feature dimension to generate a fusion feature vector that integrates multi-source information. Finally, the fused feature vector is fed into the final classifier (e.g., a lightweight multilayer perceptron or linear classification layer). The classifier calculates and outputs a scalar value representing the predicted probability, and then determines the degree of impact of the lesion on the patient as "mild" or "severe" based on a preset threshold, thus completing the end-to-end inference from multimodal data to clinical assessment.

[0062] Furthermore, refer to Figure 7 As shown, Figure 7 This is an optional flowchart for filling in missing items in clinical physiological data provided in the embodiments of this application. The method may include, but is not limited to, steps S601 to S603.

[0063] Step S601: Fill the missing items in the clinical physiological data with preset placeholder values ​​to form initial imputation data.

[0064] Step S602: For each physiological indicator of the clinical physiological data, generate a binary mask vector to indicate whether the data has been filled.

[0065] Step S603: Concatenate the initial padding data with the binary mask vector.

[0066] In steps S601 to S603, when the system receives structured clinical physiological data containing multi-dimensional information such as age, laboratory test indicators, and vital signs... Then, missing items were identified, among which, This indicates the dimension of clinical physiological data. For any missing items identified in the table feature dimensions, the system fills them with a predefined placeholder value that is co-optimized with the model. This value is typically the mean, median, or an explicit learnable parameter of the feature on the training set, thus forming a complete initial imputed data vector. However, simple imputation may confuse the model's judgment of data authenticity. Therefore, the system simultaneously generates a corresponding binary mask value for each physiological indicator in the original clinical physiological data. If the original value of the indicator exists, the mask value is 0; if it is a missing item filled by the system, the mask value is 1. This mask explicitly tells the model which missing items have been filled. The mask values ​​of all indicators are combined into a binary mask vector that matches the dimension of the initial imputed data vector. Finally, the initial imputed data vector and the binary mask vector are concatenated along the feature dimension to generate the final enhanced feature vector that integrates the original (or imputed) numerical information with the information about the missing data state. This design enables the subsequent multilayer perceptron feature extractor to clearly distinguish between true observations and imputed values. This allows the model to learn to more reasonably weigh the credibility of information from different sources, significantly improving its generalization ability and decision robustness when dealing with incomplete and heterogeneous clinical and physiological data.

[0067] Furthermore, refer to Figure 8 As shown, Figure 8 This is an optional flowchart of a classifier providing an embodiment of the present application for predicting the impact of fused features on lesions. The method may include, but is not limited to, steps S701 to S702.

[0068] Step S701: The fusion features are nonlinearly transformed and mapped by a classifier to output the predicted value of the impact of lesions in the chest CXR image on the patient.

[0069] Step S702: Compare the predicted value with the preset decision threshold. If the predicted value is greater than or equal to the decision threshold, generate a characterization of mild impact. If the predicted value is less than the decision threshold, generate a characterization of severe impact.

[0070] In steps S701 to S702, the multimodal classification module ultimately determines the severity of the lesion through its classifier. This classifier typically consists of a lightweight multilayer perceptron, whose core function is to perform nonlinear transformation and high-level feature mapping on the input fused feature vector. Specifically, the fused features pass sequentially through the classifier's fully connected layer and activation function (such as ReLU). After multiple layers of computation and abstraction, the output layer passes through a sigmoid function, ultimately generating a scalar predicted value between 0 and 1. This value comprehensively represents the probability that the model judges the lesion's impact on the patient to be "severe." Subsequently, the system compares this predicted value with a preset and adjustable decision threshold (e.g., 0.5), generating the final classification label based on the comparison result: if the predicted value is greater than or equal to the decision threshold, the impact is determined to be "severe"; conversely, if the predicted value is less than the decision threshold, it is determined to be "mild." This decision-making process transforms continuous model confidence into clear clinical classification conclusions, thereby providing doctors with qualitative judgments that can be directly used to assist in diagnostic and treatment decisions.

[0071] Furthermore, refer to Figure 9 and Figure 10 As shown, Figure 9 This is an optional flowchart for training a multimodal model provided in the embodiments of this application. Figure 10 This is another optional flowchart for training a multimodal model provided in the embodiments of this application. The method may include, but is not limited to, steps S801 to S804.

[0072] Step S801: Obtain the first sample dataset and the second sample dataset.

[0073] The first sample dataset includes multiple first chest CXR sample images with labeled lesion areas and multiple second chest CXR sample images with labeled non-lesion areas. In a preferred embodiment, the first sample dataset includes 1364 chest CXR sample images from different patients, of which 580 (approximately 43%) are from pneumonia-positive patients (i.e., first chest CXR sample images) and 784 (approximately 57%) are from pneumonia-negative patients (i.e., second chest CXR sample images). Each chest CXR sample image includes a lesion segmentation mask indicating whether or not lesion areas are present. And a binary label for whether it is a positive case of pneumonia. The first sample dataset can be represented as .

[0074] Furthermore, the second sample dataset includes multiple third chest CXR sample images labeled with lesion areas and their degree of impact, along with the corresponding clinical physiological data of the patients associated with each third chest CXR sample image. In a preferred embodiment, the second sample dataset includes 844 chest CXR sample images of patients diagnosed with pneumonia and their corresponding clinical physiological data. The chest CXR sample images are stored in DICOM format, and each image is labeled with a "mild" or "severe" degree of impact. The clinical physiological data covers symptoms, laboratory test results, underlying disease information, and disease outcomes. reactive protein , Key indicators, such as the dimer, are highly correlated with disease severity, reflecting inflammatory response, coagulation status, and pulmonary oxygenation, and have strong predictive value for disease progression. Precise pairing of imaging and clinical information forms high-quality multimodal samples, which is beneficial for joint modeling. This second sample dataset can be represented as... .

[0075] Step S802: Initialize the original model containing the image encoder, image segmentation module and multimodal classification module.

[0076] Step S803: Input the first sample dataset into the original model to iteratively train the image encoder and image segmentation module to obtain an intermediate model.

[0077] It should be noted that the multimodal model is trained using a phased strategy. The first phase aims to fully utilize a large number of labeled first and second chest CXR sample images to initially construct image segmentation functionality. During training, the image encoder and image segmentation module in the original model are activated, while the paths involving clinical data processing are temporarily disabled. Specifically, the first and second chest CXR sample images are input into the image encoder to extract multi-scale spatial feature maps; these features... Figure 1 The image data is fed into the image segmentation module, decoded to generate pixel-level prediction maps, and optimized using a segmentation loss function. This optimizes the model in the forward propagation stage, allowing the image encoder to learn feature representations rich in localization information. Through extensive iterative training on this pure image dataset, a robust intermediate model for visual tasks is obtained, laying a high-quality visual feature foundation for subsequent multimodal joint fine-tuning using clinical data in the second stage.

[0078] The loss function in step S803 is: ,in, Segmentation mask for lesions Empirical weight parameters, A binary label for whether or not the patient is positive for pneumonia. Empirical weight parameters, Let cross-entropy be the loss function. Binary cross-entropy loss function.

[0079] Step S804: Input the second sample dataset into the intermediate model to iteratively train the image encoder, image segmentation module, and multimodal classification module to obtain the multimodal model.

[0080] Specifically, the multimodal model achieves final optimization from a pure visual model to a multimodal joint inference model through a crucial second-stage training. This stage uses a second sample dataset, which not only includes chest CXR images but also incorporates corresponding real clinical physiological data. Starting with the intermediate model obtained from the previous stage, the multimodal classification module is unlocked and activated, while the image encoder and image segmentation modules remain trainable. This process enables the model to learn to comprehensively utilize visual features and structured clinical information for decision-making: the image encoder, based on the strong visual representations learned in the previous stage, further adjusts to adapt to synergy with clinical features; the classifier learns how to weigh and integrate evidence from both modalities. Finally, after thorough fine-tuning on this stage's dataset, the model is optimized into a final multimodal model capable of accurate lesion localization and robust severity assessment relying on multi-source information, achieving an advancement from perception to comprehensive diagnosis.

[0081] It is worth noting that the loss function in step S804 is , where the classification output The segmentation output is a fusion representation derived from image and clinical features. To use pseudo tags Pseudo-labels obtained through training The pseudo-mask generated for the image by the intermediate model is represented as .

[0082] Furthermore, in one possible embodiment, referring to Figure 11 As shown, Figure 11 This is an optional flowchart for training an intermediate model provided in the embodiments of this application. The method may include, but is not limited to, steps S901 to S906.

[0083] Step S901: Obtain a batch sample from the second sample dataset. The batch sample includes a third chest CXR sample image labeled with the lesion area and the degree of influence, as well as the corresponding clinical physiological data of the third chest CXR sample image.

[0084] Step S902: Input the third chest CXR sample image from the batch samples into the image encoder, and input the multiple sample feature maps extracted by the image encoder into the image segmentation module and calculate the segmentation gradient loss value of the image segmentation module.

[0085] In this process, lesion segmentation is a pixel-by-pixel binary classification problem. To alleviate the foreground / background imbalance, the original gradient loss value needs to be... Gradient loss calculated with binary cross-entropy By combining these values, we obtain the segmentation gradient loss value, which is expressed as follows: ,in, Used to measure the degree of overlap between the predicted mask and the true mask. Used to penalize pixel-level classification errors.

[0086] Step S903: Input the multiple sample feature maps and clinical physiological data extracted by the image encoder into the multimodal classification module, and calculate the classification gradient loss value of the multimodal classification module.

[0087] Step S904: The segmentation gradient loss value and the classification gradient loss value are weighted and summed to obtain the total gradient loss value of the intermediate model.

[0088] Specifically, the gradient loss value will be segmented. With classification gradient loss value Perform a weighted summation, represented as hyperparameters and The total gradient loss is used as a summation weight to balance the image segmentation module and the multimodal classification module. By minimizing this total gradient loss value, the shared image encoder and task branches (image segmentation module and multimodal classification module) can be trained end-to-end to achieve joint optimization.

[0089] Step S905: Based on the total gradient loss value, update the parameters of the image encoder, image segmentation module and multimodal classification module in reverse using the backpropagation algorithm.

[0090] Step S906: If the parameters of the image encoder, image segmentation module, and multimodal classification module all meet the training conditions, terminate the training of the intermediate model and obtain the multimodal model.

[0091] In steps S901 to S906, i.e., in each iteration, the model performs a complete multimodal forward propagation: image data is processed by the encoder to extract features, which are used by the segmentation module to predict lesion regions and calculate segmentation loss; on the other hand, the top-level semantic features are concatenated and fused with the clinical features that have undergone imputation and masking, and input into the multimodal classification module to obtain severity prediction and calculate classification loss. The total loss function in this stage is a weighted sum of the segmentation loss and the multimodal classification loss. Through the backpropagation algorithm, this joint loss simultaneously updates all parameters of the image encoder, the segmentation module, and the newly introduced multimodal classification module.

[0092] To verify the effectiveness of the multimodal model of this application, in a preferred embodiment, the first sample dataset includes 1364 chest CXR sample images from different patients, of which 580 (approximately 43%) are from pneumonia-positive patients (i.e., the first chest CXR sample images), and 784 (approximately 57%) are from pneumonia-negative patients (i.e., the second chest CXR sample images). The second sample dataset includes 844 chest CXR sample images of confirmed pneumonia-positive patients and corresponding clinical physiological data. The multimodal model trained using the first and second sample datasets is compared with a series of baseline models based on existing architectures. All methods are trained and evaluated under the same settings. (Refer to...) Figure 12 As shown, Figure 12 This is a performance score graph of a multimodal model trained on the same second sample dataset and other models provided in this application embodiment. In the graph, the model named 3M-Net is the multimodal model of this application. The performance score of the multimodal model of this application is significantly better than the performance scores of other models.

[0093] Furthermore, to verify the effectiveness of the image segmentation module and the multimodal classification module, ablation experiments were conducted by gradually adding the image segmentation module and the multimodal classification module, starting with the image encoder (SwinTransformer). Figure 13 As shown, Figure 13 This is a performance rating chart of the ablation experiment model provided in the embodiments of this application. In the chart, Swin+UPerNet is the model for lesion region identification using an image encoder and a conventional encoder; Swin+UPerNet+Multi-Task is the model for lesion region identification using an image encoder and an image segmentation module; Swin+UPerNet+Multi-Modal is the model for lesion region identification using an image encoder and a multimodal classification module; and 3M-Net is the multimodal model of this application. As can be seen from the chart, the model configured with the image segmentation module and multimodal classification module of this application has a significantly better performance rating than other models.

[0094] Furthermore, to visually demonstrate the performance of the multimodal model of this application, a comparison is made between the output of the multimodal model of this application and the output of a baseline model based on an existing architecture, as an example. Figure 14 As shown, Figure 14 This is a comparison of the outputs of the multimodal model provided in this application and an existing baseline model. The figure includes (a) a labeled sub-image, (b) a labeled image, and (c) a labeled sub-image. (a) a labeled sub-image is the original chest CXR image, in which lung texture is unclear and lesion signs are subtle. (b) a labeled image is the output image (CAM heatmap) of the existing baseline model, which misclassifies the severity of the case as "mild" due to insufficient visual feature extraction. (c) a labeled image is the output image (CAM heatmap) of the multimodal model of this application, which focuses its attention on the lesion and classifies it as "severe". The comparison between (b) and (c) a labeled sub-image shows that the performance of the multimodal model of this application is better than that of the baseline model based on the existing architecture.

[0095] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for predicting lung lesions based on multimodal data. This electronic device can be any smart terminal, including mobile phones, tablets, and in-vehicle computers.

[0096] Please see Figure 15 , Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. The electronic device includes: The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the lung lesion prediction method based on multimodal data provided in the embodiments of this application. The memory 1502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 using the lung lesion prediction method based on multimodal data provided in the embodiments of this application. The input / output interface 1503 is used to implement information input and output; The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504); The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0097] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, provides the lung lesion prediction method based on multimodal data.

[0098] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0099] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0100] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0103] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0104] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0105] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0106] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for predicting lung lesions based on multimodal data, characterized in that, include: Acquire chest CXR images of the patient, as well as the patient's clinical physiological data; The chest CXR image and clinical physiological data are input into a multimodal model. The multimodal model predicts lung lesions in the chest CXR image based on the clinical physiological data, and obtains the prediction results of lesions in the chest CXR image. The prediction results include the lesion area and the degree of impact of the lesion on the patient. The degree of impact includes mild and severe. The multimodal model includes an image encoder, an image segmentation module, and a multimodal classification module. The prediction of lung lesions includes the following steps: The chest CXR image is input into the image encoder, and the image encoder performs multi-scale spatial semantic feature extraction on the chest CXR image to obtain multiple spatial feature maps. The multiple spatial feature maps are input into the image segmentation module, and the image segmentation module performs lesion segmentation on the multiple spatial feature maps to obtain the lesion region; The clinical physiological data and the multiple spatial feature maps are input into the multimodal classification module. The multimodal classification module classifies the lesion impact of the clinical physiological data and the multiple spatial feature maps to obtain the degree of impact.

2. The method for predicting lung lesions according to claim 1, characterized in that, The image encoder includes a Swing Transformer structure with multiple spatial levels. The image encoder performs multi-scale spatial feature extraction on the chest CXR image to obtain multiple spatial feature maps, including: The chest CXR image is divided into multiple image blocks by meshing and non-overlapping segments; By using linear projection, the multiple image patches are embedded with low-dimensional vectors to form an initial feature sequence; The initial feature sequence is input into the Swin Transformer structure for feature extraction to obtain the spatial feature sequence corresponding to each spatial level; Each of the spatial feature sequences is restructured into an image spatial structure to obtain spatial feature maps corresponding to each spatial level. The spatial resolutions of each of the spatial feature maps are different.

3. The method for predicting lung lesions according to claim 2, characterized in that, The step of inputting the initial feature sequence into the Swing Transformer structure for feature extraction to obtain the spatial feature sequence corresponding to each spatial level includes: The initial feature sequence is input into the bottom spatial level of the Swin Transformer structure, and feature encoding is performed through a window-based self-attention mechanism to obtain the spatial feature sequence of the bottom spatial level; After spatial downsampling and channel expansion processing of the spatial feature sequence at the bottom spatial level, it is input to the upper spatial level. Feature encoding is performed through a window-based self-attention mechanism to obtain the spatial feature sequence of the upper spatial level. The spatial feature sequence of the upper spatial level is used as the spatial feature sequence of the lower spatial level. The process of performing spatial downsampling and channel expansion on the spatial feature sequence of the lower spatial level and inputting it into the upper spatial level is repeated. Feature encoding is performed through a window-based self-attention mechanism to obtain the spatial feature sequence of the upper spatial level. This process is repeated until the upper spatial level is the top spatial level, thus obtaining the spatial feature sequence corresponding to each spatial level.

4. The method for predicting lung lesions according to claim 3, characterized in that, The image segmentation module includes an image decoder. The step of segmenting the multiple spatial feature maps using the image segmentation module to obtain the lesion region includes: The multiple spatial feature maps are input into the image decoder, and the image decoder performs multi-scale feature fusion and progressive upsampling of spatial resolution on the multiple spatial feature maps to reconstruct a pixel-level segmentation map that matches the size of the chest CXR image; Applying a first classification activation function to the pixel-level segmentation map yields a probability distribution map representing that each pixel belongs to the lesion region. The probability distribution map is binarized to obtain the lesion region.

5. The method for predicting lung lesions according to claim 4, characterized in that, The multimodal classification module includes a multilayer perceptron and a classifier. The step of classifying the lesion impact on the clinical physiological data and the multiple spatial feature maps using the multimodal classification module to obtain the degree of impact includes: Missing data were filled in for the clinical physiological data; The imputed clinical physiological data is input into the multilayer perceptron, and the vector features of the imputed clinical physiological data are extracted by the multilayer perceptron to obtain the clinical physiological features. In the multiple spatial feature maps, the spatial feature map at the top spatial level is subjected to global averaging to obtain the image semantic features. The clinical physiological features and the image semantic features are concatenated to obtain fused features; The fused features are input into the classifier, and the classifier predicts the lesion impact of the fused features to obtain the degree of impact.

6. The method for predicting lung lesions according to claim 5, characterized in that, The imputation of missing items in the clinical physiological data includes: The missing items in the clinical physiological data are filled with preset placeholder values ​​to form initial imputation data; For each physiological indicator of the clinical physiological data, a binary mask vector is generated to indicate whether the data has been filled. The initial padding data is concatenated with the binary mask vector.

7. The method for predicting lung lesions according to claim 5, characterized in that, The step of predicting the lesion impact by using the classifier to obtain the degree of impact includes: The classifier performs nonlinear transformation and feature mapping on the fused features to output a predicted value of the impact of lesions in the chest CXR image on the patient. The predicted value is compared with a preset decision threshold. If the predicted value is greater than or equal to the decision threshold, a mild impact level is generated. If the predicted value is less than the decision threshold, a severe impact level is generated.

8. The method for predicting lung lesions according to claim 1, characterized in that, The multimodal model is obtained through the following steps: Obtain a first sample dataset and a second sample dataset; wherein, the first sample dataset includes multiple first chest CXR sample images with lesion areas marked, multiple second chest CXR sample images with no lesion areas marked, and the second sample dataset includes multiple third chest CXR sample images with lesion areas marked and the degree of influence marked, and clinical physiological data corresponding to the patients to which each third chest CXR sample image belongs. Initialize the original model containing the image encoder, the image segmentation module, and the multimodal classification module; The first sample dataset is input into the original model to iteratively train the image encoder and the image segmentation module to obtain an intermediate model. The second sample dataset is input into the intermediate model to iteratively train the image encoder, the image segmentation module, and the multimodal classification module to obtain the multimodal model.

9. The method for predicting lung lesions according to claim 8, characterized in that, The step of inputting the second sample dataset into the intermediate model to iteratively train the image encoder, the image segmentation module, and the multimodal classification module to obtain the multimodal model includes: A batch sample is obtained from the second sample dataset. The batch sample includes a third chest CXR sample image labeled with the lesion area and the degree of influence, and the clinical physiological data corresponding to the third chest CXR sample image. The third chest CXR sample image from the batch of samples is input into the image encoder, and the multiple sample feature maps extracted by the image encoder are input into the image segmentation module and the segmentation gradient loss value of the image segmentation module is calculated. The multiple sample feature maps extracted by the image encoder and the clinical physiological data are input into the multimodal classification module, and the classification gradient loss value of the multimodal classification module is calculated. The segmentation gradient loss value and the classification gradient loss value are weighted and summed to obtain the total gradient loss value of the intermediate model; Based on the total gradient loss value, the parameters of the image encoder, the image segmentation module, and the multimodal classification module are updated in reverse using the backpropagation algorithm; If the parameters of the image encoder, the image segmentation module, and the multimodal classification module all meet the training conditions, the training of the intermediate model is terminated, and the multimodal model is obtained.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a processor-executable program, which, when executed by a processor, is used to implement the lung lesion prediction method based on multimodal data as described in any one of claims 1 to 9.

Images