A risk assessment and prediction system based on viral pneumonia

Abstract

The present invention relates to the technical field of risk assessment and prediction of viral pneumonia, and in particular to a risk assessment and prediction system based on viral pneumonia. The system is used to perform the following steps: collect original data of viral pneumonia from target patients with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; perform lung region segmentation on the CT image data to obtain a lung mask image; obtain an unlabeled lung CT image; use the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map. The present invention avoids the interference of irrelevant information by focusing the system's attention on key lesion areas, and captures the dynamic development process of the patient's condition over time, thereby more accurately predicting the patient's future condition development trend, thereby improving the system's sensitivity and the accuracy of risk assessment and the system's predictive ability.

Description

Technical Field

[0001] The present invention relates to the technical field of viral pneumonia risk assessment and prediction, and in particular to a viral pneumonia risk assessment and prediction system. Background Art

[0002] The traditional risk assessment and prediction system based on viral pneumonia lacks focus on the lesion area, lacks model sensitivity, ignores time dynamic information, and has limited prediction ability. Lack of focus on the lesion area, the manifestation of insufficient model sensitivity: Early viral pneumonia risk assessment mainly relies on doctors' visual evaluation of imaging data, or uses some simple image processing techniques to extract features. These methods often directly input the original imaging data into the model, without effective positioning and feature extraction of the lesion area, and a large amount of normal tissue area information will interfere with the model's recognition of the lesion area. Due to the lack of focus on the lesion area, the traditional model is not sensitive enough to the lesion area, and it is easy to miss subtle lesion changes, resulting in a decrease in the accuracy of risk assessment. Ignoring time dynamic information and limited prediction ability: Traditional risk assessment models mostly only focus on the patient's static information, such as imaging data and clinical indicators at a single time point, ignoring the dynamic development process of the patient's condition over time, which leads to limitations in the construction process of traditional models. Summary of the invention

[0003] Based on this, it is necessary to provide a risk assessment and prediction system based on viral pneumonia to solve at least one of the above technical problems.

[0004] To achieve the above purpose, a viral pneumonia risk assessment and prediction system includes:

[0005] Control circuit;

[0006] A multimodal data characterization module is electrically connected to the control circuit, and the multimodal data characterization module includes: a CT scanner, a hospital information device and an image processing component. The multimodal data characterization module is used to collect original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; perform lung region segmentation on the CT image data to obtain a lung mask image; obtain an unlabeled lung CT image; use the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; perform lesion region annotation on the initial lesion region prediction map to obtain a lesion annotation image; perform image feature extraction on the lesion annotation image to obtain an image feature vector; construct a multimodal feature matrix according to the image feature vector to obtain a multimodal feature matrix;

[0007] A lesion attention learning module is electrically connected to the control circuit, and the lesion attention learning module includes: a deep learning server, an image processing working component and a data storage device, and the lesion attention learning module is used to generate an attention heat map according to the multimodal feature matrix and the lung mask image and normalize the heat map to obtain a normalized heat map; and construct a lesion enhancement feature matrix according to the normalized heat map, CT image data and image feature vector to obtain a lesion enhancement feature matrix;

[0008] A spatiotemporal feature fusion module is electrically connected to the control circuit, and the spatiotemporal feature fusion module includes: a Transformer encoder, and the spatiotemporal feature fusion module is used to perform feature vector splicing on the multimodal feature matrix and the lesion enhancement feature matrix to obtain a patient feature sequence; and perform spatiotemporal fusion feature multilayer processing according to the patient feature sequence and a preset Transformer encoder to obtain a spatiotemporal fusion feature vector;

[0009] A risk prediction modeling module is electrically connected to the control circuit, and the risk prediction modeling module is used to label the risk level of the target patient with viral pneumonia using predefined risk assessment standard data to obtain a risk level label; a viral pneumonia risk prediction model is constructed according to the risk level label and the spatiotemporal fusion feature vector to obtain a viral pneumonia risk prediction model;

[0010] An interpretability analysis module is electrically connected to the control circuit, and the interpretability analysis module includes: a display and a drawing library. The interpretability analysis module is used to use the viral pneumonia risk prediction model to predict the risk level of target patients with viral pneumonia to obtain risk level prediction data; identify key features based on the risk level prediction data to obtain key feature data; calculate feature contribution of the key feature data to obtain a feature contribution vector; perform risk factor analysis based on the feature contribution vector and the key feature data and generate a risk assessment report to obtain a risk assessment report.

[0011] The present invention can obtain the key information contained in the patient's multimodal data by collecting the patient's CT images and clinical data, and performing lung area segmentation, lesion area prediction and annotation, and image feature extraction, so as to lay a data foundation for the subsequent construction of a comprehensive and accurate risk assessment prediction model. By generating an attention heat map and using it to enhance the lesion area of ​​the CT image, the lesion area of ​​the model can be highlighted, and the model's sensitivity and extraction ability to lesion information can be improved, thereby constructing a more accurate lesion enhancement feature matrix. By performing time alignment, feature splicing, and position encoding on the multimodal feature matrix and the lesion enhancement feature matrix, and using the Transformer encoder to extract spatiotemporal fusion features, the patient's temporal information and multimodal features can be effectively integrated, the temporal dynamic characteristics of the patient's disease development can be captured, and a more comprehensive and accurate spatiotemporal fusion feature vector can be generated. Using predefined risk assessment standard data to label the patient's risk level, and combining the labeling information with the spatiotemporal fusion feature vector, the viral pneumonia risk prediction model can be trained, so that the model can learn the correlation between different features and risk levels, and finally obtain a prediction model with good performance. Using the trained model to predict the risk level of new patients, and combining feature importance analysis, local interpretation model and visualization technology to generate a rich and interpretative risk assessment report, can help doctors understand the patient's risk level, key risk factors and disease development trend more comprehensively and intuitively, provide effective support for clinical decision-making, and improve the accuracy and effectiveness of risk assessment prediction. Therefore, the present invention provides a risk assessment prediction system based on viral pneumonia. To solve the problem that the traditional risk assessment prediction system based on viral pneumonia lacks lesion area focus, the model is not sensitive enough, ignores time dynamic information, and has limited prediction ability. Through the steps of lung area segmentation, lesion area prediction and labeling, and lesion enhancement guided by attention heat map, the model's attention is effectively focused on the key lesion area, avoiding the interference of irrelevant information, and extracting more effective lesion features, thereby improving the sensitivity of the model and the accuracy of risk assessment. Through the steps of time alignment, time window division and Transformer encoder for spatiotemporal feature fusion, the dynamic development process of the patient's condition over time is effectively captured, so as to more accurately predict the patient's future disease development trend, such as the absorption of the lesion, whether complications will occur, etc., and improve the prediction ability of the model.

[0012] Preferably, the multimodal data characterization module is used to perform the following steps:

[0013] Step S11: collecting original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data;

[0014] Step S12: performing image data preprocessing on the CT image data to obtain image preprocessing data; performing lung region segmentation on the image preprocessing data to obtain a lung mask image;

[0015] Step S13: obtaining an unlabeled lung CT image; using the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; annotating the lesion region on the initial lesion region prediction map to obtain a lesion annotated image;

[0016] Step S14: extracting image features from the lesion annotated image to obtain an image feature vector;

[0017] Step S15: performing clinical data cleaning on the original clinical data to obtain patient clinical data; performing time series construction on the patient clinical data to obtain a clinical time series; performing clinical feature extraction on the clinical time series to obtain a clinical feature vector;

[0018] Step S16: performing feature vector combination on the clinical feature vector and the imaging feature vector to obtain a patient feature vector; constructing a multimodal feature matrix according to the patient feature vector to obtain a multimodal feature matrix.

[0019] The present invention obtains CT image data and original clinical data of target patients with viral pneumonia from the hospital information system, and provides the necessary data basis for the subsequent construction of a risk assessment prediction model. Preprocessing the CT image data can improve the image quality, reduce the influence of factors such as noise on the subsequent lung region segmentation, thereby improving the segmentation accuracy, and providing more accurate lung region information for subsequent lesion region prediction. Using unlabeled lung CT images for lesion region prediction and annotation can effectively utilize existing resources, reduce the workload of manual annotation, and improve the efficiency and accuracy of lesion region annotation, providing more reliable annotation data for subsequent image feature extraction. Image feature extraction of lesion annotated images can convert the original image data into a feature vector that can be processed by the model, extract the key information of the lesion, and provide image feature expression for the subsequent construction of a multimodal feature matrix. Cleaning and time series construction of the original clinical data can solve problems such as data missing and abnormalities, and capture the changing trend of the patient's clinical indicators over time, extract more comprehensive clinical information, and provide clinical feature expression for the subsequent construction of a multimodal feature matrix. Combining clinical feature vectors and imaging feature vectors into patient feature vectors and constructing a multimodal feature matrix can effectively integrate the patient's imaging information and clinical information, construct a more comprehensive patient feature expression, and provide richer information for subsequent model training and risk assessment.

[0020] Preferably, step S13 comprises the following steps:

[0021] Step S131: obtaining an unlabeled lung CT image; fine-tuning a pre-trained image segmentation model using the unlabeled lung CT image to obtain an image segmentation model;

[0022] Step S132: generating a lesion probability map for the lung mask image using the image segmentation model to obtain a lesion probability map;

[0023] Step S133: performing probability map threshold segmentation on the lesion probability map using a preset probability map threshold to obtain an initial lesion region prediction map;

[0024] Step S134: performing connected area analysis on the initial lesion region prediction map to obtain independent lesion region data; performing lesion area filtering on the initial lesion region prediction map according to the independent lesion region data to obtain a lesion region prediction map;

[0025] Step S135: Perform doctor annotation correction on the lesion area prediction image to obtain a lesion annotation image.

[0026] The present invention uses unlabeled lung CT images to fine-tune the pre-trained image segmentation model, so that the model can better adapt to the characteristics of the target data set and improve the accuracy and generalization ability of the model in lesion area prediction. The fine-tuned image segmentation model is used to predict the lung mask image and generate a lesion probability map, which can intuitively show the possibility that each pixel belongs to the lesion area, providing a basis for the precise positioning of the subsequent lesion area. The lesion probability map is segmented using a preset probability map threshold, and the probability information can be converted into a clear lesion area prediction result, and the scope of the lesion area is preliminarily determined, providing a basis for subsequent processing. The initial lesion area prediction map is subjected to Unicom area analysis and lesion area filtering, which can eliminate isolated noise points and areas that are too small or too large, and improve the accuracy and reliability of lesion area prediction. The doctor annotates and corrects the lesion area prediction map, and can correct the model prediction results in combination with professional knowledge, further improve the accuracy of the annotation results, and provide high-quality annotation data for subsequent image feature extraction.

[0027] Preferably, the lesion attention learning module is used to perform the following steps:

[0028] Step S21: generating an attention heat map according to the multimodal feature matrix and the lung mask image and normalizing the heat map to obtain a normalized heat map;

[0029] Step S22: using the normalized heat map to enhance the lesion area of ​​the CT image data to obtain an enhanced CT image;

[0030] Step S23: performing multi-scale feature extraction on the CT image data and the enhanced CT image to obtain a multi-scale feature map; performing feature pyramid pooling on the multi-scale feature map to obtain a pooled feature vector;

[0031] Step S24: performing lesion feature enhancement on the image feature vector and the pooled feature vector to obtain a lesion enhanced feature vector;

[0032] Step S25: constructing a lesion enhancement feature matrix according to the lesion enhancement feature vector to obtain a lesion enhancement feature matrix.

[0033] The present invention generates an attention heat map based on a multimodal feature matrix and a lung mask image, and normalizes it, so as to highlight the lesion area that the model focuses on when performing risk prediction, provide guidance for subsequent lesion area enhancement, avoid interference from irrelevant information, and improve the sensitivity of the model. By using the normalized heat map to enhance the lesion area of ​​CT image data, the features of the lesion area that the model focuses on can be amplified, making it more prominent in the subsequent feature extraction process, thereby improving the model's ability to extract lesion information. Multi-scale feature extraction and feature pyramid pooling are performed on the original CT image and the enhanced CT image, which can capture lesion information at different scales, and fuse multi-scale features to obtain a more comprehensive lesion feature expression, thereby improving the robustness of the model. The image feature vector is fused with the pooled feature vector to obtain a lesion enhancement feature vector, which can integrate lesion information from different sources, further enhance the expression ability of lesion features, and improve the prediction accuracy of the model. By constructing a lesion enhancement feature matrix based on the lesion enhancement feature vector, the lesion enhancement features at each time point can be organized in chronological order, providing richer and more detailed lesion information for subsequent spatiotemporal fusion feature extraction.

[0034] Preferably, step S21 includes the following steps:

[0035] Step S211: reconstructing image data according to the multimodal feature matrix to obtain reconstructed image data;

[0036] Step S212: inputting the reconstructed image data into the pre-trained 3D convolutional neural network model, performing model gradient calculation, and obtaining a gradient map;

[0037] Step S213: performing global average pooling on the gradient map to obtain a feature map weight value; performing feature map multiplication and summation calculation according to the feature map weight value and generating an attention heat map to obtain an attention heat map;

[0038] Step S214: using the lung mask image to perform heat map space constraints on the attention heat map to obtain a constrained heat map;

[0039] Step S215: normalize the constraint heat map to obtain a normalized heat map.

[0040] By reconstructing image data according to a multimodal feature matrix, the present invention can convert abstract feature information into intuitive image information, providing a basis for subsequent more refined analysis using a pre-trained model. The reconstructed image data is input into a pre-trained 3D convolutional neural network model, and the model gradient is calculated. The powerful feature extraction capability of the pre-trained model can be used to capture the degree of attention of the model to different regions when making predictions, providing a basis for generating an attention heat map. Global average pooling of the gradient map and weighted summation of the feature map can generate an attention heat map, which intuitively displays the areas that the model focuses on when making predictions, providing more accurate guidance for subsequent enhancement of the lesion area. Using the lung mask image to spatially constrain the attention heat map can exclude interference information outside the lung area, so that the attention heat map is more focused on the lung area, and the sensitivity of the model to the lesion area is improved. Normalization of the constrained heat map can scale the thermal value to a uniform range, which is convenient for subsequent fusion with the original image to achieve accurate enhancement of the lesion area.

[0041] Preferably, the spatiotemporal feature fusion module is used to perform the following steps:

[0042] Step S31: performing feature matrix time alignment on the multimodal feature matrix and the lesion enhancement feature matrix according to the acquisition time data in the original clinical data to obtain feature matrix time alignment data;

[0043] Step S32: dividing the feature matrix time-aligned data into feature matrix time windows to obtain time slice data;

[0044] Step S33: performing feature vector concatenation on the multimodal feature matrix and the lesion enhancement feature matrix according to the time slice data to obtain a patient feature sequence;

[0045] Step S34: adding position coding to the patient feature sequence to obtain a position coded patient feature sequence;

[0046] Step S35: Input the position-coded patient feature sequence into the preset Transformer encoder, perform multi-layer processing of spatiotemporal fusion features, and obtain a spatiotemporal fusion feature vector.

[0047] The present invention can ensure the consistency of image features and clinical features in time by time-aligning the multimodal feature matrix and the lesion enhancement feature matrix according to the acquisition time data in the original clinical data, and provide an accurate time basis for the subsequent analysis of the patient's condition change trend. The time-aligned feature matrix data is divided into time windows, and the continuous disease development process can be divided into multiple time periods, which is convenient for the model to learn the feature changes in each time period respectively, and capture the law of disease development more carefully. The multimodal feature matrix and the lesion enhancement feature matrix in each time slice are spliced ​​with feature vectors, and the image information and clinical information at the same time point can be fused together to form a more comprehensive patient feature expression, providing richer information for subsequent model training. Adding position coding information to the patient feature sequence can enable the model to learn the temporal relationship between different time points, such as the stage of disease development, so as to more accurately predict the patient's future risks. The position-coded patient feature sequence is input into a preset Transformer encoder, and the powerful spatiotemporal feature extraction capability of the Transformer model can be used to capture the temporal dynamic information of the patient's disease development and the complex interactive relationship between the features, and obtain a more expressive spatiotemporal fusion feature vector.

[0048] Preferably, step S35 includes the following steps:

[0049] Step S351: input the position-encoded patient feature sequence into a preset Transformer encoder, perform self-attention mechanism calculation, and obtain a time-step weighted feature vector; perform weighted average processing on the time-step weighted feature vector to obtain a global feature vector;

[0050] Step S352: Perform multi-head attention mechanism fusion according to the global feature vector to obtain a comprehensive global feature vector;

[0051] Step S353: performing feedforward network enhancement on the comprehensive global feature vector to obtain feedforward network enhanced data;

[0052] Step S354: Output the spatiotemporal fusion feature vector according to the feedforward network enhanced data to obtain the spatiotemporal fusion feature vector.

[0053] The present invention inputs the position-encoded patient feature sequence into the Transformer encoder and calculates the time-step weighted feature vector through the self-attention mechanism, so that the model can pay attention to the importance of each time step in the entire sequence, and obtain the global feature vector that integrates all time step information through weighted average, providing a more comprehensive information basis for subsequent feature enhancement. Using the multi-head attention mechanism to fuse the global feature vector, the relationship between features can be captured from multiple angles, richer and more comprehensive information can be extracted, and the feature expression ability can be further enhanced. The feedforward network enhancement of the comprehensive global feature vector can perform nonlinear transformation on the features, extract more abstract and more advanced semantic information, and further enhance the feature expression ability and prediction effect. The data enhanced by the feedforward network is output as the final spatiotemporal fusion feature vector, which integrates the images and clinical information of all time steps of the patient, and captures the temporal dynamic characteristics of the disease progression, providing a more powerful and accurate feature expression for subsequent risk prediction.

[0054] Preferably, the risk prediction modeling module is used to perform the following steps:

[0055] Step S41: Using predefined risk assessment standard data, labeling the risk level of the target patient with viral pneumonia to obtain a risk level label;

[0056] Step S42: performing risk label matching on the risk level label and the spatiotemporal fusion feature vector to obtain a label feature data set;

[0057] Step S43: using the label feature data set to perform model training and hyperparameter adjustment to obtain an initial risk prediction model;

[0058] Step S44: Perform model evaluation on the initial risk prediction model to obtain a viral pneumonia risk prediction model.

[0059] The present invention uses predefined risk assessment standard data to label the risk level of target patients with viral pneumonia, which can provide a clear learning goal for model training, so that the model can learn the correlation between different features and risk levels, and lay the foundation for building an accurate and reliable risk prediction model. The risk level label is matched with the spatiotemporal fusion feature vector to construct a label feature data set, which can associate the patient's feature information with its corresponding risk level, and provide a data sample for learning for model training. Using the label feature data set for model training and hyperparameter adjustment, the model can learn effective feature combinations and prediction rules from the data, and optimize the performance of the model through hyperparameter adjustment to obtain a preliminary risk prediction model. Model evaluation of the initial risk prediction model can objectively evaluate the prediction performance of the model, such as accuracy, precision, recall and other indicators, and further optimize and improve the model according to the evaluation results, and finally obtain a viral pneumonia risk prediction model with good performance.

[0060] Preferably, the interpretability analysis module is used to perform the following steps:

[0061] Step S51: using the viral pneumonia risk prediction model to predict the risk level of the target viral pneumonia patient to obtain risk level prediction data;

[0062] Step S52: sorting the feature importance according to the risk level prediction data and the viral pneumonia risk prediction model to obtain a feature importance list; identifying key features according to the feature importance list to obtain key feature data;

[0063] Step S53: constructing a local explanation model for the target patient with viral pneumonia to obtain a local explainable model; calculating feature contribution of key feature data according to the local explainable model to obtain a feature contribution vector;

[0064] Step S54: Perform risk factor analysis based on the feature contribution vector and key feature data and generate a risk assessment report to obtain a risk assessment report.

[0065] The present invention uses a trained viral pneumonia risk prediction model to predict the risk level of the target patient, which can quickly identify high-risk patients and provide early warning information to doctors so that intervention measures can be taken in a timely manner. Feature importance sorting and key feature identification based on risk level prediction data and models can help doctors understand which factors have a greater impact on the patient's risk level. Constructing a local interpretable model for the target patient and calculating the contribution of key features can more intuitively explain the reasons for the model's prediction results and improve the interpretability and transparency of the model. Finally, risk factor analysis is performed based on the feature contribution vector and key feature data, and a risk assessment report is generated to obtain a risk assessment report.

[0066] Preferably, step S54 includes the following steps:

[0067] Step S541: generating risk visualization interpretation of the feature contribution vector and key feature data to obtain risk visualization interpretation data;

[0068] Step S542: performing risk factor analysis based on the feature contribution vector and key feature data to obtain risk factor analysis data;

[0069] Step S543: Generate a risk assessment report based on the risk level prediction data, risk visualization interpretation data, and risk factor analysis data to obtain a risk assessment report.

[0070] The present invention can more clearly present the basis of the model prediction results by visually displaying the feature contribution and key feature data, helping doctors better understand the model prediction results and explain the condition to patients. Risk factor analysis based on feature contribution and key feature data can summarize the key factors that lead to high risk for patients. Integrate risk level prediction data, visual explanation and risk factor analysis to generate a risk assessment report.

[0071] The present invention can obtain the key information contained in the patient's multimodal data by collecting the patient's CT images and clinical data, and performing lung area segmentation, lesion area prediction and annotation, and image feature extraction, so as to lay a data foundation for the subsequent construction of a comprehensive and accurate risk assessment prediction model. By generating an attention heat map and using it to enhance the lesion area of ​​the CT image, the lesion area of ​​the model can be highlighted, and the model's sensitivity and extraction ability to lesion information can be improved, thereby constructing a more accurate lesion enhancement feature matrix. By performing time alignment, feature splicing, and position encoding on the multimodal feature matrix and the lesion enhancement feature matrix, and using the Transformer encoder to extract spatiotemporal fusion features, the patient's temporal information and multimodal features can be effectively integrated, the temporal dynamic characteristics of the patient's disease development can be captured, and a more comprehensive and accurate spatiotemporal fusion feature vector can be generated. Using predefined risk assessment standard data to label the patient's risk level, and combining the labeling information with the spatiotemporal fusion feature vector, the viral pneumonia risk prediction model can be trained, so that the model can learn the correlation between different features and risk levels, and finally obtain a prediction model with good performance. Using the trained model to predict the risk level of new patients, and combining feature importance analysis, local interpretation model and visualization technology to generate a rich and interpretative risk assessment report, can help doctors understand the patient's risk level, key risk factors and disease development trend more comprehensively and intuitively, provide effective support for clinical decision-making, and improve the accuracy and effectiveness of risk assessment prediction. Therefore, the present invention provides a risk assessment prediction system based on viral pneumonia. To solve the problem that the traditional risk assessment prediction system based on viral pneumonia lacks lesion area focus, the model is not sensitive enough, ignores time dynamic information, and has limited prediction ability. Through the steps of lung area segmentation, lesion area prediction and labeling, and lesion enhancement guided by attention heat map, the model's attention is effectively focused on the key lesion area, avoiding the interference of irrelevant information, and extracting more effective lesion features, thereby improving the sensitivity of the model and the accuracy of risk assessment. Through the steps of time alignment, time window division and Transformer encoder for spatiotemporal feature fusion, the dynamic development process of the patient's condition over time is effectively captured, so as to more accurately predict the patient's future disease development trend, such as the absorption of the lesion, whether complications will occur, etc., and improve the prediction ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of a module block diagram of a viral pneumonia risk assessment and prediction system;

[0073] Figure 2 Detailed implementation flow chart of step S2;

[0074] Figure 3 Detailed implementation flow chart of step S3.

[0075] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0076] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0077] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0078] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0079] To achieve this, please refer to Figures 1 to 3 , a viral pneumonia risk assessment and prediction system, comprising:

[0080] Control circuit;

[0081] A multimodal data characterization module is electrically connected to the control circuit, and the multimodal data characterization module includes: a CT scanner, a hospital information device and an image processing component. The multimodal data characterization module is used to collect original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; perform lung region segmentation on the CT image data to obtain a lung mask image; obtain an unlabeled lung CT image; use the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; perform lesion region annotation on the initial lesion region prediction map to obtain a lesion annotation image; perform image feature extraction on the lesion annotation image to obtain an image feature vector; construct a multimodal feature matrix according to the image feature vector to obtain a multimodal feature matrix;

[0082] A lesion attention learning module is electrically connected to the control circuit, and the lesion attention learning module includes: a deep learning server, an image processing working component and a data storage device, and the lesion attention learning module is used to generate an attention heat map according to the multimodal feature matrix and the lung mask image and normalize the heat map to obtain a normalized heat map; and construct a lesion enhancement feature matrix according to the normalized heat map, CT image data and image feature vector to obtain a lesion enhancement feature matrix;

[0083] A spatiotemporal feature fusion module is electrically connected to the control circuit, and the spatiotemporal feature fusion module includes: a Transformer encoder, and the spatiotemporal feature fusion module is used to perform feature vector splicing on the multimodal feature matrix and the lesion enhancement feature matrix to obtain a patient feature sequence; and perform spatiotemporal fusion feature multilayer processing according to the patient feature sequence and a preset Transformer encoder to obtain a spatiotemporal fusion feature vector;

[0084] A risk prediction modeling module is electrically connected to the control circuit, and the risk prediction modeling module is used to label the risk level of the target patient with viral pneumonia using predefined risk assessment standard data to obtain a risk level label; a viral pneumonia risk prediction model is constructed according to the risk level label and the spatiotemporal fusion feature vector to obtain a viral pneumonia risk prediction model;

[0085] An interpretability analysis module is electrically connected to the control circuit, and the interpretability analysis module includes: a display and a drawing library. The interpretability analysis module is used to use the viral pneumonia risk prediction model to predict the risk level of target patients with viral pneumonia to obtain risk level prediction data; identify key features based on the risk level prediction data to obtain key feature data; calculate feature contribution of the key feature data to obtain a feature contribution vector; perform risk factor analysis based on the feature contribution vector and the key feature data and generate a risk assessment report to obtain a risk assessment report.

[0086] In the embodiment of the present invention, reference Figure 1 The above is a schematic diagram of the steps of the viral pneumonia risk assessment and prediction system of the present invention. In this example, the viral pneumonia risk assessment and prediction system includes:

[0087] Control circuit;

[0088] A multimodal data characterization module is electrically connected to the control circuit, and the multimodal data characterization module includes: a CT scanner, a hospital information device and an image processing component. The multimodal data characterization module is used to collect original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; perform lung region segmentation on the CT image data to obtain a lung mask image; obtain an unlabeled lung CT image; use the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; perform lesion region annotation on the initial lesion region prediction map to obtain a lesion annotation image; perform image feature extraction on the lesion annotation image to obtain an image feature vector; construct a multimodal feature matrix according to the image feature vector to obtain a multimodal feature matrix;

[0089] In an embodiment of the present invention, CT image data and original clinical data of the patient are obtained. The U-Net model is used to segment the lung region of the CT image to obtain a lung mask image. A lesion region prediction model is obtained by training the public data set and the Mask R-CNN model, and the lung mask image is predicted. After a series of processing, an accurate lesion annotation image is obtained. Finally, the ResNet model is used to extract features from the lesion annotation image to obtain an image feature vector, and a multimodal feature matrix is ​​constructed.

[0090] A lesion attention learning module is electrically connected to the control circuit, and the lesion attention learning module includes: a deep learning server, an image processing working component and a data storage device, and the lesion attention learning module is used to generate an attention heat map according to the multimodal feature matrix and the lung mask image and normalize the heat map to obtain a normalized heat map; and construct a lesion enhancement feature matrix according to the normalized heat map, CT image data and image feature vector to obtain a lesion enhancement feature matrix;

[0091] In an embodiment of the present invention, the original CT image data is reconstructed using a multimodal feature matrix, and the pre-trained 3D convolutional neural network model is input to calculate the gradient map, and a normalized thermal map is obtained after global average pooling and spatial constraints. The thermal map is multiplied with the original CT image to obtain an enhanced CT image. The multi-scale features of the original and enhanced CT images are extracted respectively, and feature pyramid pooling is performed to obtain a pooled feature vector. Finally, the image feature vector is fused with the pooled feature vector to obtain a lesion enhancement feature vector, and a lesion enhancement feature matrix is ​​constructed.

[0092] A spatiotemporal feature fusion module is electrically connected to the control circuit, and the spatiotemporal feature fusion module includes: a Transformer encoder, and the spatiotemporal feature fusion module is used to perform feature vector splicing on the multimodal feature matrix and the lesion enhancement feature matrix to obtain a patient feature sequence; and perform spatiotemporal fusion feature multilayer processing according to the patient feature sequence and a preset Transformer encoder to obtain a spatiotemporal fusion feature vector;

[0093] In the embodiment of the present invention, the multimodal feature matrix and the lesion enhancement feature matrix are time-aligned according to the clinical data acquisition time, and the time slice data is obtained by dividing the time window. The multimodal features and lesion enhancement features of each time slice are spliced ​​into a patient feature sequence, and the position coding information is added. Finally, the sequence is input into the Transformer encoder for multi-layer processing of spatiotemporal fusion features to obtain a fused feature vector.

[0094] A risk prediction modeling module is electrically connected to the control circuit, and the risk prediction modeling module is used to label the risk level of the target patient with viral pneumonia using predefined risk assessment standard data to obtain a risk level label; a viral pneumonia risk prediction model is constructed according to the risk level label and the spatiotemporal fusion feature vector to obtain a viral pneumonia risk prediction model;

[0095] In the embodiment of the present invention, the doctor labels the patient's risk level according to the predefined risk assessment criteria. The risk level label is matched with the fusion feature vector to construct a data set, and a suitable machine learning model is selected for training and hyperparameter adjustment, and finally a viral pneumonia risk prediction model is obtained.

[0096] An interpretability analysis module is electrically connected to the control circuit, and the interpretability analysis module includes: a display and a drawing library. The interpretability analysis module is used to use the viral pneumonia risk prediction model to predict the risk level of target patients with viral pneumonia to obtain risk level prediction data; identify key features based on the risk level prediction data to obtain key feature data; calculate feature contribution of the key feature data to obtain a feature contribution vector; perform risk factor analysis based on the feature contribution vector and the key feature data and generate a risk assessment report to obtain a risk assessment report.

[0097] In the embodiment of the present invention, the trained model is used to predict the risk level of new patients. Key features are identified through feature importance analysis, and the contribution of key features is calculated using a local interpretable model. Finally, the prediction results, key feature contributions, and visual explanations are integrated to generate a risk assessment report.

[0098] Preferably, the multimodal data characterization module is used to perform the following steps:

[0099] Step S11: collecting original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data;

[0100] Step S12: performing image data preprocessing on the CT image data to obtain image preprocessing data; performing lung region segmentation on the image preprocessing data to obtain a lung mask image;

[0101] Step S13: obtaining an unlabeled lung CT image; using the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; annotating the lesion region on the initial lesion region prediction map to obtain a lesion annotated image;

[0102] Step S14: extracting image features from the lesion annotated image to obtain an image feature vector;

[0103] Step S15: performing clinical data cleaning on the original clinical data to obtain patient clinical data; performing time series construction on the patient clinical data to obtain a clinical time series; performing clinical feature extraction on the clinical time series to obtain a clinical feature vector;

[0104] Step S16: performing feature vector combination on the clinical feature vector and the imaging feature vector to obtain a patient feature vector; constructing a multimodal feature matrix according to the patient feature vector to obtain a multimodal feature matrix.

[0105] In an embodiment of the present invention, the electronic medical records of the target patients with viral pneumonia are retrieved from the hospital's information system to obtain the patient's CT image data (DICOM format) and original clinical data (e.g., age, gender, symptoms, laboratory test results, etc.). The acquired CT image data is preprocessed, including: image grayscale normalization, image size adjustment, noise removal, and other operations to obtain image preprocessing data. Then, the image preprocessing data is segmented into lung regions using a pretrained U-Net model to obtain a lung mask image, wherein the pixel value in the mask image represents the probability that the pixel belongs to the lung region. Unlabeled lung CT images are obtained from a public medical image database. These unlabeled images are used to fine-tune the pretrained image segmentation model (e.g., Mask R-CNN) to obtain an image segmentation model that can predict the lesion area. The model is used to generate a lesion probability map for the lung mask image, and the probability map is segmented using a preset probability threshold to obtain an initial lesion area prediction map. Finally, the predicted map is subjected to connected area analysis and lesion area filtering, and is annotated and corrected by a professional radiologist to obtain an accurate lesion annotated image. Use a pre-trained convolutional neural network model (e.g., ResNet) to extract features from the lesion annotated image to obtain an image feature vector, which contains information such as the shape, size, and texture of the lesion. Perform data cleaning on the original clinical data, including missing value filling, outlier processing, and data format unification, to obtain structured patient clinical data. Construct a time series based on the chronological order of the patient's clinical data collection, and use a time series analysis method (e.g., LSTM) to extract features from the clinical time series to obtain a clinical feature vector, which reflects the changing trend of the patient's clinical indicators over time. Combine the image feature vector with the clinical feature vector to obtain a patient feature vector containing both the patient's image information and clinical information. Finally, construct a multimodal feature matrix based on the patient feature vector, where the rows of the matrix represent different features and the columns represent different time points.

[0106] Preferably, step S13 comprises the following steps:

[0107] Step S131: obtaining an unlabeled lung CT image; fine-tuning a pre-trained image segmentation model using the unlabeled lung CT image to obtain an image segmentation model;

[0108] Step S132: generating a lesion probability map for the lung mask image using the image segmentation model to obtain a lesion probability map;

[0109] Step S133: performing probability map threshold segmentation on the lesion probability map using a preset probability map threshold to obtain an initial lesion region prediction map;

[0110] Step S134: performing connected area analysis on the initial lesion region prediction map to obtain independent lesion region data; performing lesion area filtering on the initial lesion region prediction map according to the independent lesion region data to obtain a lesion region prediction map;

[0111] Step S135: Perform doctor annotation correction on the lesion area prediction image to obtain a lesion annotation image.

[0112] In an embodiment of the present invention, a certain number of unlabeled lung CT images are obtained from a public medical image database. These images are used as training data to fine-tune the pre-trained image segmentation model. During the fine-tuning process, the main structure of the pre-trained model is kept unchanged, and only the parameters of some network layers are adjusted to make it more suitable for the characteristics of the target data set. The lung mask image is predicted using the fine-tuned image segmentation model in step S131 to obtain the probability value of each pixel belonging to the lesion area, and form a lesion probability map. A probability threshold (for example: 0.5) is set in advance, and the pixels with probability values ​​greater than or equal to the threshold in the lesion probability map are marked as lesion areas, and the pixels less than the threshold are marked as background areas, thereby obtaining an initial lesion area prediction map. The initial lesion area prediction map is subjected to a connected area analysis, and the interconnected lesion pixels are aggregated into an independent area to obtain the size, shape and other information of each lesion area. According to the pre-set lesion area threshold, the lesion areas with too small or too large areas are filtered out to further improve the accuracy of the prediction results and obtain the final lesion area prediction map. The lesion area prediction map is handed over to professional radiologists for manual annotation and correction. The doctors can judge the accuracy of the prediction results based on their experience and correct the erroneous prediction results to finally obtain an accurate lesion annotated image.

[0113] In order to verify the performance of the lesion area prediction model, this embodiment uses a test set containing 100 CT images of COVID-19 patients for evaluation. Each image in the test set has been manually annotated by a radiologist as the gold standard. The lesion area prediction results of this embodiment are compared with the gold standard, and the following evaluation indicators are calculated:

[0114] The pixel-level accuracy is 0.92, indicating that the predicted lesion area has a high degree of overlap with the actual lesion area.

[0115] The average IoU at the region level is 0.85, indicating that the predicted lesion region boundary is consistent with the actual boundary.

[0116] The false positive rate was 0.05 and the false negative rate was 0.03, indicating that the model rarely predicted normal areas as lesions or missed lesion areas.

[0117] Preferably, the lesion attention learning module is used to perform the following steps:

[0118] Step S21: generating an attention heat map according to the multimodal feature matrix and the lung mask image and normalizing the heat map to obtain a normalized heat map;

[0119] Step S22: using the normalized heat map to enhance the lesion area of ​​the CT image data to obtain an enhanced CT image;

[0120] Step S23: performing multi-scale feature extraction on the CT image data and the enhanced CT image to obtain a multi-scale feature map; performing feature pyramid pooling on the multi-scale feature map to obtain a pooled feature vector;

[0121] Step S24: performing lesion feature enhancement on the image feature vector and the pooled feature vector to obtain a lesion enhanced feature vector;

[0122] Step S25: constructing a lesion enhancement feature matrix according to the lesion enhancement feature vector to obtain a lesion enhancement feature matrix.

[0123] As an example of the present invention, refer to Figure 2 As shown, the lesion attention learning module described in this example includes:

[0124] Step S21: generating an attention heat map according to the multimodal feature matrix and the lung mask image and normalizing the heat map to obtain a normalized heat map;

[0125] In an embodiment of the present invention, the original CT image data is reconstructed using the image features in the multimodal feature matrix, and the reconstructed image data is input into a pre-trained 3D convolutional neural network model for forward propagation to calculate the gradient map of the last convolutional layer of the model. The gradient map is globally averaged pooled to obtain the feature map weight value, and the weight is multiplied and summed with the feature map to obtain the attention heat map. The attention heat map is spatially constrained using a lung mask image to retain only the thermal value of the lung area, and finally normalized to obtain a normalized heat map.

[0126] Step S22: using the normalized heat map to enhance the lesion area of ​​the CT image data to obtain an enhanced CT image;

[0127] In the embodiment of the present invention, the normalized heat map is multiplied with the original CT image data at the pixel level to enhance the lesion area and obtain an enhanced CT image.

[0128] Step S23: performing multi-scale feature extraction on the CT image data and the enhanced CT image to obtain a multi-scale feature map; performing feature pyramid pooling on the multi-scale feature map to obtain a pooled feature vector;

[0129] In the embodiment of the present invention, convolution operations are performed on the original CT image data and the enhanced CT image using convolution kernels of different sizes to extract multi-scale features and obtain a multi-scale feature map. Then, a feature pyramid pooling operation is performed on the multi-scale feature map to aggregate features of different scales into a feature vector of a fixed length to obtain a pooled feature vector.

[0130] Step S24: performing lesion feature enhancement on the image feature vector and the pooled feature vector to obtain a lesion enhanced feature vector;

[0131] In the embodiment of the present invention, the image feature vector is concatenated with the pooled feature vector, and a fully connected neural network is used to perform feature fusion to obtain a fused lesion enhancement feature vector.

[0132] Step S25: constructing a lesion enhancement feature matrix according to the lesion enhancement feature vector to obtain a lesion enhancement feature matrix;

[0133] In the embodiment of the present invention, the lesion enhancement feature vectors at each time point are arranged in chronological order to construct a lesion enhancement feature matrix, wherein the rows of the matrix represent different features and the columns represent different time points.

[0134] Preferably, step S21 includes the following steps:

[0135] Step S211: reconstructing image data according to the multimodal feature matrix to obtain reconstructed image data;

[0136] Step S212: inputting the reconstructed image data into the pre-trained 3D convolutional neural network model, performing model gradient calculation, and obtaining a gradient map;

[0137] Step S213: performing global average pooling on the gradient map to obtain a feature map weight value; performing feature map multiplication and summation calculation according to the feature map weight value and generating an attention heat map to obtain an attention heat map;

[0138] Step S214: using the lung mask image to perform heat map space constraints on the attention heat map to obtain a constrained heat map;

[0139] Step S215: normalize the constraint heat map to obtain a normalized heat map.

[0140] In an embodiment of the present invention, feature vectors related to the image are extracted from the multimodal feature matrix, and image data is reconstructed using a pre-trained variational autoencoder (VAE) or a deep learning model such as a generative adversarial network (GAN), and the feature vectors are mapped back to the original CT image data space to obtain reconstructed image data. The image data is input into a pre-trained 3D convolutional neural network model for forward propagation calculation. Then, a specific layer related to the classification task in the model is selected, and the gradient of the output of the layer relative to the input is calculated to obtain a gradient map, which reflects the degree of attention of the model to different spatial positions. The gradient map is subjected to a global average pooling operation, that is, all pixel values ​​on each feature channel are averaged to obtain a weight value representing the importance of the channel, and finally a feature map weight value is obtained. The feature map weight value is multiplied element by element with the corresponding feature map, and then the results of all channels are added to obtain an attention heat map. The heat map highlights the areas that the model focuses on when making predictions. The lung mask image is used to spatially constrain the attention heat map, and the heat value of the area outside the mask image is set to 0, and the heat value of the lung area is retained to obtain a constrained heat map. The constrained heat map is normalized and the heat value is scaled to the [0, 1] interval to obtain a normalized heat map.

[0141] Preferably, the spatiotemporal feature fusion module is used to perform the following steps:

[0142] Step S31: performing feature matrix time alignment on the multimodal feature matrix and the lesion enhancement feature matrix according to the acquisition time data in the original clinical data to obtain feature matrix time alignment data;

[0143] Step S32: dividing the feature matrix time-aligned data into feature matrix time windows to obtain time slice data;

[0144] Step S33: performing feature vector concatenation on the multimodal feature matrix and the lesion enhancement feature matrix according to the time slice data to obtain a patient feature sequence;

[0145] Step S34: adding position coding to the patient feature sequence to obtain a position coded patient feature sequence;

[0146] Step S35: Input the position-coded patient feature sequence into the preset Transformer encoder, perform multi-layer processing of spatiotemporal fusion features, and obtain a spatiotemporal fusion feature vector.

[0147] As an example of the present invention, refer to Figure 3 As shown, the spatiotemporal feature fusion module described in this example includes:

[0148] Step S31: performing feature matrix time alignment on the multimodal feature matrix and the lesion enhancement feature matrix according to the acquisition time data in the original clinical data to obtain feature matrix time alignment data;

[0149] In an embodiment of the present invention, the multimodal feature matrix and the lesion enhancement feature matrix are time-aligned according to the acquisition time of each indicator recorded in the original clinical data to ensure that the image features and clinical features acquired at the same time point are in the same column in the matrix, thereby obtaining the time-aligned feature matrix data.

[0150] Step S32: dividing the feature matrix time-aligned data into feature matrix time windows to obtain time slice data;

[0151] In an embodiment of the present invention, the time-aligned feature matrix data is divided according to a preset time window size (e.g., 7 days) and a sliding step size (e.g., 1 day), and the feature data in each time window is used as a time slice to obtain multiple time slice data.

[0152] Step S33: performing feature vector concatenation on the multimodal feature matrix and the lesion enhancement feature matrix according to the time slice data to obtain a patient feature sequence;

[0153] In the embodiment of the present invention, for each time slice, the feature vectors of the corresponding time point in the multimodal feature matrix and the lesion enhancement feature matrix are concatenated to obtain a patient feature sequence containing all feature information at the time point.

[0154] Step S34: adding position coding to the patient feature sequence to obtain a position coded patient feature sequence;

[0155] In an embodiment of the present invention, in order to enable the model to learn the temporal relationship between different time points, position coding information is added to each feature vector in the patient feature sequence, for example, a sine function is used to generate a position coding vector to obtain a position coded patient feature sequence.

[0156] Step S35: inputting the position-coded patient feature sequence into a preset Transformer encoder, performing multi-layer processing of spatiotemporal fusion features, and obtaining a spatiotemporal fusion feature vector;

[0157] In an embodiment of the present invention, the position-encoded patient feature sequence is input into a pre-built Transformer encoder, and the Transformer encoder performs multi-layer processing of spatiotemporal fusion features on the feature sequence through a self-attention mechanism and a multi-layer network structure, extracts the temporal dynamic information of the patient's disease progression, and finally obtains a fused feature vector containing rich spatiotemporal information.

[0158] Preferably, step S35 includes the following steps:

[0159] Step S351: input the position-encoded patient feature sequence into a preset Transformer encoder, perform self-attention mechanism calculation, and obtain a time-step weighted feature vector; perform weighted average processing on the time-step weighted feature vector to obtain a global feature vector;

[0160] Step S352: Perform multi-head attention mechanism fusion according to the global feature vector to obtain a comprehensive global feature vector;

[0161] Step S353: performing feedforward network enhancement on the comprehensive global feature vector to obtain feedforward network enhanced data;

[0162] Step S354: Output the spatiotemporal fusion feature vector according to the feedforward network enhanced data to obtain the spatiotemporal fusion feature vector.

[0163] In an embodiment of the present invention, after the position-encoded patient feature sequence is input into a preset Transformer encoder, a self-attention mechanism calculation is first performed. The self-attention mechanism calculates the correlation between any two time steps in the feature sequence to obtain the importance of each time step in the entire sequence, and generates a time step weighted feature vector. Then, all time step weighted feature vectors are weighted averaged to obtain a global feature vector, which integrates the information of each time step. The global feature vector is further integrated using a multi-head attention mechanism. The multi-head attention mechanism can focus on different aspects of the global feature vector from different angles, thereby extracting more comprehensive information. The outputs of multiple attention heads are spliced ​​together to obtain a comprehensive global feature vector. The comprehensive global feature vector is enhanced using a feedforward neural network, and the enhanced feature vector contains more abstract and advanced semantic information. Finally, the data enhanced by the feedforward network is output as the final spatiotemporal fusion feature vector. This vector integrates the imaging and clinical information of all time steps of the patient and captures the temporal dynamic characteristics of the disease progression.

[0164] In order to verify the effectiveness of spatiotemporal fusion features, this example uses a dataset of 500 COVID-19 patients for experiments. Each patient in the dataset has imaging data and clinical indicator data at 3-5 time points. The patients were randomly divided into a training set (400 cases) and a test set (100 cases), and the following three methods were used for risk prediction modeling:

[0165] Method 1: Using only multimodal features at a single time point;

[0166] Method 2: Concatenate multimodal features at each time point as model input;

[0167] Method 3: Using the spatiotemporal fusion features of this embodiment;

[0168] The prediction performance of the three methods is evaluated on the test set, and the results are as follows:

[0169] Method 1: 0.81 (AUC), 0.76 (accuracy), 0.72 (precision), 0.79 (recall);

[0170] Method 2: 0.85 (AUC), 0.80 (accuracy), 0.78 (precision), 0.82 (recall);

[0171] Method 2: 0.92 (AUC), 0.88 (accuracy), 0.85 (precision), 0.90 (recall);

[0172] The results show that the use of spatiotemporal fusion features can significantly improve the performance of risk prediction, outperforming methods using single time point features and simple splicing features.

[0173] Preferably, the risk prediction modeling module is used to perform the following steps:

[0174] Step S41: using predefined risk assessment standard data to label the risk level of the target viral pneumonia patient to obtain a risk level label;

[0175] Step S42: performing risk label matching on the risk level label and the spatiotemporal fusion feature vector to obtain a label feature data set;

[0176] Step S43: using the label feature data set to perform model training and hyperparameter adjustment to obtain an initial risk prediction model;

[0177] Step S44: Perform model evaluation on the initial risk prediction model to obtain a viral pneumonia risk prediction model.

[0178] In an embodiment of the present invention, according to the predefined risk assessment standard data (for example, CURB-65 score, pneumonia severity score, etc.), a professional doctor labels the risk level of each viral pneumonia target patient, for example, the risk level is divided into three levels: low, medium, and high, and the corresponding level labels (for example, 0, 1, 2) are recorded to obtain the risk level label of each patient. The risk level label is matched with the spatiotemporal fusion feature vector to construct a label feature data set, in which each sample contains a spatiotemporal fusion feature vector of a patient and a corresponding risk level label. The risk prediction model is trained using the label feature data set. A suitable machine learning model can be selected, and the hyperparameters of the model can be adjusted using methods such as grid search and random search to obtain an initial risk prediction model. The initial risk prediction model is evaluated using the test set data, and the accuracy, precision, recall rate, F1 value and other indicators of the model are calculated, and the model is adjusted and optimized according to the evaluation results, and finally a viral pneumonia risk prediction model with good performance is obtained. The viral pneumonia risk prediction model is used to predict the risk of viral pneumonia, providing reference data for doctors to predict the risk of viral pneumonia.

[0179] In order to evaluate the performance of the viral pneumonia risk prediction system, this example uses an independent test set of 200 COVID-19 patients for verification. Each patient in the test set has complete clinical and imaging data, and the doctor gives the final risk level label (low, medium, high). Use the trained risk prediction model to predict the test set and obtain the risk level prediction results for each patient. Compare the prediction results with the doctor's annotations and calculate the following evaluation indicators:

[0180] Accuracy: 0.90, indicating that the risk level predicted by the model is highly consistent with the doctor's judgment.

[0181] The weighted average precision is 0.89, which indicates that the model has high prediction accuracy at all risk levels.

[0182] The weighted average recall rate is 0.90, which indicates that the model has high sensitivity at each risk level.

[0183] The weighted average F1 value is 0.89, which takes into account both precision and recall, indicating that the overall performance of the model is good at all risk levels.

[0184] Cohen's Kappa coefficient: 0.85, indicating that the model prediction results are highly consistent with the doctor's judgment results, exceeding the possibility of random consistency.

[0185] The above experimental results show that the viral pneumonia risk prediction model constructed in this embodiment has achieved excellent prediction performance on an independent test set and can more accurately assess the risk level of COVID-19 patients. The viral pneumonia risk prediction data output by the model provides data reference for doctors to predict viral pneumonia risks.

[0186] Preferably, the interpretability analysis module is used to perform the following steps:

[0187] Step S51: using the viral pneumonia risk prediction model to predict the risk level of the target viral pneumonia patient to obtain risk level prediction data;

[0188] Step S52: sorting the feature importance according to the risk level prediction data and the viral pneumonia risk prediction model to obtain a feature importance list; identifying key features according to the feature importance list to obtain key feature data;

[0189] Step S53: constructing a local explanation model for the target patient with viral pneumonia to obtain a local explainable model; calculating feature contribution of key feature data according to the local explainable model to obtain a feature contribution vector;

[0190] Step S54: Perform risk factor analysis based on the feature contribution vector and key feature data and generate a risk assessment report to obtain a risk assessment report.

[0191] In an embodiment of the present invention, a trained viral pneumonia risk prediction model is used to predict the risk level of a new viral pneumonia target patient to obtain risk level prediction data for each patient, such as low risk, medium risk, high risk, etc. According to the risk level prediction data and the viral pneumonia risk prediction model, a feature importance analysis method is used to rank the importance of all features in the model to obtain a feature importance list. According to a preset threshold or selecting the top-ranked features as key features, and extracting the patient's data on these key features to obtain key feature data. For each target patient with viral pneumonia, a local interpretable model is used to explain its prediction results. The local interpretable model can quantify the contribution of each key feature to the final prediction result, and obtain the feature contribution vector of each key feature. Risk factor analysis is performed based on the feature contribution vector and key feature data, and a risk assessment report is generated to obtain a risk assessment report.

[0192] Preferably, step S54 includes the following steps:

[0193] Step S541: generating risk visualization interpretation of the feature contribution vector and key feature data to obtain risk visualization interpretation data;

[0194] Step S542: performing risk factor analysis based on the feature contribution vector and key feature data to obtain risk factor analysis data;

[0195] Step S543: Generate a risk assessment report based on the risk level prediction data, risk visualization interpretation data, and risk factor analysis data to obtain a risk assessment report.

[0196] In an embodiment of the present invention, based on the feature contribution vector and key feature data, a visualization method such as a chart or image is used to display the impact of each key feature on the prediction of the patient's risk level, such as using a bar chart to display the contribution of each feature, using a heat map to display the interaction between features, etc., to generate risk visualization explanatory data. Based on the feature contribution vector and key feature data, the key factors leading to the patient's risk level prediction results are analyzed, such as which imaging features or clinical indicators play a major role in predicting the patient's risk level, and the analysis results are recorded in the form of a text description to form risk factor analysis data. The risk level prediction data, risk visualization explanatory data, and risk factor analysis data are integrated to automatically generate a risk assessment report, which includes the patient's risk level prediction results, key risk factor analysis, risk visualization explanation, etc., to provide decision support for doctors.

[0197] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0198] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A risk assessment and prediction system based on viral pneumonia, characterized in that: include: Control circuit; A multimodal data characterization module is electrically connected to the control circuit, and the multimodal data characterization module includes: a CT scanner, a hospital information device and an image processing component. The multimodal data characterization module is used to collect original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; perform lung region segmentation on the CT image data to obtain a lung mask image; obtain an unlabeled lung CT image; use the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; perform lesion region annotation on the initial lesion region prediction map to obtain a lesion annotation image; perform image feature extraction on the lesion annotation image to obtain an image feature vector; construct a multimodal feature matrix according to the image feature vector to obtain a multimodal feature matrix; A lesion attention learning module is electrically connected to the control circuit, and the lesion attention learning module includes: a deep learning server, an image processing working component and a data storage device, and the lesion attention learning module is used to generate an attention heat map according to the multimodal feature matrix and the lung mask image and normalize the heat map to obtain a normalized heat map; and construct a lesion enhancement feature matrix according to the normalized heat map, CT image data and image feature vector to obtain a lesion enhancement feature matrix; A spatiotemporal feature fusion module is electrically connected to the control circuit, and the spatiotemporal feature fusion module includes: a Transformer encoder, and the spatiotemporal feature fusion module is used to perform feature vector splicing on the multimodal feature matrix and the lesion enhancement feature matrix to obtain a patient feature sequence; and perform spatiotemporal fusion feature multilayer processing according to the patient feature sequence and a preset Transformer encoder to obtain a spatiotemporal fusion feature vector; A risk prediction modeling module is electrically connected to the control circuit, and the risk prediction modeling module is used to label the risk level of the target patient with viral pneumonia using predefined risk assessment standard data to obtain a risk level label; a viral pneumonia risk prediction model is constructed according to the risk level label and the spatiotemporal fusion feature vector to obtain a viral pneumonia risk prediction model; An interpretability analysis module is electrically connected to the control circuit, and the interpretability analysis module includes: a display and a drawing library. The interpretability analysis module is used to use the viral pneumonia risk prediction model to predict the risk level of target patients with viral pneumonia to obtain risk level prediction data; identify key features based on the risk level prediction data to obtain key feature data; calculate feature contribution of the key feature data to obtain a feature contribution vector; perform risk factor analysis based on the feature contribution vector and the key feature data and generate a risk assessment report to obtain a risk assessment report.

2. The viral pneumonia risk assessment and prediction system according to claim 1, characterized in that: The multimodal data characterization module is used to perform the following steps: Step S11: collecting original data of viral pneumonia from a target patient with viral pneumonia to obtain original target patient data, wherein the original target patient data includes CT image data and original clinical data; Step S12: performing image data preprocessing on the CT image data to obtain image preprocessing data; performing lung region segmentation on the image preprocessing data to obtain a lung mask image; Step S13: obtaining an unlabeled lung CT image; using the unlabeled lung CT image to predict the lesion region to obtain an initial lesion region prediction map; annotating the lesion region on the initial lesion region prediction map to obtain a lesion annotated image; Step S14: extracting image features from the lesion annotated image to obtain an image feature vector; Step S15: performing clinical data cleaning on the original clinical data to obtain patient clinical data; performing time series construction on the patient clinical data to obtain a clinical time series; performing clinical feature extraction on the clinical time series to obtain a clinical feature vector; Step S16: performing feature vector combination on the clinical feature vector and the imaging feature vector to obtain a patient feature vector; constructing a multimodal feature matrix according to the patient feature vector to obtain a multimodal feature matrix.

3. The viral pneumonia risk assessment and prediction system according to claim 2, characterized in that: Step S13 includes the following steps: Step S131: obtaining an unlabeled lung CT image; fine-tuning a pre-trained image segmentation model using the unlabeled lung CT image to obtain an image segmentation model; Step S132: generating a lesion probability map for the lung mask image using the image segmentation model to obtain a lesion probability map; Step S133: performing probability map threshold segmentation on the lesion probability map using a preset probability map threshold to obtain an initial lesion region prediction map; Step S134: performing connected area analysis on the initial lesion region prediction map to obtain independent lesion region data; performing lesion area filtering on the initial lesion region prediction map according to the independent lesion region data to obtain a lesion region prediction map; Step S135: Perform doctor annotation correction on the lesion area prediction image to obtain a lesion annotation image.

4. The viral pneumonia risk assessment and prediction system according to claim 1, characterized in that: The lesion attention learning module is used to perform the following steps: Step S21: generating an attention heat map according to the multimodal feature matrix and the lung mask image and normalizing the heat map to obtain a normalized heat map; Step S22: using the normalized heat map to enhance the lesion area of ​​the CT image data to obtain an enhanced CT image; Step S23: performing multi-scale feature extraction on the CT image data and the enhanced CT image to obtain a multi-scale feature map; performing feature pyramid pooling on the multi-scale feature map to obtain a pooled feature vector; Step S24: performing lesion feature enhancement on the image feature vector and the pooled feature vector to obtain a lesion enhanced feature vector; Step S25: constructing a lesion enhancement feature matrix according to the lesion enhancement feature vector to obtain a lesion enhancement feature matrix.

5. The viral pneumonia risk assessment and prediction system according to claim 4, characterized in that: Step S21 includes the following steps: Step S211: reconstructing image data according to the multimodal feature matrix to obtain reconstructed image data; Step S212: inputting the reconstructed image data into the pre-trained 3D convolutional neural network model, performing model gradient calculation, and obtaining a gradient map; Step S213: performing global average pooling on the gradient map to obtain a feature map weight value; performing feature map multiplication and summation calculation according to the feature map weight value and generating an attention heat map to obtain an attention heat map; Step S214: using the lung mask image to perform heat map space constraints on the attention heat map to obtain a constrained heat map; Step S215: normalize the constraint heat map to obtain a normalized heat map.

6. The viral pneumonia risk assessment and prediction system according to claim 1, characterized in that: The spatiotemporal feature fusion module is used to perform the following steps: Step S31: performing feature matrix time alignment on the multimodal feature matrix and the lesion enhancement feature matrix according to the acquisition time data in the original clinical data to obtain feature matrix time alignment data; Step S32: dividing the feature matrix time-aligned data into feature matrix time windows to obtain time slice data; Step S33: performing feature vector concatenation on the multimodal feature matrix and the lesion enhancement feature matrix according to the time slice data to obtain a patient feature sequence; Step S34: adding position coding to the patient feature sequence to obtain a position coded patient feature sequence; Step S35: Input the position-coded patient feature sequence into the preset Transformer encoder, perform multi-layer processing of spatiotemporal fusion features, and obtain a spatiotemporal fusion feature vector.

7. The viral pneumonia risk assessment and prediction system according to claim 6, characterized in that: Step S35 includes the following steps: Step S351: input the position-encoded patient feature sequence into a preset Transformer encoder, perform self-attention mechanism calculation, and obtain a time-step weighted feature vector; perform weighted average processing on the time-step weighted feature vector to obtain a global feature vector; Step S352: Perform multi-head attention mechanism fusion according to the global feature vector to obtain a comprehensive global feature vector; Step S353: performing feedforward network enhancement on the comprehensive global feature vector to obtain feedforward network enhanced data; Step S354: Output the spatiotemporal fusion feature vector according to the feedforward network enhanced data to obtain the spatiotemporal fusion feature vector.

8. The viral pneumonia risk assessment and prediction system according to claim 1, characterized in that: The risk prediction modeling module is used to perform the following steps: Step S41: using predefined risk assessment standard data to label the risk level of the target viral pneumonia patient to obtain a risk level label; Step S42: performing risk label matching on the risk level label and the spatiotemporal fusion feature vector to obtain a label feature data set; Step S43: using the label feature data set to perform model training and hyperparameter adjustment to obtain an initial risk prediction model; Step S44: Perform model evaluation on the initial risk prediction model to obtain a viral pneumonia risk prediction model.

9. The viral pneumonia risk assessment and prediction system according to claim 1, characterized in that: The interpretability analysis module is used to perform the following steps: Step S51: using the viral pneumonia risk prediction model to predict the risk level of the target viral pneumonia patient to obtain risk level prediction data; Step S52: sorting the feature importance according to the risk level prediction data and the viral pneumonia risk prediction model to obtain a feature importance list; Identify key features based on the feature importance list to obtain key feature data; Step S53: constructing a local explanation model for the target patient with viral pneumonia to obtain a local explainable model; calculating feature contribution of key feature data according to the local explainable model to obtain a feature contribution vector; Step S54: Perform risk factor analysis based on the feature contribution vector and key feature data and generate a risk assessment report to obtain a risk assessment report.

10. The viral pneumonia risk assessment and prediction system according to claim 9, characterized in that: Step S54 includes the following steps: Step S541: generating risk visualization interpretation of the feature contribution vector and key feature data to obtain risk visualization interpretation data; Step S542: performing risk factor analysis based on the feature contribution vector and key feature data to obtain risk factor analysis data; Step S543: Generate a risk assessment report based on the risk level prediction data, risk visualization interpretation data, and risk factor analysis data to obtain a risk assessment report.

Images