A pneumonia prognosis evaluation system combining automatic segmentation and multi-modal features

Abstract

The application provides a kind of pneumonia prognosis evaluation system combining automatic segmentation and multi-modal feature, the system includes the server end and at least one client connected with each other, the server end is deployed with prognosis evaluation engine, the prognosis evaluation engine includes data preprocessing module, lung infection area automatic segmentation module, multi-modal feature extraction and fusion module and prognosis evaluation and risk stratification module connected in turn.The application automatically completes the accurate segmentation and quantitative measurement of lung infection area through deep learning model, completely overcomes the strong subjectivity of artificial reading, poor reproducibility and low efficiency, provides unprecedented image data precision, and through attention mechanism, imageomics, clinical indicators and laboratory data are deeply fused instead of simply stacked, so that the evaluation model can fully and stereoscopically understand the condition and capture the complex cross-modal feature interaction that the human brain cannot detect.

Description

Technical Field

[0001] This invention relates to the field of medical auxiliary technology, and in particular to a pneumonia prognosis assessment system that combines automatic segmentation with multimodal features. Background Technology

[0002] As a common infectious disease, the prognostic assessment of pneumonia has always been a core and challenging aspect of clinical work. Traditional assessment methods heavily rely on the personal experience of physicians, relying on subjective visual evaluation of chest CT images combined with limited clinical signs and laboratory indicators for comprehensive judgment. This model has significant limitations: Firstly, the assessment of the extent and severity of pneumonia infection areas in CT images is usually qualitative or semi-quantitative, such as simple descriptions of mild, moderate, and severe, lacking precise quantitative indicators. Significant differences may exist between different observers, and the human eye cannot accurately capture subtle textural changes and heterogeneity within lesions. Secondly, clinical data, imaging findings, and laboratory test results are often analyzed in isolation, failing to be effectively and deeply integrated. The potential correlations between information are not fully utilized, leading to potentially incomplete and inaccurate judgments of the condition. This is especially true when facing sudden, large-scale respiratory infectious disease outbreaks, where medical resources are strained, experienced physicians may be scarce, and rapid and accurate triage and prognostic assessment of a large number of patients becomes particularly difficult.

[0003] Therefore, there is an urgent clinical need for a technology that can objectively, quantitatively, and automatically analyze multi-source information to help doctors identify high-risk patients earlier and more accurately, thereby optimizing the allocation of medical resources and providing personalized and precise treatment. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a pneumonia prognosis assessment system that combines automatic segmentation with multimodal features.

[0005] The technical solution of this invention is: a pneumonia prognostic assessment system combining automatic segmentation and multimodal features. The system includes a server and at least one client connected in communication. The server is equipped with a prognostic assessment engine, which includes a data preprocessing module, an automatic lung infection region segmentation module, a multimodal feature extraction and fusion module, and a prognostic assessment and risk stratification module connected sequentially. The data preprocessing module is configured to receive and standardize multimodal input data from the client. The multimodal input data includes at least the patient's chest CT image sequence, structured clinical text data, and laboratory test data. The automatic lung infection region segmentation module incorporates a depthwise convolution-based... An improved automatic segmentation model based on neural networks and the U-Net architecture is pre-trained and fine-tuned using a CT image dataset with pixel-level infection region annotations. This model performs layer-by-layer analysis on input standardized CT image sequences, automatically identifying and accurately segmenting ground-glass opacities, consolidation, and other pneumonia-related abnormal lesions within the lung parenchyma. It generates corresponding binary segmentation masks and quantitative indicators of the infection region, including the total volume of the infection region, its percentage of the total lung volume, average density value, and a three-dimensional spatial distribution descriptor. The multimodal feature extraction and fusion module is configured to perform the following operations in parallel: extracting radiomics features from the segmentation masks and quantitative indicators generated by the automatic segmentation module. The radiomics features include first-order statistical features, shape features, texture features, and wavelet features. Clinical features are extracted from the structured clinical text data, including patient age, history of underlying diseases, duration of clinical symptoms, and oxygenation index. Laboratory features are extracted from the laboratory test data, including white blood cell count, lymphocyte count, C-reactive protein, procalcitonin, and lactate dehydrogenase levels. Then, a multimodal feature fusion network based on an attention mechanism is used to deeply fuse the extracted radiomics features, clinical features, and laboratory features. This network first maps feature vectors from different modalities to a unified high-dimensional feature space, and then calculates the modality through cross-modal attention. The module dynamically learns the importance weights of each modality feature for the prognostic assessment task and performs weighted fusion accordingly to generate a comprehensive and complementary multimodal fusion feature vector. The prognostic assessment and risk stratification module has a built-in prognostic prediction model based on gradient boosting decision trees or deep neural networks. This model takes the multimodal fusion feature vector as input and outputs a comprehensive assessment result that includes at least short-term prognostic risk probability, predicted probability of severe illness conversion, and recommended treatment strategy classification. The short-term prognostic risk probability is used to predict the risk of adverse outcome events in patients within a specific time window in the future. The recommended treatment strategy classification includes at least routine supportive care, high-flow oxygen therapy, invasive mechanical ventilation recommendations, or recommendations for transfer to the intensive care unit.The system also includes a results visualization and report generation module, used to display the comprehensive evaluation results, key input features, and segmentation results on the client side in a graphical interface, and automatically generate a structured prognostic evaluation report.

[0006] As a preferred embodiment of the present invention, the automatic segmentation model based on the improved deep convolutional neural network and U-Net architecture is specifically a U-Net++ variant that integrates residual connections and attention gating mechanisms. Its encoder part uses a ResNet-50 or ResNet-101 network pre-trained on the ImageNet dataset as the backbone feature extraction network to enhance the ability to extract features of multi-scale infected areas in CT images. In the decoder part, each upsampling stage integrates an attention gating unit, which can suppress background information of non-lung and non-infected areas in CT images, thereby focusing on the segmentation of lung infected areas. During the training process, the model adopts a hybrid loss function that combines the Dice loss function and the Focal loss function to effectively address the class imbalance problem between infected areas and normal lung tissue.

[0007] As a preferred embodiment of the present invention, the specific process of extracting radiomics features from the segmentation mask and quantitative indicators includes: registering the original CT image sequence with the segmentation mask generated by the automatic segmentation module to ensure accurate correspondence of each pixel; extracting more than 1,000 quantitative features from each infected area using PyRadiomics or a similar open-source radiomics toolkit; and through a series of feature selection steps, including removing features with low variance based on variance thresholding, removing highly correlated redundant features based on Pearson correlation coefficient, and selecting features using LASSO regression or random forest feature importance ranking, ultimately retaining a core radiomics feature subset of 50 to 200 features that are significantly related to pneumonia prognosis for subsequent feature fusion.

[0008] As a preferred embodiment of the present invention, the specific structure of the attention-based multimodal feature fusion network includes: three independent feature encoding sub-networks, respectively used to process radiomics features, clinical features, and laboratory features. Each encoding sub-network consists of a fully connected layer and an activation function, mapping the original features of different dimensions to a unified 512-dimensional feature space to obtain feature vectors; a cross-modal attention calculation module, which first concatenates the feature vectors of the three modalities, and then calculates the attention weight score of each modality through a multilayer perceptron. The attention weight is calculated by inputting the concatenated features into a two-layer neural network, followed by a Softmax function for normalization to ensure that the sum of the three weights is 1; finally, the feature vectors of each modality and their corresponding attention weights are weighted and summed to obtain the final fused feature vector.

[0009] As a preferred embodiment of the present invention, the prognostic prediction model in the prognostic assessment and risk stratification module is a gradient boosting decision tree ensemble model based on the XGBoost or LightGBM framework. This model is trained using a large dataset containing multimodal fusion feature vectors and corresponding real clinical prognostic outcomes, including whether the patient died, whether invasive mechanical ventilation was required, and whether the patient was admitted to the ICU for more than 7 days. The model can output the probability value of each predicted outcome and classify patients into three levels—low risk, medium risk, and high risk—according to a preset risk threshold. In addition, the model provides feature importance ranking to explain the key factors affecting prognostic assessment and assist clinicians in making decisions.

[0010] As a preferred embodiment of the present invention, the system further includes a continuous learning and model optimization module, which is configured to: after obtaining patient authorization and desensitizing the data, collect the patient's long-term prognostic data as model feedback; combine the newly generated data with the original training data, and periodically or when the performance decay reaches a certain threshold, perform incremental learning or retraining on the automatic segmentation model, the multimodal feature fusion network, and the prognostic prediction model to dynamically update the model parameters, so that the system can adapt to changes in the characteristics of pneumonia and treatment plans in different periods and different populations, and maintain the accuracy of prognostic assessment.

[0011] As a preferred embodiment of the present invention, the data preprocessing module performs the following standardization processing on the chest CT image sequence: reading image data according to the DICOM standard; adjusting the window width and window level to the lung window setting; performing isotropic resampling to ensure consistent spatial resolution; applying the z-score standardization method to normalize pixel values ​​to eliminate differences between different scanning devices; and for clinical text data and laboratory test index data, performing missing value imputation, outlier handling, and standardization based on mean and standard deviation, and performing one-hot encoding on all non-numerical categorical variables.

[0012] As a preferred embodiment of the present invention, the graphical interface displayed on the client by the result visualization and report generation module includes at least: coronal, sagittal, and transverse views of the original chest CT image, overlaid with the segmentation mask generated by the automatic segmentation module, highlighting different types of infection areas with different colors; displaying dynamic trend graphs of quantitative indicators of infection areas; showing the contribution of each modality feature to the current patient prognostic risk assessment in the form of radar charts or bar charts; displaying the final risk stratification results and treatment strategy recommendations in the form of eye-catching color cards; the structured prognostic assessment report follows the HL7 or FHIR standard, includes basic patient information, input data summary, segmentation and feature analysis results, prognostic assessment conclusions, and model confidence indicators, and supports one-click export to PDF format.

[0013] As a preferred embodiment of the present invention, the system's workflow is as follows: After logging into the system, the client-side doctor uploads the target patient's CT image sequence, clinical data form, and laboratory test results through a secure interface; the server-side prognostic assessment engine automatically triggers the data preprocessing module to standardize the data, then the automatic lung infection area segmentation module completes lesion segmentation and quantitative analysis, followed by the multimodal feature extraction and fusion module to extract and fuse multi-source features, and finally the prognostic assessment and risk stratification module generates a comprehensive assessment result; the entire process is completed within minutes, and the results are returned to the client interface through the result visualization and report generation module for the doctor's reference.

[0014] As a preferred embodiment of the present invention, it further includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operation method of all functional modules of the server-side prognostic assessment engine in a pneumonia prognostic assessment system that combines automatic segmentation and multimodal features.

[0015] By adopting the above technical solution, the present invention has the following advantages: 1. This invention automatically completes the precise segmentation and quantitative measurement of lung infection areas through a deep learning model, completely overcoming the drawbacks of strong subjectivity, poor repeatability, and low efficiency of manual image interpretation. It provides unprecedented accuracy in imaging data. At the same time, it deeply integrates radiomics, clinical indicators, and laboratory data through an attention mechanism, rather than simply stacking them. This enables the assessment model to gain a comprehensive and three-dimensional understanding of the disease and capture complex cross-modal feature interactions that are difficult for the human brain to perceive. This greatly improves the accuracy and reliability of prognostic prediction and provides a solid data-driven foundation for clinical decision-making.

[0016] 2. This invention integrates a complex multimodal data analysis process into one, enabling fully automated analysis from data input to result output within minutes, significantly shortening the evaluation time. It is especially suitable for scenarios requiring rapid decision-making, such as emergency rooms and intensive care units. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0018] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] A pneumonia prognostic assessment system combining automatic segmentation and multimodal features is disclosed. The system includes a server and at least one client connected in communication. The server deploys a prognostic assessment engine, which includes a data preprocessing module, an automatic lung infection region segmentation module, a multimodal feature extraction and fusion module, and a prognostic assessment and risk stratification module, all connected sequentially. The data preprocessing module is configured to receive and standardize multimodal input data from the client, which includes at least the patient's chest CT image sequence, structured clinical text data, and laboratory test data. The automatic lung infection region segmentation module incorporates an automatic segmentation model based on a deep convolutional neural network and an improved U-Net architecture. This model uses... A CT image dataset with pixel-level infection region annotations is pre-trained and fine-tuned to perform layer-by-layer analysis on input standardized CT image sequences. This automatically identifies and accurately segments ground-glass opacities, consolidation, and other pneumonia-related abnormal lesions within the lung parenchyma, generating corresponding binary segmentation masks and quantitative indicators of the infection region. These quantitative indicators include the total volume of the infection region, its percentage of the total lung volume, average density value, and a three-dimensional spatial distribution descriptor. A multimodal feature extraction and fusion module is configured to perform the following operations in parallel: extracting radiomics features from the segmentation masks and quantitative indicators generated by the automatic segmentation module. These radiomics features include first-order statistical features, shape features, texture features, and wavelet features; and extracting clinical features from structured clinical text data. Clinical features include patient age, history of underlying diseases, duration of clinical symptoms, and oxygenation index. Laboratory features are extracted from laboratory test data, including white blood cell count, lymphocyte count, C-reactive protein, procalcitonin, and lactate dehydrogenase levels. Then, a multimodal feature fusion network based on an attention mechanism is used to deeply fuse the extracted radiomics features, clinical features, and laboratory features. This network first maps feature vectors from different modalities to a unified high-dimensional feature space, then dynamically learns the importance weights of each modality's features for the prognostic assessment task through a cross-modal attention computation module, and performs weighted fusion accordingly to generate a comprehensive and complementary multimodal fused feature vector. Prognostic assessment and risk assessment are then performed. The hierarchical module incorporates a prognostic prediction model built on gradient boosting decision trees or deep neural networks. This model takes multimodal fusion feature vectors as input and outputs a comprehensive assessment result that includes at least short-term prognostic risk probability, predicted probability of progression to severe illness, and recommended treatment strategy classification. The short-term prognostic risk probability is used to predict the risk of adverse outcomes for patients within a specific future time window. The recommended treatment strategy classification includes at least routine supportive care, high-flow oxygen therapy, invasive mechanical ventilation recommendations, or recommendations for transfer to the intensive care unit. The system also includes a result visualization and report generation module, which displays the comprehensive assessment result, along with key input features and segmentation results, in a graphical interface on the client side and automatically generates a structured prognostic assessment report.The system also includes a continuous learning and model optimization module, configured to: collect patients' long-term prognostic data as model feedback after obtaining patient authorization and anonymizing the data; combine newly generated data with existing training data, and periodically or when performance degradation reaches a certain threshold, incrementally learn or retrain the automatic segmentation model, multimodal feature fusion network, and prognostic prediction model to dynamically update model parameters, enabling the system to adapt to changes in pneumonia disease characteristics and treatment plans in different periods and populations, maintaining the accuracy of prognostic assessment; and include a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the operation methods of all functional modules of the server-side prognostic assessment engine in the pneumonia prognostic assessment system combining automatic segmentation and multimodal features.

[0020] Specifically, the automatic segmentation model based on deep convolutional neural networks and the improved U-Net architecture is a U-Net++ variant that integrates residual connections and attention gating mechanisms. Its encoder uses a ResNet-50 or ResNet-101 network pre-trained on the ImageNet dataset as the backbone feature extraction network to enhance the extraction of multi-scale infection region features from CT images. In the decoder, each upsampling stage integrates an attention gating unit, which suppresses background information from non-lung and non-infected areas in CT images, thereby focusing on the segmentation of lung infection regions. During training, the model employs a hybrid loss function combining Dice and Focal loss functions. The loss function effectively addresses the class imbalance between infected areas and normal lung tissue. The specific process for extracting radiomics features from the segmentation mask and quantitative indicators includes: registering the original CT image sequence with the segmentation mask generated by the automatic segmentation module to ensure accurate correspondence for each pixel; extracting over 1000 quantitative features from each infected area using PyRadiomics or a similar open-source radiomics toolkit; and employing a series of feature selection steps, including removing features with excessively low variance based on variance thresholding, removing highly correlated redundant features based on Pearson correlation coefficient, and using LASSO regression or random forest feature importance ranking for feature selection, ultimately retaining features significantly correlated with pneumonia prognosis. A subset of core radiomics features, ranging from 50 to 200, is used for subsequent feature fusion. The specific structure of the attention-based multimodal feature fusion network includes: three independent feature encoding subnetworks, used to process radiomics features, clinical features, and laboratory features respectively. Each encoding subnetwork consists of a fully connected layer and an activation function, mapping the original features of different dimensions to a unified 512-dimensional feature space to obtain feature vectors; and a cross-modal attention calculation module, which first concatenates the feature vectors of the three modalities, and then calculates the attention weight score of each modality through a multilayer perceptron. The attention weight is calculated by inputting the concatenated features into a two-layer neural network, followed by S... The oftmax function is used for normalization to ensure that the sum of the three weights is 1. Finally, the feature vectors of each modality are weighted and summed with their corresponding attention weights to obtain the final fused feature vector. The prognostic prediction model in the prognostic assessment and risk stratification module is a gradient boosting decision tree ensemble model based on the XGBoost or LightGBM framework. This model is trained using a large dataset containing multimodal fused feature vectors and corresponding real clinical prognostic outcomes. Real clinical prognostic outcomes include whether the patient died, whether invasive mechanical ventilation was required, and whether the patient was admitted to the ICU for more than 7 days. The model can output the probability value of each predicted outcome and classify patients into three levels: low risk, medium risk, and high risk according to a preset risk threshold.Furthermore, the model provides feature importance ranking to explain key factors influencing prognostic assessment, assisting clinicians in decision-making. The data preprocessing module standardizes chest CT image sequences by: reading image data according to DICOM standards; adjusting window width and level to lung window settings; performing isotropic resampling to ensure consistent spatial resolution; and applying z-score normalization to eliminate differences between different scanning devices. For clinical text data and laboratory test data, missing value imputation, outlier handling, and standardization based on mean and standard deviation are performed, and all non-numerical categorical variables are one-hot encoded. The results visualization and report generation module allows clients to... The graphical interface displayed on the client side includes at least: coronal, sagittal, and transverse views of the original chest CT images, overlaid with a segmentation mask generated by the automatic segmentation module, highlighting different types of infection areas with different colors; dynamic trend graphs showing the quantitative indicators of the infection area; radar charts or bar graphs showing the contribution of each modality feature to the current patient's prognostic risk assessment; final risk stratification results and treatment strategy recommendations displayed in a striking color card format; and a structured prognostic assessment report following HL7 or FHIR standards, including basic patient information, a summary of input data, segmentation and feature analysis results, prognostic assessment conclusions, and model confidence indicators, and supports one-click export to PDF format.

[0021] The system's workflow is as follows: After logging into the system, the client-side doctor user uploads the target patient's CT image sequence, clinical data form, and laboratory test results through a secure interface; The server-side prognostic assessment engine automatically triggers the data preprocessing module to standardize the data. Then, the automatic lung infection area segmentation module completes lesion segmentation and quantitative analysis. Next, the multimodal feature extraction and fusion module completes the extraction and fusion of multi-source features. Finally, the prognostic assessment and risk stratification module generates a comprehensive assessment result. The entire process is completed within minutes, and the results are returned to the client interface through the results visualization and report generation module for doctors' reference.

[0022] It should be noted that doctors input multimodal data of the patient to be evaluated into the client interface. This data forms the foundation for the system's intelligent analysis. It mainly includes high-resolution chest CT image sequences, structured electronic medical record forms containing information such as patient age, underlying diseases (e.g., chronic obstructive pulmonary disease, diabetes, hypertension), clinical symptoms (e.g., duration of fever, cough, dyspnea), and current oxygenation index, as well as key laboratory test results, such as white blood cell count, lymphocyte count, C-reactive protein, procalcitonin, and lactate dehydrogenase—indicators reflecting the severity of infection and the body's inflammatory response. This heterogeneous data is sent to the server via a secure network transmission protocol and immediately processed by the data preprocessor. The processing module receives and processes the data, serving as the primary step to ensure the accuracy of subsequent analyses. It performs DICOM standard parsing on the CT image data, adjusts the window width and level to standard lung windows to optimize the display of infected areas, and performs isotropic resampling and z-score standardization of pixel values ​​to eliminate variations introduced by different scanning equipment and parameters. Simultaneously, it performs missing value imputation, outlier detection and correction on structured clinical and laboratory data, and performs one-hot encoding on all non-numerical categorical variables. Finally, it transforms all multimodal data into a standardized dataset with a uniform format and scale, preparing the input materials for downstream advanced analysis modules. The standardized CT image sequences are then fed into the system. The core automatic lung infection area segmentation module utilizes a deep learning model based on an improved U-Net architecture. Its encoder efficiently extracts multi-level features from images using a pre-trained ResNet backbone, from edges and textures to more complex anatomical structures. The decoder, through an integrated attention gating mechanism, focuses on the lung parenchyma during layer-by-layer upsampling, effectively suppressing interference from irrelevant tissues such as the chest wall and mediastinum. Trained on massive amounts of pneumonia CT data with pixel-level precise annotations, the model can accurately identify and delineate the boundaries of typical pneumonia lesions such as ground-glass opacities and consolidation within the lungs. The final output is a binarized representation corresponding layer-by-layer to the original CT sequence. A segmentation mask, acting like a highlighter, precisely marks the location and extent of the infected area. Based on this mask, the module automatically calculates a series of crucial quantitative indicators, including the total volume of the infected area, the percentage of the infected volume to the total lung volume, the average CT value of the lesions, and descriptors describing the distribution of the lesions in three-dimensional space. With the precise segmentation of the infected area completed, the system workflow triggers a multimodal feature extraction and fusion module in parallel. This module acts as a highly efficient information integration center, simultaneously performing three feature extraction tasks: First, from the obtained segmentation mask and original CT images, it extracts massive amounts of quantitative features using advanced radiomics algorithms. These features surpass human visual perception.The three main approaches are: First, first-order features describing the statistical characteristics of pixel value distribution within lesions; shape features describing the morphological complexity of lesions; texture features describing the heterogeneity within lesions (such as gray-level co-occurrence matrix features); and deeper pattern features revealed at different frequency scales after wavelet transform. These features undergo a rigorous feature selection process (such as removing features with low variance, eliminating highly correlated redundant features, and selecting the most predictive feature subset based on algorithms like LASSO), ultimately retaining dozens to hundreds of core radiomics features strongly correlated with pneumonia prognosis. Second, key clinical features are extracted from structured clinical text data, such as patient age, presence of immunosuppression, duration of symptoms, and current oxygenation index. These features directly reflect the patient's physiological reserve and the immediate impact of the disease on the body. Third, biomarker features reflecting infection and inflammation levels are extracted from laboratory data. Subsequently, the most crucial innovative step of the system, multimodal fusion based on an attention mechanism, begins. The extracted radiomics feature vectors, clinical feature vectors, and laboratory feature vectors are first mapped into a unified high-dimensional feature space by three independent feature encoding sub-networks. These vectors then enter a cross-modal attention computation unit, which dynamically analyzes the patient's specific situation and calculates the importance weight of each feature for the final prognostic judgment. For example, for an elderly patient with multiple underlying diseases, clinical features may be assigned a higher weight; while for a young patient with particularly severe radiographic findings but relatively good laboratory indicators and general condition, the weight of radiomics features may be significantly increased. These weights are not fixed but are adaptively learned by the neural network based on input information, ensuring the flexibility and specificity of the fusion strategy. Finally, each modal feature vector is multiplied by its respective learned weight and then weighted and summed to generate a comprehensive, complementary, and focused multimodal fusion feature vector. This vector maximizes the... Including prognostic information from three dimensions—imaging, clinical, and laboratory—the fused feature vector is then fed into the prognostic assessment and risk stratification module. This module's built-in predictive model, constructed using gradient boosting decision trees or deep neural networks, performs the final intelligent judgment. Utilizing knowledge trained on large historical case datasets, the model deeply analyzes the current patient's fused features, outputting a comprehensive, quantitative prognostic assessment. This result typically includes the short-term risk probability of the patient experiencing adverse outcomes such as death or requiring life support within a specific future time window, the predicted probability of progression to severe illness (e.g., requiring intubation), and personalized treatment strategy recommendations based on risk level. The system automatically categorizes patients into low-risk, medium-risk, or high-risk levels based on the calculated risk probability values ​​and matches corresponding clinical decision-making references for each level. For example, low-risk patients may be advised to receive routine supportive care and close observation, while medium-risk patients may be advised to receive intensive monitoring and consider high-flow oxygen therapy.For high-risk patients, preparations for transfer to the intensive care unit and invasive mechanical ventilation are strongly advised. This module not only provides predictive results but also offers interpretability, revealing which key features (such as extremely high infection volume percentage and extremely low lymphocyte count combined with advanced age) dominate the high-risk assessment, assisting physicians in understanding the model's decision-making basis. Ultimately, all analysis results, including the original CT images, fused views overlaid with the infection area segmentation contours, key quantitative indicator charts, visualizations of the contribution of each modality feature, and the final risk stratification conclusions and treatment recommendations, are clearly and intuitively presented to physicians on the client's graphical interface through the results visualization and report generation module. Simultaneously, the system automatically generates a structured prognostic assessment report conforming to medical documentation standards, detailing the analysis process and results. The system supports one-click export for easy storage in electronic medical record systems or for clinical consultations and discussions. Furthermore, it incorporates a continuous learning and model optimization module. With informed consent and ethical approval from patients, the system can anonymize and collect actual clinical outcome data as feedback labels. It periodically uses new data to incrementally learn or retrain the segmentation and prediction models, enabling the system to continuously adapt to new pathogen prevalence characteristics and changes in diagnostic and treatment technologies, achieving continuous performance evolution and improvement. The system deeply integrates objective quantitative analysis from radiology, rich clinical experience, and precise laboratory testing indicators, providing physicians with a powerful, objective, and efficient decision-making tool. This significantly enhances the ability to make accurate prognostic judgments and personalized interventions in complex pneumonia cases, demonstrating extremely high clinical practical value.

[0023] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

Claims

1. A pneumonia prognostic assessment system combining automatic segmentation and multimodal features, characterized in that, The system includes a server and at least one client connected to each other. The server is equipped with a prognostic assessment engine, which includes a data preprocessing module, an automatic lung infection region segmentation module, a multimodal feature extraction and fusion module, and a prognostic assessment and risk stratification module connected in sequence. The data preprocessing module is configured to receive and standardize multimodal input data from the client, which includes at least the patient's chest CT image sequence, structured clinical text data, and laboratory test data. The automatic lung infection region segmentation module incorporates an automatic segmentation model based on a deep convolutional neural network and an improved U-Net architecture, which uses pixel-level... The CT image dataset labeled with infected areas is pre-trained and fine-tuned to perform layer-by-layer analysis on the input standardized CT image sequence, automatically identifying and accurately segmenting ground-glass opacities, consolidation, and other pneumonia-related abnormal lesions within the lung parenchyma, and generating corresponding binary segmentation masks and quantitative indicators of the infected area. The quantitative indicators include the total volume of the infected area, its percentage of the total lung volume, its average density value, and a three-dimensional spatial distribution descriptor. The multimodal feature extraction and fusion module is configured to perform the following operations in parallel: extracting radiomics features from the segmentation mask and quantitative indicators generated by the automatic segmentation module. The radiomics features include first-order statistical features, shape features, texture features, and wavelet features. Clinical features are extracted from the structured clinical text data, including patient age, history of underlying diseases, duration of clinical symptoms, and oxygenation index. Laboratory features are extracted from the laboratory test data, including white blood cell count, lymphocyte count, C-reactive protein, procalcitonin, and lactate dehydrogenase levels. Then, a multimodal feature fusion network based on an attention mechanism is used to deeply fuse the extracted radiomics features, clinical features, and laboratory features. This network first maps feature vectors from different modalities to a unified high-dimensional feature space, then dynamically learns the importance weights of each modality feature for the prognostic assessment task through a cross-modal attention calculation module, and performs weighted fusion accordingly to generate a comprehensive and complementary multimodal fused feature vector. The prognostic assessment and risk stratification module incorporates a prognostic prediction model built on gradient boosting decision trees or deep neural networks. This model takes the multimodal fusion feature vector as input and outputs a comprehensive assessment result that includes at least short-term prognostic risk probability, predicted probability of progression to severe illness, and recommended treatment strategy classification. The short-term prognostic risk probability is used to predict the risk of adverse outcomes for patients within a specific future time window. The recommended treatment strategy classification includes at least routine supportive care, high-flow oxygen therapy, invasive mechanical ventilation recommendations, or recommendations for transfer to the intensive care unit. The system also includes a result visualization and report generation module, which displays the comprehensive assessment result, along with key input features and segmentation results, in a graphical interface on the client side and automatically generates a structured prognostic assessment report.

2. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The improved automatic segmentation model based on deep convolutional neural networks and the U-Net architecture is specifically a U-Net++ variant that integrates residual connections and attention gating mechanisms. Its encoder uses a ResNet-50 or ResNet-101 network pre-trained on the ImageNet dataset as the backbone feature extraction network to enhance the extraction of multi-scale infected area features from CT images. In the decoder, each upsampling stage integrates an attention gating unit, which suppresses background information from non-lung and non-infected areas in CT images, thereby focusing on the segmentation of infected lung areas. During training, the model employs a hybrid loss function combining Dice and Focal loss functions to effectively address the class imbalance problem between infected areas and normal lung tissue.

3. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The specific process of extracting radiomics features from the segmentation mask and quantitative indicators includes: registering the original CT image sequence with the segmentation mask generated by the automatic segmentation module to ensure accurate correspondence of each pixel; extracting more than 1,000 quantitative features from each infected area using PyRadiomics or a similar open-source radiomics toolkit; and selecting features through a series of feature screening steps, including removing features with low variance based on variance thresholding, removing highly correlated redundant features based on Pearson correlation coefficient, and using LASSO regression or random forest feature importance ranking to select features. Finally, a subset of core radiomics features, numbering between 50 and 200, that are significantly related to pneumonia prognosis are retained for subsequent feature fusion.

4. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The specific structure of the attention-based multimodal feature fusion network includes: three independent feature encoding sub-networks, used to process radiomics features, clinical features, and laboratory features respectively. Each encoding sub-network consists of a fully connected layer and an activation function, mapping the original features of different dimensions to a unified 512-dimensional feature space to obtain feature vectors; a cross-modal attention calculation module, which first concatenates the feature vectors of the three modalities, and then calculates the attention weight score of each modality through a multilayer perceptron. The attention weight is calculated by inputting the concatenated features into a two-layer neural network, followed by a Softmax function for normalization to ensure that the sum of the three weights is 1; finally, the feature vectors of each modality are weighted and summed with their corresponding attention weights to obtain the final fused feature vector.

5. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The prognostic prediction model in the prognostic assessment and risk stratification module is a gradient boosting decision tree ensemble model based on the XGBoost or LightGBM framework. This model is trained using a large dataset containing multimodal fusion feature vectors and corresponding real clinical prognostic outcomes, including whether the patient died, whether invasive mechanical ventilation was required, and whether the patient was admitted to the ICU for more than 7 days. The model can output the probability value of each predicted outcome and classify patients into three levels—low-risk, medium-risk, and high-risk—according to a preset risk threshold. In addition, the model provides feature importance ranking to explain the key factors affecting prognostic assessment and assist clinicians in making decisions.

6. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The system also includes a continuous learning and model optimization module, which is configured to: after obtaining patient authorization and desensitizing the data, collect the patient's long-term prognostic data as model feedback; combine the newly generated data with the original training data, and periodically or when the performance decay reaches a certain threshold, perform incremental learning or retraining on the automatic segmentation model, multimodal feature fusion network, and prognostic prediction model to dynamically update the model parameters, so that the system can adapt to changes in pneumonia disease characteristics and treatment plans in different periods and different populations, and maintain the accuracy of prognostic assessment.

7. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The data preprocessing module performs the following standardization processing on chest CT image sequences: reading image data according to the DICOM standard; adjusting window width and window level to the lung window setting; performing isotropic resampling to ensure consistent spatial resolution; applying z-score standardization to normalize pixel values ​​to eliminate differences between different scanning devices; and for clinical text data and laboratory test indicator data, performing missing value imputation, outlier handling, and standardization based on mean and standard deviation, and performing one-hot encoding on all non-numerical categorical variables.

8. The pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The graphical interface displayed on the client side by the results visualization and report generation module includes at least: coronal, sagittal, and transverse views of the original chest CT images, overlaid with the segmentation mask generated by the automatic segmentation module, highlighting different types of infection areas with different colors; dynamic trend charts of quantitative indicators of the infection area; radar charts or bar charts showing the contribution of each modality feature to the current patient's prognostic risk assessment; and final risk stratification results and treatment strategy recommendations displayed in a striking color card format. The structured prognostic assessment report follows the HL7 or FHIR standard, includes basic patient information, input data summary, segmentation and feature analysis results, prognostic assessment conclusions, and model confidence indicators, and supports one-click export to PDF format.

9. A pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, The system's workflow is as follows: After logging into the system, the client-side doctor uploads the target patient's CT image sequence, clinical data forms, and laboratory test results through a secure interface. The server-side prognostic assessment engine automatically triggers the data preprocessing module to standardize the data. Then, the automatic lung infection area segmentation module completes lesion segmentation and quantitative analysis. Next, the multimodal feature extraction and fusion module extracts and fuses multi-source features. Finally, the prognostic assessment and risk stratification module generates a comprehensive assessment result. The entire process is completed within minutes, and the results are returned to the client interface through the result visualization and report generation module for the doctor's reference.

10. A pneumonia prognostic assessment system combining automatic segmentation and multimodal features according to claim 1, characterized in that, It also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operation method of all functional modules of the server-side prognostic assessment engine in the pneumonia prognostic assessment system combining automatic segmentation and multimodal features as described in any one of claims 1 to 9.

Images