Artificial intelligence-based rapid assessment method and system for ultrasound images of pneumonia
By constructing an AI assessment system for lung ultrasound and circulatory volume ultrasound based on deep convolutional neural networks, the problems of ionizing radiation and difficulty in re-examination of CT images in the diagnosis of COVID-19 have been solved. This system enables rapid and convenient pneumonia assessment and remote monitoring, especially for the assessment and early warning of severe and critically ill COVID-19 patients.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IMABOT SHENZHEN MEDICAL CO LTD
- Filing Date
- 2021-01-22
- Publication Date
- 2026-07-24
Smart Images
Figure CN114767153B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of biotechnology, and more specifically relates to a method and system for rapid assessment of pneumonia ultrasound images based on artificial intelligence. Background Technology
[0002] Currently, high-resolution CT images are widely recognized as the primary medical imaging basis for diagnosing COVID-19. However, in clinical use, CT's detection capabilities are limited, making it unable to quickly respond to a large number of patients, especially severe cases.
[0003] The clinical role of lung ultrasound in the diagnosis of pulmonary exudative diseases, assessment of pleural effusion, identification of pneumothorax, and in patients with dyspnea and acute respiratory failure has been recognized by medical experts both domestically and internationally. Ultrasound imaging is a versatile, low-cost, and convenient imaging method widely used in most modern healthcare systems. Ultrasound imaging utilizes the propagation of sound waves in a medium, generating images based on the pulse-echo principle. An inherent characteristic of this method is that at the interface between two media with very different acoustic impedances (e.g., tissue and air), almost all incident energy is reflected, preventing further penetration into the tissue and producing artifacts that can obscure the visibility of internal structures. For air-filled organs (such as the lungs or bones), these artifacts negatively impact the intuitiveness of the analyzed images, limiting the effectiveness of ultrasound in clinical practice. Furthermore, because ultrasound examinations are highly dependent on the physician's skill, their accuracy is often limited by the sonographer's actual operating techniques and clinical experience, thus restricting its value in assessing the condition. Nevertheless, the effectiveness of ultrasound in the diagnosis of pneumonia has been continuously confirmed in numerous studies in recent years. Lichtenstein et al.'s study evaluated the accuracy of lung ultrasound imaging (LUS) in diagnosing alveolar consolidation in critically ill subjects in the emergency department. This study aimed to eliminate the inconvenience and time loss of transferring subjects from the emergency department or intensive care unit (ICU) to the radiology department. The method was found to have a sensitivity of 90% and a specificity of 98% in 65 cases of alveolar comorbidities, demonstrating the potential of LUS as a non-invasive diagnostic method. Volpicelli et al.'s study found that LUS had a sensitivity of 85.3% and a specificity of 96.8% in diagnosing adenocarcinoma in situ (AIS) in cases of mild pneumonia. Their study showed that comet tail artifacts (B-lines) can not only be used to accurately diagnose AIS but also to rule out pneumothorax and pulmonary edema. In 2009, Parlamento et al. compared LUS with chest X-ray (CXR) and CT imaging, reporting an accuracy of 96.9%. Although their work was limited to 32 subjects, the subjects established a standard for LUS diagnosis of pneumonia. For rapid and accurate diagnosis, identifying specific features indicative of pneumonia in LUS is crucial. Xirouchaki et al. studied the manifestations of LUS and identified four clinical signs most frequently observed in LUS images of pneumonia subjects: pulmonary consolidation, positive air bronchus, pleural abnormalities, and pleural effusion. Pagano et al. expanded these criteria to include subpleural lung, alveolar syndrome with dynamic tracheography, and interstitial syndrome with three or more B lines. With a larger sample size and updated criteria, they obtained superiority over CT with a positive predictive value (PPV) of 0.838, a negative predictive value (NPV) of 0.960, a sensitivity of 0.985, and a specificity of 0.649.
[0004] Recently, research has attempted to build artificial intelligence models for COVID-19, using deep learning models for computer-aided diagnosis. The table below describes the main application directions of existing artificial intelligence models for COVID-19:
[0005]
[0006] The table below describes existing artificial intelligence products / platforms developed for COVID-19:
[0007]
[0008] The table below summarizes representative papers published from early to mid-2020 on computer-aided diagnosis of COVID-19 using deep learning models:
[0009]
[0010]
[0011] literature:
[0012] 1. Wang et al. 2020 "A deep learning algorithm using CT images to screen for Corona Virus Disease (COVID-19)."
[0013] 2. Chen et al. 2020 "Deep learning-based model for detecting 2019novelcoronavirus pneumonia on high-resolution computed tomography: a prospective study."
[0014] 3. Song et al. 2020 "Deep learning Enables Accurate Diagnosis of NovelCoronavirus (COVID-19) with CT images."
[0015] The aforementioned computer-aided diagnostic techniques for COVID-19 using deep learning models are applications of artificial intelligence in CT images. However, it's important to note that CT scans pose risks such as ionizing radiation, the need to transport critically ill patients, and the cumbersome process of instrument sterilization, thus requiring cautious clinical application. Furthermore, due to the rapid progression of COVID-19, repeated CT scans to monitor changes in the patient's condition are impractical, especially for isolated severely or critically ill patients. Therefore, how to achieve rapid and convenient dynamic imaging monitoring of pneumonia, particularly COVID-19, is an urgent problem that needs to be solved. Summary of the Invention
[0016] The purpose of this invention is to develop a remotely operated ultrasound robot system that integrates ultrasound artificial intelligence assessment and decision-making systems for pneumonia, especially COVID-19, by constructing an AI assessment and decision-making system for lung ultrasound and circulatory volume ultrasound and optimizing the ultrasound robot system.
[0017] Therefore, in a first aspect, the present invention provides a method for rapid assessment of pneumonia ultrasound images based on artificial intelligence, the method comprising:
[0018] 1) Obtain lung ultrasound images and circulatory volume ultrasound images of the subject, wherein the circulatory volume ultrasound images include left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images.
[0019] 2) Input the lung ultrasound image and circulatory volume ultrasound image of the subject into the training model, which includes a trained lung ultrasound image scoring model and a trained circulatory volume ultrasound image scoring model, to obtain the pneumonia assessment result of the subject.
[0020] In one implementation, the pneumonia is COVID-19.
[0021] In one implementation, in 2), the training model is a deep convolutional neural network model.
[0022] In a second aspect, the present invention provides a method for constructing a trained deep convolutional neural network model as described in the first aspect, the method comprising:
[0023] a) Obtain lung ultrasound images and circulatory volume ultrasound images from multiple subjects, the circulatory volume ultrasound images including left ventricular outflow tract ultrasound, left ventricular long-axis section ultrasound, and inferior vena cava long-axis section ultrasound.
[0024] b) The lung ultrasound images of the subject were scored on three dimensions: pleural line, solid lung area, and pleural effusion, and the scores were weighted to obtain the lung ultrasound score of the subject; the grading assessment of the left ventricular outflow tract, left ventricular long axis section, and inferior vena cava long axis section was obtained from the circulatory volume ultrasound images of the subject.
[0025] c) Constructing a lung ultrasound image scoring model: Using the lung ultrasound images of the multiple subjects as the original images, input the deep convolutional neural network model for feature extraction. The extracted feature maps are input into three sub-tasks: classification of pleural lines, segmentation of solid lung areas, and segmentation of pleural effusion. The parameters of the model are adjusted based on the feedback results of the sub-tasks. The process is iterated until the loss function converges. Finally, the output is obtained and the model parameters are fixed to obtain a deep convolutional neural network model that can provide lung ultrasound scores for input ultrasound images.
[0026] d) Constructing a circulatory volume ultrasound image scoring model: Using left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images as raw image inputs, the input images are segmented, and the left ventricular outflow tract Vmax, VTI, and SV are obtained based on the segmentation results; left ventricular volume (LV) is calculated. d The study calculated the inferior vena cava (LVs) and the inferior vena cava IVCd and ΔIVC, and obtained a weighted score of circulatory volume ultrasound for the subjects.
[0027] In one implementation, in b), the subject's lung ultrasound images are scored and the circulatory volume ultrasound images are graded by combining the subject's lung CT images and diagnostic data scores.
[0028] In one implementation, in c), the feature extraction is performed through multiple convolutional layers and activation layers, which are interconnected to extract more abstract and discriminative features from the input image step by step.
[0029] In one implementation, in d), the UNet structure is used to segment the image by combining feature extraction and upsampling.
[0030] In one implementation, in 1) or a), the lung ultrasound images are obtained from parallel intercostal multi-point scanning of multiple lung regions.
[0031] In one implementation, the plurality of lung regions are 12 lung regions.
[0032] In one implementation, the plurality of lung regions are located on the left and right anterior chest walls, lateral chest walls, and posterior chest walls.
[0033] In one embodiment, in 1) or a), the circulatory volume ultrasound image includes a dynamic graph.
[0034] In one implementation, the dynamic graph includes three cardiac cycles.
[0035] In one implementation, in c), the deep convolutional neural network is a lightweight deep convolutional neural network.
[0036] In one implementation, in c), the deep convolutional neural network uses a ResNet-50 model, consisting of parameter-shared convolutional layers and a region proposal network, with the original image fed into the convolutional layers to generate feature representations for all subsequent tasks.
[0037] In one implementation, the region proposal network uses bounding boxes with predefined size ratios to search feature maps and then outputs multiple sets of rectangular candidate regions. The candidate regions are resized to the same size using bilinear interpolation and fed into the head structure, which outputs classification and segmentation results respectively.
[0038] In one implementation, the diagnostic data includes clinical data.
[0039] In one implementation, the clinical data includes data collected according to three scales: the Pulmonary Infection Score (CPIS score), the Sequential Organ Failure Score (SOFA score), and the Acute Physiology and Chronic Health Evaluation II (APACHE II score).
[0040] In a third aspect, the present invention provides a trained deep convolutional neural network model, which is constructed by the method described in the second aspect of the present invention.
[0041] In a fourth aspect, the present invention provides a remotely operated ultrasound robot suitable for an integrated ultrasound artificial intelligence assessment and decision-making system for pneumonia, comprising a deep convolutional neural network model trained according to a third aspect of the present invention.
[0042] This invention constructs a deep convolutional neural network model for rapid assessment of pneumonia ultrasound images based on artificial intelligence. It simplifies and rapidly assesses lung ultrasound features in pneumonia and dynamically evaluates the patient's condition using a lung ultrasound score (LUS), assisting in clinical treatment decisions. This invention can easily and rapidly assess changes in circulating volume in patients with pneumonia, especially severe and critically ill patients with COVID-19, dynamically monitor the progression and outcome of hypoxic respiratory failure and acute respiratory distress syndrome (ARDS), and enable real-time ultrasound examinations remotely controlled in isolation environments. While effectively protecting medical personnel, it provides rapid assessment of the condition of patients with pneumonia, especially COVID-19, and offers a timely intelligent early warning model for critically ill patients with multiple organ dysfunction. Attached Figure Description
[0043] Figure 1 A schematic diagram of the location of a 12-zone lung ultrasound examination method according to one implementation is shown: the lungs are divided into bilateral anterior superior (1), anterior inferior (2), lateral superior (3), lateral inferior (4), posterosuperior (5), and posteroinferior (6) lungs with the parasternal line, anterior axillary line, posterior axillary line, and paravertebral line as the longitudinal axis and the horizontal line below the nipple as the transverse axis.
[0044] Figure 2The schematic diagram illustrates the architecture of a pneumonia ultrasound image scoring model according to one implementation scheme: the model consists of a feature extractor (multiple convolutional pooling layers) and a head structure; the overall task is divided into three sub-tasks: automatic B-line classification, lung solid region segmentation, and automatic dark fluid region segmentation, and the area of the segmented region is calculated and the corresponding classification judgment is given based on Table 1, and finally the weighted total score of the three sub-tasks is given.
[0045] Figure 3 The diagram illustrates the structure of a cyclic volumetric ultrasound image segmentation model according to one implementation: the UNet network consists of a contraction path (left) and an expansion path (right). The fully symmetrical U-shaped structure allows for better fusion of image features. The upsampling portion contains a large number of feature channels, enabling the propagation of global features to higher resolution layers, resulting in more accurate segmentation. The corresponding area and proportion data are calculated based on the segmentation results. "Acquisition" refers to directly utilizing the segmentation results, while "calculation" refers to calculating the area of the segmented region after the segmentation results are obtained. The left ventricular outflow tract parameters Vmax, VTI, and SV are acquired; left ventricular LV is calculated. d LVs and calculation of inferior vena cava IVCd and ΔIVC.
[0046] Figure 4 This paper illustrates the overall technical approach of using artificial intelligence scoring of cardiopulmonary ultrasound in subjects to determine the severity of their diseases in real time, according to one implementation scheme.
[0047] Figure 5 An exemplary Acute Physiology and Chronic Health Status (APACHE II) score is shown. Detailed Implementation
[0048] The inventors investigated the application of ultrasound imaging in COVID-19 positive subjects during the progression, severe, and recovery phases. There is a need for lung ultrasound examinations and remote diagnosis of COVID-19 patients, and clinically, there is a need for real-time, dynamic, and effective image monitoring of lung lesions in severely ill COVID-19 patients. Adhering to the principles of "problem-oriented, information technology-supported, and model-innovative," the inventors integrated various resources to propose a targeted solution for implementing lung ultrasound examinations and remote diagnosis of COVID-19, enabling the widespread and efficient application of lung ultrasound technology in a short period. This invention applies lung ultrasound to the diagnosis and treatment of COVID-19 in different regions and utilizes a 5G remote consultation platform to achieve rapid collaboration across institutions, regions, and specialties.
[0049] The inventors discovered that severely or critically ill patients are prone to acute respiratory distress syndrome (ARDS) in addition to lung lesions. In such cases, besides the lung lesions themselves, systemic circulatory volume assessment is particularly important for monitoring the condition and guiding treatment. Therefore, the inventors proposed using ultrasound examination to dynamically assess the pulmonary edema and consolidation, pleural, peritoneal, and pericardial effusions, cardiac function and structure, and vascular capacity and patency in COVID-19 patients in real time. First, a comprehensive ultrasound examination of the lungs, heart, inferior vena cava, liver, and kidneys is performed on each patient. Then, the focus is on collecting images related to lung ultrasound and circulatory volume ultrasound. Simultaneously, multiple time points are used for each patient, with lung ultrasound generally performed every 3 days. If there are sudden changes in the condition, additional examinations are performed as needed to dynamically track and observe changes in the patient's condition. Figure 4 This paper illustrates the overall technical approach for using artificial intelligence scoring of cardiopulmonary ultrasound in subjects to assess the severity of their diseases in real time, according to one implementation scheme. The following provides a detailed description of data acquisition and data processing.
[0050] The training of the model of this invention used the following data: it was conducted at Zhejiang Provincial People's Hospital / Hainan 301 Hospital, with a total of 48 subjects.
[0051] 1. Acquisition and annotation of ultrasound data of lungs and circulatory volume, and clinical data.
[0052] Frames were extracted from all ultrasound videos from the subjects, and doctor-labeled and AI model learned. After the ultrasound images were input into the model, the model scored each image. If the input was an ultrasound image of a subject, all frames of the ultrasound image were scored, and the maximum score was selected and added in different lung areas to obtain the total score. If the input video was a cardiac ultrasound, the region of interest (ROI) was segmented for each frame and scored according to the segmentation results. The maximum score is given in Table 2.
[0053] 1) Acquisition of lung ultrasound image data.
[0054] The subjects underwent examinations of 12 lung regions, including the upper and lower parts of the left and right anterior chest walls, lateral chest walls, and posterior chest walls (e.g., ...). Figure 1 (As shown); each zone is scanned using a convex array probe with parallel intercostal scanning at multiple points, and the most typical and severe intercostal areas are selected for image storage. The ultrasound two-dimensional images must be clear and uniform, and should not contain color Doppler or any form of measurement data or annotations.
[0055] Each region was scored according to three categories: pleural line, solid lung area, and pleural effusion (as shown in Table 1). The scores of the 12 lung regions (upper and lower parts of the bilateral anterior, lateral, and posterior chest walls) on the A / B line, the depth of the solid lung area, and the depth of the pleural effusion were weighted to obtain the lung ultrasound score (LUS).
[0056] Table 1 Lung ultrasound scoring criteria
[0057]
[0058] 2) Acquisition of circulatory volume ultrasound data.
[0059] Dynamic ultrasound images of the apical five-chamber view, left ventricular long-axis view, and inferior vena cava long-axis view, encompassing three cardiac cycles, were retained. The following relevant data were obtained and graded for evaluation (as shown in Table 2):
[0060] (1) Obtain the Doppler spectrum of the blood flow signal in the left ventricular outflow tract from the apical five-chamber view, and measure the peak velocity (Vmax), blood flow velocity integral (VTI), and stroke volume (SV) of the left ventricular outflow tract.
[0061] (2) Obtaining the diastolic diameter of the left ventricle (LV) from the long axis section of the left ventricle. d ), systolic inner diameter (LV) s );
[0062] (3) Obtain the inner diameter of the inferior vena cava (IVC) from the long axis section behind the left liver. d ) and variability (△IVC).
[0063] Table 2. Ultrasonic Evaluation Indicators of Circulatory Capacity
[0064]
[0065] 3) Collection of diagnostic data from test subjects.
[0066] Epidemiological data, clinical data, and lung CT images of the subjects were collected to provide a basis for clinical staging and treatment.
[0067] (1) Epidemiological data include gender, age, etc.
[0068] (2) For lung CT, at least two CT scan images of the upper and lower lungs corresponding to the ultrasound acquisition plane should be saved.
[0069] (3) Clinical data will be calculated according to the Pulmonary Infection Score (CPIS, Table 3), Sequential Organ Failure Score (SOFA score, Table 4), and Acute Physiology and Chronic Health Evaluation Score (APACHE II score, see [reference]). Figure 5 Data was collected using three scales: CPIS, SOFA, and APACHE II. These scores were not directly used for model parameters, but rather to assess the correlation or consistency between these scores and the model's results, thereby validating the model's effectiveness.
[0070] Table 3 Clinical Pulmonary Infection Score (CPIS)
[0071]
[0072] Note: For tracheal aspirate culture or sputum culture, 0 points are awarded for no pathogenic bacteria growth; 1 point is awarded for pathogenic bacteria growth; 2 points are awarded for two cultures showing the same bacteria or for Gram staining consistent with the culture.
[0073] Table 4 Sequential Organ Failure Assessment (SOFA)
[0074]
[0075]
[0076] SOFA rating: _________
[0077] 2. Design of the ultrasound image scoring model architecture.
[0078] 1) Lung ultrasound image scoring model: Each region is scored according to three dimensions: pleural line, solid lung area, and pleural effusion (see Table 1). A feature extractor corresponding to the above ultrasound features is established, and a lightweight deep convolutional neural network design is adopted to ensure the real-time performance of the scoring. Based on the scoring criteria, the model's tasks are divided into three sub-tasks: a classification task for the B-line (classifying data according to…). Figure 5 The tasks shown are divided into 0-3 points, segmentation of solid lung areas, and segmentation of pleural effusion. Furthermore, the system automatically classifies the solid and effusion areas based on their segmentation depth (also based on...). Figure 5 The categories shown are respectively divided into 0-3 points. Figure 2The ResNet-50 model was used as the backbone network for the lung ultrasound image scoring model. High-dimensional features were extracted from the image through stacked convolutional and pooling layers. After feature extraction, a linear classifier was added to classify the image to complete the classification task. The classification labels were provided by the physician. The backbone network consisted of parameter-shared convolutional layers (for feature extraction) and a region proposal network (for generating candidate regions for detection). The original image was fed into the convolutional layers to generate feature representations for all subsequent tasks (e.g., segmentation and classification). The region proposal network used bounding boxes with predefined size ratios to search for feature maps and then output multiple sets of rectangular candidate regions. The candidate regions (ROIs) were resized to the same size using bilinear interpolation and fed into the head structure, which then output classification and segmentation results respectively. Using lung ultrasound images as the original image input, the data is processed through a feature extractor, which consists of multiple convolutional and activation layers interconnected. This process extracts more abstract and discriminative features from the input image step by step. These learned feature maps are then input into three sub-tasks. Based on the feedback from the sub-tasks, the parameters of the feature layers are adjusted. This process iterates until the loss function converges. Finally, the output is obtained, and the model parameters are fixed to produce a deep model that can provide a comprehensive score of 0-9 for the input ultrasound image.
[0079] 2) Circulatory Capacity Ultrasound Image Scoring Model: Using left ventricular outflow tract spectral Doppler ultrasound images, left ventricular long-axis section ultrasound images, and left hepatic posterior inferior vena cava long-axis ultrasound images as raw image inputs, a UNet structure is used combined with feature extraction and upsampling to segment the images (image segmentation refers to the technique and process of distinguishing the foreground and background of an image to divide it into several specific regions with unique properties and extract the target of interest. Here, it refers to extracting anatomical structures appearing in specific ultrasound images from the image). With a certain amount of data and physician labeling, the model learns the physician's prior knowledge to segment lesions or organs in unseen medical images. It can also provide relatively accurate segmentation results with a small number of samples. Based on the segmentation results, the left ventricular outflow tract Vmax, VTI, and SV can be obtained; and the left ventricular LV can be calculated. d LVs and calculation of inferior vena cava IVCd, ΔIVC ( Figure 3 Using the above quantitative data (Vmax, VTI, SV, LV) d LV s The weighted scores of circulatory volume ultrasound for the subjects were obtained by referring to Table 2 (IVCd, ΔIVC).
[0080] 3) Development of AI-based assessment and decision-making models for lung ultrasound and circulating volume ultrasound
[0081] The model of this invention is validated through multidimensional information verification by calculating scores based on three dimensions of lung ultrasound images: pleural line (and A / B line), solid lung area, and pleural effusion; left ventricular outflow tract (Vmax), VTI, and SV; and inferior vena cava (IVCd) and ΔIVC. These scores are then compared with scores from multiple clinical follow-up data, including CT images (see Table 5), APACHE II score, SOFA score, and CPIS score, ultimately yielding an intelligent assessment result for pneumonia. The severity of the subject's illness is assessed by evaluating the correlation between the model's results and other scoring systems, thereby determining whether a positive correlation exists between the results and the final score of the model.
[0082] Table 5. Comparison of CT and Ultrasound Image Features of COVID-19
[0083]
[0084] Table 5 shows a one-to-one correspondence between the main signs of COVID-19 on CT and ultrasound, indicating that ultrasound can also be used to assess COVID-19 without missing the main features of the COVID-19 images.
[0085] Based on automated 4D quantitative assessment using multi-time point data, the changes in the number, volume, and density of lesions in subjects at different stages (disease course) are tracked for auxiliary analysis and modeling. By utilizing the correlation modeling and analysis of lesions at different time points, treatment efficacy and prognosis can be grouped and analyzed, and a medical treatment plan suggestion model can be constructed to help quickly determine the changes in the subject's condition and treatment efficacy.
[0086] Example
[0087] In the applicant's two partner hospitals, remote real-time ultrasound examinations were conducted to provide diagnostic opinions for the 2019-nCoV epidemic, avoiding the infection risks associated with ultrasound physicians performing bedside examinations and maximizing the safety of medical staff. The inventors optimized and improved the artificial intelligence assessment and decision-making model for lung ultrasound and circulatory volume ultrasound, embedding it into a remote ultrasound robot system. This enables automatic identification and judgment of lung ultrasound and circulatory volume ultrasound during the robot scan process, providing treatment suggestions. The system of this invention underwent practical testing in multiple partner hospitals in February 2020, establishing a rapid lung ultrasound assessment system for COVID-19 based on artificial intelligence and robotics technology, and completing AI application testing for cardiopulmonary ultrasound function. The remote ultrasound robot was deployed for testing in multiple partner hospitals in Wuhan, and assisted experts from hospitals in Zhejiang Province, Beijing, and Hainan Province in conducting remote consultations for subjects, achieving good results.
[0088] The inventors trained and validated the classifier on 803 ultrasound images showing B-lines (comet sign) and tested it on 189 images. After 200 iterations, they finally obtained a relatively ideal classifier with an accuracy of 98.88%. It is important to note that although the boundaries of the inventors' segmentation results differ somewhat from the actual boundaries, the inventors' ultimate goal was to classify ultrasound images containing solid lung areas and effusion areas based on region depth. In the 176 test images used, no images were misclassified.
Claims
1. A rapid assessment method for pneumonia ultrasound images based on artificial intelligence, the method comprising: 1) Obtain lung ultrasound images and circulatory volume ultrasound images of the subject, wherein the circulatory volume ultrasound images include left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images; 2) Input the lung ultrasound images and circulatory volume ultrasound images of the subject into the training model. The training model includes a trained lung ultrasound image scoring model and a trained circulatory volume ultrasound image scoring model to obtain the pneumonia assessment results of the subject. The training model is a deep convolutional neural network model, which is constructed through the following steps: a) Obtain lung ultrasound images and circulatory volume ultrasound images from multiple subjects, wherein the circulatory volume ultrasound images include left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images. b) The lung ultrasound images of the subject were scored on three dimensions: pleural line, solid lung area, and pleural effusion, and the scores were weighted to obtain the lung ultrasound score of the subject; the grading assessment of the left ventricular outflow tract, left ventricular long axis section, and inferior vena cava long axis section was obtained from the circulatory volume ultrasound images of the subject. c) Constructing a lung ultrasound image scoring model: Using the lung ultrasound images of the multiple subjects as the original images, a deep convolutional neural network model is input for feature extraction. The extracted feature maps are input into three sub-tasks: classification of pleural lines, segmentation of solid lung areas, and segmentation of pleural effusion. The parameters of the model are adjusted based on the feedback results of the sub-tasks. The process is iterated until the loss function converges. Finally, the output is obtained and the model parameters are fixed to obtain a deep convolutional neural network model that can provide a lung ultrasound score for the input ultrasound image. d) Constructing a circulatory volume ultrasound image scoring model: Using left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images as raw image inputs, the input images are segmented, and the left ventricular outflow tract Vmax, VTI, and SV are obtained based on the segmentation results; left ventricular volume (LV) is calculated. d The study calculated the inferior vena cava (LVs) and the inferior vena cava IVCd and ΔIVC, and obtained a weighted score of circulatory volume ultrasound for the subjects.
2. A method for constructing a deep convolutional neural network model trained according to claim 1, the method comprising: a) Obtain lung ultrasound images and circulatory volume ultrasound images from multiple subjects, wherein the circulatory volume ultrasound images include left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images. b) The lung ultrasound images of the subject were scored on three dimensions: pleural line, solid lung area, and pleural effusion, and the scores were weighted to obtain the lung ultrasound score of the subject; the grading assessment of the left ventricular outflow tract, left ventricular long axis section, and inferior vena cava long axis section was obtained from the circulatory volume ultrasound images of the subject. c) Constructing a lung ultrasound image scoring model: Using the lung ultrasound images of the multiple subjects as the original images, a deep convolutional neural network model is input for feature extraction. The extracted feature maps are input into three sub-tasks: classification of pleural lines, segmentation of solid lung areas, and segmentation of pleural effusion. The parameters of the model are adjusted based on the feedback results of the sub-tasks. The process is iterated until the loss function converges. Finally, the output is obtained and the model parameters are fixed to obtain a deep convolutional neural network model that can provide a lung ultrasound score for the input ultrasound image. d) Constructing a circulatory volume ultrasound image scoring model: Using left ventricular outflow tract ultrasound images, left ventricular long-axis section ultrasound images, and inferior vena cava long-axis section ultrasound images as raw image inputs, the input images are segmented, and the left ventricular outflow tract Vmax, VTI, and SV are obtained based on the segmentation results; left ventricular volume (LV) is calculated. d The study calculated the inferior vena cava (LVs) and the inferior vena cava IVCd and ΔIVC, and obtained a weighted score of circulatory volume ultrasound for the subjects.
3. The method according to claim 1 or 2, in d), the input image is segmented using a UNet structure combined with feature extraction and upsampling.
4. The method according to claim 1 or 2, in b), the lung ultrasound images of the subject are scored and the circulatory volume ultrasound images are graded in combination with the subject's lung CT images and diagnostic data scores.
5. In the method according to claim 1 or 2, in c), the feature extraction is performed through multiple convolutional layers and activation layers, which are interconnected to extract more abstract and discriminative features from the input image step by step.
6. The method according to claim 1 or 2, wherein in 1) or a), the lung ultrasound images are obtained from parallel intercostal multi-point scanning of multiple lung regions.
7. The method according to claim 6, wherein the plurality of lung regions comprises 12 lung regions.
8. The method according to claim 6, wherein the plurality of lung regions are located on the left and right anterior chest walls, lateral chest walls, and posterior chest walls.
9. The method according to claim 1 or 2, wherein in 1) or a), the circulatory volume ultrasound image includes a dynamic graph.
10. The method according to claim 9, wherein the dynamic graph comprises 3 cardiac cycles.
11. The method according to claim 1 or 2, wherein in c), the deep convolutional neural network is a lightweight deep convolutional neural network.
12. The method of claim 11, wherein the deep convolutional neural network uses a ResNet-50 model and consists of parameter-shared convolutional layers and a region proposal network, wherein the original image is fed into the convolutional layers to generate feature representations for all subsequent tasks.
13. The method according to claim 12, wherein the region proposal network uses bounding boxes with a set size ratio to search feature maps and then outputs multiple sets of rectangular candidate regions; the candidate regions are resized to the same size by bilinear interpolation and are fed into the head structure so that it outputs classification results and segmentation results respectively.
14. The method according to claim 4, wherein the diagnostic data includes clinical data, and the clinical data includes: Data were collected using three scales: pulmonary infection score, sequential organ failure score, and acute physiology and chronic health status score.
15. The method according to claim 1, wherein the pneumonia is COVID-19.
16. A remotely operated ultrasound robot for an integrated ultrasound artificial intelligence assessment and decision-making system for pneumonia, which uses the artificial intelligence-based rapid assessment method for pneumonia ultrasound images according to any one of claims 1, 3-15.
17. The remotely operated ultrasound robot according to claim 16, wherein the pneumonia is COVID-19.