Novel coronavirus infection quantitative evaluation system

Abstract

The invention relates to a novel coronavirus infection quantitative evaluation system, which comprises an image acquisition and preprocessing module used for acquiring and preprocessing a lung ultrasound image; the stealth characterization module is used for taking the preprocessed lung ultrasound image as input and extracting stealth characterization of the lung ultrasound image; the knowledge extraction module is used for labeling the preprocessed lung ultrasound image and extracting features capable of reflecting the lung condition from a labeling area as priori knowledge; and the fusion module is used for fusing the stealth representation and the priori knowledge and quantitatively evaluating the infection severity of the novel coronavirus according to a fusion result. The method can be widely applied to the field of data processing.

Description

Technical Field

[0001] The present invention relates to a novel coronavirus infection quantitative assessment system, and belongs to the field of data processing. Background Art

[0002] According to the World Health Organization (WHO) definition of the severity of novel coronavirus infection, treatment methods will vary depending on the severity of novel coronavirus infection. Accurate diagnosis of the severity of novel coronavirus infection, including mild, moderate, severe and critical, is crucial for patient treatment and resource allocation, especially in distinguishing between severe (severe and critical) and non-severe (mild and moderate) cases that require different intervention strategies.

[0003] Currently, reverse transcription polymerase chain reaction (RT-PCR) is the gold standard for confirming SARS-CoV-2 infection. However, it has the disadvantages of long detection time, low sensitivity and inability to assess disease severity. Although chest computed tomography (CT) and X-ray are relatively reliable in diagnosing novel coronavirus infection, the ionizing radiation from these methods poses risks to patient groups including children, pregnant women, and those who need multiple examinations in a short period of time. In addition, the lack of portability makes them unsuitable for bedside follow-up of critically ill patients.

[0004] In contrast, lung ultrasound (LUS) has been widely used in the assessment of the lung status of patients with 2019-nCoV infection due to its unique advantages, such as portability, cost-effectiveness, real-time imaging, and safety. In recent years, LUS has been increasingly used for monitoring patients with 2019-nCoV infection and for triage of symptomatic individuals. It is particularly useful in identifying interstitial pneumonia associated with 2019-nCoV infection and tracking its progression. LUS imaging captures pathological lung conditions through various imaging artifact patterns in B-mode images. In the context of diagnosis and treatment of 2019-nCoV infection, specific LUS modes have been used to assess the severity of lung conditions, which is reflected in changes in the pleural line (PL), fusion of B lines (BLs), and the presence of lung consolidation. However, the assessment of the severity of 2019-nCoV infection mainly relies on the qualitative and subjective judgment of clinicians, which is a limitation that requires attention.

[0005] Semi-quantitative methods based on machine learning have emerged for the diagnosis, evaluation, and monitoring of patients infected with the novel coronavirus. Some methods emphasize features related to PL and BLs separately, while ignoring the combination of PL and BLs. The combination of PL and BLs may improve the evaluation performance. Wang et al. extracted features related to PL and BLs to evaluate lung conditions and assessed the severity of novel coronavirus infection, but other information contained in LUS images has not been utilized. These studies evaluated the severity of novel coronavirus infection by utilizing prior knowledge, especially focusing on PL and BLs, without using global LUS images.

[0006] Recently, automatic image analysis based on deep learning (DL) methods has shown great promise in ultrasound image classification, segmentation, and reconstruction. In addition, automatic diagnosis systems using DL methods have shown potential in accurately diagnosing the severity of novel coronavirus infection. However, the scarcity of training data is a common challenge in DL-based medical image analysis, making it more difficult to effectively train complex models. Many models have been introduced to quantify the severity of LUS images at the frame or video level. Although these models can accurately classify LUS images or videos, they are not suitable for tracking disease progression or assessing the severity of patients. This limitation stems from the fact that there is no direct connection between the severity of clinical symptoms of patients infected with the novel coronavirus and the severity of frames or videos. An aggregation step is required to assess the severity of patients infected with the novel coronavirus.

[0007] Some studies aim to classify the severity of COVID-19 infection at the patient level or examination level. However, the accuracy of these methods is lower than that of frame-level or video-level analysis. This is because, first, learning-based methods require a large number of labeled samples from patients infected with COVID-19, which are very difficult to obtain. Compared with frame-level or video-level severity assessment, the data scale at the examination level or patient level is much smaller. In addition, LUS images are particularly susceptible to significant data heterogeneity, which is affected by various factors such as differences in clinical experience, scanning angles, and different ultrasound equipment. Second, there is a lack of prior knowledge or guidance in other fields. Although Frank et al. introduced knowledge into their model, the pixel-level knowledge embedded in the mask image was fused through splicing in the channel, which made it more difficult for the neural network to learn the implicit guidance information in the LUS image. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a quantitative assessment system for novel coronavirus infection, which evaluates the severity of novel coronavirus infection based on LUS examination data through a framework that integrates knowledge and invisible representation.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A novel coronavirus infection quantitative assessment system, comprising:

[0011] Image acquisition and preprocessing module, used to collect the i-th lung ultrasound examination E i The lung ultrasound images in the image were preprocessed;

[0012] An invisible representation module is used for extracting invisible representation of the preprocessed lung ultrasound image as input;

[0013] A knowledge extraction module is used to annotate the preprocessed lung ultrasound image and extract features that can reflect the lung condition from the annotated area as prior knowledge;

[0014] The fusion module is used to fuse the invisible representation and prior knowledge, and quantitatively evaluate the severity of the new coronavirus infection based on the fusion results.

[0015] Furthermore, the image acquisition and preprocessing module includes an image acquisition module, an image depth compensation module, an image normalization module and an image correction module;

[0016] The image acquisition module is used to screen key frames from the lung ultrasound data acquisition video to obtain the i-th lung ultrasound examination E i n i Ultrasound image of the lungs;

[0017] The image depth compensation module is used to calibrate the size of each lung ultrasound image according to the imaging depth, so that the calibrated lung ultrasound images have a consistent size;

[0018] The image normalization module is used to normalize the pixel intensity of the lung ultrasound image according to the imaging depth and the number of pixels of the central column of the detected lung area;

[0019] The image correction module is used to convert the lung ultrasound image in the xy Cartesian coordinate system into the polar coordinate system.

[0020] Furthermore, in the image correction module, the process of converting the lung ultrasound image in the xy Cartesian coordinate system into the polar coordinate system is:

[0021] The normalized lung ultrasound images acquired using a convex array probe were converted from a Cartesian coordinate system to a polar coordinate system;

[0022] Identify the sector area in the LUS image and determine the center point O' of the sector arc;

[0023] The lung region is mapped from Cartesian coordinates to polar coordinates with the center point O' as the origin.

[0024] Further, the invisible representation module includes a pre-trained model module and a representation vector mapping module;

[0025] The pre-trained model module is used to use the pre-trained model to i n i Each lung ultrasound image is represented by an m k ×n i The feature matrix of

[0026] The representation vector mapping module is used to represent the i-th lung ultrasound examination E i All feature matrices in are processed to obtain the invisible representation of Ei.

[0027] Furthermore, the pre-training model adopts RegNet, DenseNet, EfficientNet, GoogLeNet, ResNet, Swin Transformer or VGG.

[0028] Further, the representation vector mapping module includes an average pooling and concatenation layer, a channel linear layer and a dimension reduction module;

[0029] The average pooling and concatenation layers are used to transform E i All feature matrices of are concatenated on the feature dimension and processed to obtain the deep feature vector f i ;

[0030] The channel linear layer is used to transform the depth feature vector f i Perform scale normalization and screening;

[0031] The dimension reduction module is used to reduce the dimension of the output feature vector of the channel linear layer to obtain a x with a size of 900×1 i As E i Invisible representation.

[0032] Further, the knowledge extraction module includes a preliminary annotation module, an annotation verification module and a knowledge embedding module;

[0033] The preliminary annotation module is used by biomedical engineers to examine the i-th LUS i All lung ultrasound images are annotated with regions of interest, including pleural line region annotation and B-line region annotation;

[0034] The marking column verification module is used by clinicians to verify the marked region of interest to obtain the final region of interest;

[0035] The knowledge embedding module is used to extract features related to the pleural line and B line from the final region of interest of all lung ultrasound images and then take the average value to generate the i-th lung ultrasound examination Ei Prior knowledge k i .

[0036] Furthermore, the features related to the pleural line include pleural line thickness, pleural line roughness, pleural line intensity average and pleural line intensity standard deviation; the features related to the B line include the number of B lines, the cumulative width of the B line, the attenuation coefficient of the B line, the cumulative intensity of the B line and texture features.

[0037] Further, the knowledge hidden representation fusion module includes a knowledge fusion module, a self-attention module and a severity assessment module;

[0038] The knowledge fusion module is used to fuse the invisible representation and the prior knowledge to form a coding set including the knowledge and the invisible representation;

[0039] The self-attention module is used to further encode the encoding set to represent it as a whole;

[0040] The severity assessment module is used to assess the severity of the novel coronavirus infection based on the output of the self-attention module to obtain an assessment result.

[0041] Further, the knowledge fusion module includes M stacked cross-space fusion modules and a connection layer;

[0042] Each of the cross-space fusion modules captures the invisible representation x i and knowledge i The mutual and self-connection between these two inputs enables the fusion of the two;

[0043] The connection layer is used to connect the fusion features output by the M cross-space fusion modules to form a coding set containing knowledge and invisible representation.

[0044] The present invention adopts the above technical solution, which has the following advantages:

[0045] 1. This paper combines the invisible representation of LUS examination with the doctor's prior knowledge, and uses the KFLR model based on the Transformer architecture to evaluate the severity of the new coronavirus infection, thereby improving the accuracy of the evaluation;

[0046] 2. The present invention uses a pre-trained model to preliminarily encode the LUS inspection image instead of splitting the image into multiple image blocks. This method significantly improves the classification performance because the pre-trained model is trained on a large data set and has a strong feature extraction capability, thereby reducing the amount of data required for training effective encoding. The invisible representation captures global information from the entire LUS inspection, enhancing the performance of the model;

[0047] 3. The present invention uses multiple pre-trained models to encode LUS inspection images, which represent different architectures and extract diverse features;

[0048] 4. The present invention uses the KFLR model to integrate the knowledge extracted by clinicians with the implicit representation, and uses KFM with intra-spatial and cross-spatial attention mechanisms to reveal the cross-relationship and intrinsic connection between the implicit representation and knowledge. This innovative fusion significantly improves the overall performance of severity assessment.

[0049] Therefore, the present invention can be widely applied in the field of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:

[0051] Figure 1 This is a structural diagram of a quantitative assessment system for novel coronavirus infection provided by an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of an invisible characterization module provided by an embodiment of the present invention;

[0053] Figure 3 (a)~ Figure 3 (f) is a schematic diagram of knowledge extraction provided by an embodiment of the present invention, wherein: Figure 3 (a) is the normalized LUS image in the Cartesian coordinate system; Figure 3 (b) is the corrected LUS image in the polar coordinate system; Figure 3 (c) is the pleural line (PL) area; Figure 3 (d) is the B-line (BLs) area; Figure 3 (e) are features related to PL; Figure 3 (f) are the features related to the B line;

[0054] Figure 4 (a)~ Figure 4 (c) is the architecture of the knowledge and hidden representation fusion (KFLR) model provided by an embodiment of the present invention, wherein: Figure 4 (a) is the structure of KFM; Figure 4 (b) is the self-attention module; Figure 4 (c) is the severity assessment head;

[0055] Figure 5 is a Sankey diagram of the distribution of imaging parameters provided by an embodiment of the present invention;

[0056] Figure 6 (a)~ Figure 6 (b) is a schematic diagram of ROC curve comparison provided by an embodiment of the present invention;

[0057] Figure 7 (a)~ Figure 7 (b) is a receiver operating characteristic curve of the ablation experiment provided by an embodiment of the present invention, wherein: Figure 7 (a) is the four-level severity assessment of the novel coronavirus infection. Figure 7 (b) is the secondary novel coronavirus infection severity assessment; K, I, L, and F represent features from knowledge, LUS images, LRM, and KFM, respectively. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] The severity assessment of novel coronavirus infection is of great clinical significance. Lung ultrasound (LUS) plays a key role in the severity assessment of novel coronavirus infection due to its safety and portability. However, the severity assessment of novel coronavirus infection based on LUS relies on the qualitative and subjective observation of clinicians, which has certain limitations. In addition, LUS images usually show significant heterogeneity, but there is no quantitative assessment method based on LUS examination data.

[0061] Given the outstanding performance of deep learning in image understanding and considering the challenges associated with expertise requirements and data scarcity, a logical approach is to leverage the rich information in LUS images while incorporating clinician expertise to guide the assessment process.

[0062] Based on this, in some embodiments of the present invention, a quantitative assessment system for novel coronavirus infection is provided, which proposes a framework that integrates knowledge and invisible representation, and is designed specifically for assessing the severity of novel coronavirus infection based on LUS examination data. The framework converts LUS examination data into invisible representations, and with the help of clinicians' expertise, extracts features reflecting the condition of the lungs from the PL and BLs of the marked areas as knowledge to improve the accuracy of the assessment. In order to integrate knowledge into the invisible representation, we adopted the knowledge invisible representation fusion (KFLR) model. Compared with methods that lack prior knowledge, the present invention can significantly reduce assessment errors. It has application potential in monitoring disease progression and stratifying patients with novel coronavirus infection cases.

[0063] Example 1

[0064] like Figure 1 As shown, the present invention provides a novel coronavirus infection quantitative assessment system, which includes:

[0065] Image acquisition and preprocessing module, used to acquire LUS images and perform preprocessing;

[0066] An invisible representation module is used to extract the invisible representation of the preprocessed LUS image as input;

[0067] The knowledge extraction module is used to annotate the preprocessed LUS images and extract features that can reflect the lung conditions from the annotated areas as prior knowledge;

[0068] The fusion module is used to fuse the invisible representation and prior knowledge, and quantitatively evaluate the severity of the new coronavirus infection based on the fusion results.

[0069] Furthermore, the image acquisition and preprocessing module includes an image acquisition module, an image depth compensation module, an image normalization module and an image correction module. The image acquisition module is used to filter key frames from the LUS data acquisition video to obtain the i-th LUS inspection E i n i The image depth compensation module is used to calibrate the size of each LUS image according to the imaging depth, so that the calibrated LUS images have the same size; the image normalization module is used to normalize the pixel intensity of the LUS image according to the imaging depth and the number of pixels of the central column of the detected lung area, that is, to normalize the pixel intensity of the LUS image to the range of 0 to 1; the image correction module is used to convert the LUS image in the xy Cartesian coordinate system into the polar coordinate system, so as to facilitate the subsequent marking of the BLs area.

[0070] Furthermore, in this embodiment, when the image depth compensation module calibrates the size of each LUS image, each pixel is calibrated to 0.256×0.256 mm. 2 This is because, during data acquisition, various imaging parameters (such as imaging depth and overall gain) are slightly adjusted to optimize the imaging field of view and image quality for different patients using different equipment.

[0071] Furthermore, in this embodiment, the image normalization module uses d I / p I / p a The pixel intensity of the LUS image is adjusted by a factor of I is the imaging depth, p I is the number of pixels, p a is the physical distance between pixels, which is 0.256 mm to minimize the impact of different overall gains.

[0072] Furthermore, in order to mark the area containing BLs, the LUS image in the xy Cartesian coordinate system needs to be converted to the polar coordinate system. The BLs start from the PL and extend radially to the bottom of the LUS image. In the xy Cartesian coordinate system, as the BLs are farther away from the center of the image, they tilt more toward the outer edge. However, after the coordinate conversion, the LUS image is converted to the polar coordinate system. In the ρ-θ polar coordinate system, the BLs are arranged vertically, such as Figure 1 This embodiment is implemented by an image correction module.

[0073] Specifically, the calibration process is as follows: First, the normalized LUS image acquired using the convex array probe is converted from the Cartesian coordinate system to the polar coordinate system. The normalized LUS image is displayed in Cartesian coordinates, corresponding to the horizontal position (x) and the vertical position (y), respectively, as shown in Figure 1 The image preprocessing box in the middle shows the xy Cartesian coordinates on the left. Secondly, the sector area is identified in the LUS image and the center point O' of the sector arc is determined. Then, the lung area is mapped from the Cartesian coordinates to the polar coordinates with the center point O' as the origin.

[0074] In the corrected LUS image, polar coordinates are used as horizontal and vertical coordinates. For example, the axial position ρ0 and beam deflection angle θ0 of a point P in the normalized LUS image are calculated. In the corrected LUS image, the pixel intensity at (ρ0, θ0) corresponds to the intensity of P in the normalized LUS image. Subsequently, these images are converted to polar coordinates, corresponding to the axial position (ρ) and the lateral position (beam deflection angle, θ), respectively, as shown in Figure 1 As shown in the right figure with ρ-θ polar coordinates marked in the image preprocessing box.

[0075] Furthermore, if Figure 2 As shown, the hidden representation module (LRM) includes a pre-trained model module and a representation vector mapping module. The pre-trained model module is used to project each LUS image from the original image space to the high-dimensional feature space using the pre-trained model, that is, the i-th LUS inspection E i n i LUS images (usually based on the M-BLUE protocol) i =8) are represented by m k ×n i The characterization vector mapping module is used to check the i-th LUS E i All feature matrices in are further processed to obtain the invisible representation of Ei.

[0076] Furthermore, the pre-trained model in the pre-trained model module can adopt RegNet, DenseNet, EfficientNet, GoogLeNet, ResNet, Swin Transformer and VGG, etc., which will not be described in detail in the present invention.

[0077] Furthermore, the representation vector mapping module includes an average pooling and concatenation layer, a channel linear layer and a dimensionality reduction module. Among them, the average pooling and concatenation layer are used to convert E i All feature matrices of are concatenated on the feature dimension and processed to obtain the deep feature vector f i , whose size is 42080×1; the channel linear layer is used to transform the depth feature vector f i The scale is normalized and screened to obtain the features that are crucial for assessing the severity of the novel coronavirus infection. The dimension reduction module consists of a fully connected layer and a dropout layer, which is used to reduce the dimension of the output feature vector of the channel linear layer to obtain a x matrix of size 900×1. i As the output, it is E i Invisible representation.

[0078] Furthermore, since the multiple deep features extracted from the i-th LUS examination Ei by different pre-trained models have different scales and attributes, in this embodiment, the channel linear layer has learnable parameters, which can normalize the scales of deep feature vectors fi of different scales.

[0079] Furthermore, the knowledge extraction module includes a preliminary annotation module, an annotation verification module and a knowledge embedding module. Among them, the preliminary annotation module is used by biomedical engineers to analyze the i-th LUS examination E iThe ROI (usually pathological artifacts) of all LUS images are annotated, including PL (pleural line) area annotation and BLs (B-line) area annotation; the column verification module is used for clinicians (with more than 10 years of clinical experience) to verify the annotated ROI and obtain the final ROI; the knowledge embedding module is used to extract the features related to PL and BLs from the final ROI of all LUS images and take the average value to generate the i-th LUS examination E. i Prior knowledge k i .

[0080] like Figure 3 (a)~ Figure 3 (f) shows the process diagram of knowledge extraction. Figure 3 (a) is the normalized LUS image in the Cartesian coordinate system; Figure 3 (b) is the corrected LUS image in the polar coordinate system; Figure 3 (c) is the PL region; Figure 3 (d) is the BLs region; Figure 3 (e) PL-related features, including TPL (PL thickness), RPL (PL roughness), MPLI (mean PL intensity), and SDPLI (standard deviation of PL intensity); Figure 3 (f) Features related to BLs, including NBL (number of BLs), AWBL (cumulative width of BLs), ACBL (attenuation coefficient of BLs), AIBL (cumulative intensity of BLs), and GLCM (texture feature).

[0081] As an important anatomical landmark, PL (pleural line) has key significance in LUS images. In addition to indicating the interface between the chest wall and the lungs in anatomy, it is also associated with a variety of lung pathologies such as pleural thickening, pleural effusion, and multiple consolidations. Changes in PL, including thickening and destruction, will affect its thickness, brightness, brightness changes, and morphological uniformity. Figure 3 As shown in (e), the features related to PL include TPL, RPL, MPLI and SDPLI, and their calculations are introduced below.

[0082] Where TPL is the thickness of PL, and its calculation formula is as follows:

[0083] TPL=M k [d p (k)-d a (k)] (1)

[0084] Among them, M k is the mean, d p (k) and d a (k) are the front and rear edges of the PL passing through 75% of the PL peak intensity, that is, TPL is defined as dp (k) and d a (k) The average distance between

[0085] RPL is the roughness of PL, which is expressed by signal d ch The standard deviation is evaluated and the calculation formula is as follows:

[0086]

[0087] Among them, d ch (k) Through d c (k) is obtained by applying a high-pass filter to remove the PL outline, while d c (k) is the center point of the kth column of PL, which is automatically detected by locating the peak intensity within the ROI.

[0088] MPLI and SDPLI are the mean and standard deviation of PL peak intensity, respectively, and are evaluated as potential markers of disease severity. Their calculation formulas are:

[0089] MPLI=M k [I(d c (k), k)] (3)

[0090]

[0091] Among them, I(d c (k), k) is the center point d c (k) intensity value.

[0092] When annotating the relevant features of BLs, the corrected LUS image is used, that is, Figure 3 (b) shows the LUS image in the ρ-θ polar coordinate system. Figure 3 As shown in (d), based on the corrected LUS image, the regions corresponding to the B-lines are marked and various features of the B-lines are extracted. The number of B-lines (NBL) is determined from the corrected LUS image. On the corrected LUS image, a rectangular ROI ( Figure 3(d)) is used to measure the width and attenuation coefficient of B-lines. Within the ROI, for each line (axial position), the point of peak intensity is identified as the center of the B-line, and the left and right boundaries of the B-line are marked on both sides of the center point where the intensity reaches 75% of its peak value. The accumulated width of the B-line (AWBL) is calculated by accumulating the widths of different B-lines. The center position of the B-line is identified by locating the peak intensity at different axial positions. Subsequently, the attenuation coefficient of each B-line is determined by linear regression analysis between these peak intensities and their axial positions. The overall attenuation coefficient (ACBL) of the B-line of each LUS image is calculated by averaging the attenuation coefficients of the B-lines. For the intensity measurement of each B-line, a rectangular ROI is selected whose boundaries are the left and right boundaries of the B-line, 1.5 cm below the PL, and an axial length of 2 cm. The intensity of each B-line is calculated by calculating the sum of the pixel intensities within the ROI. The accumulated intensity (AIBL) of the B-line of each LUS image is the sum of the intensities of different B-lines. As Figure 3 As shown in (f), the features associated with B lines include NBL, AWBL, ACBL, and AIBL.

[0093] Furthermore, if Figure 4 (a)~ Figure 4 As shown in (c), in this embodiment, the knowledge hidden representation fusion module (KFLR) adopts a Transformer architecture, which includes a knowledge fusion module (KFM), a self-attention module and a severity assessment module. Among them, the knowledge fusion module is used to fuse the hidden representation output by the deep feature embedding module and the prior knowledge output by the artificial feature embedding module to form a coding set containing knowledge and hidden representation; the self-attention module is composed of N stacked self-attention blocks, which is used to further encode the coding set output by the knowledge fusion module to represent it as a whole and improve its ability to assess the severity of the new coronavirus infection; the severity assessment module includes an average pooling layer and a three-layer multilayer perceptron, which is used to assess the severity of the new coronavirus infection according to the output of the self-attention module, and obtain the assessment result, that is, the final prediction probability of different severity levels.

[0094] In fact, in order to integrate LUS examination E i Knowledge i and invisible representation x i , KFLR uses a multi-head cross-spatial attention (MHCSA) module to improve performance. To enhance the model’s ability to perform multi-scale encoding simultaneously, we employ a multi-head attention mechanism in both KFM and the self-attention module. This operation significantly enhances the model’s ability to capture complex relationships within the data.

[0095] Furthermore, if Figure 4As shown in (a), the knowledge fusion module consists of M stacked cross-space fusion modules and a connection layer. The cross-space fusion module captures the invisible representation x i and knowledge i The mutual connection and self-connection between the two inputs realize the fusion of the two. This novel fusion method implicitly filters the information in the invisible space, reduces redundant representation, and ensures that both the invisible representation xi and the knowledge ki can be effectively integrated into the subsequent processing stage; the connection layer is used to connect the fused features output by the M cross-space fusion modules to form a coding set containing knowledge and invisible representation.

[0096] Example 2

[0097] This embodiment mainly focuses on using LUS examination to assess the severity of novel coronavirus infection.

[0098] In this embodiment, 167 examinations from 152 patients were included, which were classified as 113 mild, 21 common, 8 severe, and 25 critical. Only critical patients underwent multiple examinations. The data set includes a total of 1447 frames. All patients were confirmed to be infected with SARS-CoV-2 by RT-PCR test and were examined at Ditan Hospital in Beijing from March 2, 2020 to November 30, 2022. The examination data set was acquired using a variety of ultrasound imaging systems, including GE Vivid iq equipped with C1-5-RS probe (center frequency: 3MHz; GE Healthcare, Chicago, Illinois, USA), Philips CX50 equipped with C5-1 probe (center frequency: 2.7MHz; Philips Healthcare, Amsterdam, Netherlands), and Hi-Vision Noblus equipped with EUP-C715 probe (center frequency: 3.5MHz; Hitachi Healthcare, Tokyo, Japan). Images were acquired at depths ranging from 6 to 16 cm, with a focal depth range of 2 to 9 cm and a mechanical index (MI) range of 1 to 1.4. During LUS examination, the choice of imaging parameters depended on the patient. The detailed distribution of imaging parameters in the dataset is shown in Figure 5 shown.

[0099] like Figure 5 The following is a Sankey diagram of the distribution of imaging parameters. From left to right, the patients are grouped by the ultrasound imaging system used, the central frequency used, the focal depth, the imaging depth, and the mechanical index (MI).

[0100] Several studies have proposed and evaluated some imaging protocols. In this study, an imaging protocol based on eight scanning areas and a four-level severity assessment system (M-BLUE protocol) was implemented. According to the M-BLUE protocol, eight areas were evaluated for each patient (bilateral upper BLUE points, M points, PLAPS points, and diaphragm points). Usually, one representative image is collected and stored for each area during the examination. Therefore, the data set for each LUS examination usually contains eight images. The severity of the examination is determined by clinical experts based on clinical symptoms.

[0101] It should be noted that the novel coronavirus infection severity assessment system used in this embodiment is consistent with the diagnosis and treatment plans of China and the World Health Organization (WHO) for novel coronavirus infection. Patients are classified into four categories: mild, ordinary, severe and critical. The four-level severity assessment of this embodiment corresponds to this classification. In the secondary severity assessment task (severe / non-severe), severe and critical cases are classified as severe patients, while mild and ordinary cases are classified as non-severe patients.

[0102] In this example, we first show the visualization of representations at different stages in our framework through t-distribution-random neighbor embedding (t-SNE). Next, we show the results of ablation studies and various comparative experiments. The ablation study contains various configurations to verify the individual contributions of each component in our framework. The comparative experiments involve machine learning methods that use hidden representations and knowledge. The evaluation metrics used in this example include accuracy, sensitivity, specificity, F1 score, and area under the receiver operating characteristic curve (AUC). For multi-class prediction, a weighted average metric is used.

[0103] It is noteworthy that although the method of the present invention and RF showed only slight differences in accuracy, specificity and AUC when evaluating the four-level severity. Figure 6 As can be seen from the ROC curve in (a), there is no significant difference in AUC between the method of the present invention and RF. Figure 7 (a) and Figure 7 The confusion matrix in (b) shows the advantage of our method. Compared with RF, our method performs better in assessing severe patients, while the performance in other types of grading is similar. In addition, the sensitivity of our method is 13% higher than that of RF. In fact, in the secondary severity assessment, despite the similar AUC values, our method outperforms RF in terms of accuracy, sensitivity, specificity, and F1 score. The ROC curve for the secondary novel coronavirus infection severity assessment is closer to the point (0,1), as shown in Figure 6(b) shows that our method achieves more accurate assessment than RF while maintaining a similar error rate. This indicates that our method has a higher true positive rate and lower false positive rate compared to RF, which is a desirable property for a severity assessment model.

[0104] Table 1 Comparative experimental results of the fourth-level and second-level novel coronavirus infection severity assessment

[0105]

[0106]

[0107] Table 2 Ablation experiment results for the fourth and second level evaluation of the severity of novel coronavirus infection

[0108]

[0109]

[0110]

[0111] Among them, K, I, L, and F represent knowledge, LUS image, LRM, and KFM respectively.

[0112] Table 1 shows that the proposed method consistently outperforms machine learning methods in terms of accuracy in assessing the severity of novel coronavirus infection. This highlights the superiority of the proposed method in capturing and utilizing relevant information required for accurate severity assessment. The ablation experiments in Table 2 show that it is challenging for DL ​​networks to automatically learn to extract expertise from images without guidance from clinicians. These findings reveal the importance of annotations provided by clinicians in improving model performance and reliability.

[0113] The present invention can directly assess the severity of COVID-19 pneumonia at the LUS examination level, utilizing the valuable prior knowledge of LUS examinations and experienced clinicians. The present invention achieved 96.4% and 87.4% accuracy rates at the LUS examination level for secondary and quaternary COVID-19 pneumonia severity assessments, respectively.

[0114] In summary, the present invention combines the doctor's prior knowledge and the invisible representation of the LUS examination. The quality of the LUS image will affect the manually extracted features. Since these features are based on manually collected and annotated LUS images, the manual features have certain instability in the characterization of patients. In contrast, the invisible representation captures the global information from the entire LUS examination and enhances the performance of the model. Another important contribution is our new model KFLR for severity assessment of COVID-19 pneumonia, which combines the doctor's prior knowledge with the invisible representation. KFLR based on the Transformer architecture has been shown to be very effective in integrating information from different fields.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A quantitative assessment system for novel coronavirus infection, characterized in that: include: Image acquisition and preprocessing module, used to collect the i-th lung ultrasound examination E i The lung ultrasound images in the image were preprocessed; An invisible representation module is used for extracting invisible representation of the preprocessed lung ultrasound image as input; A knowledge extraction module is used to annotate the preprocessed lung ultrasound image and extract features that can reflect the lung condition from the annotated area as prior knowledge; The fusion module is used to fuse the invisible representation and prior knowledge, and quantitatively evaluate the severity of the new coronavirus infection based on the fusion results.

2. A novel coronavirus infection quantitative assessment system as claimed in claim 1, characterized in that: The image acquisition and preprocessing module includes an image acquisition module, an image depth compensation module, an image normalization module and an image correction module; The image acquisition module is used to screen key frames from the lung ultrasound data acquisition video to obtain the i-th lung ultrasound examination E i n i Ultrasound image of the lungs; The image depth compensation module is used to calibrate the size of each lung ultrasound image according to the imaging depth, so that the calibrated lung ultrasound images have a consistent size; The image normalization module is used to normalize the pixel intensity of the lung ultrasound image according to the imaging depth and the number of pixels of the central column of the detected lung area; The image correction module is used to convert the lung ultrasound image in the xy Cartesian coordinate system into the polar coordinate system.

3. A novel coronavirus infection quantitative assessment system as claimed in claim 2, characterized in that: In the image correction module, the process of converting the lung ultrasound image in the xy Cartesian coordinate system into the polar coordinate system is: The normalized lung ultrasound images acquired using a convex array probe were converted from a Cartesian coordinate system to a polar coordinate system; Identify the sector area in the lung ultrasound image and determine the center point O' of the sector arc; The lung region is mapped from Cartesian coordinates to polar coordinates with the center point O' as the origin.

4. A novel coronavirus infection quantitative assessment system as claimed in claim 2, characterized in that: The invisible representation module includes a pre-training model module and a representation vector mapping module; The pre-trained model module is used to use the pre-trained model to i n i Each lung ultrasound image is represented by an m k ×n i The feature matrix of The representation vector mapping module is used to represent the i-th lung ultrasound examination E i All feature matrices in are processed to obtain the invisible representation of Ei.

5. A novel coronavirus infection quantitative assessment system as claimed in claim 4, characterized in that: The pre-training model uses RegNet, DenseNet, EfficientNet, GoogLeNet, ResNet, Swin Transformer or VGG.

6. A novel coronavirus infection quantitative assessment system as claimed in claim 2, characterized in that: The representation vector mapping module includes an average pooling and concatenation layer, a channel linear layer and a dimensionality reduction module; The average pooling and concatenation layers are used to transform E i All feature matrices of are concatenated on the feature dimension and processed to obtain the deep feature vector f i ; The channel linear layer is used to transform the depth feature vector f i Perform scale normalization and screening; The dimension reduction module is used to reduce the dimension of the output feature vector of the channel linear layer to obtain a x with a size of 900×1 i As E i invisible representation.

7. A novel coronavirus infection quantitative assessment system as claimed in claim 2, characterized in that: The knowledge extraction module includes a preliminary annotation module, an annotation verification module and a knowledge embedding module; The preliminary annotation module is used by biomedical engineers to analyze the i-th lung ultrasound examination E i All lung ultrasound images are annotated with regions of interest, including pleural line region annotation and B-line region annotation; The marking column verification module is used by clinicians to verify the marked region of interest to obtain the final region of interest; The knowledge embedding module is used to extract features related to the pleural line and B line from the final region of interest of all lung ultrasound images and then take the average value to generate the i-th lung ultrasound examination E i Prior knowledge k i .

8. A novel coronavirus infection quantitative assessment system as claimed in claim 7, characterized in that: The features related to the pleural line include pleural line thickness, pleural line roughness, pleural line intensity average and pleural line intensity standard deviation; the features related to the B line include the number of B lines, the cumulative width of the B line, the attenuation coefficient of the B line, the cumulative intensity of the B line and texture features.

9. A novel coronavirus infection quantitative assessment system as claimed in claim 1, characterized in that: The knowledge hidden representation fusion module includes a knowledge fusion module, a self-attention module and a severity assessment module; The knowledge fusion module is used to fuse the invisible representation and the prior knowledge to form a coding set including the knowledge and the invisible representation; The self-attention module is used to further encode the encoding set to represent it as a whole; The severity assessment module is used to assess the severity of the novel coronavirus infection based on the output of the self-attention module to obtain an assessment result.

10. A novel coronavirus infection quantitative assessment system as claimed in claim 9, characterized in that: The knowledge fusion module includes M stacked cross-space fusion modules and a connection layer; Each of the cross-space fusion modules captures the invisible representation x i and knowledge i The mutual and self-connection between these two inputs enables the fusion of the two; The connection layer is used to connect the fusion features output by the M cross-space fusion modules to form a coding set containing knowledge and invisible representation.

Images