A lung ultrasound image classification method, system and medical device

Abstract

The application discloses a lung ultrasonic image classification method and system and medical equipment. The application comprises: lung ultrasonic data acquisition; DGF-ResNet lung ultrasonic automatic classification model establishment; inputting the data set into the model to obtain the DGF-ResNet lung ultrasonic automatic classification model through training; and using the DGF-ResNet lung ultrasonic automatic classification model to perform lung ultrasonic automatic scoring. The application improves the lung ultrasonic automatic scoring accuracy, solves the problems of the traditional lung ultrasonic scoring method, such as dependence on subjective judgment of doctors, inconsistent scoring results and long time consumption, and provides reliable and efficient technical support for timely diagnosis of critical patients. Through the automatic scoring system, the application can quickly and accurately analyze lung ultrasonic images, provide objective and quantitative scoring results for doctors, thereby assisting the doctors in rapid and accurate diagnosis and saving valuable time for treatment of the patients.

Description

Technical Field

[0001] This invention relates to the field of medical image classification, specifically to a method, system, and medical device for classifying lung ultrasound images. Background Technology

[0002] Lung ultrasound (LUS), a chest imaging technique, offers diagnostic accuracy comparable to CT scans, while being portable and easy to use. Recent studies have shown that LUS demonstrates great potential for rapid assessment of critically ill patients in primary healthcare settings and developing countries. Given the scarcity of lung ultrasound specialists, automated scoring of LUS images has become a key approach to expanding the technology's application, both compensating for the shortage of experts and significantly improving diagnostic efficiency and accuracy.

[0003] Currently, lung ultrasound scoring methods can be mainly categorized into three types: manual judgment, machine learning, and deep learning. Manual lung ultrasound scoring relies heavily on the subjective judgment of clinicians, which can easily affect the accuracy and reliability of the ultrasound assessment due to differences in the doctor's interpretation level and clinical experience, potentially leading to misdiagnosis and serious consequences. Machine learning algorithms utilize pixels, texture, and other features to classify lung ultrasound LUS images. Some researchers have designed four pleural line-related features and four B-line-related features to analyze lung ultrasound images and used support vector machines to achieve severe / non-severe binary classification. Other researchers have used eight features of the pleural line and the area below it based on support vector machines (SVM) to automatically score four types of LUS images. Another method uses dual-tree complex wavelet transform (DTCWT) to decompose the image into sub-images, then uses a combination of global statistical pixel intensity distribution, gray-level co-occurrence matrix, gray-level running length matrix, and linear binary pattern features to extract texture and morphological features, which are then input into a linear discriminant analysis (LDA)-based classifier for six types of LUS image classification. Machine learning algorithms rely on manually extracted local features from images, failing to adequately consider the characteristics of various indicators in the entire lung ultrasound image and their correlations, resulting in poor generalization. In the field of deep learning, researchers have proposed an automatic quantitative scoring algorithm for LUS (Lung Ultrasound Image) based on a multi-layer fully connected neural network, which performs well in automatically scoring four types of LUS images. However, because this network structure only contains fully connected layers, it is insufficient in handling local correlations. For images scored from 0 to 7, local details play a crucial role in the classification process, thus the algorithm's generalization performance is poor on scores from 0 to 7. Other researchers have proposed a ResNet algorithm utilizing local phase information and multi-feature, multi-scale approaches to achieve LUS-based binary classification of COVID-19. This algorithm only considers local image information and does not take into account the overall correlation of the image, resulting in poor generalization. Still other researchers have combined a separate attention mechanism and the Mish function with the ResNet model, achieving a classification accuracy of 94.84% for four types of lung ultrasound images, but performing poorly for categories with similar features. Therefore, this algorithm may perform well in classifying 4 types of images, but its generalization performance is poor when faced with more detailed 8 types of features.

[0004] Limited by the receptive field size of convolutional layers, convolutional neural networks often struggle to fully capture the global contextual information of an image. However, in more refined rating tasks (0-7 points), global features often provide more comprehensive and discriminative information, which is crucial for improving classification accuracy and generalization performance. Furthermore, when processing images, convolutional neural networks tend to focus on low-level pixel-level features, and are relatively weak at extracting high-level semantic information. Summary of the Invention

[0005] Addressing the challenge of low accuracy in automatic lung ultrasound scoring based on a 0-7 scale, this invention discloses an automatic lung ultrasound scoring method, system, and medical device based on a Detailed-Global Fusion Residual Network (DGF-ResNet) that fuses local detailed features and high-order global features from a 0-7 scale. This invention, building upon ResNet50, designs a feature extraction method for images under a novel lung ultrasound scoring standard that integrates local detailed features and high-order global features.

[0006] This invention provides an automated lung ultrasound scoring method, comprising the following steps:

[0007] S1. Acquire lung ultrasound image data, classify the image data, and obtain a training set;

[0008] Lung ultrasound videos of different patients were acquired using ultrasound instruments. The lung ultrasound videos were then processed by frame extraction to form a lung ultrasound image dataset, which was then divided into a training set and a test set.

[0009] S2. Establish an automatic lung ultrasound classification model based on DGF-ResNet;

[0010] S3. Using the training set obtained in step S1, train the lung ultrasound automatic classification model based on DGF-ResNet in step S2 to obtain a trained lung ultrasound automatic classification model.

[0011] S4. Obtain the lung ultrasound image to be classified and input it into the trained lung ultrasound automatic classification model for automatic classification of the lung ultrasound image.

[0012] Preferably, in step S3, training ends when the total loss decreases to no more than 0.001. The total loss function uses a combination of triplet loss and label smoothing cross-entropy loss. The formula for calculating triplet loss is as follows:

[0013]

[0014] In the formula, N represents the batch of lung ultrasound images during training. These are randomly selected lung ultrasound image samples. For and Similar lung ultrasound image samples, For and Different types of lung ultrasound image samples, Extracted for the automatic lung ultrasound classification model and Euclidean distance between features Extracted for the automatic lung ultrasound classification model and The Euclidean distance between features, where γ is a hyperparameter representing... The minimum interval is set to 0.3. This ensures that lung ultrasound image feature vectors of the same class are tightly clustered in the feature space, while lung ultrasound image feature vectors of different classes are far apart in the feature space, thereby learning a more effective lung ultrasound feature representation.

[0015] The formula for label smoothing cross-entropy loss is as follows:

[0016]

[0017] In the formula, ε is the random noise coefficient, with a value of 0.1; N is the number of lung ultrasound scoring categories; y true These are real labels, with 1 for positive and 0 for negative; y pred It is the predicted probability output by the automatic lung ultrasound classification model.

[0018] The final loss function is calculated as follows:

[0019] Loss=αL Triplet +βL Smoothing

[0020] In the formula, α and β are adjustable hyperparameters used to weigh the weight of each loss value. Given that this invention considers both losses to be equally important, we set both weighting coefficients α and β to 1 to ensure that their contributions to the loss function are equal.

[0021] Preferably, the automatic lung ultrasound classification model based on DGF-ResNet in step S2 includes a 7×7 Conv module, 16 HFF Blocks, and a fully connected layer; the 16 HFF Blocks are divided into four stages, and the number of HFF Blocks and channels in each stage are set to [3,4,6,3] and [C,2C,4C,8C], respectively.

[0022] The structure of the HFF Block, from top to bottom, consists of a DFE module, a GFE module, and two residual structures.

[0023] The DFE module includes one 3×3 Conv module, two 1×1 Conv modules, and one two-dimensional local detail attention mechanism (CBAM) module. For each input feature A, the DFE module first passes it through one 3×3 Conv module and then two 1×1 Conv modules to obtain the feature. Figure IThe CBAM module first sets the features Figure I The image is first processed by global max pooling and then global average pooling based on its height and width, compressing the spatial dimension and forming two branches. These branches are then fed into a multilayer perceptron with shared weights. Subsequently, the outputs of the two multilayer perceptrons are merged by element-wise summation. Finally, a sigmoid activation function is applied to generate the final channel attention matrix, which is then combined with the feature... Figure I Element-wise multiplication yields the features Figure II ; the features Figure II The mean and maximum values ​​are calculated separately along the channel dimension. The two results are then concatenated based on the channel dimension to form a feature matrix with two channels. A convolution operation is then performed to reduce the channel dimension to one. Finally, a spatial attention matrix is ​​generated using a sigmoid activation function, and the features are then... Figure II The features are obtained by element-wise multiplication with the spatial attention matrix. Figure III ; Combine input feature A with feature Figure III The local detail features B of lung ultrasound are obtained by adding the first residual connections, which increases the depth of the network.

[0024] Furthermore, the GFE module of this invention consists of a third-order recursive gated convolutional layer, a two-dimensional global attention mechanism (GAM) module, and an MLP module containing a fully connected layer. The third-order recursive gated convolutional layer first maps the input lung ultrasound local detail features B∈R through a linear mapping. C×H×W The number of channels is increased in dimensionality to obtain the increased-dimensional feature B'∈R. 2C ×H×W Then, the upgraded feature B' is divided into feature matrices p0∈R according to the channel dimension. C / 4×H×W and q 0～2 ∈R 7C / 4×H×W The characteristic matrix q 0～2 The entire dataset is processed through a single 7×7 depthwise separable convolution to obtain spatially interactive weights with input adaptation, and then the feature matrix q is used. 0～2 Split into q0∈R along the channel dimension C / 4×H×W , q1∈R C / 2×H×W and q2∈R C×H×W Finally, feature C is obtained through recursive multiplication. The GAM module calculates the global channel attention matrix I by inputting feature C into a multilayer perceptron (MLP), then generates the final global channel attention matrix II through a sigmoid activation function, and finally multiplies feature C element-wise to obtain the feature. Figure IV The features Figure IVAfter two convolutional layers, information from the feature space is fused to obtain global spatial attention matrix I. Then, a sigmoid activation function is used to generate global spatial attention matrix II. Finally, the feature... Figure IV The features are obtained by element-wise multiplication with the global spatial attention matrix II. Figure V ; the features Figure V The complex relationships between features are captured by a multilayer perceptron (MLP) to obtain the final high-order global feature D of lung ultrasound. The local detail feature B of lung ultrasound is added to the high-order global feature D of lung ultrasound through a second residual connection to obtain the lung ultrasound feature E, which is a fusion of local detail and high-order global features, and further deepens the network.

[0025] Step S1, acquiring lung ultrasound image data, specifically includes: collecting lung ultrasound images of the patient. Lung examination and classification are performed by a fixed and experienced ultrasound-trained physician to ensure clear and complete images. After anonymization, the images are classified and reviewed according to the new LUS standard to ensure accuracy. 5-15 ultrasound images are acquired for each region to fully demonstrate the lungs and surrounding tissue structures.

[0026] Furthermore, the present invention also provides a lung ultrasound image classification system, characterized in that it includes a data acquisition module, a lung ultrasound automatic classification model construction module, a lung ultrasound automatic classification model training module, and a lung ultrasound automatic classification model testing module.

[0027] Data acquisition module: Acquires lung ultrasound image data, classifies the image data, and obtains a training set; Lung ultrasound automatic classification model construction module: Establishes a lung ultrasound automatic classification model based on DGF-ResNet; Lung ultrasound automatic classification model training module: Uses the training set obtained in the data acquisition module to train the DGF-ResNet-based lung ultrasound automatic classification model, obtaining a trained lung ultrasound automatic classification model; Lung ultrasound automatic classification model testing module: Acquires lung ultrasound images to be classified, inputs them into the trained lung ultrasound automatic classification model, and performs automatic classification of the lung ultrasound images.

[0028] Furthermore, the lung ultrasound automatic classification model based on DGF-ResNet obtained by the lung ultrasound automatic classification model construction module described in this invention includes a 7×7 Conv module, 16 HFF Blocks and a fully connected layer; the 16 HFF Blocks are divided into four stages, and the number of HFF Blocks and the number of channels in each stage are set to [3,4,6,3] and [C,2C,4C,8C], respectively;

[0029] Furthermore, the structure of the HFF Block described in this invention, from top to bottom, includes: a DFE module, a GFE module, and two residual structures;

[0030] Furthermore, the present invention also provides a medical device in which the lung ultrasound image classification method as described in claims 1-6 is embedded.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention proposes an automatic lung ultrasound scoring algorithm based on the fusion of local detailed features and high-order global features from a 0-7 score using a Detailed-Global Fusion Residual Network (DGF-ResNet). The algorithm then utilizes the HFF Block to extract and fuse local detailed features and high-order global features. The core of the HFF Block consists of a DFE Module and a GFE Module. The former introduces a two-dimensional local detail attention mechanism (CBAM) module to guide the network to focus on local detailed features of the lung ultrasound image; the latter uses recursively gated convolution to achieve interaction of high-order features and combines a two-dimensional global attention mechanism (GAM) module to deeply mine high-order global features. Finally, the local detailed features and high-order global features are fused, significantly improving the network's classification ability for lung ultrasound images scored from 0-7. Attached Figure Description

[0033] Figure 1 The structure of the DGF-ResNet lung ultrasound automatic scoring algorithm;

[0034] Figure 2 This is a diagram of the HFF Block structure;

[0035] Figure 3 This is a diagram of a third-order recursive gated convolution structure;

[0036] Figure 4 A flowchart of the model training process;

[0037] Figure 5 Images from the FCSPF-13324 dataset;

[0038] Figure 6 Visualize the heatmap for features; Detailed Implementation

[0039] The invention will now be further described with reference to the accompanying drawings.

[0040] This invention discloses a method, system, and medical device for classifying lung ultrasound images. Addressing the challenge of low accuracy in automatic lung ultrasound scoring due to the limitations of traditional convolutional neural networks (CNNs) in capturing the global contextual information of images, which are often restricted by the receptive field size of convolutional layers, this invention resolves the issue of lung ultrasound image classification on a 0-7 scale. The method comprises: lung ultrasound data acquisition; establishing a DGF-ResNet automatic lung ultrasound classification model; training the model by inputting the dataset; and performing automatic lung ultrasound scoring using the DGF-ResNet model. This invention improves the accuracy of automatic lung ultrasound scoring, solving the problems of reliance on physician subjective judgment, inconsistent scoring results, and long processing times associated with traditional methods. It provides reliable and efficient technical support for the timely diagnosis of critically ill patients. Through the automated scoring system, this invention can quickly and accurately analyze lung ultrasound images, providing physicians with objective and quantitative scoring results, thereby assisting physicians in making rapid and accurate diagnoses and saving valuable time for patient treatment.

[0041] The lung ultrasound image classification method in this embodiment includes the following steps:

[0042] S1, constructing lung ultrasound data:

[0043] S1-1: Obtain lung ultrasound images from lung ultrasound videos of different patients to form the dataset FCSPF-13324 required by this invention. The dataset contains 13,324 lung ultrasound images, of which 10,669 are used as the training set and 2,655 are used as the test set.

[0044] S1-2: Organize and annotate the above dataset, placing it into 8 folders according to a score of 0-7. Some lung ultrasound images from FCSPF-13324 are shown below. Figure 5 As shown.

[0045] S2. Establish an automatic lung ultrasound classification model based on DGF-ResNet;

[0046] The basic framework of the method proposed in this invention is as follows: Figure 1 As shown.

[0047] Figure 1The diagram illustrates the basic framework of the method of this invention. The input image is first preprocessed to extract the region of interest, and then input into a lung ultrasound feature extraction network based on DGF-ResNet. In the lung ultrasound feature extraction network based on DGF-ResNet, the input image data is first downsampled by a 7×7 Conv module to retain the main information of the original image. Then, it is processed by 16 HFF Blocks to extract local details and high-order global features of lung ultrasound, and the local detail features are fused with the global features. Finally, it is processed by a fully connected layer to output the network classification feature vector. The output feature vector is used to calculate the loss function, guiding the network to fully learn the key features of lung ultrasound images during training, and further distinguishing the details of images between different scores.

[0048] S2-1 HFF Block Module

[0049] like Figure 2 As shown, the HFF Block consists of a DFE Module, a GFE Module, and two residual structures. The DFE Module extracts local detail features from lung ultrasound images using the local receptive field of convolution and local channel and spatial attention mechanisms. The input features and the lung ultrasound local detail features extracted by the DFE Module are fused through a first residual connection to obtain a feature vector, which is then input into the GFE Module. The GFE Module performs high-order feature interaction through a third-order recursive gated convolution, and combines global channel attention and global spatial attention to capture high-order global features. Finally, a multilayer perceptron (MLP) is used to capture the complex relationships between high-order global features, resulting in the final high-order global features of the lung ultrasound image. The feature vector from the first feature fusion is then fused with the high-order global features of the lung ultrasound image output by the GFE Module through a second residual connection to obtain a lung ultrasound feature vector containing both local detail features and high-order global features.

[0050] S2-2 DFE Module

[0051] The DFE Module consists of a bottleneck composed of one 3×3 and two 1×1 convolutional layers, and a two-dimensional local detail attention mechanism (CBAM) consisting of cascaded local channel attention and local spatial attention.

[0052] The two-dimensional local detail attention mechanism consists of cascaded local channel attention and local spatial attention. For the channel attention matrix, the focus is primarily on the weight coefficients of the feature matrix in the channel dimension. First, the input feature matrix is ​​processed by both height-width-based global max pooling and global average pooling, compressing the input feature matrix in the spatial dimension to reduce its dimensionality and generate two different spatial contexts. These are then fed into a multilayer perceptron (MLP) with shared weights. Subsequently, these two processes are combined by element-wise summation, and finally, the final channel attention feature matrix M is generated by passing it through a sigmoid activation function. CA ∈R C×1×1 For the characteristic matrix M a M CA The calculation formula is:

[0053] M CA (M a )=Sigmoid(MLP(GlobalAvgPool(M a ))+MLP(GlobalMaxPool(M a )))

[0054] M CA Attention weights and M a The feature matrix M is obtained by element-wise multiplication of the corresponding channel features. b Characteristic matrix M b Not only inherited M a The original information in the matrix is ​​further enhanced through a local channel attention mechanism to improve the feature representation of key channels. Feature matrix M b The calculation formula is:

[0055] M b =M a ×M CA

[0056] For the spatial attention matrix, the goal of this invention is to extract local detail features from lung ultrasound images; therefore, the feature matrix M containing channel attention is used. b As input, we further focus on the weight coefficients of the feature matrix in both the height and width dimensions. First, we also consider the feature matrix M... b The channel dimensionality is reduced by calculating the maximum and average values ​​for each channel. These two results are then concatenated to form a feature matrix with two channels. A convolution operation is then performed to reduce the channel dimensionality to one. Finally, a sigmoid activation function is applied to generate the spatial attention feature matrix M. SA ∈R 1×H×WFor the characteristic matrix M b M SA The calculation formula is:

[0057] M SA (M b ) = Sigmoid(Conv 3×3 (mean(M b dim=1), max(M b dim=1)))

[0058] Subsequently, in order to incorporate the local spatial attention feature matrix M SA Integrate it into the existing feature representation and combine it with the feature matrix M b Perform element-wise multiplication. The resulting characteristic matrix M c It includes information from channel interactions and incorporates guidance on spatial attention. Feature matrix M c The calculation formula is:

[0059] M c =M b ×M SA

[0060] Compare the input matrix X with the local detail feature matrix M c The first residual connection is performed. This step generates a feature matrix M containing rich local details by fusing the original input with local detail features. Local .

[0061] S2-3 GFE module

[0062] The GFE Module consists of a third-order recursive gated convolutional layer, a two-dimensional global attention mechanism (GAM) composed of cascaded global channel attention and global spatial attention, and an MLP.

[0063] To enable the network to accurately capture and focus on the subtle differences between features in adjacent scored ultrasound images, a third-order recursive gated convolution is introduced to focus on the interaction between the current feature and its surrounding space. Higher-order semantic features are extracted by progressively fusing feature information. Figure 3 As shown, for a third-order recursive gated convolution, the input feature matrix M is first mapped through a linear mapping. local ∈R C×H×W Channel number upgrade M local '∈R 2C×H×W Then, for the specified order 3, the input features are split into p0∈R according to the channel dimension. C / 4×H×W and q 0～2 ∈R 7C / 4×H×W . q 0～2The entire dataset is processed through a separate 7×7 depthwise separable convolution to obtain spatially adaptive interaction weights, which are then split into q0∈R along the channel dimension. C / 4×H×W q1∈R C / 2×H×W q2∈R C×H×W Finally, the interaction of third-order features is achieved through recursive multiplication. The formula for third-order recursive gated convolution is:

[0064] p1 = p0 * q0, p k+1 =Linear(p k )*q k k = 1, 2

[0065] To further extract features resulting from interactions across different spaces, we introduce a two-dimensional global attention mechanism to learn the correlations between these spatial features. This mechanism consists of cascaded global channel attention and global spatial attention. Unlike local channel attention and local spatial attention, global channel attention and global spatial attention do not use max pooling or average pooling; their purpose is to fully utilize global contextual information.

[0066] For the global channel attention matrix, our goal is to fully utilize the feature matrix M, which contains rich semantics. x Further extract the interaction information along the channel dimension. First, we extract the interaction information from feature M. x The input is fed into a multilayer perceptron (MLP) to compute the global channel attention matrix M. GCA Then, the final global channel attention feature matrix M is generated through the sigmoid activation function. GCA ∈R C×H×W For the characteristic matrix M x M GCA The calculation formula is:

[0067] M GCA (M x =Sigmoid(MLP(M x ))

[0068] Next, M GCA Attention weights and M x The corresponding channel features are combined to generate a new feature matrix M. y Characteristic matrix M y Not only inherited M x The original information in the data is further enhanced by a global channel attention mechanism to improve the feature representation of key channels.

[0069] For the global spatial attention matrix, we want the network to further learn features with higher-order semantic interactions, thereby effectively distinguishing lung ultrasound images with similar scores and improving the model's classification ability. First, we directly use two convolutional layers to fuse information in the feature space, and then use a sigmoid activation function to generate the global spatial attention feature matrix M. GSA ∈R C×H×W For the characteristic matrices My, M GSA The calculation formula is:

[0070] M GSA (M y ) = Sigmoid(Conv 7×7 (Conv 7×7 (M y )))

[0071] Subsequently, in order to incorporate the global spatial attention feature matrix M GSA We integrate it into the existing feature representation and combine it with the feature matrix M y Perform element-wise multiplication. This step not only preserves M... y The original information in the matrix is ​​further enhanced through a spatial attention mechanism to improve the feature representation of key spatial regions. The resulting feature matrix M... z It includes both global channel information and spatial attention. Finally, in order to further extract M... z We extract high-order global features from the dataset, capture the complex relationships between features using a multilayer perceptron (MLP), and output the final high-order global feature matrix M. global .

[0072] To enhance the extraction capability of local details and global representations from lung ultrasound images, we propose a feature fusion strategy. Specifically, we will use the local detail feature matrix Ml extracted by the DFE Module... ocal The high-order global features M captured by the GFE Module global The addition operation is then performed. This step aims to effectively fuse two different types of feature information to generate a feature matrix M that combines local details with high-order global features. HFF Through this integration, M HFF It can not only preserve minute local changes in lung ultrasound images, but also reflect overall structural information and higher-order features.

[0073] S3. Using the training set described in step S1, train the automatic lung ultrasound scoring model based on DGF-ResNet in step S2. When the total loss decreases to no more than 0.001, the training ends, and the trained automatic lung ultrasound scoring model is obtained.

[0074] S4. Input the data from the test set described in step S1 into DGF-ResNet for automatic lung ultrasound scoring.

[0075] Example 1

[0076] This embodiment is based on an automatic lung ultrasound scoring method using Detailed-Global Fusion Residual Network (DGF-ResNet). The flowchart of this method is as follows: Figure 4 As shown, it includes the following steps:

[0077] S1. Constructing the dataset

[0078] S1-1: Collect 12-zone lung ultrasound images. Lung examinations and scoring are performed by a fixed and experienced ultrasound-trained physician to ensure image clarity and completeness. After anonymization, images are categorized and reviewed according to the new LUS criteria to ensure accuracy. 5-15 ultrasound images are acquired for each zone to fully visualize the lungs and surrounding tissue structures. The new LUS criteria are: 0 points, normal lung tissue (thoracic pleural line and A-line); 1 point, smooth pleural line, fewer than 3 scattered B-lines in the subpleural region; 2 points, smooth pleural line, more than 3 scattered B-lines in the subpleural region; 3 points, fused B-lines in the subpleural region, less than half the entire section; 4 points, fused B-lines in the subpleural region, occupying more than half the entire section; 5 points, irregular, fragmented echoes in the subpleural region, depth ≤1cm; 6 points, soft tissue-like hypoechoic shadows in the lung parenchyma, depth ≤3cm; 7 points, soft tissue-like hypoechoic shadows in the lung parenchyma, depth ≥3cm. As if two images appear within an entire plane, the score is calculated using a higher-level value (e.g.) Figure 5 All images were organized into folders, with eight subfolders for the new lung ultrasound scoring system (0, 1, 2, 3, 4, 5, 6, and 7 points). A final dataset of 13,324 images, FCSPF-13324, was collected for algorithm evaluation. This dataset contains images at different scoring levels, specifically distributed as follows: 0 points (3,688 images), 1 point (2,692 images), 2 points (237 images), 3 points (1,325 images), 4 points (1,088 images), 5 points (2,464 images), 6 points (1,146 images), and 7 points (684 images). The FCSPF-13324 dataset was divided into a training set (10,669 images) and a test set (2,655 images) at a 4:1 ratio to ensure a reasonable distribution of scoring levels after partitioning, for model training and testing. A portion of the lung ultrasound images in the FCSPF-13324 dataset is shown below. Figure 5 As shown.

[0079] S2. Design an automatic lung ultrasound scoring algorithm based on Detailed-Global Fusion Residual Network (DGF-ResNet); the basic framework of the automatic lung ultrasound scoring algorithm based on Detailed-Global Fusion Residual Network (DGF-ResNet) proposed in this invention is as follows: Figure 1 As shown.

[0080] Figure 1 In the example, the input image is X∈R H×W×C H, W, and C represent the height, width, and number of channels of the input image, respectively. The input image in this invention is a 3-channel lung ultrasound image with a resolution of [resolution value missing].

[0081] The network consists of a 7×7 convolutional layer, 16 HFF blocks, and a fully connected layer. The 7×7 convolutional layer is used to downsample the input image, preserving the main information of the original image. The HFF blocks are used to extract local details and high-order global semantic fusion features from the lung ultrasound image. The number of blocks is set to [3, 4, 6, 3], and the number of channels in the four stages is set to [C, 2C, 4C, 8C]. Structurally, it consists of a double residual structure formed by cascading DFE Module and GFE Module. The DFE Module introduces a two-dimensional local detail attention mechanism, enabling the network to more accurately focus on local details of the lung ultrasound. To further extract high-order global features from the lung ultrasound image, the GFE Module introduces a third-order recursive gated convolution and a global attention mechanism to extract high-order semantic features. First, the third-order recursive gated convolution performs high-order spatial interactions, and then the global attention mechanism captures the global information after the high-order interactions, enabling the network to better understand the global semantic information of the entire lung ultrasound image. Finally, the output feature matrices of the DFE Module and GFE Module are fused to obtain a fusion matrix of local detail features and high-order global features. The fully connected layer serves as the classifier of the network, and its output feature vector is used to calculate the triplet loss and the label smoothing cross-entropy loss, guiding the network to fully learn the key features of lung ultrasound images during training and further distinguish the details of images between different scores.

[0082] The models generated using the proposed algorithm, VGG16 algorithm, ResNet50 algorithm, and Vision Transformer algorithm were trained on the FCSPF-13324 dataset for 50 training rounds. The performance of the three classification algorithms on the test set was then evaluated. Visual heatmaps were created for the features extracted by the models on the test set using the four models. The red areas in the heatmaps represent the lung ultrasound features learned by the models, i.e., the regions the models focused on during classification. The visualized heatmaps are shown below. Figure 6 As shown in the figures, for images with a score of 0, the algorithm of this invention significantly focuses on the pleural line and the A-line region, effectively highlighting the characteristics of normal lung tissue. In contrast, other algorithms show a shift or dispersion in their focus, failing to fully capture relevant features. For images with a score of 1, the algorithm of this invention precisely focuses on the pleural line and the scattered B-lines below it, while other algorithms focus more on the region below the pleural line, ignoring key local information. For images with a score of 2, the algorithm of this invention focuses on the pleural line and more than three scattered B-lines below it, while other algorithms only focus on the B-line region, showing weak global perception. In images with a score of 3, the algorithm of this invention focuses on the pleural line and the fused B-lines below it, the ResNet50 algorithm focuses on the fused B-lines while also paying attention to a large number of non-feature regions, and the VGG and VisionTransformer algorithms only focus on the local B-line region. For images with a score of 4, both the algorithm of this invention and the VisionTransformer algorithm focus on the fused B-line occupying more than 1 / 2 of the slice. ResNet50, while focusing on the fused B-line, also pays attention to a large number of non-feature regions, while the VGG algorithm only focuses on local B-line regions, exhibiting poor global perception capabilities. In images with a score of 5, the algorithm of this invention accurately focuses on irregular, fragmented echoes under the pleura, significantly outperforming other algorithms. Finally, for images with scores of 6 and 7, the algorithm of this invention accurately captures and focuses on soft tissue-like hypoechoic shadows within the lung parenchyma, with performance significantly superior to other comparative algorithms. In summary, this algorithm demonstrates significant advantages and higher accuracy in focusing on and identifying lung ultrasound image features at different score levels.

[0083] S3. Input the dataset described in S1 into the Detailed-Global Fusion Residual Network (DGF-ResNet) lung ultrasound automatic scoring algorithm to train and obtain the lung ultrasound automatic scoring model.

[0084] The experimental server used in this invention was an NVIDIA TITAN RTX 3090 GPU, and the deep learning framework used was PyTorch. The input image was a 224×224 resolution 3-channel lung ultrasound image. The training batch size was 16, and the training run consisted of 50 epochs. The loss function was optimized using a rectified Adaptive Moment Estimation optimizer (RAdam) combined with the backup function of the LookAhead mechanism to accelerate model convergence. The learner rate of the optimizer was set to 10. -3 .

[0085] The following is an experimental data analysis of the algorithm proposed in this invention based on a partitioned test set database. The performance of this invention is compared with that of the ResNet50 algorithm, VGG16 algorithm, and Vision Transformer algorithm in the FCSPF-13324 test set.

[0086] This invention comprehensively evaluates model performance using five key metrics: accuracy, precision, recall, F1 score, and AUC. In multi-class classification tasks, we calculate the precision, recall, and F1 score for each class separately and integrate the results using a macro-averaging method to reflect the overall performance. The AUC value is the area under the ROC curve, reflecting the model's recognition performance. When the vertical axis of the ROC curve represents the recognition rate, the AUC value is the area between the curve and the horizontal axis; a higher AUC value indicates better model performance.

[0087] Table 1 shows the specific evaluation indicators and their calculation methods.

[0088] Table 1 Evaluation Indicators

[0089]

[0090] Among them, TP (True Positive) indicates that the actual result is positive and the prediction is positive; FP (False Positive) indicates that the actual result is negative but the prediction is positive; TN (True Negative) indicates that the actual result is negative and the prediction is negative; and FN (False Negative) indicates that the actual result is positive but the prediction is negative.

[0091] Table 2 shows the performance of different algorithms for image classification on our FCSPF-13324 dataset. The results demonstrate that our classification model significantly improves several key performance metrics. Compared to the Vision Transformer algorithm with global attention, our model improves precision, recall, and F1-score by 7.05%, 4.52%, and 5.89%, respectively; compared to the VGG16 classification algorithm, it improves by 3.06%, 4.37%, and 3.8%; and compared to the ResNet50 classification algorithm, it improves by 2.05%, 4.26%, and 3.34%. This indicates that our model is more reliable in classifying images across different scores and can more accurately classify images within each category. The significant improvement in F1-score highlights our model's excellent performance in maintaining high precision and recall, effectively reducing score matching errors and enhancing classification credibility. Furthermore, its improved accuracy and robustness make it more advantageous in lung ultrasound scoring applications.

[0092] Table 2:Comparison of different algorithm combinations in FCSPF-13324

[0093] Dataset metrics

[0094]

[0095]

[0096] This invention proposes a lung ultrasound image classification method that utilizes an HFF Block composed of a DFE Module and a GFE Module to extract and fuse local detail features and high-order global features. The DFE Module algorithm introduces a two-dimensional local detail attention mechanism to guide the network to focus on the local detail features of the lung ultrasound image. The GFE Module uses recursively gated convolution to perform high-order interactions and combines the two-dimensional global attention mechanism to deeply mine high-order global features. The results show that this method significantly improves the network's classification ability for novel lung ultrasound images (scored from 0 to 7), demonstrating the effectiveness of the proposed method.

[0097] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for classifying lung ultrasound images, characterized in that, Includes the following steps: S1. Acquire lung ultrasound image data, classify the image data, and obtain a training set; S2. Establish an automatic lung ultrasound classification model based on DGF-ResNet; The DGF-ResNet automatic lung ultrasound classification model includes a 7×7 Conv module, 16 HFF blocks, and a fully connected layer; the 16 HFF blocks are divided into four stages, and the number of HFF blocks and channels in each stage are set to [3,4,6,3] and [C,2C,4C,8C], respectively. The structure of the HFF Block, from top to bottom, includes: a DFE module, a GFE module, and two residual structures; The DFE module includes one 3×3 Conv module, two 1×1 Conv modules, and one two-dimensional local detail attention mechanism (CBAM) module. For each input feature A, the DFE module first obtains feature map I through one 3×3 Conv module and two 1×1 Conv modules. The CBAM module first performs global max pooling and global average pooling based on the image's height and width, respectively, to compress the feature map I spatially and form two branches, which are then fed into a multilayer perceptron with shared weights. Subsequently, the outputs of the two multilayer perceptrons are merged by element-wise summation, and finally processed by si. The gmoid activation function generates the final channel attention matrix, and the channel attention matrix is ​​multiplied element-wise with feature map I to obtain feature map II. The average and maximum values ​​of feature map II are calculated in the channel dimension, and the two results are concatenated based on the channel to form a feature matrix with 2 channels. Then, a convolution operation is performed to reduce the channel dimension to 1. Finally, the spatial attention matrix is ​​generated by the sigmoid activation function, and feature map II is multiplied element-wise with the spatial attention matrix to obtain feature map III. The input feature A and feature map III are added through the first residual connection to obtain the lung ultrasound local detail feature B, which also increases the depth of the network. The GFE module consists of a third-order recursive gated convolutional layer, a two-dimensional global attention mechanism (GAM) module, and an MLP module containing a fully connected layer. The third-order recursive gated convolutional layer first maps the input lung ultrasound local detail features B∈R through a linear mapping. C×H×W The number of channels is increased in dimensionality to obtain the increased-dimensional feature B'∈R. 2C×H×W Then, the upgraded feature B' is divided into feature matrices p0∈R according to the channel dimension. C / 4×H×W and q 0~2 ∈R 7C / 4×H×W ; the characteristic matrix q 0~2 The entire dataset is processed through a single 7×7 depthwise separable convolution to obtain spatially interactive weights with input adaptation, and then the feature matrix q is used. 0~2 Split into q0∈R along the channel dimension C / 4×H×W , q1∈R C / 2×H×W and q2∈R C×H×W Finally, feature C is obtained through recursive multiplication. The GAM module calculates the global channel attention matrix I by inputting feature C into a multilayer perceptron (MLP), then generates the final global channel attention matrix II through a sigmoid activation function, and multiplies the global channel attention matrix II with feature C element-wise to obtain feature map IV. Feature map IV is then fused with information from the feature space through two convolutional layers to obtain global spatial attention matrix I, which is then generated through a sigmoid activation function. Feature map IV is then multiplied with global spatial attention matrix II element-wise to obtain feature map V. Feature map V is then used by a multilayer perceptron (MLP) to capture the complex relationships between features, resulting in the final high-order global feature D of lung ultrasound. The local detail feature B of lung ultrasound is added to the high-order global feature D of lung ultrasound through a second residual connection to obtain the fused local detail and high-order global lung ultrasound feature E, further deepening the network. S3. Using the training set obtained in step S1, train the lung ultrasound automatic classification model based on DGF-ResNet in step S2 to obtain a trained lung ultrasound automatic classification model. S4. Obtain the lung ultrasound image to be classified and input it into the trained lung ultrasound automatic classification model for automatic classification of the lung ultrasound image.

2. The lung ultrasound image classification method as described in claim 1, characterized in that, In step S3, training ends when the total loss drops to no more than 0.

001. The total loss function uses a combination of triplet loss and label smoothing cross-entropy loss. The formula for calculating triplet loss is as follows: ； In the formula, N represents the batch of lung ultrasound images during training. These are randomly selected lung ultrasound image samples. For and Lung ultrasound image samples of the same batch and type, For and Lung ultrasound image samples of different types from the same batch, Extracted for the automatic lung ultrasound classification model and The Euclidean distance between the features Extracted for the automatic lung ultrasound classification model and Euclidean distance between features For hyperparameters, it means and The minimum interval is set to 0.3, which makes the lung ultrasound image feature vectors of the same class tightly clustered in the feature space, and the lung ultrasound image feature vectors of different classes far apart in the feature space, thereby learning a more effective lung ultrasound feature representation. The formula for label smoothing cross-entropy loss is as follows: ； In the formula, is the random noise coefficient, with a value of 0.1; N is the number of lung ultrasound scoring categories; These are real labels; 1 for positive and 0 for negative. It is the predicted probability output by the automatic lung ultrasound classification model; The final loss function is calculated as follows: ； In the formula, α and β are adjustable hyperparameters used to weigh the weight of each loss value.

3. The lung ultrasound image classification method as described in claim 1, characterized in that, The acquisition of lung ultrasound image data in step S1 specifically includes: collecting lung ultrasound images of the patient; having a fixed and experienced ultrasound-trained physician perform lung examination and classification to ensure clear and complete images; after anonymization, classifying and reviewing the images according to the new LUS standard to ensure classification accuracy; and acquiring 5-15 ultrasound images for each zone to fully demonstrate the lungs and surrounding tissue structures.

4. A lung ultrasound image classification system, the system being used to implement the method as described in any one of claims 1-3, characterized in that, It includes a data acquisition module, a lung ultrasound automatic classification model construction module, a lung ultrasound automatic classification model training module, and a lung ultrasound automatic classification model testing module; Data acquisition module: Acquires lung ultrasound image data, classifies the image data, and obtains a training set; Lung ultrasound automatic classification model construction module: Establishes a lung ultrasound automatic classification model based on DGF-ResNet; Lung ultrasound automatic classification model training module: Uses the training set obtained from the data acquisition module to train the DGF-ResNet-based lung ultrasound automatic classification model, obtaining a trained lung ultrasound automatic classification model; Lung ultrasound automatic classification model testing module: Acquires lung ultrasound images to be classified, inputs them into the trained lung ultrasound automatic classification model, and performs automatic classification of the lung ultrasound images.

5. A medical device, characterized in that... The device incorporates a lung ultrasound image classification method as described in any one of claims 1-3.

Images