An AS severity classification system based on lightweight model and semi-supervised learning

Abstract

A system for AS severity classification based on a lightweight model and semi-supervised learning belongs to the field of medical image processing. The present invention solves the problems of existing methods such as waste of unlabeled data, poor interpretability of patient-level AS severity classification due to lack of view classification and image-level AS severity classification, and high computing power requirements due to large number of model parameters. The present invention designs a lightweight model LWM based on structural reparameterization, deep large kernel convolution and wavelet downsampling, and trains LWM through semi-supervised learning. Semi-supervised learning uses targeted strong enhancement and selective total loss calculation. The model LWM can be trained using labeled data and a large amount of unlabeled data. The trained LWM realizes view classification and image-level AS severity classification, and then uses an aggregation mechanism based on view correlation to obtain patient-level AS severity classification results. The method of the present invention can be applied to view classification and AS severity classification of echocardiography.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to an AS severity classification system based on a lightweight model and semi-supervised learning. Background Art

[0002] The aortic valve (AV) is located between the left ventricle and the aorta and is the last barrier before blood flows out of the heart. The aortic valve can control the direction of blood flow and is an important valve. Once the aortic valve thickens and calcifies, it may cause the outlet to narrow, causing aortic stenosis (AS). AS is a degenerative valvular disease with a high mortality rate in critically ill patients. In clinical practice, the diagnosis of the severity of AS is usually based on the cardiologist's observation of relevant views in the echocardiogram. Traditional clinical diagnostic methods are expensive and subjective. Therefore, there is an urgent need for an efficient and feasible auxiliary diagnostic method. With the development of deep learning, using deep learning to assist in the diagnosis of AS severity has become a feasible method.

[0003] In reality, only a subset of echocardiograms are relevant for the diagnosis of AS, as the aortic valve is only visible in these views (referred to as relevant views). Based on clinical criteria, we consider the parasternal long-axis (PLAX) and parasternal short-axis (PSAX) views to be relevant views. In clinical practice, the first step in diagnosing AS severity is typically to correctly identify the relevant views on transthoracic echocardiography (TTE), followed by diagnosing AS severity based on the relevant views. However, current deep learning-based methods focus more on patient-level AS severity classification and related aggregation mechanisms, with less attention paid to view- and image-level AS severity classification. View and image-level AS severity classification are clinically considered prerequisites for patient-level AS severity classification. Their lack of these capabilities reduces the interpretability of patient-level AS severity classification. Furthermore, current methods fail to consider the limited computing power of healthcare systems. The models used are large in number of parameters and computationally complex. Furthermore, due to the high cost of medical image annotation, only a small number of images can be annotated, resulting in a large amount of unlabeled data. Most current methods use labeled data to perform fully supervised training on the model and cannot utilize a large amount of unlabeled data.

[0004] In summary, since existing methods pay less attention to view classification and image-level AS severity classification, the interpretability of patient-level AS severity classification results is poor; since unlabeled data is not used, there is serious data waste and it is difficult to obtain the best classification performance by relying solely on labeled data for training; and since the model used by existing methods has a large number of parameters and high computational complexity, the classification efficiency is low and the computing power requirements of the medical system are high. Therefore, it is very necessary to propose a new AS severity classification technology to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of existing classification methods such as waste of unlabeled data, lack of view classification and image-level AS severity classification, resulting in poor interpretability of patient-level AS severity classification results, and high computing power requirements for medical systems. An AS severity classification system based on lightweight models and semi-supervised learning is proposed.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an AS severity classification system based on a lightweight model and semi-supervised learning, the system comprising an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model and an aggregation module;

[0007] The echocardiogram acquisition module is used to acquire echocardiogram data of the subject to be detected;

[0008] The first LWM classification model is used to classify each echocardiogram in the acquired echocardiographic data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module, and a classifier; and the working process of the first LWM classification model is as follows:

[0009] In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module;

[0010] The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module;

[0011] The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module;

[0012] The output of the third Stage module is then used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is then passed through the classifier to obtain the view classification result;

[0013] The first Rep HWD Block module includes a DW Rep Block unit, an HWT Block unit, and a PW RepBlock unit, and the working process of the first Rep HWD Block module is as follows:

[0014] The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module;

[0015] The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module;

[0016] The first stage module includes 2 submodules, the second stage module includes 2 submodules, the third stage module includes 6 submodules, and the fourth stage module includes 2 submodules, and the structure and working process of each submodule are the same;

[0017] The submodule includes the DW Rep Block unit and Rep LKFFN. The working process in the submodule is as follows:

[0018] First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule;

[0019] The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer and the output of the flattening layer are connected. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer.

[0020] The second LWM classification model is used to perform image-level AS severity classification on the acquired echocardiographic data, and the structure of the second LWM classification model is the same as that of the first LWM classification model;

[0021] The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

[0022] Furthermore, during the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depthwise separable convolution layer and a batch normalization layer. The working process of the DW Rep Block unit during training is as follows:

[0023] y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x]

[0024] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer;

[0025] In the inference process of the first LWM classification model, the working process of the DW Rep Block unit is as follows:

[0026] y′=σ(Conv 12 (x′))

[0027] Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

[0028] Furthermore, during the training of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer. The working process of the PW Rep Block unit during training is as follows:

[0029] y″=σ[BN(Conv(x″))]

[0030] Where y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer;

[0031] During the inference process of the first LWM classification model, the working process of the PW Rep Block unit is as follows:

[0032] y″′=σ(Conv(x″′))

[0033] Among them, x″ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

[0034] Furthermore, the training data of the first LWM classification model and the second LWM classification model are obtained from the TMED-2 dataset, the training data of the first LWM classification model includes data with view labels and data without view labels, and the training data of the second LWM classification model includes data with AS severity labels and data without AS severity labels.

[0035] Furthermore, the working process of the Rep LKFFN is:

[0036] In Rep LKFFN, the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, and the output is recorded as z'. Then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

[0037] Furthermore, the two submodules in the first Stage module are respectively recorded as the first submodule and the second submodule. The working process of the first Stage module is:

[0038] In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule. The output of the Rep LKFFN of the first submodule is recorded as q2, and then q1 and q2 are concatenated to obtain the concatenation result q';

[0039] q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule. The output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q".

[0040] The connection result q" is the output of the first Stage module.

[0041] Furthermore, the first LWM classification model is trained using a semi-supervised learning method, and the specific training process is as follows:

[0042] Step 1: Initialize the data batch number t=1;

[0043] Step 2: sampling the tth batch of data from the acquired echocardiogram;

[0044] Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch, record the weak enhancement result of the unlabeled data x as WA(x), and then use the classification result P of the first LWM classification model for WA(x) as model (WA(x)) passes through Softmax to obtain the probability distribution;

[0045] If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo label of the unlabeled data x, and the unlabeled data x is added to the training process of the first LWM classification model, and the total loss is calculated based on the pseudo label;

[0046] Otherwise, the unlabeled data x does not participate in the training process;

[0047] Step 4: Stop the training process of the tth batch until the total loss converges;

[0048] Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

[0049] Furthermore, the confidence level is:

[0050]

[0051] in, is the adaptive local threshold of the c-th category during the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, and C is the total number of categories.

[0052] Furthermore, in step 3, the total loss is calculated as follows:

[0053] The supervision loss of labeled data is denoted as L s , the unsupervised loss of unlabeled data is recorded as L u , back-propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss

[0054] like and The dot product result is less than or equal to zero, then the total loss function is:

[0055] L=L s

[0056] like and The dot product result of is greater than zero, then the total loss function is:

[0057] L=L s +αL u +βL f

[0058] Unsupervised loss L u The calculation formula is:

[0059]

[0060] Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the strong enhancement result of x, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of the first LWM classification model for TSA(x), L f represents the adaptive fair regularization penalty, α represents the unlabeled loss coefficient, β represents the adaptive fair regularization penalty coefficient, and S represents the Softmax operation.

[0061] Furthermore, the working process of the aggregation module is as follows:

[0062] Step 1: The number of all echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view relevance weight of each echocardiogram is calculated:

[0063] w i =p[θ V (x i )∈RV]

[0064] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for echocardiogram x i The view classification results, RV represents the relevant view, θV (x i )∈RV indicates that the view classification result belongs to the relevant view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight;

[0065] Step 2: Aggregate the outputs of the second LWM classification model for all echocardiograms according to the weights:

[0066]

[0067] Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

[0068] The beneficial effects of the present invention are:

[0069] The present invention designs a lightweight model LWM based on structural reparameterization, deep large kernel convolution and wavelet downsampling. LWM has the characteristics of small number of parameters, low computational complexity and high classification accuracy, which can improve the efficiency of classification. In addition, semi-supervised learning is used to train LWM. Semi-supervised learning uses targeted strong enhancement and selective total loss calculation. The model LWM can be trained using labeled data and a large amount of unlabeled data. It has efficient data utilization capabilities and avoids the waste of unlabeled data. Compared with traditional semi-supervised methods, the semi-supervised learning method of the present invention has more superior performance. The trained LWM realizes view classification and image-level AS severity classification, and then uses an aggregation mechanism based on view correlation to aggregate the image-level AS severity classification results output by LWM, thereby obtaining patient-level AS severity classification results, and the classification results are highly interpretable.

[0070] On the public TMED dataset, the proposed method achieved a balanced accuracy of 97.8% in view classification and 76.7% in patient-level AS severity classification through a view correlation-based aggregation mechanism. This demonstrates that the proposed method achieves superior classification performance with a lightweight model and semi-supervised training. Furthermore, all work was performed on a single NVIDIA RTX-4070 GPU, reducing the computing power requirements of the medical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is the structural diagram of the first LWM classification model;

[0072] Figure 2 It is the structural diagram of the DW Rep Block unit;

[0073] Figure 3 It is the structural diagram of the PW Rep Block unit;

[0074] Activation is the activation function layer;

[0075] Figure 4 This is the structural diagram of the Rep LKFFN module;

[0076] Figure 5 This is the structure diagram of the Rep HWD Block module;

[0077] Figure 6a It is a PLAX type view;

[0078] Figure 6b It is a PSAX type view;

[0079] Figure 7 Flowchart of the aggregation mechanism based on view correlation;

[0080] Figure 8 Flowchart for Efficient SSL semi-supervised learning;

[0081] Figure 9 This is a heat map of LWM’s classification of view types;

[0082] In each row, the first figure is the original image in the test set, the second figure is the heat map obtained by the method of the present invention, the third figure is the heat map obtained by the WideResNet model, and the fourth figure is the heat map obtained by the FastViT-T8 model;

[0083] Figure 10a is the confusion matrix classified by three view types (PLAX, PSAX, Other);

[0084] Figure 10b is the confusion matrix classified by four view types (PLAX, PSAX, A2C, A4C);

[0085] Figure 11a It is a scatter plot of the feature matrix visualization output by LWM-4V;

[0086] Figure 11b It is a scatter plot of the feature matrix visualization output by LWM-3V;

[0087] Figure 12a This is the ROC curve when the TMED-1 dataset uses the split2 partitioning method;

[0088] The blue curve represents the ROC curve of TMED-1, and the red curve represents the ROC curve of TMED-2;

[0089] Figure 12b This is the ROC curve when the TMED-1 dataset uses the split3 partitioning method;

[0090] Figure 13a The results of the ablation experiment on the components of LWM;

[0091] In the figure, Parameters represents the number of parameters and Balanced Accuracy represents the balanced accuracy. This shows that RepLKFFN and Rep HWD Block achieve the best performance while maintaining lightweight.

[0092] Figure 13b The results of the ablation experiment on Efficient SSL are shown in the figure;

[0093] It shows that TSA strong enhancement and SLC loss calculation have the best accuracy and robustness;

[0094] Figure 14 Schematic diagram of the model's robustness to image quality degradation;

[0095] In the figure, Pixels Removed represents the percentage of pixel discarded. DETAILED DESCRIPTION

[0096] Specific implementation method 1: Combination Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 and Figure 5 This embodiment describes an AS severity classification system based on a lightweight model and semi-supervised learning, the system comprising an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model, and an aggregation module;

[0097] The echocardiogram acquisition module is used to acquire echocardiogram data of the subject to be detected, and the echocardiogram data includes multiple echocardiograms;

[0098] The first LWM classification model is used to classify each echocardiogram in the acquired echocardiographic data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module, and a classifier; and the working process of the first LWM classification model is as follows:

[0099] In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module;

[0100] The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module;

[0101] The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module;

[0102] The output of the third Stage module is then used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is then passed through the classifier to obtain the view classification result;

[0103] The first Rep HWD Block module includes a DW Rep Block unit, an HWT Block unit, and a PW RepBlock unit, and the working process of the first Rep HWD Block module is as follows:

[0104] The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module;

[0105] The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module;

[0106] The difference is that the convolution kernel sizes of the two parallel convolution branches of the DW Rep Block unit in the first Rep HWD Block module are 3×3 and 1×1 respectively. The purpose is to reduce the number of parameters and computational complexity by using small convolution kernels. The convolution kernel sizes of the two parallel convolution branches of the DW Rep Block unit in the second Rep HWD Block module, the third Rep HWD Block module, and the fourth Rep HWD Block module are 7×7 and 5×5 respectively. The HWTBlock unit can decompose the input into low-frequency sub-bands (y L ) and three high frequency sub-bands (y HL ,y LH ,y HH ), the resolution of each sub-band is half of the input image, and then the HWT Block unit splices the low-frequency sub-band and the high-frequency sub-band in the channel dimension to generate a new tensor.

[0107] The DW Rep Block unit captures the key information in the input data and extracts spatial features before downsampling. The HWT Block unit overcomes the problem of spatial information loss caused by traditional downsampling, retaining as much key information as possible while reducing the spatial resolution of the feature map. The PW Rep Block unit implements channel number conversion and channel information fusion to output the final feature map. The four Rep HWD Block modules can achieve high-quality downsampling, spatial feature extraction and channel number conversion, and have better performance and lower computational complexity than traditional strided convolution.

[0108] The first stage module includes 2 submodules, the second stage module includes 2 submodules, the third stage module includes 6 submodules, and the fourth stage module includes 2 submodules, and the structure and working process of each submodule are the same;

[0109] The submodule includes a DW Rep Block unit (where the convolution kernel sizes of the two parallel convolution branches are 3×3 and 1×1, respectively, for extracting local features) and a Rep LKFFN. The working process within the submodule is as follows:

[0110] First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule;

[0111] The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer and the output of the flattening layer are connected. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer.

[0112] The second LWM classification model is used to perform image-level AS severity classification on the acquired echocardiographic data, and the structure of the second LWM classification model is exactly the same as that of the first LWM classification model except for the fully connected layer. The only difference between the fully connected layers of the first and second LWM classification models is the number of output categories.

[0113] The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

[0114] The structure and parameter settings of the first LWM classification model are shown in Table 1;

[0115] Table 1

[0116]

[0117]

[0118] Specific implementation method 2: Combination Figure 2 This embodiment is described below. The difference between this embodiment and the first embodiment is that during the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depthwise separable convolution layer and a batch normalization layer. The DW Rep Block captures spatial information of different scales through two depthwise separable convolutions of different sizes to achieve feature extraction. The working process of the DWRep Block unit during training is as follows:

[0119] y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x]

[0120] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer;

[0121] In the inference process of the first LWM classification model, the working process of the DW Rep Block unit is as follows:

[0122] y′=σ(Conv 12 (x′))

[0123] Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

[0124] Other steps and parameters are the same as those in the first embodiment.

[0125] During the training and inference process of the second LWM classification model and the first LWM classification model, the working process of the DW Rep Block unit is the same.

[0126] Specific implementation method three: Combination Figure 3 This embodiment differs from the first or second embodiment in that, during the training of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer. The working process of the PW Rep Block unit during training is as follows:

[0127] y″=σ[BN(Conv(x″))]

[0128] Where y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer;

[0129] During the inference process of the first LWM classification model, the working process of the PW Rep Block unit is as follows:

[0130] y″′=σ(Conv(x″′))

[0131] Among them, x″ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

[0132] Other steps and parameters are the same as those in the first or second embodiment.

[0133] The PW Rep Block is a single-branch cascade structure. It consists of a point-wise convolution layer, a batch normalization layer, and an activation layer. It uses point convolution to achieve channel number conversion and channel information fusion.

[0134] During training, the Rep Block is a complex structure that integrates convolutional layers, batch normalization layers, and activation layers. During inference, the Rep Block uses structural reparameterization to reparameterize the training structure and parameters to a simpler structure consisting of only convolutional layers and activation layers, thereby reducing the number of parameters and FLOPs. Conventional composite structures of convolutional layers, batch normalization layers, and activation layers cannot achieve the computational and memory cost reductions achieved through structural reparameterization.

[0135] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the training data for the first and second LWM classification models are echocardiograms acquired from the TMED-2 dataset. Specifically, the echocardiograms acquired by the first LWM classification model include labeled data with view labels and a large amount of unlabeled data, while the echocardiograms acquired by the second LWM classification model include labeled data with AS severity labels and a large amount of unlabeled data.

[0136] The other steps and parameters are the same as those in the first to third embodiments.

[0137] TMED-2 is currently the most commonly used training and testing benchmark for AS research. TMED-2 view labels include PLAX, PSAX, A2C, A4C, and Other, and AS severity labels include No AS, Early AS, and Significant AS. The view labels provided by TMED-2 can be used to classify three view types (Other, PLAX, PSAX) and four view types (A2C, A4C, PLAX, PSAX). The TMED-2 dataset contains 17,270 dual-label images from 599 studies. These 599 studies were divided into training, validation, and test sets in a ratio of 360 / 119 / 120. In addition, the TMED-2 dataset contains 353,500 unlabeled echocardiograms (the unlabeled set). Many previous studies have used only labeled data, resulting in a large amount of unlabeled data being wasted.

[0138] The present invention effectively uses unlabeled data to train the model. Similar to the prediction process, the input of the second LWM classification model in the training process is the echocardiogram classified as belonging to the relevant view by the first LWM classification model.

[0139] Specific implementation method five: Combination Figure 1 and Figure 4 This embodiment is different from the first to fourth embodiments in that the working process of the Rep LKFFN is as follows:

[0140] In Rep LKFFN, the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, and the output is recorded as z'. Then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

[0141] The two convolution kernels of the DW Rep Block unit have sizes of 7×7 and 5×5 respectively. The DW Rep Block unit can capture a wide range of spatial information at different scales. The two PW Rep Block units realize the expansion and restoration of the number of channels within the Rep LKFFN. In general, the Rep LKFFN is used to extract global features. The submodule combines the DW Rep Block unit and the Rep LKFFN to achieve the fusion of local information, global information, and channel information.

[0142] The other steps and parameters are the same as those in the first to fourth embodiments.

[0143] Specific implementation method six: combination Figure 1 This embodiment differs from Embodiments 1 to 5 in that the two submodules within the first Stage module are respectively referred to as the first submodule and the second submodule, and the working process of the first Stage module is as follows:

[0144] In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule. The output of the Rep LKFFN of the first submodule is recorded as q2, and then q1 and q2 are concatenated to obtain the concatenation result q';

[0145] q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule. The output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q".

[0146] The connection result q" is the output of the first Stage module.

[0147] The other steps and parameters are the same as those in the first to fifth embodiments.

[0148] The DW Rep Block unit is mainly used to extract local spatial features, while the Rep LKFFN is used to extract global spatial features and fuse channel information. By combining the DW Rep Block unit and Rep LKFFN, LWM simultaneously focuses on information in both spatial and channel dimensions, enabling a more comprehensive understanding and representation of the characteristics of the input data.

[0149] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that the first LWM classification model is trained using a semi-supervised learning (Efficient SSL) method. The process of semi-supervised learning is as follows: Figure 8 As shown in Figure 2, the specific process of training is:

[0150] Step 1: Initialize the data batch number t=1;

[0151] Step 2: sampling the tth batch of data from the acquired echocardiogram;

[0152] Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch (weak enhancement can be achieved through rotation and displacement algorithms), record the weak enhancement result of the unlabeled data x as WA(x), and then calculate the classification result P of WA(x) by the first LWM classification model. model (WA(x)) passes through Softmax to obtain the probability distribution;

[0153] If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo label of the unlabeled data x, the first LWM classification model is trained using the unlabeled data x, and the total loss is calculated based on the pseudo label;

[0154] Otherwise, the unlabeled data x does not participate in the training process;

[0155] Step 4: Stop the training process of the tth batch until the total loss function converges;

[0156] Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

[0157] The other steps and parameters are the same as those in the first to sixth embodiments.

[0158] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the confidence level is:

[0159]

[0160] in, is the adaptive local threshold of the c-th category during the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, C is the total number of categories, and τ t It can be set according to actual situation.

[0161] The other steps and parameters are the same as those in the first to seventh embodiments.

[0162] Mainstream semi-supervised learning uses a fixed threshold to determine whether a pseudo-label is available, which will discard a large number of available pseudo-labels, resulting in low utilization of unlabeled data. The present invention adopts a dynamic threshold in the pseudo-label generation process. The semi-supervised training model with a dynamic threshold sets a lower confidence threshold for the pseudo-labels of samples of different categories in the early stage, and the confidence threshold of the pseudo-label increases dynamically as the training progresses. Therefore, the semi-supervised training model with a dynamic threshold can introduce more pseudo-labels to participate in training, improve the utilization efficiency of unlabeled data, and can use labeled data and a large amount of unlabeled data to train the model at the same time, which has better performance than the fully supervised method and the traditional semi-supervised method.

[0163] Specific embodiment 9: This embodiment differs from any one of specific embodiments 1 to 8 in that, in step 3, the selectivity total loss calculation (SLC) method is:

[0164] The supervision loss of labeled data is denoted as L s (calculated by CrossEntropyLoss), the unsupervised loss of unlabeled data is recorded as L u (calculated by ConsistencyLoss), back propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss

[0165] like and The dot product result is less than or equal to zero, then the total loss function is:

[0166] L= L s

[0167] like and The dot product result of is greater than zero, then the total loss function is:

[0168] L=L s +αL u +βL f

[0169] The formula for calculating the unsupervised loss is:

[0170]

[0171] Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the result of strong enhancement of x using TSA, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of the first LWM classification model for TSA(x), L f represents the adaptive fairness regularization penalty (SAF) proposed by FreeMatch, α represents the unlabeled loss coefficient, β represents the adaptive fairness regularization penalty coefficient, and S represents the Softmax operation.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Introducing L f The purpose is to offset the negative impact of the imbalanced distribution of pseudo labels. The default value of α is 1.0, and the default value of β is 0.01. The targeted strong enhancement (TSA) method used in this embodiment includes three local enhancements and three global enhancements. The local algorithm acts on a random rectangular area in the echocardiogram, and the side length of the random rectangular area does not exceed 1 / 2 of the side length of the echocardiogram. The global algorithm acts on the entire echocardiogram. The three local data enhancement algorithms are: injecting random salt and pepper noise spots into the random rectangular area; randomly disrupting the order of pixels in the random rectangular area to achieve local pixel shuffling; directly deleting the random rectangular area and filling it with all black pixels. The three global data enhancement algorithms are: changing the overall brightness of the echocardiogram with a brightness offset of 50%; using a 5×5 Gaussian kernel to perform Gaussian blur on the entire echocardiogram; and randomly discarding the original pixels in the echocardiogram with a probability of no more than 50%. When performing strong enhancement on each batch of unlabeled data, TSA first randomly shuffles the order of the six data enhancement algorithms and randomly selects 1-6 as the enhancement algorithms for this batch. During strong enhancement, each algorithm is randomly applied with a 50% probability. This solves the problem that traditional strong enhancement methods such as color transformation and contrast transformation are not suitable for echocardiograms stored in grayscale form and have poor adaptability. By improving the loss function, this embodiment solves the problem that the loss of traditional SSL is affected by OOD (out-of-distribution) data.

[0174] Specific implementation method ten: Combination Figure 7This embodiment is different from the first embodiment to the ninth embodiment in that the operation process of the aggregation module is as follows:

[0175] Step 1: The number of all echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view relevance weight of each echocardiogram is calculated:

[0176] w i =p[θ V (x i )∈RV]

[0177] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for echocardiogram x i The view classification results are as follows: RV represents related views (related views include PLAX and PSAX type views, such as Figure 6a and Figure 6b shown), θ V (x i )∈RV indicates that the view classification result belongs to the relevant view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight;

[0178] Step 2: Aggregate the outputs of the second LWM classification model for all echocardiograms according to the weights:

[0179]

[0180] Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0182] The polymerization mechanism of this embodiment is as follows Figure 7 As shown in FIG, the view correlation weight is used to perform weighted averaging on all image-level AS severity classification results of the patient, and the sequence number corresponding to the maximum value after the weighted averaging is taken as the patient-level AS severity classification result.

[0183] Recent research has used two aggregation mechanisms: relevant view-prioritization and attention-based aggregation. Traditional relevant view-prioritization simply assigns higher weights or priorities to relevant views, without a rational and interpretable weight calculation formula. The attention-based aggregation mechanism significantly increases computational cost. Furthermore, because the attention layer is directly embedded within the model, it only outputs patient-level AS severity classification, failing to achieve view- and image-level AS severity classification. Therefore, using the attention-based aggregation mechanism to achieve patient-level AS severity classification is incomplete and lacks interpretability.

[0184] Unlike the previous weighted average aggregation mechanism based on view relevance, the present invention provides a clear view relevance weight calculation formula based entirely on view classification results, as well as an aggregation formula based entirely on image-level AS severity classification results. Compared with the traditional weighted average aggregation mechanism, the aggregation mechanism of the present invention fully balances the weights of relevant views and irrelevant views. Compared with the attention aggregation mechanism, the aggregation mechanism of the present invention does not require model modifications and hardly increases computational costs. In addition, the fundamental basis of the aggregation mechanism of the present invention is the view classification results and the image-level AS severity classification results. Therefore, the aggregation mechanism of the present invention has better interpretability than the attention aggregation mechanism.

[0185] Experimental part

[0186] By modifying the fully connected layer of the classifier, multiple LWMs were pre-configured. These LWMs differed only in the output categories of the fully connected layer. The LWM that implements classification for three view types (PLAX, PSAX, and Other) was designated LWM-3V, while the LWM that implements classification for four view types (PLAX, PSAX, A2C, and A4C) was designated LWM-4V. The LWM that implements image-level AS severity classification for TMED-1 was designated LWM-AS1, while the LWM that implements image-level AS severity classification for TMED-2 was designated LWM-AS2.

[0187] We trained multiple LWMs for view classification and image-level AS severity classification using both fully supervised and semi-supervised (Efficient SSL) training methods. Fully supervised training used only images from the official training set of the TMED dataset (labeled data), while Efficient SSL used all images from the official training set of the TMED dataset and the unlabeled dataset.

[0188] Furthermore, we used a view-correlation-based aggregation mechanism to aggregate image-level AS severity classification results to obtain patient-level AS severity. All of the above work was performed using a single consumer-grade NVIDIA RTX-4070 GPU. It should be noted that we did not use any pre-training or transfer learning techniques for LWM.

[0189] The TMED-2 dataset is used as the training and testing benchmark for view classification. LWM-3V and LWM-4V are trained using both fully supervised and semi-supervised (Efficient SSL) training methods to achieve view classification. The maximum learning rate for fully supervised training is 5×10 -4 , the minimum learning rate is 5×10 -6 The fully supervised training uses the cosine annealing learning rate update strategy. The maximum learning rate of the semi-supervised training is set to 10 -3 , the minimum learning rate is set to 10 -5 Semi-supervised training uses a cosine annealing learning rate update strategy with a warmup epoch of 5. The batch size of labeled data for both training methods is set to 16. The AdamW optimizer is used to update model parameters, and the exponential moving average (EMA) is used to improve model robustness. Detailed hyperparameters for the training process are shown in Table 2.

[0190] Table 2 Training hyperparameters for the TMED-2 view type classification experiment

[0191]

[0192]

[0193] The TMED-1 and TMED-2 datasets were used as training and testing benchmarks for AS severity classification. AS severity classification includes both image-level and patient-level AS severity classification. The method of the present invention not only performs image-level AS severity classification, but also patient-level AS severity classification.

[0194] LWM-AS1 was trained using both fully supervised and semi-supervised (Efficient SSL) training methods to achieve image-level AS severity classification for TMED-1. Due to the low resolution of echocardiograms in the TMED-1 dataset (only 64×64), we used CutMix (S. Yun, D. Han, SJ Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 6023-6032.) and data augmentation methods such as Resize to reduce the risk of overfitting. The maximum learning rate for fully supervised training was 10 -3 , the minimum learning rate is 10 -5 The fully supervised training uses the cosine annealing learning rate update strategy. The maximum learning rate of the semi-supervised training is 5×10 -3 , the minimum learning rate is 5×10 -5 Semi-supervised training uses a cosine annealing learning rate update strategy with a warmup epoch of 5. Both training methods use the AdamW optimizer to update model parameters, and both use the exponential moving average (EMA) to improve model robustness. The detailed hyperparameter settings for the training process are shown in Table 3.

[0195] Table 3 Training hyperparameters for image-level AS severity classification of TMED-1

[0196]

[0197] We train LWM-AS2 to achieve image-level AS severity classification for TMED-2 using both fully supervised and semi-supervised (Efficient SSL) training methods. The maximum learning rate for fully supervised training is 10 -3 , the minimum learning rate is 10 -5 The fully supervised training uses the cosine annealing learning rate update strategy. The maximum learning rate of the semi-supervised training is 5×10 -3 , the minimum learning rate is 5×10 -5 Semi-supervised training uses a cosine annealing learning rate update strategy with a warmup epoch of 5. Both training methods use the AdamW optimizer to update model parameters, and both use the exponential moving average (EMA) to improve model robustness. The detailed hyperparameter settings for the training process are shown in Table 4.

[0198] Table 4 Training hyperparameters for image-level AS severity classification of TMED-2

[0199]

[0200]

[0201] This paper uses a view-correlation-based aggregation mechanism to aggregate the image-level classification results of LWM-AS2, thereby achieving patient-level AS severity classification using TMED-2. Specifically, we use LWM-4V to output the view prediction results for each echocardiogram and calculate the view-correlation weight. We also use LWM-AS2 to output the image-level AS severity prediction results for each echocardiogram. Furthermore, we obtain the patient-level AS severity prediction results by taking a weighted average based on view-correlation for all image-level AS severity prediction results for the same patient.

[0202] Table 5 compares the performance of our method and previous methods on the view classification task. The balanced accuracy in Table 5 and the following tables is calculated using the metrics library in scikit-learn (F. Pedregosa et al., "Scikit-learn: Machine learning in Python," vol. 12, pp. 2825-2830, 2011). Clearly, our proposed LWM achieves state-of-the-art accuracy even with fully supervised training alone, and LWM trained with Efficient SSL further improves performance.

[0203] Table 5 Comparison of view classification performance between the proposed method and previous methods on the TMED-2 dataset

[0204]

[0205] N.Ahmadi et al. refers to the document "H.Vaseli et al.,"ProtoASNet:DynamicPrototypes for Inherently Interpretable and Uncertainty-Aware Aortic StenosisClassification in Echocardiography,"in International Conference on MedicalImage Computing and Computer-Assisted Intervention,2023,pp.368-378:Springer."; Huang et al. al. refers to the document "Z.Huang,M.-J.Sidhom,BSWessler,andM.CJapaHughes,"Fix-a-step: Semi-supervised learning from uncuratedunlabeled data,"2022."; the training method VAT comes from "T.Miyato,S.-i.Maeda,M.Koyama,SJItopaIshii,and m.intelligence,"Virtual adversarial training:aregularization method for supervised and semi-supervised learning,"vol.41,no.8,pp.1979-1993,2018."; the model WideResNet comes from "S.Zagoruyko andN.JapaKomodakis,"Wide residual networks,"2016.".

[0206] The view classification performance of LWM and FastViT-T8 is compared in Table 6. Under the same fully supervised training and evaluation conditions, our proposed LWM has superior performance while having fewer parameters and computational complexity.

[0207] Table 6 Comparison of view type classification performance between LWM and FastViT-T8 models on the TMED-2 dataset

[0208]

[0209] Model FastViT-T8 comes from "PKAVasu, J.Gabriel, J.Zhu, O.Tuzel, andA.Ranjan, "FastViT: A fast hybrid vision transformer using structuralreparameterization," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp.5785-5795."

[0210] Grad-CAM (RRSelvaraju, M.Cogswell, A.Das, R.Vedantam, D.Parikh, and D.Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," in Proceedings of the IEEE international conference on computer vision, 2017, pp.618-626) is applied to the last layer of each Stage module of LWM to visualize the prediction process and improve the interpretability of the prediction results. The heat map generated by Grad-CAM is as follows: Figure 9 As shown in FIG. 1 , it can be seen that the method of the present invention can accurately focus on the area related to the task, so the method of the present invention has excellent classification performance.

[0211] We also plotted the following based on the view classification results: Figure 10a and Figure 10b The confusion matrix shown in Figure 2 shows that when the classification accuracy is very high, the errors are mostly due to confusion between similar categories (PLAX and PSAX, A2C and A4C), which does not particularly affect the final result.

[0212] We use UMAP (L.McInnes, J.Healy, and JJapaMelville, "Umap: Uniform manifold approximation and projection for dimension reduction," 2018.) to reduce the dimension and visualize the feature tensor output by LWM, which indirectly reflects the effectiveness of the combination of LWM and Efficient SSL proposed in this paper from another perspective. Figure 11a and Figure 11b As shown in Figure 3, data points of the same category are clearly clustered, while data points of different categories have clear boundaries. Therefore, the LWM trained with Efficient SSL learns meaningful features of the data and can effectively distinguish different categories.

[0213] Table 7 compares the performance of the proposed method and previous methods on the TMED-1 image-level AS severity classification task. It can be seen that the proposed method is better than the previous methods in terms of both performance and parameter count.

[0214] Table 7 Comparison of image-level AS severity classification performance between the proposed method and previous methods on the TMED-1 dataset

[0215]

[0216]

[0217] MixMatch is a semi-supervised method from the paper "D.Berthelot,N.Carlini,I.Goodfellow,N.Papernot,A.Oliver,and CAJAinipsRaffel,"Mixmatch:A holistic approach to semi-supervised learning,"vol.32,2019."

[0218] The present invention achieved an 82.2% accuracy rate in image-level AS severity classification on the TMED-2 dataset. The ROC curve, which is not affected by the sample category distribution, was used to evaluate the performance of image-level AS severity classification. Positive and negative samples were divided based on the criteria of no AS and AS (including mild, moderate and severe, etc.) with clinical application value. Figure 12a and Figure 12b The ROC curve is shown.

[0219] As shown in Table 8, the method of the present invention achieves a good patient-level AS severity classification effect with an extremely lightweight model and semi-supervised training method.

[0220] Table 8

[0221]

[0222] In Table 8, the method Huang et al. in the first row refers to "Z. Huang, G. Long, B. Wessler, and M. C. Hughes," TMED 2: a dataset for semi-supervised classification of echocardiograms," in Data Perf: Benchmarking Data for Data-Centric AI Workshop, 2022.", and the method Huang et al. in the second row refers to "Z. Huang, B S Wessler, and M C Hughes," Detecting Heart Disease from Multi-View Ultrasound Images via Supervised Attention Multiple Instance Learning," in Machine Learning for Healthcare Conference, 2023, pp. 285-307: PMLR.", and the VAT model comes from "T. Miyato, S. -i. Maeda, M. Koyama, S J Itopa Ishii, and m. intelligence," Virtual adversarial training: a regularization method for supervised and semi-supervised learning,"vol.41,no.8,pp.1979-1993,2018."

[0223] The method of the present invention achieves patient-level AS severity classification based on view classification and image-level AS severity classification. Therefore, the method of the present invention has better interpretability than the method of directly achieving patient-level AS severity classification using the attention mechanism.

[0224] Previous experiments have demonstrated the effectiveness of the Lightweight Model (LWM) and Efficient Semi-Supervised Learning (Efficient SSL) proposed in this paper. Furthermore, we conducted ablation experiments on the components of the LWM and the improved Efficient SSL approach. We used view classification performance (PLAX, PSAX, A2C, and A4C) as the evaluation metric for the ablation experiments. All hyperparameters in the ablation experiments were consistent with those in the view classification experiments.

[0225] The LWM proposed in the present invention is designed based on structural reparameterization and deep large kernel convolution. LWM uses the large convolution kernel feedforward neural network (Rep LKFFN) and the reparameterizable wavelet downsampling module (Rep HWDBlock) proposed in the present invention. In order to verify the effectiveness of the model components proposed in the present invention, we replaced the DW Rep Block (convolution kernel is 7×7+5×5) in Rep LKFFN with DW Rep Block (convolution kernel is 3×3+1×1) and named it reparameterizable small convolution kernel feedforward neural network (Rep SKFFN). We replaced Rep LKFFN and Rep HWD Block with Rep SKFFN and depthwise convolution with a stride of 2 for ablation experiments.

[0226] In order to verify the effectiveness of our proposed improved method, we replaced ESA and SLC (i.e., the loss calculation method of the present invention) with RandAugment and the conventional loss calculation method to conduct ablation experiments.

[0227] We are Figure 13a and Figure 13b The results of the ablation experiments are presented in the form of bar charts and box plots. We evaluate the components of LWM by balancing accuracy, FLOPs, and parameter count. Efficient SSL methods are evaluated by balancing accuracy and the degree of fluctuation in balancing accuracy.

[0228] In terms of model components, the LWM of Rep LKFFN and Rep HWD Block proposed in this invention has the highest classification accuracy. Rep LKFFN is replaced by Rep SKFFN with a structure using a small convolution kernel. The number of parameters and FLOPs are only reduced by 0.07M and 0.02G respectively (almost negligible), but the classification balance accuracy is reduced by 0.4%. Rep HWDBlock replaces the traditional strided convolution downsampling. The FLOPs increased significantly by 0.4G, but the classification balance accuracy was reduced by 0.5%. In general, Rep LKFFN and Rep HWD Block achieve the best performance while maintaining lightweight.

[0229] In terms of the improvement method of Efficient SSL, Efficient SSL using TSA strong enhancement and SLC loss calculation method has the highest classification accuracy. Replacing TSA strong enhancement with traditional RandAugment, the classification balance accuracy is reduced by 0.3%. Replacing the SLC loss calculation method with the conventional loss calculation method, the classification balance accuracy is reduced by 0.5% (lower than fully supervised training). This shows that the conventional loss calculation method is negatively affected by OOD data. Compared with traditional RandAugment, the TSA strong enhancement proposed in the present invention can effectively improve the performance of semi-supervised learning and has better robustness (small accuracy fluctuation). The SLC loss calculation method can effectively alleviate the negative impact caused by OOD data, and therefore has better performance than the conventional loss calculation method.

[0230] The pixels in the TMED test image are randomly removed at a certain ratio (10% to 50%) and replaced with pure black pixels to simulate different degrees of image quality loss. Figure 14 As shown in Figure 3, even discarding 50% of the pixels only slightly affects the classification accuracy of LWM. Therefore, the LWM trained with Efficient SSL is very robust.

[0231] The present invention proposes a lightweight model (LWM) based on structural reparameterization and deep large kernel convolution design. LWM has a lower number of parameters and computational complexity, and is more suitable for medical systems with lower computing power than previous models. The present invention also proposes an Efficient Semi-Supervised Learning (Efficient SSL) using Targeted Strong Augment (TSA) and Selective Total Loss Calculation (SLC). Efficient SSL can use labeled data and a large amount of unlabeled data to train the model at the same time. Compared with the fully supervised method, Efficient SSL makes full use of the unlabeled AS dataset, thus having more efficient data utilization capabilities; compared with the traditional semi-supervised method, Efficient SSL has superior performance.

[0232] The method presented in this paper uses Efficient SSL to train a LWM for view classification and image-level AS severity classification. Furthermore, the method also uses a view-correlation-based aggregation mechanism to achieve patient-level AS severity classification. Experimental results demonstrate that the method presented in this paper performs well in view classification, image-level AS severity classification, and patient-level AS severity classification, demonstrating its potential for clinical application.

[0233] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. An AS severity classification system based on lightweight model and semi-supervised learning, characterized by: The system includes an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model, and an aggregation module; The echocardiogram acquisition module is used to acquire echocardiogram data of the subject to be detected; The first LWM classification model is used to classify each echocardiogram in the acquired echocardiographic data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module, and a classifier; and the working process of the first LWM classification model is as follows: In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module; The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module; The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module; The output of the third Stage module is then used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is then passed through the classifier to obtain the view classification result; The first Rep HWD Block module includes a DW Rep Block unit, an HWT Block unit, and a PW Rep Block unit, and the working process of the first Rep HWD Block module is as follows: The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module; The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module; The first stage module includes 2 submodules, the second stage module includes 2 submodules, the third stage module includes 6 submodules, and the fourth stage module includes 2 submodules, and the structure and working process of each submodule are the same; The submodule includes the DW Rep Block unit and Rep LKFFN. The working process in the submodule is as follows: First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule; The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer and the output of the flattening layer are connected. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer. The second LWM classification model is used to perform image-level AS severity classification on the acquired echocardiographic data, and the structure of the second LWM classification model is the same as that of the first LWM classification model; The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

2. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 1 is characterized in that: During the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depthwise separable convolution layer and a batch normalization layer. The working process of the DW Rep Block unit during training is as follows: y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer; In the inference process of the first LWM classification model, the working process of the DW Rep Block unit is as follows: y′=σ(Conv 12 (x′)) Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

3. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 2 is characterized in that: During the training of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer. The working process of the PW Rep Block unit during training is as follows: y″=σ[BN(Conv(x″))] Where y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer; During the inference process of the first LWM classification model, the working process of the PW Rep Block unit is as follows: y″′=σ(Conv(x″′)) Among them, x″′ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

4. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 1 is characterized in that: The training data of the first LWM classification model and the second LWM classification model are obtained from the TMED-2 dataset. The training data of the first LWM classification model includes data with view labels and data without view labels. The training data of the second LWM classification model includes data with AS severity labels and data without AS severity labels.

5. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 3 is characterized in that: The work of Rep LKFFN is: In Rep LKFFN, the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, and the output is recorded as z'. Then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

6. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 5, characterized in that: The two submodules in the first Stage module are respectively recorded as the first submodule and the second submodule. The working process of the first Stage module is as follows: In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule. The output of the Rep LKFFN of the first submodule is recorded as q2, and then q1 and q2 are connected to obtain the connection result q'; q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule. The output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q". The connection result q" is the output of the first Stage module.

7. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 6, characterized in that: The first LWM classification model is trained using a semi-supervised learning method. The specific training process is as follows: Step 1: Initialize the data batch number t=1; Step 2: sampling the tth batch of data from the acquired echocardiogram; Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch, record the weak enhancement result of the unlabeled data x as WA(x), and then use the classification result P of the first LWM classification model for WA(x) as model (WA(x)) passes through Softmax to obtain the probability distribution; If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo label of the unlabeled data x, and the unlabeled data x is added to the training process of the first LWM classification model, and the total loss is calculated based on the pseudo label; Otherwise, the unlabeled data x does not participate in the training process; Step 4: Stop the training process of the tth batch until the total loss converges; Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

8. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 7, characterized in that: The confidence level is: in, is the adaptive local threshold of the c-th category during the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, and C is the total number of categories.

9. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 8, characterized in that: In step 3, the total loss is calculated as follows: The supervision loss of labeled data is denoted as L s , the unsupervised loss of unlabeled data is recorded as L u , back-propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss like and The dot product result is less than or equal to zero, then the total loss function is: L=L s like and The dot product result of is greater than zero, then the total loss function is: L=L s +αL u +βL f Unsupervised loss L u The calculation formula is: Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the strong enhancement result of x, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of the first LWM classification model for TSA(x), L f represents the adaptive fair regularization penalty, α represents the unlabeled loss coefficient, β represents the adaptive fair regularization penalty coefficient, and S represents the Softmax operation.

10. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 9, characterized in that: The working process of the aggregation module is as follows: Step 1: The number of all echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view relevance weight of each echocardiogram is calculated: w i =p[θ V (x i )∈RV] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for echocardiogram x i The view classification results, RV represents the relevant view, θ V (x i )∈RV indicates that the view classification result belongs to the relevant view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight; Step 2: Aggregate the outputs of the second LWM classification model for all echocardiograms according to the weights: Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

Images