AS severity classification system based on lightweight model and semi-supervised learning

Abstract

The invention discloses an AS severity classification system based on a lightweight model and semi-supervised learning, and belongs to the field of medical image processing. According to the method, the problems of waste of unlabeled data, poor interpretability of patient-level AS severity classification caused by lack of view classification and image-level AS severity classification, high calculation requirement caused by large model parameter quantity and the like in the existing method are solved. According to the method, a lightweight model LWM is designed based on structure re-parameterization, deep large kernel convolution and wavelet down-sampling, and the LWM is trained through semi-supervised learning. The semi-supervised learning uses targeted strong enhancement and selective total loss calculation, a model LWM can be trained by using label data and a large amount of label-free data, view classification and image-level AS severity classification are realized by the trained LWM, and a patient-level AS severity classification result is obtained by using an aggregation mechanism based on view correlation. The method can be applied to view classification and AS severity classification of echocardiograms.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to an AS severity classification system based on a lightweight model and semi-supervised learning. Background Art

[0002] The aortic valve (AV) is located between the left ventricle and the aorta and is the last barrier before blood flows out of the heart. The aortic valve can control the direction of blood flow and is an important valve. Once the aortic valve thickens and calcifies, it may cause the outlet to narrow, causing aortic stenosis (AS). AS is a degenerative valvular disease with a high mortality rate in critically ill patients. In clinical practice, the diagnosis of the severity of AS is usually based on the cardiologist's observation of relevant views in echocardiograms. Traditional clinical diagnostic methods are expensive and subjective. Therefore, there is an urgent need for an efficient and feasible auxiliary diagnostic method. With the development of deep learning, using deep learning to assist in the diagnosis of AS severity has become a feasible method.

[0003] In fact, only a part of the echocardiogram is relevant for the diagnosis of AS, because the aortic valve is only visible in these views (called relevant views). According to clinical standards, we regard the parasternal long axis view (PLAX) and the parasternal short axis view (PSAX) as relevant views. In clinical practice, the first step to diagnose the severity of AS is usually to correctly identify the relevant views on the transthoracic echocardiogram (TTE), and the second step is to diagnose the severity of AS based on the relevant views. However, current deep learning-based methods pay more attention to patient-level AS severity classification and related aggregation mechanisms, and less attention to view classification and image-level AS severity classification. View classification and image-level AS severity classification are clinically regarded as the prerequisite tasks of patient-level AS severity classification. The lack of view classification and image-level AS severity classification will reduce the interpretability of patient-level AS severity classification. In addition, the current methods do not consider the limited computing power of the medical system. The parameters of the model used are large and the computational complexity is high. In addition, due to the high cost of annotating medical images, only a small number of medical images can be annotated, which generates a large amount of unlabeled data. Most current methods use labeled data to perform fully supervised training on the model and cannot utilize a large amount of unlabeled data.

[0004] In summary, since the existing methods pay less attention to view classification and image-level AS severity classification, the interpretability of patient-level AS severity classification results is poor; since unlabeled data is not used, there is serious data waste and it is difficult to obtain the best classification performance by training only on labeled data; and since the model used by the existing methods has a large number of parameters and high computational complexity, the classification efficiency is low and the computing power requirements of the medical system are high. Therefore, it is very necessary to propose a new AS severity classification technology to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of existing classification methods, such as waste of unlabeled data, lack of view classification and image-level AS severity classification, poor interpretability of patient-level AS severity classification results, and high computing power requirements for medical systems. An AS severity classification system based on lightweight models and semi-supervised learning is proposed.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an AS severity classification system based on a lightweight model and semi-supervised learning, the system comprising an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model and an aggregation module;

[0007] The ultrasound cardiogram acquisition module is used to acquire ultrasound cardiogram data of the object to be detected;

[0008] The first LWM classification model is used to classify the views of each echocardiogram in the acquired echocardiogram data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module and a classifier; and the working process of the first LWM classification model is:

[0009] In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module;

[0010] The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module;

[0011] The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module;

[0012] The output of the third Stage module is used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is passed through the classifier to obtain the view classification result.

[0013] The first Rep HWD Block module includes a DW Rep Block unit, a HWT Block unit and a PW RepBlock unit, and the working process of the first Rep HWD Block module is as follows:

[0014] The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module;

[0015] The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module;

[0016] The first stage module includes 2 sub-modules, the second stage module includes 2 sub-modules, the third stage module includes 6 sub-modules, and the fourth stage module includes 2 sub-modules, and the structure and working process of each sub-module are the same;

[0017] The submodule includes DW Rep Block unit and Rep LKFFN. The working process in the submodule is as follows:

[0018] First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule;

[0019] The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer is connected with the output of the flattening layer. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer.

[0020] The second LWM classification model is used to classify the severity of AS at the image level on the acquired echocardiographic data, and the structure of the second LWM classification model is the same as that of the first LWM classification model;

[0021] The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

[0022] Furthermore, during the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depth-separable convolution layer and a batch normalization layer. The working process of the DW Rep Block unit during training is:

[0023] y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x]

[0024] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer;

[0025] In the reasoning process of the first LWM classification model, the working process of the DW Rep Block unit is:

[0026] y′=σ(Conv 12 (x′))

[0027] Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

[0028] Furthermore, during the training process of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer. The working process of the PW Rep Block unit during training is as follows:

[0029] y″=σ[BN(Conv(x″))]

[0030] Among them, y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer;

[0031] In the reasoning process of the first LWM classification model, the working process of the PW Rep Block unit is:

[0032] y″′=σ(Conv(x″′))

[0033] Among them, x″ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

[0034] Furthermore, the training data of the first LWM classification model and the second LWM classification model are obtained from the TMED-2 data set, the training data of the first LWM classification model includes data with view labels and data without view labels, and the training data of the second LWM classification model includes data with AS severity labels and data without AS severity labels.

[0035] Furthermore, the working process of the Rep LKFFN is:

[0036] In Rep LKFFN, after the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, the output is recorded as z', and then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

[0037] Furthermore, the two submodules in the first Stage module are respectively recorded as the first submodule and the second submodule, and the working process of the first Stage module is:

[0038] In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule, and the output of the Rep LKFFN of the first submodule is recorded as q2, and then q1 and q2 are connected to obtain the connection result q';

[0039] q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule, and the output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q".

[0040] The connection result q" is the output of the first Stage module.

[0041] Furthermore, the first LWM classification model is trained by semi-supervised learning, and the specific training process is as follows:

[0042] Step 1, initialize the data batch number t=1;

[0043] Step 2: sampling the tth batch of data from the acquired echocardiogram;

[0044] Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch, record the weak enhancement result of the unlabeled data x as WA(x), and then calculate the classification result P of WA(x) by the first LWM classification model. model (WA(x)) passes through Softmax to obtain probability distribution;

[0045] If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo-label of the unlabeled data x, the unlabeled data x is added to the training process of the first LWM classification model, and the total loss is calculated based on the pseudo-label;

[0046] Otherwise, the unlabeled data x does not participate in the training process;

[0047] Step 4: Stop the training process of the tth batch until the total loss converges;

[0048] Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

[0049] Furthermore, the confidence level is:

[0050]

[0051] in, is the adaptive local threshold of the c-th category in the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, and C is the total number of categories.

[0052] Furthermore, in step 3, the total loss is calculated as follows:

[0053] The supervision loss of labeled data is denoted as L s , the unsupervised loss of unlabeled data is recorded as L u , back propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss

[0054] like and The dot product result is less than or equal to zero, then the total loss function is:

[0055] L=L s

[0056] like and The dot product result of is greater than zero, then the total loss function is:

[0057] L=L s +αL u +βL f

[0058] Unsupervised loss L u The calculation formula is:

[0059]

[0060] Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the strong enhancement result of x, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of TSA(x) by the first LWM classification model, L f represents the adaptive fair regularization penalty, α represents the unlabeled loss coefficient, β represents the adaptive fair regularization penalty coefficient, and S represents the Softmax operation.

[0061] Furthermore, the working process of the aggregation module is:

[0062] Step 1: The number of all the echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view correlation weight of each echocardiogram is calculated respectively:

[0063] w i =p[θ V (x i )∈RV]

[0064] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for the echocardiogram x i The view classification results of , RV represents the relevant view, θV (x i )∈RV means that the view classification result belongs to the related view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight;

[0065] Step 2: Aggregate the outputs of all echocardiograms after the second LWM classification model according to the weights:

[0066]

[0067] Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

[0068] The beneficial effects of the present invention are:

[0069] The present invention designs a lightweight model LWM based on structural re-parameterization, deep large kernel convolution and wavelet downsampling. LWM has the characteristics of small number of parameters, low computational complexity and high classification accuracy, which can improve the efficiency of classification. Moreover, semi-supervised learning is used to train LWM. Semi-supervised learning uses targeted strong enhancement and selective total loss calculation. The model LWM can be trained with labeled data and a large amount of unlabeled data. It has efficient data utilization capabilities and avoids the waste of unlabeled data. Compared with traditional semi-supervised methods, the semi-supervised learning method of the present invention has superior performance. The trained LWM realizes view classification and image-level AS severity classification, and then uses an aggregation mechanism based on view correlation to aggregate the image-level AS severity classification results output by LWM, thereby obtaining patient-level AS severity classification results, and the classification results are highly interpretable.

[0070] On the public TMED dataset, the method of the present invention achieved a balanced accuracy of 97.8% in view classification, and achieved a balanced accuracy of 76.7% in patient-level AS severity classification through an aggregation mechanism based on view correlation, indicating that the method of the present invention achieved superior classification performance with a lightweight model and semi-supervised training. Moreover, all the work was completed by a single NVIDIA RTX-4070 GPU, reducing the requirements for medical system computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the structural diagram of the first LWM classification model;

[0072] Figure 2 It is the structural diagram of the DW Rep Block unit;

[0073] Figure 3 It is the block diagram of the PW Rep Block unit;

[0074] Activation is the activation function layer;

[0075] Figure 4 It is the structural diagram of the Rep LKFFN module;

[0076] Figure 5 This is the structure diagram of the Rep HWD Block module;

[0077] Figure 6a It is a PLAX type view;

[0078] Figure 6b is a PSAX type view;

[0079] Figure 7 is a flow chart of the aggregation mechanism based on view correlation;

[0080] Figure 8 Flowchart for semi-supervised learning of Efficient SSL;

[0081] Fig. 9 This is the heat map of LWM’s classification of view types;

[0082] In each row, the first figure is the original image in the test set, the second figure is the heat map obtained by the method of the present invention, the third figure is the heat map obtained by the WideResNet model, and the fourth figure is the heat map obtained by the FastViT-T8 model;

[0083] Fig.10a is the confusion matrix classified by three view types (PLAX, PSAX, Other);

[0084] Fig.10b is the confusion matrix classified by four view types (PLAX, PSAX, A2C, A4C);

[0085] Fig.11a It is a scatter plot of the feature matrix visualization output by LWM-4V;

[0086] Fig.11b It is a scatter plot of the feature matrix visualization output by LWM-3V;

[0087] Fig.12a This is the ROC curve of the TMED-1 dataset using the split2 partitioning method;

[0088] The blue curve represents the ROC curve of TMED-1, and the red curve represents the ROC curve of TMED-2;

[0089] Figure 12b This is the ROC curve when the TMED-1 dataset is divided using the split3 method;

[0090] Fig.13a The results of the ablation experiment on the components of LWM are shown in Figure 2.

[0091] In the figure, Parameters represents the parameter quantity and Balanced Accuracy represents the balanced accuracy; it shows that RepLKFFN and Rep HWD Block achieve the best performance while maintaining lightweight;

[0092] Fig.13b This is the result of the ablation experiment on Efficient SSL;

[0093] It is shown that TSA strong enhancement and SLC loss calculation have the best accuracy and robustness;

[0094] Fig.14 Schematic diagram of the model's robustness to image quality degradation;

[0095] In the figure, Pixels Removed represents the percentage of pixel discarded. DETAILED DESCRIPTION

[0096] Specific implementation method 1: Combination Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 The present embodiment is described as follows: The present embodiment describes an AS severity classification system based on a lightweight model and semi-supervised learning, the system comprising an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model, and an aggregation module;

[0097] The echocardiogram acquisition module is used to acquire echocardiogram data of the object to be detected, and the echocardiogram data includes a plurality of echocardiograms;

[0098] The first LWM classification model is used to classify the views of each echocardiogram in the acquired echocardiogram data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module and a classifier; and the working process of the first LWM classification model is:

[0099] In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module;

[0100] The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module;

[0101] The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module;

[0102] The output of the third Stage module is used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is passed through the classifier to obtain the view classification result.

[0103] The first Rep HWD Block module includes a DW Rep Block unit, an HWT Block unit and a PW RepBlock unit, and the working process of the first Rep HWD Block module is as follows:

[0104] The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module;

[0105] The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module;

[0106] The difference is that the convolution kernel sizes of the two parallel convolution branches of the DW Rep Block unit in the first Rep HWD Block module are 3×3 and 1×1 respectively. The purpose is to reduce the number of parameters and computational complexity through small convolution kernels. The convolution kernel sizes of the two parallel convolution branches of the DW Rep Block unit in the second Rep HWD Block module, the third Rep HWD Block module, and the fourth Rep HWD Block module are 7×7 and 5×5 respectively. The HWTBlock unit can decompose the input into low-frequency sub-bands (y L ) and three high frequency sub-bands (y HL ,y LH ,y HH ), the resolution of each sub-band is half of the input image, and then the HWT Block unit concatenates the low-frequency sub-band and the high-frequency sub-band in the channel dimension to generate a new tensor.

[0107] The DW Rep Block unit captures the key information in the input data and extracts the spatial features before downsampling. The HWT Block unit overcomes the problem of spatial information loss caused by traditional downsampling, and retains as much key information as possible while reducing the spatial resolution of the feature map. The PW Rep Block unit implements channel number conversion and channel information fusion, and outputs the final feature map. The four Rep HWD Block modules can achieve high-quality downsampling, spatial feature extraction and channel number conversion, and have superior performance and lower computational complexity compared to traditional strided convolution;

[0108] The first stage module includes 2 sub-modules, the second stage module includes 2 sub-modules, the third stage module includes 6 sub-modules, and the fourth stage module includes 2 sub-modules, and the structure and working process of each sub-module are the same;

[0109] The submodule includes a DW Rep Block unit (where the convolution kernel sizes of the two parallel convolution branches are 3×3 and 1×1, respectively, for extracting local features) and a Rep LKFFN. The working process in the submodule is as follows:

[0110] First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule;

[0111] The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer is connected with the output of the flattening layer. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer.

[0112] The second LWM classification model is used to classify the severity of AS at the image level for the acquired echocardiographic data, and the structure of the second LWM classification model is exactly the same as that of the first LWM classification model except for the fully connected layer, and the difference between the fully connected layers of the first and second LWM classification models is only the number of output categories;

[0113] The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

[0114] The structure and parameter settings of the first LWM classification model are shown in Table 1;

[0115] Table 1

[0116]

[0117]

[0118] Specific implementation method 2: Combination Figure 2 This embodiment is described. The difference between this embodiment and the first embodiment is that during the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depthwise separable convolution layer and a batch normalization layer. DW Rep Block captures spatial information of different scales through two depthwise separable convolutions of different sizes to achieve feature extraction. The working process of the DWRep Block unit during training is:

[0119] y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x]

[0120] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer;

[0121] In the reasoning process of the first LWM classification model, the working process of the DW Rep Block unit is:

[0122] y′=σ(Conv 12 (x′))

[0123] Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

[0124] The other steps and parameters are the same as those in the first embodiment.

[0125] During the training and reasoning process of the second LWM classification model and the first LWM classification model, the working process of the DW Rep Block unit is the same.

[0126] Specific implementation method three: Combination Figure 3 The present embodiment is different from the first or second embodiment in that, during the training process of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer, and the working process of the PW Rep Block unit during training is as follows:

[0127] y″=σ[BN(Conv(x″))]

[0128] Among them, y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer;

[0129] In the reasoning process of the first LWM classification model, the working process of the PW Rep Block unit is as follows:

[0130] y″′=σ(Conv(x″′))

[0131] Among them, x″ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

[0132] The other steps and parameters are the same as those in the first or second embodiment.

[0133] PW Rep Block is a single-branch cascade structure. PW Rep Block consists of: a point-by-point convolution layer, a batch normalization layer, and an activation layer. PW Rep Block achieves channel number transformation and channel information fusion through point convolution.

[0134] During the training phase, Rep Block is a complex structure that integrates convolutional layers, batch normalization layers, and activation layers. During the inference phase, Rep Block uses structural reparameterization technology to reparameterize the structure and parameters during training into a simple structure with only convolutional layers and activation layers, thereby reducing the number of parameters and FLOPs. However, the conventional composite structure of convolutional layers, batch normalization layers, and activation layers cannot reduce the computational and memory costs through structural reparameterization.

[0135] Specific implementation mode 4: This implementation mode is different from any one of specific implementation modes 1 to 3 in that the training data of the first LWM classification model and the second LWM classification model are echocardiograms obtained from the TMED-2 data set. Specifically, the echocardiograms obtained by the first LWM classification model include labeled data with view labels and a large amount of unlabeled data, and the echocardiograms obtained by the second LWM classification model include labeled data with AS severity labels and a large amount of unlabeled data.

[0136] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0137] TMED-2 is the most common training and testing benchmark for AS research. The view labels of TMED-2 include PLAX, PSAX, A2C, A4C, Other, and the AS severity labels include No AS, Early AS, and Significant AS. Among them, the view labels provided by TMED-2 can be used for three view types (Other, PLAX, PSAX) classification and four view types (A2C, A4C, PLAX, PSAX) classification. The double-label images of the TMED-2 dataset come from 599 studies, totaling 17,270 images. These 599 studies are divided into training sets, validation sets, and test sets in a ratio of 360 / 119 / 120. In addition, there are 353,500 echocardiograms without any labels (unlabeled set) in the TMED-2 dataset. Many previous studies only used labeled data, which resulted in a large amount of waste of unlabeled data.

[0138] The present invention effectively uses unlabeled data to train the model. Similar to the prediction process, the input of the second LWM classification model in the training process is the echocardiogram classified as belonging to the relevant view by the first LWM classification model.

[0139] Specific implementation method five: Combination Figure 1 and Figure 4 This embodiment is different from the first to fourth embodiments in that the working process of the Rep LKFFN is:

[0140] In Rep LKFFN, after the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, the output is recorded as z', and then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

[0141] The two convolution kernel sizes of the DW Rep Block unit are 7×7 and 5×5 respectively. The DW Rep Block unit can capture a wide range of spatial information at different scales. The two PW Rep Block units realize the expansion and restoration of the number of channels inside Rep LKFFN. In general, Rep LKFFN is used to extract global features. The sub-module combines the DW Rep Block unit and Rep LKFFN to achieve the fusion of local information, global information and channel information.

[0142] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0143] Specific implementation method six: Combination Figure 1 This embodiment is different from the first to fifth embodiments in that the two submodules in the first Stage module are respectively recorded as the first submodule and the second submodule, and the working process of the first Stage module is:

[0144] In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule, and the output of the Rep LKFFN of the first submodule is recorded as q2, and then q1 and q2 are connected to obtain the connection result q';

[0145] q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule, and the output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q".

[0146] The connection result q" is the output of the first Stage module.

[0147] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0148] The DW Rep Block unit is mainly used to extract local spatial features, and the Rep LKFFN is used to extract global spatial features and fuse channel information. By combining the DW Rep Block unit and Rep LKFFN, LWM pays attention to information in both spatial and channel dimensions, and can more comprehensively understand and represent the characteristics of the input data.

[0149] Specific implementation method 7: This implementation method is different from any one of the specific implementation methods 1 to 6 in that the first LWM classification model is trained by using a semi-supervised learning (Efficient SSL) method. The process of semi-supervised learning is as follows: Figure 8 As shown in the figure, the specific process of training is:

[0150] Step 1, initialize the data batch number t=1;

[0151] Step 2: sampling the tth batch of data from the acquired echocardiogram;

[0152] Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch (weak enhancement can be achieved through rotation and displacement algorithms), record the weak enhancement result of the unlabeled data x as WA(x), and then calculate the classification result P of WA(x) by the first LWM classification model. model (WA(x)) passes through Softmax to obtain probability distribution;

[0153] If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo-label of the unlabeled data x, the first LWM classification model is trained using the unlabeled data x, and the total loss is calculated based on the pseudo-label;

[0154] Otherwise, the unlabeled data x does not participate in the training process;

[0155] Step 4: Stop the training process of the tth batch until the total loss function converges;

[0156] Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

[0157] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0158] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the confidence level is:

[0159]

[0160] in, is the adaptive local threshold of the c-th category in the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, C is the total number of categories, and τ t It can be set according to actual situation.

[0161] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0162] Mainstream semi-supervised learning uses a fixed threshold to determine whether a pseudo-label is available, which will discard a large number of available pseudo-labels, resulting in low utilization of unlabeled data. The present invention uses a dynamic threshold in the pseudo-label generation process. The semi-supervised training model with a dynamic threshold sets a lower confidence threshold for the pseudo-labels of samples of different categories in the early stage, and the confidence threshold of the pseudo-label increases dynamically as the training progresses. Therefore, the semi-supervised training model with a dynamic threshold can introduce more pseudo-labels to participate in the training, improve the utilization efficiency of unlabeled data, and can use labeled data and a large amount of unlabeled data training models at the same time, which has more superior performance than the fully supervised method and the traditional semi-supervised method.

[0163] Specific embodiment 9: This embodiment is different from any one of specific embodiments 1 to 8 in that in step 3, the selectivity total loss calculation (SLC) method is:

[0164] The supervision loss of labeled data is denoted as L s (calculated by CrossEntropyLoss), the unsupervised loss of unlabeled data is recorded as L u (calculated by ConsistencyLoss), back propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss

[0165] like and The dot product result is less than or equal to zero, then the total loss function is:

[0166] L= L s

[0167] like and The dot product result of is greater than zero, then the total loss function is:

[0168] L=L s +αL u +βL f

[0169] The formula for calculating the unsupervised loss is:

[0170]

[0171] Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the result of strong enhancement of x using TSA, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of TSA(x) by the first LWM classification model, L f represents the adaptive fair regularization penalty (SAF) proposed by FreeMatch, α represents the unlabeled loss coefficient, β represents the adaptive fair regularization penalty coefficient, and S represents the Softmax operation.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Introducing L f The purpose is to offset the negative impact of the imbalanced distribution of pseudo labels. The default value of α is 1.0, and the default value of β is 0.01. The targeted strong enhancement (TSA) method used in this embodiment includes 3 local enhancements and 3 global enhancements. The local algorithm acts on a random rectangular area in the echocardiogram, and the side length of the random rectangular area does not exceed 1 / 2 of the side length of the echocardiogram. The global algorithm acts on the entire echocardiogram. The three local data enhancement algorithms are: injecting random salt and pepper noise spots into the random rectangular area; randomly disrupting the order of pixels in the random rectangular area to achieve local pixel shuffling; directly deleting the random rectangular area and filling it with all black pixels. The three global data enhancement algorithms are: changing the overall brightness of the echocardiogram with a brightness offset of 50%; using a Gaussian kernel of 5×5 to perform Gaussian blur on the entire echocardiogram; and randomly discarding the original pixels in the echocardiogram with a probability of no more than 50%. When performing strong enhancement on each batch of unlabeled data, TSA first randomly disrupts the order of the 6 data enhancement algorithms and randomly selects 1-6 as the enhancement algorithms for this batch. During strong enhancement, each algorithm is randomly applied with a probability of 50%. This solves the problem that traditional strong enhancement methods such as color transformation and contrast transformation are not suitable for echocardiograms stored in grayscale form and have poor adaptability. This implementation solves the problem that the loss of traditional SSL is affected by OOD (out-of-distribution, unlabeled data) data by improving the loss function.

[0174] Specific implementation example 10: This implementation example is described in conjunction with FIG13. This implementation example is different from the specific implementation examples 1 to 9 in that the working process of the aggregation module is as follows:

[0175] Step 1: The number of all the echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view correlation weight of each echocardiogram is calculated respectively:

[0176] w i =p[θ V (x i )∈RV]

[0177] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for the echocardiogram x i RV represents the related view (related views include PLAX and PSAX type views, such as Figure 6a and Figure 6b As shown), θ V (x i )∈RV means that the view classification result belongs to the related view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight;

[0178] Step 2: Aggregate the outputs of all echocardiograms after the second LWM classification model according to the weights:

[0179]

[0180] Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0182] The polymerization mechanism of this embodiment is as follows Figure 7 As shown, the view correlation weight is used to perform weighted averaging on all image-level AS severity classification results of the patient, and the serial number corresponding to the maximum value after the weighted averaging is taken as the patient-level AS severity classification result.

[0183] The aggregation mechanisms used in related studies in recent years can be summarized into two types: related view priority aggregation mechanism and attention aggregation mechanism. The traditional related view priority aggregation mechanism only gives related views higher weights or priorities, and there is no reasonable and interpretable weight calculation formula. The attention aggregation mechanism will significantly increase the computational cost. In addition, since the attention layer is directly embedded in the model, the model can only output patient-level AS severity classification results, and cannot achieve view classification and image-level AS severity classification. Therefore, using the attention aggregation mechanism to achieve patient-level AS severity classification is one-sided and lacks explanation.

[0184] Different from the previous weighted average aggregation mechanism based on view relevance, the present invention provides a clear view relevance weight calculation formula based entirely on view classification results and an aggregation formula based entirely on image-level AS severity classification results. Compared with the traditional weighted average aggregation mechanism, the aggregation mechanism of the present invention fully weighs the weights of relevant views and irrelevant views. Compared with the attention aggregation mechanism, the aggregation mechanism of the present invention does not require modification of the model and hardly increases the computational cost. In addition, the fundamental basis of the aggregation mechanism of the present invention is the view classification results and the image-level AS severity classification results. Therefore, the aggregation mechanism of the present invention has better interpretability than the attention aggregation mechanism.

[0185] Experimental Section

[0186] Multiple LWMs are preset by modifying the fully connected layer of the classifier. These LWMs do not differ except for the output categories of the fully connected layer. The LWM that implements the classification of three view types (PLAX, PSAX, Other) is denoted as LWM-3V, and the LWM that implements the classification of four view types (PLAX, PSAX, A2C, A4C) is denoted as LWM-4V. The LWM that implements the image-level AS severity classification of TMED-1 is denoted as LWM-AS1, and the LWM that implements the image-level AS severity classification of TMED-2 is denoted as LWM-AS2.

[0187] Multiple LWMs were trained for view classification and image-level AS severity classification using both fully supervised and semi-supervised (Efficient SSL) training methods. Fully supervised training uses only images in the training set of the official partition of the TMED dataset (labeled data), while Efficient SSL uses all images in the training set of the official partition of the TMED dataset and the unlabeled dataset.

[0188] Furthermore, we use a view-correlation-based aggregation mechanism to aggregate the image-level AS severity classification results to obtain the patient-level AS severity. All of the above work was completed using a single consumer-grade NVIDIA RTX-4070 GPU. It should be noted that we did not use any form of pre-training or transfer learning techniques for LWM.

[0189] The TMED-2 dataset is used as the training and testing benchmark for view classification. LWM-3V and LWM-4V are trained using both fully supervised and semi-supervised (Efficient SSL) training methods to achieve view classification. The maximum learning rate for fully supervised training is 5×10 -4 , the minimum learning rate is 5×10 -6 The fully supervised training uses the cosine annealing learning rate update strategy. The maximum learning rate of the semi-supervised training is set to 10 -3 , the minimum learning rate is set to 10 -5 The semi-supervised training adopts the cosine annealing learning rate update strategy with a warm-up epoch number of 5. The batch size of labeled data for both training methods is set to 16, and the AdamW optimizer is used to update the model parameters. The exponential moving average (EMA) is used to improve the robustness of the model. The detailed hyperparameters of the training process are shown in Table 2.

[0190] Table 2 Training hyperparameters of TMED-2 view type classification experiment

[0191]

[0192]

[0193] The TMED-1 and TMED-2 datasets are used as training and testing benchmarks for AS severity classification. AS severity classification includes image-level AS severity classification and patient-level AS severity classification. The method of the present invention not only completes image-level AS severity classification, but also completes patient-level AS severity classification.

[0194] LWM-AS1 is trained by both fully supervised and semi-supervised (Efficient SSL) training methods to achieve image-level AS severity classification for TMED-1. Since the echocardiogram resolution of the TMED-1 dataset is low (only 64×64), we use CutMix (S. Yun, D. Han, S. J Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 6023-6032.) and data augmentation methods such as Resize to reduce the risk of overfitting. The maximum learning rate for fully supervised training is 10 -3 , the minimum learning rate is 10 -5 The fully supervised training uses a cosine annealing learning rate update strategy. The maximum learning rate for semi-supervised training is 5×10 -3 , the minimum learning rate is 5×10 -5 The semi-supervised training uses a cosine annealing learning rate update strategy with a warm-up epoch number of 5. Both training methods use the AdamW optimizer to update model parameters, and both use the exponential moving average (EMA) to improve the robustness of the model. The detailed hyperparameter settings of the training process are shown in Table 3.

[0195] Table 3. Training hyperparameters for image-level AS severity classification of TMED-1

[0196]

[0197] We train LWM-AS2 to achieve image-level AS severity classification for TMED-2 using both fully supervised and semi-supervised (Efficient SSL) training methods. The maximum learning rate for fully supervised training is 10 -3 , the minimum learning rate is 10 -5 The fully supervised training uses a cosine annealing learning rate update strategy. The maximum learning rate for semi-supervised training is 5×10 -3 , the minimum learning rate is 5×10 -5 The semi-supervised training uses a cosine annealing learning rate update strategy with a warm-up epoch number of 5. Both training methods use the AdamW optimizer to update model parameters, and both use the exponential moving average (EMA) to improve the robustness of the model. The detailed hyperparameter settings of the training process are shown in Table 4.

[0198] Table 4 Training hyperparameters for image-level AS severity classification of TMED-2

[0199]

[0200]

[0201] The present invention uses an aggregation mechanism based on view relevance to aggregate the image-level classification results of LWM-AS2, thereby realizing the patient-level AS severity classification of TMED-2. Specifically, we use LWM-4V to output the view prediction results of each echocardiogram and calculate the view relevance weight. We use LWM-AS2 to output the image-level AS severity prediction results of each echocardiogram. Furthermore, we obtain the patient-level AS severity prediction results by performing a weighted average based on view relevance on all image-level AS severity prediction results of the same patient.

[0202] Table 5 compares the performance of the proposed method and previous methods in the view classification task. The balanced accuracy in Table 5 and the subsequent tables is calculated by the metrics library of scikit-learn (F. Pedregosa et al., "Scikit-learn: Machine learning in Python," vol. 12, pp. 2825-2830, 2011.). Obviously, the proposed LWM can achieve state-of-the-art accuracy even with only full supervision training, and the LWM trained with Efficient SSL can further improve the performance.

[0203] Table 5 Comparison of view classification performance between the proposed method and previous methods on the TMED-2 dataset

[0204]

[0205] N.Ahmadi et al. refers to the document "H.Vaseli et al.,"ProtoASNet:DynamicPrototypes for Inherently Interpretable and Uncertainty-Aware Aortic StenosisClassification in Echocardiography,"in International Conference on MedicalImage Computing and Computer-Assisted Intervention,2023,pp.368-378:Springer."; Huang et al. al. refers to the document "Z.Huang,M.-J.Sidhom,BSWessler,andM.CJapaHughes,"Fix-a-step: Semi-supervised learning from uncuratedunlabeled data,"2022."; the training method VAT comes from "T.Miyato,S.-i.Maeda,M.Koyama,SJItopaIshii,and m.intelligence,"Virtual adversarial training:aregularization method for supervised and semi-supervised learning,"vol.41,no.8,pp.1979-1993,2018."; the WideResNet model comes from "S.Zagoruyko andN.JapaKomodakis,"Wide residual networks,"2016.".

[0206] The view classification performance of LWM and FastViT-T8 is compared in Table 6. Under the same fully supervised training and evaluation conditions, our proposed LWM has superior performance with lower parameter amount and computational complexity.

[0207] Table 6 Comparison of view type classification performance between LWM and FastViT-T8 models on the TMED-2 dataset

[0208]

[0209] Model FastViT-T8 comes from "PKAVasu, J.Gabriel, J.Zhu, O.Tuzel, andA.Ranjan, "FastViT: A fast hybrid vision transformer using structuralreparameterization," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp.5785-5795."

[0210] Grad-CAM (RRSelvaraju, M.Cogswell, A.Das, R.Vedantam, D.Parikh, and D.Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," in Proceedings of the IEEE international conference on computer vision, 2017, pp.618-626) is applied to the last layer of each Stage module of LWM to visualize the prediction process and improve the interpretability of the prediction results. The heat map generated by Grad-CAM is shown in Fig. 9 As shown, it can be seen that the method of the present invention can accurately focus on the area related to the task, so the method of the present invention has excellent classification performance.

[0211] We also plotted the following based on the view classification results: Fig.10a and Fig.10b The confusion matrix is ​​shown. It is obvious that when the classification accuracy is extremely high, the errors are mostly caused by confusing similar categories (PLAX and PSAX, A2C and A4C), which will not particularly affect the final result.

[0212] We use UMAP (L.McInnes, J.Healy, and JJapaMelville, "Umap: Uniform manifold approximation and projection for dimension reduction," 2018.) to reduce the dimension and visualize the feature tensor output by LWM, which indirectly reflects the effectiveness of the combination of LWM and Efficient SSL proposed in this paper from another perspective. Fig.11a and Fig.11b As shown in Figure 1, data points of the same category have obvious clusters, while data points of different categories have clear boundaries. Therefore, LWM trained with Efficient SSL learns meaningful features of the data and can effectively distinguish different categories.

[0213] The performance difference between the method of the present invention and the previous method on the image-level AS severity classification task of TMED-1 is compared in Table 7. It can be seen that the method of the present invention is better than the previous method in both performance and parameter quantity.

[0214] Table 7 Comparison of image-level AS severity classification performance between the proposed method and the previous method on the TMED-1 dataset

[0215]

[0216]

[0217] MixMatch is a semi-supervised method from the literature "D.Berthelot,N.Carlini,I.Goodfellow,N.Papernot,A.Oliver,and CAJ AnipsRaffel,"Mixmatch:A holistic approach to semi-supervised learning,"vol.32,2019."

[0218] The present invention achieved an 82.2% accuracy rate in image-level AS severity classification on the TMED-2 dataset. The ROC curve, which is not affected by the sample category distribution, was used to evaluate the performance of image-level AS severity classification. Positive and negative samples were divided based on the criteria of no AS and AS (including mild, moderate and severe, etc.) with clinical application value, and the following curves were plotted: Fig.12a and Figure 12b The ROC curve is shown.

[0219] As shown in Table 8, the method of the present invention achieves a good patient-level AS severity classification effect with an extremely lightweight model and semi-supervised training method.

[0220] Table 8

[0221]

[0222] In Table 8, the method Huang et al. in the first row refers to "Z. Huang, G. Long, B. Wessler, and M. C. Hughes," TMED 2: a dataset for semi-supervised classification of echocardiograms," in Data Perf: Benchmarking Data for Data-Centric AI Workshop, 2022.", and the method Huang et al. in the second row refers to "Z. Huang, B S Wessler, and M C Hughes," Detecting Heart Disease from Multi-View Ultrasound Images via Supervised Attention Multiple Instance Learning," in Machine Learning for Healthcare Conference, 2023, pp. 285-307: PMLR.", and the VAT model comes from "T. Miyato, S. -i. Maeda, M. Koyama, S J Itopa Ishii, and m. intelligence," Virtual adversarial training: a regularization method for supervised and semi-supervised learning,"vol.41,no.8,pp.1979-1993,2018."

[0223] The method of the present invention achieves patient-level AS severity classification based on view classification and image-level AS severity classification. Therefore, the method of the present invention has better interpretability than the method of directly achieving patient-level AS severity classification using the attention mechanism.

[0224] The previous experiments have proved the effectiveness of the Lightweight Model (LWM model) and Efficient Semi-Supervised Learning (Efficient SSL) proposed in this paper. Furthermore, we conducted ablation experiments on the components of LWM and the improved method of Efficient SSL. We used the view (PLAX, PSAX, A2C, A4C) classification performance as the evaluation criterion for the ablation experiment. All hyperparameters of the ablation experiment are consistent with the view type classification experiment.

[0225] The LWM proposed in the present invention is designed based on structural reparameterization and deep large kernel convolution. LWM uses the large convolution kernel feedforward neural network (Rep LKFFN) and the reparameterizable wavelet downsampling module (Rep HWDBlock) proposed in the present invention. In order to verify the effectiveness of the model components proposed in the present invention, we replaced the DW Rep Block (the convolution kernel is 7×7+5×5) in Rep LKFFN with DW Rep Block (the convolution kernel is 3×3+1×1) and named it the reparameterizable small convolution kernel feedforward neural network (Rep SKFFN). We replaced Rep LKFFN and Rep HWD Block with Rep SKFFN and depthwise convolution with a stride of 2 for ablation experiments.

[0226] In order to verify the effectiveness of the improved method we proposed, we replaced ESA and SLC (i.e., the loss calculation method of the present invention) with RandAugment and the conventional loss calculation method to conduct ablation experiments.

[0227] We are Fig.13a and Fig.13b The results of the ablation experiments are shown in the form of bar graphs and box plots. We evaluate the components of LWM by balancing accuracy, FLOPs, and number of parameters. We evaluate the Efficient SSL method by balancing accuracy and the degree of fluctuation of balancing accuracy.

[0228] In terms of model components, the LWM of Rep LKFFN and Rep HWD Block proposed in the present invention has the highest classification accuracy. Rep LKFFN is replaced by Rep SKFFN with a structure using a small convolution kernel. The number of parameters and FLOPs are only reduced by 0.07M and 0.02G respectively (almost negligible), but the classification balance accuracy is reduced by 0.4%. Rep HWDBlock replaces the traditional strided convolution downsampling, and FLOPs increases by 0.4G, but the classification balance accuracy is reduced by 0.5%. In general, Rep LKFFN and Rep HWD Block achieve the best performance while maintaining lightweight.

[0229] In terms of the improvement method of Efficient SSL, Efficient SSL using TSA strong enhancement and SLC loss calculation method has the highest classification accuracy. By replacing TSA strong enhancement with traditional RandAugment, the classification balance accuracy is reduced by 0.3%. By replacing the SLC loss calculation method with the conventional loss calculation method, the classification balance accuracy is reduced by 0.5% (lower than full supervised training). This shows that the conventional loss calculation method is negatively affected by OOD data. Compared with traditional RandAugment, the TSA strong enhancement proposed in the present invention can effectively improve the performance of semi-supervised learning, and has better robustness (small accuracy fluctuation). The SLC loss calculation method can effectively alleviate the negative impact caused by OOD data, and therefore has superior performance than the conventional loss calculation method.

[0230] The pixels in the TMED test image are randomly removed at a certain ratio (10% to 50%) and replaced with pure black pixels to simulate different degrees of image quality loss. Fig.14 As shown in Figure 2, even discarding 50% of the pixels will only slightly affect the classification accuracy of LWM. Therefore, the LWM trained with Efficient SSL has good robustness.

[0231] The present invention proposes a lightweight model (LWM) based on structural reparameterization and deep large kernel convolution design. LWM has lower parameter quantity and computational complexity, and is more suitable for medical systems with lower computing power than previous models. The present invention also proposes an Efficient Semi-Supervised Learning (Efficient SSL) using Targeted Strong Augment (TSA) and Selective Total Loss Calculation (SLC). Efficient SSL can use labeled data and a large amount of unlabeled data to train the model at the same time. Compared with the fully supervised method, Efficient SSL makes full use of the unlabeled AS dataset, thus having a more efficient data utilization capability; compared with the traditional semi-supervised method, Efficient SSL has superior performance.

[0232] The method of the present invention uses Efficient SSL to train LWM to achieve view classification and image-level AS severity classification. In addition, the method of the present invention also uses an aggregation mechanism based on view correlation to achieve patient-level AS severity classification. Experimental results show that the method of the present invention performs well in view classification, image-level AS severity classification, and patient-level AS severity classification, and has potential for clinical application.

[0233] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. An AS severity classification system based on lightweight model and semi-supervised learning, characterized in that: The system includes an echocardiogram acquisition module, a first LWM classification model, a second LWM classification model, and an aggregation module; The ultrasound cardiogram acquisition module is used to acquire ultrasound cardiogram data of the object to be detected; The first LWM classification model is used to classify the views of each echocardiogram in the acquired echocardiogram data, and the first LWM classification model includes a first Rep HWD Block module, a first Stage module, a second Rep HWDBlock module, a second Stage module, a third Rep HWD Block module, a third Stage module, a fourth Rep HWD Block module, a fourth Stage module and a classifier; and the working process of the first LWM classification model is: In the first LWM classification model, the input echocardiographic data first passes through the first Rep HWD Block module, and then the output of the first Rep HWD Block module is used as the input of the first Stage module; The output of the first Stage module is then used as the input of the second Rep HWD Block module, and the output of the second Rep HWDBlock module is used as the input of the second Stage module; The output of the second Stage module is then used as the input of the third Rep HWD Block module, and the output of the third Rep HWDBlock module is used as the input of the third Stage module; The output of the third Stage module is used as the input of the fourth Rep HWD Block module, and the output of the fourth Rep HWDBlock module is used as the input of the fourth Stage module. The output of the fourth Stage module is passed through the classifier to obtain the view classification result. The first Rep HWD Block module includes a DW Rep Block unit, a HWT Block unit and a PW Rep Block unit, and the working process of the first Rep HWD Block module is as follows: The input of the first Rep HWD Block module first passes through the DW Rep Block unit, the output of the DW Rep Block unit is then used as the input of the HWT Block unit, the output of the HWT Block unit is then used as the input of the PW Rep Block unit, and the output of the PW Rep Block unit is used as the output of the first Rep HWD Block module; The structures and working processes of the second Rep HWD Block module, the third Rep HWD Block module and the fourth Rep HWD Block module are the same as those of the first Rep HWD Block module; The first stage module includes 2 sub-modules, the second stage module includes 2 sub-modules, the third stage module includes 6 sub-modules, and the fourth stage module includes 2 sub-modules, and the structure and working process of each sub-module are the same; The submodule includes DW Rep Block unit and Rep LKFFN. The working process in the submodule is as follows: First, the input of the submodule is used as the input of the DW Rep Block unit, and then the output of the DW Rep Block unit is used as the input of the Rep LKFFN. Finally, the output of the DW Rep Block unit is connected with the output of the Rep LKFFN, and the connection result is used as the output of the submodule; The classifier includes an average pooling layer, a flattening layer, and a fully connected layer. The input of the classifier passes through the average pooling layer and the flattening layer respectively, and then the output of the average pooling layer is connected with the output of the flattening layer. The connection result passes through the fully connected layer, and the classification result is output through the fully connected layer. The second LWM classification model is used to classify the severity of AS at the image level on the acquired echocardiographic data, and the structure of the second LWM classification model is the same as that of the first LWM classification model; The aggregation module is used to aggregate the classification results of the first LWM classification model and the classification results of the second LWM classification model to obtain a patient-level AS severity classification result.

2. According to claim 1, an AS severity classification system based on lightweight model and semi-supervised learning is characterized in that: During the training process of the first LWM classification model, the DW Rep Block unit includes two parallel convolution branches and a skip layer connection, and each convolution branch includes a depth-separable convolution layer and a batch normalization layer. The working process of the DW Rep Block unit during training is: y=σ[BN1(Conv1(x))+BN2(Conv2(x))+x] Among them, y is the output of the DW Rep Block unit during training, x is the input of the DW Rep Block unit during training, Conv1 and Conv2 are depth-separable convolutional layers, BN1 and BN2 are batch normalization layers, and σ is the activation function layer; In the reasoning process of the first LWM classification model, the working process of the DW Rep Block unit is: y′=σ(Conv 12 (x′)) Among them, y′ is the output of the DW Rep Block unit during the reasoning process, x′ is the input of the DW Rep Block unit during the reasoning process, and Conv 12 is a depth-wise separable convolutional layer.

3. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 2 is characterized in that: During the training process of the first LWM classification model, the PW Rep Block unit includes a point-by-point convolution layer and a batch normalization layer. The working process of the PW Rep Block unit during training is as follows: y″=σ[BN(Conv(x″))] Among them, y″ is the output of the PW Rep Block unit during training, x″ is the input of the PW Rep Block unit during training, Conv is the point-by-point convolution layer, BN is the batch normalization layer, and σ is the activation function layer; In the reasoning process of the first LWM classification model, the working process of the PW Rep Block unit is: y″′=σ(Conv(x″′)) Among them, x″ is the input of the PW Rep Block unit during the reasoning process, and y″′ is the output of the PW Rep Block unit during the reasoning process.

4. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 1, characterized in that: The training data of the first LWM classification model and the second LWM classification model are obtained from the TMED-2 data set. The training data of the first LWM classification model includes data with view labels and data without view labels. The training data of the second LWM classification model includes data with AS severity labels and data without AS severity labels.

5. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 3 is characterized in that: The working project of Rep LKFFN is: In Rep LKFFN, after the input z passes through the DW Rep Block unit and two PW Rep Block units in sequence, the output is recorded as z', and then the input z and the output z' are connected, and the connection result is used as the output of Rep LKFFN.

6. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 5, characterized in that: The two submodules in the first Stage module are respectively recorded as the first submodule and the second submodule. The working process of the first Stage module is: In the first Stage module, the input q first passes through the DW Rep Block unit of the first submodule, and then the output q1 of the DW Rep Block unit of the first submodule is used as the input of the Rep LKFFN of the first submodule, and the output of the RepLKFFN of the first submodule is recorded as q2, and then q1 and q2 are connected to obtain the connection result q'; q' then passes through the DW Rep Block unit of the second submodule, and the output q'1 of the DW Rep Block unit of the second submodule is used as the input of the Rep LKFFN of the second submodule, and the output of the Rep LKFFN of the second submodule is recorded as q'2, and then q'1 and q'2 are connected to obtain the connection result q". The connection result q" is the output of the first Stage module.

7. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 6, characterized in that: The first LWM classification model is trained by semi-supervised learning, and the specific training process is as follows: Step 1, initialize the data batch number t=1; Step 2: sampling the tth batch of data from the acquired echocardiogram; Step 3: Perform weak enhancement on any unlabeled echocardiographic data x in the tth batch, record the weak enhancement result of the unlabeled data x as WA(x), and then calculate the classification result P of WA(x) by the first LWM classification model. midel (WA(x)) passes through Softmax to obtain probability distribution; If the maximum probability in the probability distribution is greater than the confidence level, then P model (WA(x)) is used as the pseudo-label of the unlabeled data x, the unlabeled data x is added to the training process of the first LWM classification model, and the total loss is calculated based on the pseudo-label; Otherwise, the unlabeled data x does not participate in the training process; Step 4: Stop the training process of the tth batch until the total loss converges; Step 5: Set t=t+1, and then return to step 2 until the entire training process is completed to obtain the trained first LWM classification model.

8. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 7, characterized in that: The confidence level is: in, is the adaptive local threshold of the c-th category in the t-th batch training, τ t is the adaptive global threshold for the t-th batch training, τ t (c) is the confidence of the t-th batch training, and C is the total number of categories.

9. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 8, characterized in that: In step 3, the total loss is calculated as follows: The supervision loss of labeled data is denoted as L s , the unsupervised loss of unlabeled data is recorded as L u , back propagation loss L s Get the gradient corresponding to the supervised loss of labeled data Back propagation loss L u Get the gradient corresponding to the unsupervised loss like and The dot product result is less than or equal to zero, then the total loss function is: L=L s like and The dot product result of is greater than zero, then the total loss function is: L=L s +αL u +βL f Unsupervised loss L u The calculation formula is: Where D represents the consistency regularization loss function, x represents unlabeled data, WA(x) represents the weak enhancement result of x, TSA(x) represents the strong enhancement result of x, and P model (WA(x)) represents the classification result of WA(x) by the first LWM classification model, P model (TSA(x)) represents the classification result of TSA(x) by the first LWM classification model, L f represents the adaptive fair regularization penalty, α represents the unlabeled loss coefficient, β represents the adaptive fair regularization penalty coefficient, and S represents the Softmax operation.

10. The AS severity classification system based on lightweight model and semi-supervised learning according to claim 9, characterized in that: The working process of the aggregation module is as follows: Step 1: The number of all the echocardiograms of a subject to be detected is recorded as n. After classifying all the echocardiograms of the subject to be detected using the first LWM classification model, the view correlation weight of each echocardiogram is calculated respectively: w i =p[θ V (x i )∈RV] Among them, x i represents the input echocardiogram, θ V (x i ) represents the first LWM classification model for the echocardiogram x i The view classification results of , RV represents the relevant view, θ V (x i )∈RV means that the view classification result belongs to the related view, p[θ V (x i )∈RV] represents the probability that the view classification result belongs to the relevant view, that is, the view relevance weight; Step 2: Aggregate the outputs of all echocardiograms after the second LWM classification model according to the weights: Where D′ represents the classification result of patient-level AS severity, θ AS (x i ) represents x i After the output of the second LWM classification model, S represents the Softmax operation.

Images