An echocardiogram angle classification method and system based on mixed data enhancement and open set semi-supervised learning

Abstract

The application discloses an echocardiogram angle classification method and system based on mixed data enhancement and open set semi-supervised learning, and belongs to the technical field of medical image analysis.The application solves the problems that the existing method is difficult to capture effective information in an echocardiogram, a detection process of out-of-class distribution data is complex and low in detection efficiency, and feature information of an unmarked data set cannot be fully utilized.The application adds a mixed data enhancement unit between the third layer and the fourth layer of a teacher model coding part; adds an open set semi-supervised unit as a parallel branch of a classifier in a student model coding part; carries out preprocessing on obtained echocardiograms with labels and without labels, trains a constructed model by using the preprocessed echocardiograms, and finally classifies to-be-classified echocardiograms by using the trained student model.The method can be applied to echocardiogram angle classification.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis technology, specifically relating to an echocardiogram angle classification method and system based on hybrid data augmentation and open set semi-supervised learning. Background Technology

[0002] Echocardiography plays a vital role in the diagnosis of heart diseases, widely used in medicine. When diagnosing heart conditions, physicians need to analyze echocardiograms from specific angles. However, current supervised learning-based echocardiogram angle classification methods require significant training costs, and the extensive labeling of echocardiogram data demands substantial time from cardiologists. Some classification methods attempt to utilize semi-supervised learning, but these methods struggle to balance training costs with model accuracy, resulting in poor generalization. Mixed data augmentation, which augments datasets through linear interpolation, is widely used in deep learning to improve model generalization. However, current data augmentation methods often fail to capture effective information from echocardiograms.

[0003] Open-set semi-supervised learning is a novel semi-supervised learning paradigm designed to address the problem of out-of-class data compromising semi-supervised learning. In recent years, the idea of ​​detecting and excluding out-of-class data during training has been widely applied in computer vision. However, existing semi-supervised classification methods suffer from two main problems: (1) Current open-set semi-supervised classification methods are limited to using complex computational processes to detect out-of-class data, resulting in complex and inefficient algorithms; (2) Current classification methods do not yet address how to utilize the information from out-of-class data, potentially failing to fully leverage the feature information of unlabeled datasets, thus limiting classification performance. Summary of the Invention

[0004] The purpose of this invention is to address the problems of existing methods being unable to capture effective information in echocardiography, having a complex and inefficient detection process for out-of-class distributed data, and being unable to fully utilize the feature information of unlabeled datasets. Therefore, this invention proposes an echocardiography angle classification method and system based on hybrid data augmentation and open set semi-supervised learning.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] An echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning, the method specifically includes the following steps:

[0007] Step S1: Obtain echocardiogram datasets with and without category labels, where the category label is the angle of the echocardiogram.

[0008] Step S2: Preprocess each echocardiogram obtained in step S1. The size of the preprocessed image is N×N.

[0009] Step S3: Construct the teacher model and student model respectively;

[0010] In the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers; in the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier.

[0011] Step S4: Use the preprocessed images from step S2 to perform end-to-end joint training on the constructed teacher model and student model;

[0012] Step S5: Use the trained student model to perform angle classification on the echocardiogram to be classified.

[0013] Furthermore, the specific process of step S2 is as follows:

[0014] After randomly flipping the acquired echocardiogram horizontally, pixel filling operations of arbitrary size and method are performed, and finally, N×N images are obtained by random cropping.

[0015] Furthermore, the hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module;

[0016] The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-of-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier.

[0017] Furthermore, the semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model;

[0018] During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map.

[0019] Furthermore, the training process of the teacher model is as follows:

[0020] For any batch of data:

[0021] Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. H is the height of semantic feature map X, W is the width of semantic feature map X, and C is the number of channels in semantic feature map X.

[0022] The feature token acquisition module divides the semantic feature map X into S×S sub-feature maps, and the size of each sub-feature map is [missing information]. Each obtained sub-feature map is used as a feature token of the semantic feature map X.

[0023] Step S42: Use the semantic feature guidance module to process each feature token obtained in step S41 to generate a weighted feature map;

[0024] Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram.

[0025] Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask:

[0026]

[0027] in, and These are weighted feature maps corresponding to two unlabeled echocardiograms, where Q represents... The output after a 1×1 convolutional layer, K represents The output after a 1×1 convolutional layer: D is the dimension of the semantic feature map, T represents the transpose, h is a constant, sigmoid is the activation function, upsample represents the upsampling operation, and mask is the mixing mask. Represents matrix multiplication;

[0028] Step S45: The mixed data generation module uses a mixing mask to generate mixed data X. m1x :

[0029] X mix =Mask⊙X1+(1-Mask)⊙X2

[0030] Where X1 represents The corresponding echocardiogram, X2 represents The corresponding echocardiogram shows that ⊙ represents element-wise matrix multiplication;

[0031] Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model.

[0032] Training for the current batch is stopped when the loss function converges.

[0033] Furthermore, the specific process of step S42 is as follows:

[0034] Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: and

[0035] Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys

[0036] Step S423: For the i-th feature token, calculate the Query′ after average pooling of the i-th feature token. i Key′ after average pooling with any other feature token k The dot product of i ≠ k is used as the element in the i-th row of the weight matrix P.

[0037] Similarly, after processing each feature token separately, the weight matrix P is obtained;

[0038] Step S424: Retrieve the (Key′) corresponding to the first k' largest element values ​​from the weight matrix P. k Query′ i Yes, then based on the retrieved (Key′) k Wuery' i ) to obtain (Key′ k Value′ i )right;

[0039] Step S425: For each (Key′) obtained in step S424 k Value′ i The data is collected and spliced ​​to obtain the spliced ​​results Key′ and Value′;

[0040] Key′=Gather(Key′ k )

[0041] Value′=Gather(Value′ i )

[0042] Among them, Key′ is for each (Key′ k Value′ i Key′ in the pair k The result of the collection and splicing is that Value′ is for each (Key′) k Value′ i Value′ in the pair i The result of the collection and splicing is given by Gather(·), which represents the function for collecting and splicing tensors, i = 1, 2, ..., k′, k = 1, 2, ..., k′;

[0043] Step S426: Calculate the feature map X after attention score weighting using scaled dot product attention. out :

[0044]

[0045] Specifically, a 1×1 convolution is used to perform a linear transformation on the feature map X to obtain the Query. It is the scaling factor.

[0046] Furthermore, the loss function in step S46 includes the classification loss of the classifier for the echocardiogram with class labels. ce (c,z) and the generation loss of mixed data l mix (X1,X2,X mix );

[0047]

[0048] Where z = [p1, p2, p3, p4, ..., p C '] represents the output of the classifier for echocardiograms with category labels, p j Let C' represent the probability that the echocardiogram belongs to the j-th category, C' represent the number of known categories, c represent the category label of the echocardiogram, and p represent the probability that the echocardiogram belongs to the j-th category. c This represents the probability that the classifier predicts the echocardiogram belongs to category label c;

[0049]

[0050] Among them, c X1 c represents the class with the highest probability among the classification results of X1 output by the classifier; X2 p represents the class with the highest probability among the classification results of X2 output by the classifier; Xmix This represents the classifier output for the mixed data X. mix The classification results, p umix=[p′1,p′2,p′3,p′4,...,p′ C’ ], p′1, p′2, p′3, p′4,...,p′ C’ Represents mixed data X mix The probability of belonging to the 1st, 2nd, 3rd, 4th, ..., C'th category.

[0051] Furthermore, the training process of the student model is as follows:

[0052] For any batch of data:

[0053] Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model;

[0054] Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point;

[0055] Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point;

[0056] Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution probability and out-of-distribution probability of the data.

[0057] Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data.

[0058] The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

[0059] Furthermore, the probability within the distribution is:

[0060] P in =Linear(AVGPOOL(RELU(BN(x))))

[0061] Where x is the input of the probability calculation module within the distribution, P in It is the probability of x within the distribution. BN is a batch normalization layer, ReLU is an activation function, AVGPOOL is an average pooling layer, and Linear is a fully connected layer;

[0062] The out-of-distribution probability calculation module is a multi-head classifier composed of C' binary classifiers, where C' represents the number of known label categories; the output of x through the out-of-distribution probability calculation module is denoted as P. outout-of-distribution probability

[0063] Using the in-distribution probability P in and out-of-distribution probability P out The calculation of open set pseudo-labels is as follows:

[0064] Step (1): Calculate the dot product P of the in-distribution probability and the out-of-distribution probability. sp :

[0065] P sp =P in P out

[0066] Step (2): Calculate the weight of each category within the class:

[0067]

[0068] Among them, z j It is the probability P within the distribution. in In the expression, α represents the probability that x belongs to the j-th category. j It represents the weight of the j-th category;

[0069] Step (3): Calculate the open set pseudo-label corresponding to x:

[0070] SP = [α1P] sp [0],α2P sp [0],…,α C′ P sp [0],P sp [1]

[0071] Among them, P sp [0] is P sp The first element in, P sp [1] is P sp The second element in.

[0072] Furthermore, the total open set loss includes the loss from generating pseudo-labels and the classification loss from the open set, i.e., the total open set loss L = l sp +l op ;

[0073] Classification loss of open sets l sp for:

[0074] l sp =l in (c,z)+l out (P in ,P out )

[0075]

[0076] Among them, z b [c] represents the probability that the labeled data belongs to the true category c, calculated by the in-distribution probability calculation module. b [j] represents the probability that the labeled data belongs to category j, calculated by the probability calculation module within the distribution. B represents the number of labeled samples in each batch. This represents the first element in the c-th row of the output of the out-of-distribution probability calculation module for labeled data. This represents the second element in the c′-th row of the output of the out-of-distribution probability calculation module for labeled data.

[0077] The generation loss of pseudo-labels op for:

[0078]

[0079] V u1 Z u1 =max(SP u1 )

[0080] V u2 Z u2 =max(SP u2 )

[0081] Among them, SP u1 SP represents the pseudo-label of unlabeled data U1. u2 V represents the pseudo-label of the unlabeled data U2. u1 and Z u1 These represent pseudo-tags SP respectively. u1 The maximum probability in the probability distribution and the category index corresponding to the maximum probability, V u2 and Z u2 These represent pseudo-tags SP respectively. u2 The maximum probability in the probability distribution and the category index corresponding to the maximum probability. This represents the prediction result of the mixed data obtained from U1 and U2 by the open set classifier, where μ is the mixing ratio.

[0082] An echocardiographic angle classification system based on hybrid data augmentation and open-set semi-supervised learning, the system comprising an echocardiographic dataset acquisition module, an echocardiographic preprocessing module, and a neural network module; wherein:

[0083] An echocardiogram dataset acquisition module is used to acquire both labeled and unlabeled echocardiogram datasets, where the label represents the angle at which the echocardiogram was taken.

[0084] The echocardiography preprocessing module is used to preprocess each acquired echocardiogram separately.

[0085] The specific process of preprocessing is as follows:

[0086] After randomly flipping the acquired echocardiogram horizontally, perform pixel filling operations of arbitrary size and method, and finally obtain N×N images by random cropping.

[0087] The neural network module includes a teacher model and a student model. In the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers. In the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier.

[0088] The hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module;

[0089] The semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model.

[0090] During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map;

[0091] The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier;

[0092] The teacher model and the student model are jointly trained end-to-end based on the preprocessed images. That is, after each batch of data is trained, the parameters of the teacher model are updated according to the parameters of the student model. The final trained student model is used to classify the echocardiogram to be classified into angles.

[0093] The training process for the teacher model is as follows:

[0094] For any batch of data:

[0095] Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. H is the height of semantic feature map X, W is the width of semantic feature map X, and C is the number of channels in semantic feature map X.

[0096] The feature token acquisition module divides the semantic feature map X into S×S sub-feature maps, and the size of each sub-feature map is [missing information]. Each obtained sub-feature map is used as a feature token of the semantic feature map X.

[0097] Step S42: Process each feature token obtained in step S41 using the semantic feature guidance module to generate a weighted feature map; specifically:

[0098] Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: and

[0099] Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys

[0100] Step S423: For the i-th feature token, calculate the Query′ after average pooling of the i-th feature token. i Key′ after average pooling with any other feature token k The dot product of i ≠ k is used as the element in the i-th row of the weight matrix P.

[0101] Similarly, after processing each feature token separately, the weight matrix P is obtained;

[0102] Step S424: Retrieve the (Key′) corresponding to the first k' largest element values ​​from the weight matrix P. k Query′ i Yes, then based on the retrieved (Key′) k Query′ i ) to obtain (Key′ k Value′ i )right;

[0103] Step S425: For each (Key′) obtained in step S424 k Value′ i The data is collected and spliced ​​to obtain the spliced ​​results Key′ and Value′;

[0104] Key′=Gather(Key′ k )

[0105] Value′=Gather(Value′ i )

[0106] Among them, Key′ is for each (Key′ k Value′ i Key′ in the pair kThe result of the collection and splicing is that Value′ is for each (Key′) k Value′ i Value′ in the pair i The result of the collection and splicing is given by Gather(·), which represents the function for collecting and splicing tensors, i = 1, 2, ..., k′, k = 1, 2, ..., k′;

[0107] Step S426: Calculate the feature map X after attention score weighting using scaled dot product attention. out :

[0108]

[0109] Specifically, a 1×1 convolution is used to perform a linear transformation on the feature map X to obtain the Query. It is the scaling factor;

[0110] Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram.

[0111] Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask:

[0112]

[0113] in, and These are weighted feature maps corresponding to two unlabeled echocardiograms, where Q represents... The output after a 1×1 convolutional layer, K represents The output after a 1×1 convolutional layer: D is the dimension of the semantic feature map, T represents the transpose, h is a constant, sigmoid is the activation function, upsample represents the upsampling operation, and mask is the mixing mask. Represents matrix multiplication;

[0114] Step S45: The mixed data generation module uses a mixing mask to generate mixed data X. mix :

[0115] X mix =Mask⊙X1+(1-Mask)⊙X2

[0116] Where X1 represents The corresponding echocardiogram, X2 represents The corresponding echocardiogram shows that ⊙ represents element-wise matrix multiplication;

[0117] Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model.

[0118] The loss function includes the classification loss of the classifier for the echocardiogram with class labels. ce (c,z) and the generation loss of mixed data l mix (X1,X2,X mix );

[0119]

[0120] Where z = [p1, p2, p3, p4, ..., p C’ [p] represents the output of the classifier for echocardiograms with category labels. j Let C' represent the probability that the echocardiogram belongs to the j-th category, C' represent the number of known categories, c represent the category label of the echocardiogram, and p represent the probability that the echocardiogram belongs to the j-th category. c This represents the probability that the classifier predicts the echocardiogram belongs to category label c;

[0121]

[0122] Among them, c X1 c represents the class with the highest probability among the classification results of X1 output by the classifier; X2 This represents the category with the highest probability among the classification results of X2 output by the classifier; This represents the classifier output for the mixed data X. mix The classification results Represents mixed data X mix The probability of belonging to the 1st, 2nd, 3rd, 4th, ..., C'th category;

[0123] Training of the current batch is stopped when the loss function converges;

[0124] The training process for the student model is as follows:

[0125] For any batch of data:

[0126] Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model;

[0127] Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point;

[0128] The probability within the distribution is:

[0129] P in=Linear(AVGPOOL(RELU(BN(x))))

[0130] Where x is the input of the probability calculation module within the distribution, P in It is the probability of x within the distribution. BN is a batch normalization layer, ReLU is an activation function, AVGPOOL is an average pooling layer, and Linear is a fully connected layer;

[0131] Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point;

[0132] The out-of-distribution probability calculation module is a multi-head classifier composed of C' binary classifiers, where C' represents the number of known label categories; the output of x after passing through the out-of-distribution probability calculation module is denoted as P. out out-of-distribution probability

[0133] Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution and out-of-distribution probabilities of the data; specifically:

[0134] Step (1): Calculate the dot product P of the in-distribution probability and the out-of-distribution probability. sp :

[0135] P sp =P in P out

[0136] Step (2): Calculate the weight of each category within the class:

[0137]

[0138] Among them, z j It is the probability P within the distribution. in In the expression, α represents the probability that x belongs to the j-th category. j It represents the weight of the j-th category;

[0139] Step (3): Calculate the open set pseudo-label corresponding to x:

[0140] SP = [α1P] sp [0],α2P sp [0],…,α C′ P sp [0],P sp [1]

[0141] Among them, P sp [0] is P sp The first element in, P sp [1] is Psp The second element in;

[0142] Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data.

[0143] The total open-set loss includes the loss from generating pseudo-labels and the classification loss from the open set, i.e., the total open-set loss L = e sp +e op ;

[0144] Classification loss e of open set sp for:

[0145] l sp =l in (c,z)+l out (P in ,P out )

[0146]

[0147] Among them, z b [c] represents the probability that the labeled data belongs to the true category c, calculated by the in-distribution probability calculation module. b [j] represents the probability that the labeled data belongs to category j, calculated by the probability calculation module within the distribution. B represents the number of labeled samples in each batch. This represents the first element in the c-th row of the output of the out-of-distribution probability calculation module for labeled data. This represents the second element in the c′-th row of the output of the out-of-distribution probability calculation module for labeled data.

[0148] The generation loss of pseudo-labels op for:

[0149]

[0150] V u1 Z u1 =max(SP u1 )

[0151] V u2 Z u2 =max(SP u2 )

[0152] Among them, SP u1 SP represents the pseudo-label of unlabeled data U1. u2 V represents the pseudo-label of the unlabeled data U2. u1 and Zu1 These represent pseudo-tags SP respectively. u1 The maximum probability in the probability distribution and the category index corresponding to the maximum probability, V u2 and Z u2 These represent pseudo-tags SP respectively. u2 The maximum probability in the probability distribution and the category index corresponding to the maximum probability. This represents the prediction result of the mixed data obtained from U1 and U2 by the open set classifier, where μ is the mixing ratio;

[0153] The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

[0154] The beneficial effects of this invention are:

[0155] This invention, based on hybrid data augmentation technology, addresses the problem of current methods' inability to efficiently and accurately capture effective features from echocardiograms, significantly improving the robustness of classification results. Simultaneously, by introducing open-set pseudo-labels, it solves the problem of out-of-class distributed data impairing the classification performance of semi-supervised algorithms. Unlike traditional open-set semi-supervised algorithms that detect and filter out-of-distribution data, this invention constructs open-set pseudo-labels by calculating the probabilities of unlabeled data belonging to the in-distribution and out-of-distribution ranges, and utilizes these pseudo-labels in model training, thus fully leveraging the feature information of the unlabeled dataset. Furthermore, this invention innovatively proposes a hybrid loss mechanism using open-set pseudo-labels, significantly improving the accuracy of echocardiogram angle classification. The strategy of co-training the hybrid data augmentation unit and the open-set classification unit proposed in this invention synchronizes data augmentation and model classification training, making the trained classification model more reliable in practical clinical applications. The novel echocardiographic classification framework combining open-set semi-supervised and hybrid data augmentation provided by this invention can improve the training efficiency of the classification model, while having good accuracy and robustness, and has the potential for widespread application in clinical practice. In addition, compared with existing methods, this invention has the advantages of simple process and high detection efficiency for out-of-class distribution data. Attached Figure Description

[0156] Figure 1 This is a workflow diagram of the efficient hybrid data augmentation unit;

[0157] Figure 2 This is a flowchart of the open-set semi-supervised unit;

[0158] Figure 3 This is a schematic diagram of the structure of the teacher model and student model after adding the hybrid data augmentation unit and the open set semi-supervised unit. Detailed Implementation

[0159] Specific implementation method one: Combining Figure 3 This embodiment describes an echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning. The method specifically includes the following steps:

[0160] Step S1: Obtain echocardiogram datasets with and without category labels, where the category label is the angle of the echocardiogram.

[0161] Step S2: Preprocess each echocardiogram obtained in step S1. The size of the preprocessed image is N×N.

[0162] Step S3: Construct the teacher model and student model respectively;

[0163] like Figure 3 As shown, in the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers (that is, based on the existing teacher model, this invention improves the structure of the encoder part by adding a hybrid data augmentation unit between the third and fourth layers of the encoder); in the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier (that is, based on the existing student model, this invention improves the structure of the encoder part by adding an open-set semi-supervised unit at the end of the encoder, that is, an open-set semi-supervised unit is added after the last layer before the classifier, and the open-set semi-supervised unit is a branch parallel to the classifier).

[0164] Step S4: Use the preprocessed images from step S2 to perform end-to-end joint training on the constructed teacher model and student model;

[0165] The parameters of the student model are updated using the optimizer, and the parameters of the teacher model are updated using the momentum update method.

[0166] θ′ s =SGD(θ) s )

[0167] θ′ t =mθ t +(1-m)θ′ s

[0168] Where, θ s θ represents the original parameters of the student model. t Represents the original parameters of the teacher model, SGD(θ) s ) indicates that the parameters of the student model are updated using the stochastic gradient descent algorithm, where m is the momentum parameter and θ′ is the parameter. t These are the updated teacher model parameters;

[0169] Step S5: Use the trained student model (i.e., the original student model after removing the open set semi-supervised units) to perform angle classification on the echocardiogram to be classified.

[0170] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the specific process of step S2 is as follows:

[0171] After randomly flipping the acquired echocardiogram horizontally, pixel filling operations of arbitrary size and method are performed, and finally, N×N images are obtained by random cropping.

[0172] The other steps and parameters are the same as in Specific Implementation Method 1.

[0173] In this embodiment, the size of the cropped image is fixed at 112×112, that is, the value of N is 112.

[0174] Specific Implementation Method 3: This implementation method differs from Specific Implementation Method 1 or 2 in that the hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module;

[0175] The semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model.

[0176] During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map;

[0177] The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-of-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier.

[0178] Other steps and parameters are the same as in specific implementation method one or two.

[0179] The models that can be used in this implementation include, but are not limited to, WideResNet, FastViT, RepViT, and visual transformer. The model used is a four-layer network model, with the feature maps output by each layer having sizes of 64×64, 128×128, 256×256, and 512×512, respectively.

[0180] Specific implementation method four: Combination Figure 1 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to three is that the training process of the teacher model is as follows:

[0181] For any batch of data:

[0182] Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. H is the height of semantic feature map X, W is the width of semantic feature map X, and C is the number of channels in semantic feature map X.

[0183] The feature token acquisition module divides the semantic feature map X into S×S sub-feature maps, and the size of each sub-feature map is [missing information]. Each obtained sub-feature map is used as a feature token of the semantic feature map X, and each feature token...

[0184] Step S42: Use the semantic feature guidance module to process each feature token obtained in step S41 to generate a weighted feature map;

[0185] Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram.

[0186] Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask:

[0187]

[0188]

[0189] in, and These are weighted feature maps corresponding to two unlabeled echocardiograms, where Q represents... The output after a 1×1 convolutional layer, K indicates The output after a 1×1 convolutional layer, D is the dimension of the semantic feature map, T represents transpose, h is a constant, sigmoid is the activation function, upsample represents the upsampling operation, and mask is the mixing mask. Represents matrix multiplication;

[0190] Step S45: The mixed data generation module uses a mixing mask to generate mixed data X. mix :

[0191] X mix =Mask⊙X1+(1-Mask)⊙X2

[0192] Where X1 represents The corresponding echocardiogram, X2 represents The corresponding echocardiogram shows that ⊙ represents element-wise matrix multiplication;

[0193] Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model.

[0194] Training for the current batch is stopped when the loss function converges.

[0195] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0196] It should be noted that after each batch of training is completed, the parameters of the teacher model are updated based on the parameters of the student model after training.

[0197] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the specific process of step S42 is as follows:

[0198] Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: and

[0199] Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys

[0200] Step S423: For the i-th feature token, calculate the Query′ after average pooling of the i-th feature token. i Key′ after average pooling with any other feature token k The dot product of i ≠ k is used as the element in the i-th row of the weight matrix P.

[0201] Similarly, after processing each feature token separately, the weight matrix P is obtained;

[0202] Step S424: Retrieve the (Key′) corresponding to the first k' largest element values ​​from the weight matrix P. k Query′ i Yes, then based on the retrieved (Key′) k Query′ i ) to obtain (Key′ k Value′ i Yes, soon (Key′) k Query′i ) in the Query′ i Replace with Value′ i ;

[0203] Step S425: For each (Key′) obtained in step S424 k Value′ i The data is collected and spliced ​​to obtain the spliced ​​results Key′ and Value′;

[0204] Key′=Gather(Key′ k )

[0205] Value′=Gather(Value′ i )

[0206] Among them, Key′ is for each (Key′ k Value′ i Key′ in the pair k The result of the collection and splicing is that Value′ is for each (Key′) k Value′ i Value′ in the pair i The result of the collection and splicing is given by Gather(·), which represents the function for collecting and splicing tensors, where i = 1, 2, ..., k′, and k = 1, 2, ..., k′.

[0207] Step S426: Calculate the feature map X after attention score weighting using scaled dot product attention. out :

[0208]

[0209] in, The feature map X is linearly transformed using a 1×1 convolution to obtain the Query. It is the scaling factor.

[0210] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0211] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the loss function in step S46 includes the classification loss of the echocardiogram after passing through the classifier with class labels. ce (c,z) and the generation loss of mixed data l mix (X1,X2,X mix );

[0212]

[0213] Where z = [p1, p2, p3, p4, ..., p C’ [p] represents the output of the classifier for echocardiograms with category labels. j Let C' represent the probability that the echocardiogram belongs to the j-th category, C' represent the number of known categories, c represent the category label of the echocardiogram, and p represent the probability that the echocardiogram belongs to the j-th category. c This represents the probability that the classifier predicts the echocardiogram belongs to category label c;

[0214]

[0215] Among them, c X1 c represents the class with the highest probability among the classification results of X1 output by the classifier; X2 p represents the class with the highest probability among the classification results of X2 output by the classifier; xmix This represents the classifier output for the mixed data X. mix The classification results Represents mixed data X mix The probability of belonging to the 1st, 2nd, 3rd, 4th, ..., C'th category.

[0216] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0217] The gradient descent algorithm is used to update the parameters of the hybrid data augmentation unit.

[0218] Specific implementation method seven: Combining Figure 2 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one through six is ​​that the training process of the student model is as follows:

[0219] For any batch of data:

[0220] Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model;

[0221] Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point;

[0222] Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point;

[0223] Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution probability and out-of-distribution probability of the data.

[0224] Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data.

[0225] The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

[0226] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0227] Specific Implementation Method Eight: This implementation method differs from one of Specific Implementation Methods One to Seven in that the probability within the distribution is:

[0228] P in =Linear(AVGPOOL(RELU(BN(x))))

[0229] Where x is the input to the probability calculation module (i.e., the output of the mixed data after passing through the earlier part of the student model), and P... in It is the probability of x within the distribution. BN is a batch normalization layer, ReLU is an activation function, AVGPOOL is an average pooling layer, and Linear is a fully connected layer;

[0230] The out-of-distribution probability calculation module is a multi-head classifier composed of C' binary classifiers, where C' represents the number of known label categories; the output of x through the out-of-distribution probability calculation module is denoted as P. out out-of-distribution probability

[0231] Using the in-distribution probability P in and out-of-distribution probability P out The calculation of open set pseudo-labels is as follows:

[0232] Step (1): Calculate the dot product P of the in-distribution probability and the out-of-distribution probability. sp :

[0233] P sp =P in P out

[0234] Step (2): Calculate the weight of each category within the class:

[0235]

[0236] Among them, z j It is the probability P within the distribution. in In the expression, α represents the probability that x belongs to the j-th category. jIt represents the weight of the j-th category;

[0237] Step (3): Calculate the open set pseudo-label corresponding to x:

[0238] SP = [α1P] sp [0],α2P sp [0],…,α C′ P sp [0],P sp [1]

[0239] Among them, P so [0] is P sp The first element in, P sp [1] is P sp The second element in.

[0240] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0241] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the total open set loss includes the generation loss of pseudo-labels and the classification loss of the open set, i.e., the total open set loss L = l sp +l op ;

[0242] Classification loss of open sets l sp for:

[0243] l sp =l in (c,z)+l out (P in ,P out )

[0244]

[0245] Among them, z b [c] represents the probability that the labeled data belongs to the true category c, calculated by the in-distribution probability calculation module. b [j] represents the probability that the labeled data belongs to category j, calculated by the probability calculation module within the distribution. B represents the number of labeled samples in each batch. P outb [c][0] represents the first element in the c-th row of the output of the out-of-distribution probability calculation module for labeled data, P. outb [c′][1] represents the second element in the c′-th row of the output of the out-of-distribution probability calculation module for labeled data;

[0246] The generation loss of pseudo-labels op for:

[0247]

[0248] V u1 Z u1 =max(SP u1 )

[0249] V u2 Z u2 =max(SP u2 )

[0250] Among them, SP u1 SP represents the pseudo-label of unlabeled data U1. u2 V represents the pseudo-label of the unlabeled data U2. u1 and Z u1 These represent pseudo-tags SP respectively. u1 The maximum probability in the probability distribution and the category index corresponding to the maximum probability, V u2 and Z u2 These represent pseudo-tags SP respectively. u2 The maximum probability in the probability distribution and the category index corresponding to the maximum probability. This represents the prediction result of the mixed data obtained from U1 and U2 by the open set classifier, where μ is the mixing ratio.

[0251] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0252] Specific Implementation Method Ten: This implementation method describes an echocardiographic angle classification system based on hybrid data augmentation and open-set semi-supervised learning. The system includes an echocardiographic dataset acquisition module, an echocardiographic preprocessing module, and a neural network module; wherein:

[0253] An echocardiogram dataset acquisition module is used to acquire both labeled and unlabeled echocardiogram datasets, where the label represents the angle at which the echocardiogram was taken.

[0254] The echocardiography preprocessing module is used to preprocess each acquired echocardiogram separately.

[0255] The specific process of preprocessing is as follows:

[0256] After randomly flipping the acquired echocardiogram horizontally, perform pixel filling operations of arbitrary size and method, and finally obtain N×N images by random cropping.

[0257] The neural network module includes a teacher model and a student model. In the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers. In the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier.

[0258] The hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module;

[0259] The semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model.

[0260] During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map;

[0261] The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier;

[0262] The teacher model and the student model are jointly trained end-to-end based on the preprocessed images. That is, after each batch of data is trained, the parameters of the teacher model are updated based on the parameters of the student model (the teacher model starts the training of the next batch based on the updated parameters, until all batches of data are trained). The final trained student model is used to classify the echocardiograms to be classified into angles.

[0263] The training process for the teacher model is as follows:

[0264] For any batch of data:

[0265] Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. H is the height of semantic feature map X, W is the width of semantic feature map X, and C is the number of channels in semantic feature map X.

[0266] The feature token acquisition module divides the semantic feature map X into S×S sub-feature maps, and the size of each sub-feature map is [missing information]. Each obtained sub-feature map is used as a feature token of the semantic feature map X.

[0267] Step S42: Process each feature token obtained in step S41 using the semantic feature guidance module to generate a weighted feature map; specifically:

[0268] Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: and

[0269] Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys

[0270] Step S423: For the i-th feature token, calculate the Query′ after average pooling of the i-th feature token. i Key′ after average pooling with any other feature token k The dot product of i ≠ k is used as the element in the i-th row of the weight matrix P.

[0271] Similarly, after processing each feature token separately, the weight matrix P is obtained;

[0272] Step S424: Retrieve the (Key′) corresponding to the first k' largest element values ​​from the weight matrix P. k Query′ i Yes, then based on the retrieved (Key′) k Query′ i ) to obtain (Key′ k Value′ i )right;

[0273] Step S425: For each (Key′) obtained in step S424 k Value′ i The data is collected and spliced ​​to obtain the spliced ​​results Key′ and Value′;

[0274] Key′=Gather(Key′ k )

[0275] Value′=Gather(Value′ i )

[0276] Among them, Key′ is for each (Key′ k Value′ i Key′ in the pair k The result of the collection and splicing is that Value′ is for each (Key′) k Value′ i Value′ in the pair i The result of the collection and splicing is given by Gather(·), which represents the function for collecting and splicing tensors, i = 1, 2, ..., k′, k = 1, 2, ..., k′;

[0277] Step S426: Calculate the feature map X after attention score weighting using scaled dot product attention.out :

[0278]

[0279] Specifically, a 1×1 convolution is used to perform a linear transformation on the feature map X to obtain the Query. It is the scaling factor;

[0280] Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram.

[0281] Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask:

[0282]

[0283] in, and These are weighted feature maps corresponding to two unlabeled echocardiograms, where Q represents... The output after a 1×1 convolutional layer, K represents The output after a 1×1 convolutional layer: D is the dimension of the semantic feature map, T represents the transpose, h is a constant, sigmoid is the activation function, upsample represents the upsampling operation, and mask is the mixing mask. Represents matrix multiplication;

[0284] Step S45: The mixed data generation module uses a mixing mask to generate mixed data X. mix :

[0285] X mix =Mask⊙X1+(1-Mask)⊙X2

[0286] Where X1 represents The corresponding echocardiogram, X2 represents The corresponding echocardiogram shows that ⊙ represents element-wise matrix multiplication;

[0287] Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model.

[0288] The loss function includes the classification loss of the classifier for the echocardiogram with class labels. ce (c,z) and the generation loss of mixed data l mix (X1,X2,X mix );

[0289]

[0290] Where z = [p1, p2, p3, p4, ..., p C’ [p] represents the output of the classifier for echocardiograms with category labels. j Let C' represent the probability that the echocardiogram belongs to the j-th category, C' represent the number of known categories, c represent the category label of the echocardiogram, and p represent the probability that the echocardiogram belongs to the j-th category. c This represents the probability that the classifier predicts the echocardiogram belongs to category label c;

[0291]

[0292] Among them, c X1 c represents the class with the highest probability among the classification results of X1 output by the classifier; X2 This represents the category with the highest probability among the classification results of X2 output by the classifier; This represents the classifier output for the mixed data X. mix The classification results Represents mixed data X mix The probability of belonging to the 1st, 2nd, 3rd, 4th, ..., C'th category;

[0293] Training of the current batch is stopped when the loss function converges;

[0294] The training process for the student model is as follows:

[0295] For any batch of data:

[0296] Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model;

[0297] Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point;

[0298] The probability within the distribution is:

[0299] P in =Linear(AVGPOOL(RELU(BN(x))))

[0300] Where x is the input of the probability calculation module within the distribution, P in It is the probability of x within the distribution. BN is a batch normalization layer, ReLU is an activation function, AVGPOOL is an average pooling layer, and Linear is a fully connected layer;

[0301] Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point;

[0302] The out-of-distribution probability calculation module is a multi-head classifier composed of C' binary classifiers, where C' represents the number of known label categories; the output of x after passing through the out-of-distribution probability calculation module is denoted as P. out out-of-distribution probability

[0303] Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution and out-of-distribution probabilities of the data; specifically:

[0304] Step (1): Calculate the dot product P of the in-distribution probability and the out-of-distribution probability. sp :

[0305] P sp =P in P out

[0306] Step (2): Calculate the weight of each category within the class:

[0307]

[0308] Among them, z j It is the probability P within the distribution. in In the expression, α represents the probability that x belongs to the j-th category. j It represents the weight of the j-th category;

[0309] Step (3): Calculate the open set pseudo-label corresponding to x:

[0310] SP = [α1P] sp [0],α2P sp [0],…,α C′ P sp [0],P sp [1]

[0311] Among them, P sp [0] is P sp The first element in, P sp [1] is P sp The second element in;

[0312] Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data.

[0313] The total open-set loss includes the loss from generating pseudo-labels and the classification loss from the open set, i.e., the total open-set loss L = l sp +l op ;

[0314] Classification loss of open sets l sp for:

[0315] l sp =l in (c,z)+l out (P in ,P out )

[0316]

[0317] Among them, z b [c] represents the probability that the labeled data belongs to the true category c, calculated by the in-distribution probability calculation module. b [j] represents the probability that the labeled data belongs to category j, calculated by the probability calculation module within the distribution. B represents the number of labeled samples in each batch. P outb [c][0] represents the first element in the c-th row of the output of the out-of-distribution probability calculation module for labeled data, P. outb [c′][1] represents the second element in the c′-th row of the output of the out-of-distribution probability calculation module for labeled data;

[0318] The generation loss of pseudo-labels op for:

[0319]

[0320] V u1 Z u1 =max(SP u1 )

[0321] V u2 Z u2 =max(SP u2 )

[0322] Among them, SP u1 SP represents the pseudo-label of unlabeled data U1. u2 V represents the pseudo-label of the unlabeled data U2. u1 and Z u1 These represent pseudo-tags SP respectively. u1 The maximum probability in the probability distribution and the category index corresponding to the maximum probability, V u2 and Z u2 These represent pseudo-tags SP respectively. u2 The maximum probability in the probability distribution and the category index corresponding to the maximum probability. This represents the prediction result of the mixed data obtained from U1 and U2 by the open set classifier, where μ is the mixing ratio;

[0323] The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

[0324] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for echocardiographic angle classification based on hybrid data augmentation and open-set semi-supervised learning, characterized in that, The method specifically includes the following steps: Step S1: Obtain echocardiogram datasets with and without category labels, where the category label is the angle of the echocardiogram. Step S2: Preprocess each echocardiogram obtained in step S1. The size of the preprocessed image is N×N. The specific process of step S2 is as follows: After randomly flipping the acquired echocardiogram horizontally, perform pixel filling operations of arbitrary size and method, and finally obtain N×N images by random cropping. Step S3: Construct the teacher model and student model respectively; In the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers; in the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier. The hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module; The semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model. During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map; The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier; Step S4: Use the preprocessed images from step S2 to perform end-to-end joint training on the constructed teacher model and student model; Step S5: Use the trained student model to perform angle classification on the echocardiogram to be classified.

2. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 1, characterized in that, The training process of the teacher model is as follows: For any batch of data: Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. , It is a semantic feature map height, It is a semantic feature map width, It is a semantic feature map The number of channels; The feature token acquisition module will obtain the semantic feature map Divided into There are 10 sub-feature maps, and the size of each sub-feature map is 1. The obtained sub-feature maps are used as semantic feature maps. Each characteristic token; Step S42: Use the semantic feature guidance module to process each feature token obtained in step S41 to generate a weighted feature map; Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram. Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask: in, and These are weighted feature maps corresponding to two unlabeled echocardiograms. express go through The output after the convolutional layer, express go through The output after the convolutional layer, It is the dimension of the semantic feature map. Represents transpose. It is a constant. For activation function, This represents an upsampling operation. It is a hybrid mask. Represents matrix multiplication; Step S45: The mixed data generation module uses a mixing mask to generate mixed data. : in, express The corresponding echocardiogram, express The corresponding echocardiogram, This represents element-wise multiplication of matrices; Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model. Training for the current batch is stopped when the loss function converges.

3. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 2, characterized in that, The specific process of step S42 is as follows: Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: , and ; Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys ; Step S423: For the i-th feature token, calculate the average pooled value of the i-th feature token. Average pooling with any other feature token The dot product, i≠k, is used as the weight matrix. The element in the i-th row of the array, ; Similarly, after processing each feature token separately, a weight matrix is ​​obtained. ; Step S424, from the weight matrix Retrieve the values ​​corresponding to the first k' largest elements from the ( , Yes, then based on the retrieved ( , ) for obtaining right; Step S425: For each of the results obtained in step S424 The data is collected and pieced together to obtain the pieced-together result. and ; in, For each In China The results of collecting and splicing are For each In China The results of collecting and splicing are This represents a function for collecting spliced ​​tensors. , ; Step S426: Calculate the feature map after attention score weighting using scaled dot product attention. : Among them, 1×1 convolution pairs are used for the feature maps. Perform a linear transformation to obtain , It is the scaling factor.

4. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 3, characterized in that, The loss function in step S46 includes the classification loss of the classified echocardiogram after passing through the classifier. and generation loss of mixed data ; in, This is the result of a classifier processing echocardiograms with category labels. This represents the probability that the echocardiogram belongs to the j-th category. This indicates the number of known categories, where c represents the category label of the echocardiogram. This indicates that the classifier predicts the echocardiogram belongs to a category label. The probability of; in, This indicates the output of the classifier. The category with the highest probability in the classification results; This indicates the output of the classifier. The category with the highest probability in the classification results; This represents the output of the classifier for the mixed data. The classification results , Representing mixed data Belonging to the 1st, 2nd, 3rd, 4th, ..., the The probability of each category.

5. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 4, characterized in that, The training process for the student model is as follows: For any batch of data: Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model; Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point; Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point; Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution probability and out-of-distribution probability of the data. Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data. The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

6. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 5, characterized in that, The probability within the distribution is: in, It is the input to the probability calculation module within the distribution. yes The probability within the distribution, , For batch normalization layer, For activation function, For average pooling, Linear is a fully connected layer; The out-of-distribution probability calculation module is composed of A multi-head classifier is composed of combinations of binary classifiers, where... This indicates the number of known label categories; The output of the out-of-distribution probability calculation module is denoted as... out-of-distribution probability ; Using in-distribution probability and out-of-distribution probability The calculation of open set pseudo-labels is as follows: Step (1): Calculate the dot product of the in-distribution probability and the out-of-distribution probability. : Step (2): Calculate the weight of each category within the class: in, It is the probability within the distribution. middle, The probability of belonging to the j-th category. It represents the weight of the j-th category; Step (3) Calculation Corresponding open set pseudo-tags: in, yes The first element in yes The second element in.

7. The echocardiographic angle classification method based on hybrid data augmentation and open-set semi-supervised learning according to claim 6, characterized in that, The total open set loss includes the pseudo-label generation loss and the open set classification loss, i.e., the total open set loss. ; Classification loss of open sets for: in, This indicates that the labeled data, calculated by the probability calculation module within the distribution, belongs to the true category. The probability, This represents the probability that the labeled data belongs to category j, calculated by the in-distribution probability calculation module. B represents the number of labeled samples in each batch. This indicates that the labeled data, after passing through the out-of-distribution probability calculation module, is in the output of the [missing information]. The first element of the row, This indicates that the labeled data, after passing through the out-of-distribution probability calculation module, is in the output of the [missing information]. The second element in the row; Loss of pseudo-label generation for: in, Indicates unlabeled data pseudo-tags Indicates unlabeled data pseudo-tags as well as These represent pseudo tags. The maximum probability in the probability distribution and the category index corresponding to the maximum probability. as well as These represent pseudo tags. The maximum probability in the probability distribution and the category index corresponding to the maximum probability. Indicates according to and The prediction results of the obtained mixed data by the open set classifier It is the mixing ratio.

8. A system for classifying echocardiographic angles based on hybrid data augmentation and open-set semi-supervised learning, characterized in that, The system includes an echocardiogram dataset acquisition module, an echocardiogram preprocessing module, and a neural network module; wherein: An echocardiogram dataset acquisition module is used to acquire both labeled and unlabeled echocardiogram datasets, where the label represents the angle at which the echocardiogram was taken. The echocardiography preprocessing module is used to preprocess each acquired echocardiogram separately. The specific process of preprocessing is as follows: After randomly flipping the acquired echocardiogram horizontally, perform pixel filling operations of arbitrary size and method, and finally obtain N×N images by random cropping. The neural network module includes a teacher model and a student model. In the encoding part of the teacher model, a hybrid data augmentation unit is added between the third and fourth layers. In the encoding part of the student model, an open-set semi-supervised unit is added as a parallel branch of the classifier. The hybrid data enhancement unit includes a semantic feature map acquisition module, a feature token acquisition module, a semantic feature guidance module, a hybrid mask learning module, and a hybrid data generation module; The semantic feature map acquisition module uses WideResNet, FastViT, RepViT, or a visual transformer model. During the forward propagation process, the features output from the penultimate layer of the model are saved as the extracted semantic feature map; The open-set semi-supervised unit includes an in-distribution probability calculation module, an out-distribution probability calculation module, a superclass pseudo-label generation module, and an open-set classifier; The teacher model and the student model are jointly trained end-to-end based on the preprocessed images. That is, after each batch of data is trained, the parameters of the teacher model are updated according to the parameters of the student model. The final trained student model is used to classify the echocardiogram to be classified into angles. The training process for the teacher model is as follows: For any batch of data: Step S41: For any echocardiogram without a category label in the current batch, after the echocardiogram is input into the teacher model, the semantic feature map acquisition module outputs a semantic feature map. , It is a semantic feature map height, It is a semantic feature map width, It is a semantic feature map The number of channels; The feature token acquisition module will obtain the semantic feature map Divided into There are 10 sub-feature maps, and the size of each sub-feature map is 1. The obtained sub-feature maps are used as semantic feature maps. Each characteristic token; Step S42: Process each feature token obtained in step S41 using the semantic feature guidance module to generate a weighted feature map; specifically: Step S421: After performing a 1×1 convolution on each feature token, record the key, query, and value corresponding to the i-th feature token as follows: , and ; Step S422: Perform average pooling on the query and key of each feature token to obtain the query corresponding to the i-th feature token after average pooling. s and keys ; Step S423: For the i-th feature token, calculate the average pooled value of the i-th feature token. Average pooling with any other feature token The dot product, i≠k, is used as the weight matrix. The element in the i-th row of the array, ; Similarly, after processing each feature token separately, a weight matrix is ​​obtained. ; Step S424, from the weight matrix Retrieve the values ​​corresponding to the first k' largest elements from the ( , Yes, then based on the retrieved ( , ) for obtaining right; Step S425: For each of the results obtained in step S424 The data is collected and pieced together to obtain the pieced-together result. and ; in, For each In China The results of collecting and splicing are For each In China The results of collecting and splicing are This represents a function for collecting spliced ​​tensors. , ; Step S426: Calculate the feature map after attention score weighting using scaled dot product attention. : Among them, 1×1 convolution pairs are used for the feature maps. Perform a linear transformation to obtain , It is the scaling factor; Step S43: Process each unlabeled echocardiogram using the methods in steps S41 and S42 to obtain the weighted feature map corresponding to each unlabeled echocardiogram. Step S44: Use the hybrid mask learning module to process the weighted feature maps corresponding to any two unlabeled echocardiograms to obtain the hybrid mask: in, and These are weighted feature maps corresponding to two unlabeled echocardiograms. express go through The output after the convolutional layer, express go through The output after the convolutional layer, It is the dimension of the semantic feature map. Represents transpose. It is a constant. For activation function, This represents an upsampling operation. It is a hybrid mask. Represents matrix multiplication; Step S45: The mixed data generation module uses a mixing mask to generate mixed data. : in, express The corresponding echocardiogram, express The corresponding echocardiogram, This represents element-wise multiplication of matrices; Step S46: After the echocardiogram data with category labels in the current batch and the generated mixed data are processed by the teacher model, the loss function is calculated based on the classifier output of the teacher model. The loss function includes the classification loss of the classified echocardiogram with class labels. and generation loss of mixed data ; in, This is the result of a classifier processing echocardiograms with category labels. This represents the probability that the echocardiogram belongs to the j-th category. This indicates the number of known categories, where c represents the category label of the echocardiogram. This indicates that the classifier predicts the echocardiogram belongs to a category label. The probability of; in, This indicates the output of the classifier. The category with the highest probability in the classification results; This indicates the output of the classifier. The category with the highest probability in the classification results; This represents the output of the classifier for the mixed data. The classification results , Representing mixed data Belonging to the 1st, 2nd, 3rd, 4th, ..., the The probability of each category; Training for the current batch is stopped when the loss function converges. The training process for the student model is as follows: For any batch of data: Step 1: Input the mixed data obtained from the teacher model and the current batch data into the student model; Step 2: Use the in-distribution probability calculation module to output the in-distribution probability of each unlabeled data point and the in-distribution probability of each labeled data point; The probability within the distribution is: in, It is the input to the probability calculation module within the distribution. yes The probability within the distribution, , For batch normalization layer, For activation function, For average pooling, Linear is a fully connected layer; Step 3: Use the out-of-distribution probability calculation module to output the out-of-distribution probability of each unlabeled data point and the out-of-distribution probability of each labeled data point; The out-of-distribution probability calculation module is composed of A multi-head classifier is composed of combinations of binary classifiers, where... This indicates the number of known label categories; The output of the out-of-distribution probability calculation module is denoted as... out-of-distribution probability ; Step 4: For any unlabeled data point, calculate the open set pseudo-label using the in-distribution and out-of-distribution probabilities of the data; specifically: Step (1): Calculate the dot product of the in-distribution probability and the out-of-distribution probability. : Step (2): Calculate the weight of each category within the class: in, It is the probability within the distribution. middle, The probability of belonging to the j-th category. It represents the weight of the j-th category; Step (3) Calculation Corresponding open set pseudo-tags: in, yes The first element in yes The second element in; Step 5: Calculate the classification loss of the open set based on the in-distribution probability and out-distribution probability of the labeled data. Calculate the generation loss of the pseudo-label based on the classification results predicted by the open set classifier for the open set pseudo-labels obtained in Step 4 and the mixed data. The total open set loss includes the pseudo-label generation loss and the open set classification loss, i.e., the total open set loss. ; Classification loss of open sets for: in, This indicates that the labeled data, calculated by the probability calculation module within the distribution, belongs to the true category. The probability, This represents the probability that the labeled data belongs to category j, calculated by the in-distribution probability calculation module. B represents the number of labeled samples in each batch. This indicates that the labeled data, after passing through the out-of-distribution probability calculation module, is in the output of the [missing information]. The first element of the row, This indicates that the labeled data, after passing through the out-of-distribution probability calculation module, is in the output of the [missing information]. The second element in the row; Loss of pseudo-label generation for: in, Indicates unlabeled data pseudo-tags Indicates unlabeled data pseudo-tags as well as These represent pseudo tags. The maximum probability in the probability distribution and the category index corresponding to the maximum probability. as well as These represent pseudo tags. The maximum probability in the probability distribution and the category index corresponding to the maximum probability. Indicates according to and The prediction results of the obtained mixed data by the open set classifier It is the mixing ratio; The sum of the open set classification loss and the pseudo-label generation loss is used as the total open set loss. Training for the current batch is stopped when the total open set loss converges.

Images