Semi-supervised medical image classification method based on space-time structure consistency

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a space-time structure consistency-based semi-supervised image classification method which effectively improves the classification precision of medical images under the condition of limited annotation data by combining space and time structure information. The unlabeled data is effectively utilized, the learning ability of the model for the unlabeled data is improved, the dependence on the labeled data is remarkably reduced, and meanwhile, the method can be applied to different types of medical image tasks, such as skin lesion classification and lung disease classification, and shows excellent performance in single-label and multi-label tasks.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically is a semi-supervised image classification method based on spatiotemporal structural consistency, which effectively improves the classification accuracy of medical images when labeled data is limited by combining spatial and temporal structural information. Background Art

[0002] With the development of medical imaging technology, deep learning has been widely used in medical image analysis, especially with the support of large-scale annotated data sets and manual annotation, which has significantly improved the accuracy of various medical image classification and detection tasks. However, obtaining high-quality medical image annotations requires reliance on clinical experts, which makes the annotation process time-consuming and expensive, especially when dealing with large-scale data sets. This high dependence on annotated data has greatly limited the scope of application of deep learning models in medical image analysis.

[0003] In order to reduce the dependence on labeled data, semi-supervised learning (SSL) methods have emerged. By combining limited labeled data with a large amount of unlabeled data, the generalization ability of the model has been significantly improved. SSL methods are generally divided into two categories: self-training methods and consistency regularization methods. Self-training methods use models to predict unlabeled data and add high-confidence prediction results to the training set to improve the performance of the model; consistency regularization methods introduce data augmentation and perturbations to maintain the consistency of the model output under different perturbations of the input data, thereby improving the model's utilization of unlabeled data.

[0004] Although existing SSL methods have made some progress in reducing the reliance on labeled data, existing technologies mainly focus on the processing of static images, ignoring the spatiotemporal structural relationship between different samples in medical images. In medical image analysis, clinicians often refer to historical images or other relevant samples of patients to improve the accuracy of diagnosis. In order to further improve the utilization efficiency of unlabeled data, there is an urgent need for a semi-supervised learning framework that can simultaneously consider the spatial and temporal relationship between samples, so as to better capture the structural information between samples. Summary of the invention

[0005] In view of the shortcomings and deficiencies in the prior art, the present invention proposes a method of improving the accuracy of image classification by combining unlabeled data with limited labeled data, thereby solving the problem of the prior art's semi-supervised image classification method based on spatiotemporal structural consistency, which is highly dependent on a large amount of labeled data.

[0006] The present invention is achieved by the following measures:

[0007] A semi-supervised image classification method based on spatiotemporal structural consistency is characterized in that the spatiotemporal structural consistency between modeling samples is used to improve the efficiency of the model in utilizing unlabeled data, and the method includes the following steps:

[0008] Step 1: In the spatial dimension, the similarity between samples is captured by constructing the Gram matrix between samples;

[0009] Step 2: In the time dimension, a time substructure consistency method is proposed to maintain the stability of samples in different time steps;

[0010] Step 3: Combine the spatiotemporal structural consistency method with semi-supervised learning, adopt a teacher-student model structure, and jointly optimize the model through supervised loss and unsupervised loss;

[0011] Step 4: Apply the optimized model to classify the image.

[0012] In the spatiotemporal structure consistency method of the present invention, in step 1, the following operations are performed in the spatial dimension:

[0013] Step 1-1: Calculate the feature representation of the sample and calculate the similarity between samples through the inner product of the Gram matrix: Extract the feature representation of the small batch sample Where B is the batch size, H and W are the spatial dimensions of the feature map, and C is the number of channels. l Reshape into a 2D matrix And calculate the Gram matrix M l ,

[0014] M l =D l ·(D l ) T ;

[0015] Step 1-2: Convert the Gram matrix into an adjacency matrix to describe the graph structure of the sample: l Perform L2 normalization to obtain the relationship matrix R l ,

[0016]

[0017] Step 1-3: Define the spatial structure consistency loss function L sc , ensuring that the relationship between samples remains stable under different perturbations:

[0018]

[0019] Among them, the X generation data samples, Represents N with label y i Sample xi ,and Represents M unlabeled samples x i , θ and θ ` represents the learnable parameters in the teacher-student model f(·,θ), while η and η ` Represents the perturbations added to the input of the teacher and student models respectively.

[0020] In the spatiotemporal structure consistency method of the present invention, in step 2, in the time dimension, the adjacency matrix is ​​constructed by binarizing the Gram matrix to describe the connection relationship between samples. During the training process, the consistency of the time substructure is continuously monitored to capture the stable substructure in the graph structure to ensure the consistency of the sample relationship at different time steps:

[0021] Step 2-1: By binarizing the Gram matrix, construct the adjacency matrix A between samples, and its element A(i,j) is defined as:

[0022]

[0023] Among them, τ is the threshold for filtering weak semantic relationships, and R(i,j) is used to measure the sample x i With sample x j The Gram matrix of the semantic relationship between them.

[0024] Step 2-2: Based on the adjacency matrix A, construct a graph structure G(X,E), where X represents the sample and E represents the edge between samples. During the training process, identify the substructure that remains stable between time t and t+1

[0025] Step 2-3: Define the temporal structure consistency loss function L tc , ensuring that the relationship between samples at different times remains stable:

[0026]

[0027] Where k represents the number of stable substructures, s i Represents the i th A stable substructure, and Represents the relationship matrix between time t and time t-1 during the training process,

[0028] The parameters of the teacher model in step 3 of the present invention are updated by the exponential moving average EMA of the parameters of the student model, and the updating formula is:

[0029] θ′ t =αθ′ t-1 +(1-α)θ t ,

[0030] Among them, α is a hyperparameter that controls the update rate, θ′ t and θ′ t-1 is the weight parameter at time t and t-1 during the teacher model training process, and θ t is the weight parameter of the student model at time t. The teacher model and the student model maintain the prediction consistency under the same input image through the following consistency loss:

[0031]

[0032] Among them, E η′,η is a consistency constraint, f(x i ,θ′,η′) is the teacher model for sample x i The prediction result after adding the perturbation η`, f(x i ,θ,η) is the student model for sample x i Prediction results after adding the perturbation η.

[0033] The total loss function consists of supervised loss and unsupervised loss, which is defined as follows:

[0034] L=L s +λL u ,L u =L c +βL sc +γL tc ,

[0035] Among them, L s To monitor the loss, L u is the unsupervised loss, including the consistency loss L c , spatial structure consistency loss L sc and the temporal substructure consistency loss L tc , β and γ are balance hyperparameters, usually set to 1, and λ controls the trade-off between supervised loss and unsupervised loss.

[0036] Compared with the prior art, the present invention effectively utilizes unlabeled data by introducing a spatiotemporal structural consistency mechanism, thereby improving the model's learning ability for unlabeled data and significantly reducing dependence on labeled data. At the same time, the present invention can be applied to different types of medical image tasks, such as skin lesion classification, lung disease classification, etc., and exhibits excellent performance in single-label and multi-label tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Attached Figure 1 It is a flow chart of the present invention.

[0038] Attached Figure 2 It is a heat map visualization between different relationship matrices in the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0040] Example:

[0041] like Figure 1 , 2 As shown in the figure, this example proposes a semi-supervised learning model based on spatiotemporal structural consistency (STSC) to improve the performance of medical image classification tasks. The model uses the teacher-student model structure and combines the spatial and temporal consistency between samples to achieve effective utilization of unlabeled data.

[0042] like Figure 1 As shown in Figure 2, in this example, the teacher model is used as the exponential moving average (EMA) of the student model to update. In this example, in order to ensure the effective learning of the model for unlabeled data, the framework introduces spatiotemporal consistency, including the spatial relationship loss L sc and time relationship loss L tc ;

[0043] In order to capture the spatial relationship between samples, the present invention describes the similarity between samples by calculating the Gram matrix. The specific steps are as follows:

[0044] Step 1: Extract feature representation of small batch samples Where B is the batch size, H and W are the spatial dimensions of the feature map, and C is the number of channels. Step 2: Convert the feature map F l Reshape into a 2D matrix And calculate the Gram matrix M l :

[0045] M l =D l ·(D l ) T ,

[0046] Step 3: Gram matrix M l Perform L2 normalization to obtain the relationship matrix R l :

[0047] Step 4: Define the spatial structure consistency loss function L sc , ensuring that the relationship between samples remains stable under different perturbations:

[0048]

[0049] like Figure 2 As shown, in order to ensure the consistency of samples in the time dimension, the present invention proposes a time substructure consistency method, and the specific steps are as follows:

[0050] Step 1: By binarizing the Gram matrix, construct the adjacency matrix A between samples, and its element A(i,j) is defined as:

[0051]

[0052] Among them, τ is the threshold used to filter weak semantic relations;

[0053] Step 2: Based on the adjacency matrix A, construct a graph structure G(X,E), where X represents the sample and E represents the edge between samples. During the training process, identify the substructure that remains stable between time t and t+1.

[0054] Step 3: Define the temporal structure consistency loss function L tc , ensuring that the relationship between samples at different times remains stable:

[0055]

[0056] like Figure 1 As shown in Figure 2, the parameters of the teacher model are updated by the exponential moving average (EMA) of the parameters of the student model, and the update formula is:

[0057] θ′ t =αθ′ t-1 +(1-α)θ t

[0058] Among them, α is a hyperparameter that controls the update rate. The teacher model and the student model maintain the prediction consistency under the same input image through the following consistency loss:

[0059]

[0060] The total loss function in this example consists of supervised loss and unsupervised loss, which are defined as follows:

[0061] L=L s +λL u ,L u =L c +βL sc +γL tc ,

[0062] Among them, L s To monitor the loss, L u is the unsupervised loss, including the consistency loss L c , spatial structure consistency loss L sc and the temporal substructure consistency loss L tc , β and γ are balance hyperparameters, usually set to 1, and λ controls the trade-off between supervised loss and unsupervised loss.

[0063] The following example is applied to the public medical image datasets ISIC 2018 and ChestX-ray14 for experiments. The results show that the spatiotemporal structural consistency framework has better classification performance than the existing methods. Figure 2 The heat map visualization results of different relationship matrices between the student model and the teacher model are shown, further demonstrating the effectiveness of spatiotemporal consistency in maintaining the stability of sample structure;

[0064] By introducing spatial and temporal consistency, the present invention significantly improves the generalization ability of the model on unlabeled data and reduces the dependence on labeled data.

Claims

1. A semi-supervised image classification method based on spatiotemporal structural consistency, characterized in that: The model improves the efficiency of using unlabeled data by modeling the spatiotemporal structure consistency between samples, which includes the following steps: Step 1: In the spatial dimension, the similarity between samples is captured by constructing the Gram matrix between samples; Step 2: In the time dimension, a time substructure consistency method is proposed to maintain the stability of samples in different time steps; Step 3: Combine the spatiotemporal structural consistency method with semi-supervised learning, adopt a teacher-student model structure, and jointly optimize the model through supervised loss and unsupervised loss; Step 4: Apply the optimized model to classify the image.

2. The semi-supervised image classification method based on spatiotemporal structural consistency according to claim 1, characterized in that: In the spatiotemporal structure consistency method, in step 1, the following operations are performed in the spatial dimension: Step 1-1: Calculate the feature representation of the sample and calculate the similarity between samples through the inner product of the Gram matrix: Extract the feature representation of the small batch sample Where B is the batch size, H and W are the spatial dimensions of the feature map, and C is the number of channels. l Reshape into a 2D matrix And calculate the Gram matrix M l , M l =D l ·(D l ) R ; Step 1-2: Convert the Gram matrix into an adjacency matrix to describe the graph structure of the sample: l Perform L2 normalization to obtain the relationship matrix R l : Step 1-3: Define the spatial structure consistency loss function L sc , ensuring that the relationship between samples remains stable under different perturbations: Among them, the X generation data samples, Represents N with label y i Sample x i ,and Represents M unlabeled samples x i , θ and θ ` represents the learnable parameters in the teacher-student model f(·,θ), while η and η ` Represents the perturbations added to the input of the teacher and student models respectively.

3. The semi-supervised image classification method based on spatiotemporal structural consistency according to claim 1, characterized in that: In the spatiotemporal structural consistency method, in step 2, in the time dimension, the adjacency matrix is ​​constructed by binarizing the Gram matrix to describe the connection relationship between samples. During the training process, the consistency of the time substructure is continuously monitored to capture the stable substructure in the graph structure to ensure the consistency of the sample relationship at different time steps: Step 2-1: By binarizing the Gram matrix, construct the adjacency matrix A between samples, and its element A(i,j) is defined as: Among them, τ is the threshold for filtering weak semantic relationships, and R(i,j) is used to measure the sample x i With sample x j The Gram matrix of the semantic relationship between them. Step 2-2: Based on the adjacency matrix A, construct a graph structure G(X,E), where X represents the sample and E represents the edge between samples. During the training process, identify the substructure that remains stable between time t and t+1 Step 2-3: Define the temporal structure consistency loss function L tc , ensuring that the relationship between samples at different times remains stable: Where k represents the number of stable substructures, s i Represents the i th A stable substructure, and Represents the relationship matrix between time t and time t-1 during the training process, 4. The semi-supervised image classification method based on spatiotemporal structural consistency according to claim 1, characterized in that: In step 3, the parameters of the teacher model are updated by the exponential moving average EMA of the parameters of the student model. The update formula is: θ′ t =θ′ t-1 +(1-α)θ t , Among them, α is a hyperparameter that controls the update rate, θ′ t and θ′ t-1 is the weight parameter at time t and t-1 during the teacher model training process, and θ t is the weight parameter of the student model at time t. The teacher model and the student model maintain the prediction consistency under the same input image through the following consistency loss: Among them, E η′,η is a consistency constraint, f(x i ,θ′,η′) is the teacher model for sample x i When adding the perturbation η ` The subsequent prediction results, f(x i ,θ,η) is the student model for sample x i The prediction results after adding the perturbation η are: The total loss function consists of supervised loss and unsupervised loss, which is defined as follows: L=L s +λL u ,L u =L c +βL sc +γL tc , Among them, L s To monitor the loss, L u is the unsupervised loss, including the consistency loss L c , spatial structure consistency loss L sc and the temporal substructure consistency loss L tc , β and γ are balance hyperparameters, usually set to 1, and λ controls the trade-off between supervised loss and unsupervised loss.

Images