Semi-supervised semantic segmentation model training method and application based on multi-level label correction

Abstract

The present invention discloses a semi-supervised semantic segmentation model training method and application based on multi-level label correction. The training method includes: constructing student and teacher networks; performing labeled training on the student network; generating pseudo labels and intermediate features for unlabeled data sets; constructing a semantic affinity matrix using a semantic level graph, and constructing a classification consistency matrix using a category level graph; exchanging the structural information corresponding to the level graph, gradually correcting the level graph, updating the intermediate features and correcting the pseudo labels; performing unlabeled training, and iterating the training process to obtain a semantic segmentation model. In the technical solution provided by the present invention, the semantic level graph represents the semantic affinity in the feature space, the category level graph represents the category consistency in the category space, the semantic level graph and the category level graph learn from each other and evolve together, and the category level graph and the semantic level graph are gradually corrected, thereby correcting some pseudo labels and achieving significant improvements in training efficiency and model prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a semi-supervised semantic segmentation model training method based on multi-level label correction and its application. Background Art

[0002] Semantic segmentation is a fundamental task in computer vision, which aims to assign a label to each pixel of an image. The development of deep convolutional neural networks has led to great progress in semantic segmentation in many public benchmarks. However, supervised semantic segmentation requires a large amount of pixel-level annotated data, but pixel-level annotation is costly and time-consuming, which is the bottleneck of semantic segmentation.

[0003] Semi-supervised learning can alleviate this problem by extracting useful knowledge from both labeled and unlabeled samples. Consistency regularization is a commonly used method in semi-supervised learning. It encourages the network to produce similar predictions for different augmented versions of a single unlabeled image. Another semi-supervised learning method is self-training, which can use the trained network to generate pseudo labels for unlabeled images and use them for supervised learning.

[0004] However, since predictions on unlabeled data are inevitably error-prone, the network is prone to overfitting incorrect predictions, leading to confirmation bias. To alleviate this problem, existing methods use thresholds to filter predictions, retaining high-confidence predictions and discarding low-confidence predictions. This indiscriminately wastes pseudo-labels that may be correct. Some methods correct incorrect pseudo-labels by learning auxiliary networks, for example, error correction networks learned from the differences between network predictions and ground truth annotations. However, in practice, when only a few labeled data are used for training, such networks are prone to overfitting problems and have difficulty covering a wide range of prediction errors. In other learning paradigms, there are also some graph-based pseudo-label correction methods. However, these methods usually rely on a single graph using very basic feature propagation methods, and they are not designed for semi-supervised learning, have poor matching, and it is difficult to obtain better training efficiency / effects. Summary of the invention

[0005] In view of the deficiencies in the prior art, the object of the present invention is to provide a semi-supervised semantic segmentation model training method and application based on multi-level label correction.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0007] In a first aspect, the present invention provides a semi-supervised semantic segmentation model training method based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, and comprises:

[0008] Construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the structure of the student network is the same as that of the teacher network, the labeled data set includes labeled data and its corresponding labels, and the unlabeled data set includes unlabeled data;

[0009] Using a labeled data set to perform labeled training on the student network;

[0010] Generate pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network;

[0011] Using a semantic level graph to construct a semantic affinity matrix of pixel feature pairs in the intermediate features, using a category level graph to construct a classification consistency matrix in the pseudo labels, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively;

[0012] Interchanging structural information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels;

[0013] The updated intermediate features and the corrected pseudo-labels are used to update the parameters of the student network and the teacher network, and unlabeled training is performed; and the labeled training and unlabeled training processes are iterated until the end condition is reached to obtain a semantic segmentation model.

[0014] In a second aspect, the present invention further provides an image semantic segmentation method, which comprises:

[0015] Providing a semantic segmentation model obtained by the above-mentioned semi-supervised semantic segmentation model training method;

[0016] The semantic segmentation model is used to perform semantic segmentation on the image to be segmented.

[0017] In a third aspect, the present invention further provides a semi-supervised semantic segmentation model training system based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, and comprises:

[0018] A basic module, used to construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the structure of the student network is the same as that of the teacher network, the labeled data set contains labeled data and its corresponding labels, and the unlabeled data set contains unlabeled data;

[0019] A labeled training module, used for performing labeled training on the student network using a labeled data set;

[0020] A module for generating pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network;

[0021] A matrix construction module, for constructing a semantic affinity matrix of pixel feature pairs in the intermediate features using a semantic level graph, constructing a classification consistency matrix in the pseudo labels using a category level graph, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively;

[0022] A structure exchange module, used for exchanging the structure information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels;

[0023] The iterative update module is used to update the parameters of the student network and the teacher network using the updated intermediate features and the corrected pseudo labels to perform unlabeled training; and iteratively perform the process of labeled training and unlabeled training until the end condition is reached to obtain a semantic segmentation model.

[0024] In a fourth aspect, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the above-mentioned semi-supervised semantic segmentation model training method are executed.

[0025] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0026] In the technical solution provided by the present invention, the semantic-level graph represents the semantic affinity in the feature space, and the category-level graph represents the category consistency in the category space. The semantic-level graph and the category-level graph learn from each other and evolve together. Furthermore, during the training process, the category-level graph uses the semantic affinity in the semantic-level graph to aggregate similar category predictions. In this process, the noise will be smoothed by the neighbors, and the semantic-level graph uses the category consistency in the category-level graph to aggregate features of the same category to obtain more representative features. More representative features can generate more correct semantic affinities. Such a process can be executed multiple times, and the category-level graph and the semantic-level graph are gradually corrected, thereby correcting some pseudo-labels and achieving significant improvements in training efficiency and model prediction accuracy.

[0027] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of a semi-supervised semantic segmentation model training method based on multi-level label correction provided by a typical implementation case of the present invention;

[0029] Figure 2It is a schematic diagram of the network structure of a semi-supervised semantic segmentation model training method based on multi-level label correction provided by a typical implementation case of the present invention;

[0030] Figure 3 It is a comparison diagram of semantic segmentation effects of a semi-supervised semantic segmentation model training method based on multi-level label correction provided by a typical implementation case of the present invention and a control baseline;

[0031] Figure 4 It is a comparison diagram of the training effects of a semi-supervised semantic segmentation model training method based on multi-level label correction and a supervised control baseline provided by a typical implementation case of the present invention;

[0032] Figure 5 This is a comparison chart of the impact of the number of graph convolutions on the performance of a semi-supervised semantic segmentation model training method based on multi-level label correction provided in a typical implementation case of the present invention. DETAILED DESCRIPTION

[0033] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0035] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0036] The purpose of the present invention is to propose a semi-supervised semantic segmentation method based on multi-level label correction, and its process can be summarized as follows: use labeled data to train a student network; use a teacher network to generate pseudo labels and intermediate features for unlabeled data; use pseudo labels and intermediate features to construct category-level graphs and semantic-level graphs; the semantic-level graph and the category-level graph update each other and evolve together; and update network parameters.

[0037] In addition, some preferred embodiments of the present invention also propose a contrastive learning method that combines intra-image and inter-image comparison to train semantic-level graphs to promote feature learning and a cross-entropy with dynamic weights to train category-level graphs to further alleviate the impact of noisy labels.

[0038] See also Figure 1-Figure 2 (For ease of observation, the present invention will Figure 2The embodiment of the present invention provides a semi-supervised semantic segmentation model training method based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, and the method comprises the following steps:

[0039] Construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the structure of the student network is the same as that of the teacher network, the labeled data set includes labeled data and its corresponding labels, and the unlabeled data set includes unlabeled data;

[0040] Using a labeled data set to perform labeled training on the student network;

[0041] Generate pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network;

[0042] Using a semantic level graph to construct a semantic affinity matrix of pixel feature pairs in the intermediate features, using a category level graph to construct a classification consistency matrix in the pseudo labels, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively;

[0043] Interchanging structural information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels;

[0044] The updated intermediate features and the corrected pseudo-labels are used to update the parameters of the student network and the teacher network, and unlabeled training is performed; and the labeled training and unlabeled training processes are iterated until the end condition is reached to obtain a semantic segmentation model.

[0045] Regarding the meaning of the symbols in the figure, Figure 2 In the figure, SLG and CLG represent semantic-level graphs and category-level graphs, respectively. W.Aug and S.Aug represent weak image enhancement and strong image enhancement, respectively. GCN represents graph convolution operation. Emb and Cls represent feature embedding layer and embedding layer, respectively.

[0046] Compared with the prior art, the training method provided by the present invention can effectively correct incorrect pseudo-labels and avoid the interference of incorrect pseudo-labels on the training process. The method uses graph neural networks to capture the structural relationship between semantic-level graphs and class-level graphs to correct incorrect pseudo-labels. It can significantly improve the performance of semi-supervised semantic segmentation and easily achieve new state-of-the-art performance on popular semi-supervised semantic segmentation benchmarks under different data partitioning protocols. This framework can also be easily used for different network frameworks.

[0047] As some typical implementation examples of the above-mentioned overall training process, the above-mentioned training method can be implemented by adopting the following specific process:

[0048] S1. Design a set of student-teacher networks;

[0049] S2, train the student network using labeled data and its labels;

[0050] S3, use the teacher network to generate pseudo labels and intermediate features for unlabeled data;

[0051] S4, using the semantic level graph to construct the semantic affinity matrix of pixel feature pairs in the intermediate features, and using the category level graph to construct the classification consistency matrix in the pseudo labels;

[0052] S5, the structural information in the semantic level graph and the category level graph are exchanged to obtain more accurate semantic level graph and category level graph, and the wrong pseudo labels are gradually corrected;

[0053] S6. Use gradient descent to update the student network parameters, use exponential moving average to update the teacher network parameters, and repeat S2-S5 until convergence.

[0054] As for the specific training details, in some embodiments, the training of the student network is preferably performed by cross entropy training;

[0055] In some embodiments, the labeled loss function for training the student network is expressed as:

[0056]

[0057] in Indicates label loss, X l Represents the labeled data, Y l represents a hot vector obtained by the label transformation, f θ Denotes that the parameter corresponding to the student network is θ, and H represents the cross entropy loss function, which is defined as:

[0058]

[0059] Where i, j represent pixel indices, c represents category index, and Y l represents a one-hot vector transformed by the ground truth or pseudo label, and P represents the prediction vector of the student network.

[0060] In some embodiments, the teacher network is input with weak image enhancement, and the student network is input with strong image enhancement.

[0061] In some embodiments, the process of constructing the semantic level graph may specifically include the following steps:

[0062] Retain a first number of nearest neighbor pixels of any pixel, and construct a first undirected graph of all pixels in the same batch of the unlabeled data using a first affinity matrix;

[0063] Normalizing the first affinity matrix to obtain a first normalized matrix;

[0064] The process of constructing the category-level graph specifically includes:

[0065] Using a second affinity matrix to represent the relationship between label confidence scores of all pixels in the unlabeled data of the same batch, constructing a second undirected graph;

[0066] The second affinity matrix is ​​normalized to obtain a second normalized matrix.

[0067] In some embodiments, the construction process of the semantic level graph can be expressed as:

[0068]

[0069] in, represents the first affinity matrix, represents the feature vector corresponding to pixel i, γ represents the scaling parameter, express The set of k nearest neighbors of

[0070]

[0071] Wherein, A represents the first normalized matrix, express The diagonal matrix of

[0072] The first undirected graph in, represents the vertex set consisting of generated features; ε (S) represents a set of edges, measuring the affinity between pixel-level feature pairs, corresponding to the first normalized matrix; is the feature matrix;

[0073] The construction process of the category-level graph can be expressed as:

[0074]

[0075] in, represents the second affinity matrix, represents the predicted probability vector corresponding to pixel i, express The corresponding maximum probability category;

[0076]

[0077] Wherein, W represents the second normalized matrix, express The diagonal matrix of

[0078] The second undirected graph in represents the set of vertices consisting of the output label predictions; ε (C) represents a set of edges, used to measure the affinity between prediction nodes, corresponding to the second normalized matrix; is the confidence matrix, each row corresponds to the predicted probability vector.

[0079] In some embodiments, the gradual modification process of the semantic level graph and the category level graph may specifically include:

[0080] In one iteration, the semantic level graph is subjected to a first graph convolution operation several times, and the category level graph is subjected to a second graph convolution operation several times;

[0081] In the kth convolution operation of the second graph, the first node of the category-level graph is updated, and the propagation rule is expressed as:

[0082]

[0083] in, represents the first node, represents a parameter of the first graph convolution operation Composed of nonlinear functions, [·,·] is a joint function;

[0084] In the kth convolution operation of the first graph, the second node of the semantic level graph is updated, and the update rule is expressed as:

[0085]

[0086]

[0087] in, represents the second node, α is a hyperparameter that controls the proportion of nodes before and after label propagation, η (t，c) represents the dynamic threshold, σ is the fixed threshold, represents the learning effect of the current category c in the current small batch t, ​​pi is the prediction vector output by the original segmentation head, is a function that returns 1 if its input is true, and 0 otherwise;

[0088] After the second node is updated, the edge connection W(k) is updated by constructing a formula matrix through the second affinity matrix of the category-level graph.

[0089] In some embodiments, the loss function of the unlabeled training may include a semantic-level graph loss that reflects the similarity of the intermediate features provided by the teacher network and the student network, and a category-level graph loss that reflects the similarity of the provided pseudo-labels;

[0090] In some embodiments, the calculation method of the semantic level graph loss can be expressed as:

[0091]

[0092] in, represents the semantic-level graph loss, and λ represents the weight factor. if and Have the same tags, is 1, otherwise it is 0; A strongly data-augmented version representing the features in the k-th semantic level graph; Represents a prototype; It is the pseudo label of the weakly enhanced image at pixel i predicted by the network, which determines the positive prototype τ represents the temperature constant;

[0093] The calculation method of the category-level graph loss can be expressed as:

[0094]

[0095] in, represents the category-level graph loss, is the pseudo label generated by weakly enhanced data, and Pk is the confidence matrix of the k-th category-level graph, that is, Pseudo Labels It is obtained by aggregating the confidence matrices of K category-level graphs. Represents dynamic weight, and the calculation method can be expressed as:

[0096]

[0097] In some embodiments, the prototype may be calculated using feature averaging, and the resulting prototype is used as the feature center of each class;

[0098] In some embodiments, the calculation of the prototype can be expressed as:

[0099]

[0100] in is the feature in the k-th semantic level graph, is the pseudo label of the unlabeled data at position i;

[0101] In some embodiments, the prototype is estimated as an exponential moving average of small batch clustering. Specifically, in each round of iteration, the estimate of the prototype is expressed as:

[0102]

[0103] where β is a hyperparameter that controls the speed at which the prototype is updated.

[0104] In some embodiments, the unlabeled loss function of the unlabeled training is a weighted addition of the semantic level graph loss and the category level graph loss;

[0105] The total loss function of iterative training is the sum of the unlabeled loss function and the labeled loss function;

[0106] In some embodiments, during multiple rounds of iterations, the student network is updated by gradient descent, and the parameters of the teacher network are derived from the exponential moving average of the corresponding parameters of the student network;

[0107] In some embodiments, the gradient descent process can be expressed as:

[0108]

[0109] The parameter calculation method of the teacher network can be expressed as:

[0110] ξ(t+1)←αξ(t)+(1-α)θ(t+1)

[0111] in, represents the total loss function, β is the learning rate, ξ represents the parameters of the teacher network, θ represents the parameters of the student network, α represents the hyperparameter, and 0<α<1.

[0112] As an example, in some typical implementation cases, the specific steps of the training method are as follows:

[0113] (1) Set up a set of teacher-student networks, where the teacher network and the student network have the same structure;

[0114] (2) Use labeled data and its labels to train the student network in the teacher-student network;

[0115] (3) Use the classification head of the teacher network to generate pseudo labels for unlabeled data, and use the mapping head to generate intermediate features for unlabeled data;

[0116] (4) Use the intermediate features to construct a semantic-level graph, where the semantic affinity matrix represents the similarity between pixel pairs, and use the pseudo-labels to construct a category-level graph, where the classification consistency matrix represents the consistency between pixel-level pseudo-labels;

[0117] (5) The structural information between the semantic-level graph and the category-level graph is exchanged to obtain more accurate semantic-level graph and category-level graph, that is, to obtain more representative features and gradually corrected pseudo-labels;

[0118] (6) Use gradient descent to update the student network parameters and use exponential moving average to update the teacher network parameters. Repeat steps (2) to (5) until convergence.

[0119] The step (1) is a common initialization step, with a set of teacher-student networks, and the teacher network and the student network have the same structure. In each iteration, the student network parameters are updated using gradient descent, and the parameters of the teacher network come from the exponential moving average of the corresponding student network. The corresponding formula is as follows:

[0120] ξ=αξ+(1-α)θ

[0121] where ξ represents the parameters of the teacher, θ represents the parameters of the student network, and τ is a hyperparameter with a value between 0 and 1. Therefore, the teacher model can be considered as a temporal ensemble of student models across training time t, resulting in more stable predictions for unlabeled images.

[0122] Furthermore, in step (2), cross entropy is used to train the student network.

[0123] Furthermore, in step (3), each teacher-student network has a classification detection head and a mapping head, which generate category predictions and intermediate feature representations respectively.

[0124] Furthermore, the construction of the semantic level graph and the category level graph in step (4) specifically includes the following steps:

[0125] (i) The semantic level graph records the semantic relationships between features. By aggregating semantic neighborhood knowledge, noisy pseudo labels can be effectively corrected.

[0126] (ii) The label-level graph records the consistency information of the predicted categories, that is, the edges between nodes represent pixel pairs belonging to the same class. By aggregating the similarity of the predicted category labels, the relationship between feature nodes belonging to the same category can be strengthened. In addition, this aggregation can make node features context-aware and obtain more accurate feature representation;

[0127] (iii) Semantic-level graphs and classification-level graphs have their own unique advantages.

[0128] Furthermore, the updating process in step (5) pays more attention to low-confidence predictions, because low-confidence predictions are more likely to be noisy.

[0129] As a further application of the above technical solution, an embodiment of the present invention further provides an image semantic segmentation method, which includes the following steps:

[0130] Providing a semantic segmentation model obtained by the semi-supervised semantic segmentation model training method provided in any of the above implementation schemes;

[0131] The semantic segmentation model is used to perform semantic segmentation on the image to be segmented.

[0132] Correspondingly, an embodiment of the present invention further provides a semi-supervised semantic segmentation model training system based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, and includes the following modules:

[0133] A basic module, used to construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the structure of the student network is the same as that of the teacher network, the labeled data set contains labeled data and its corresponding labels, and the unlabeled data set contains unlabeled data;

[0134] A labeled training module, used for performing labeled training on the student network using a labeled data set;

[0135] A module for generating pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network;

[0136] A matrix construction module, for constructing a semantic affinity matrix of pixel feature pairs in the intermediate features using a semantic level graph, constructing a classification consistency matrix in the pseudo labels using a category level graph, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively;

[0137] A structure exchange module, used for exchanging the structure information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels;

[0138] The iterative update module is used to update the parameters of the student network and the teacher network using the updated intermediate features and the corrected pseudo labels to perform unlabeled training; and iteratively perform the process of labeled training and unlabeled training until the end condition is reached to obtain a semantic segmentation model.

[0139] Correspondingly, an embodiment of the present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the semi-supervised semantic segmentation model training method provided in any of the above-mentioned implementation schemes are executed.

[0140] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0141] Example 1

[0142] This embodiment illustrates the process of using the above technical solution to perform semi-supervised training of a semantic segmentation model, as shown below:

[0143] (1) Set up a set of teacher-student networks, where the teacher network and the student network have the same structure. In each iteration, the student network parameters are updated using gradient descent, and the parameters of the teacher network come from the exponential moving average of the corresponding student network. The corresponding formula is as follows:

[0144] ξ=αξ+(1-α)θ

[0145] Where ξ represents the parameters of the teacher, θ represents the parameters of the student network, and α is a hyperparameter with a value between 0 and 1. Therefore, the teacher model can be considered as a temporal collection of student models across training time t, resulting in more stable predictions for unlabeled images;

[0146] (2) Use cross entropy to train the student network. The formula is as follows:

[0147]

[0148] Where X l Indicates labeled data, Y l represents a hot vector transformed by the label, f θ The parameter corresponding to the student network is θ, and H represents the cross entropy loss function, which is defined as follows:

[0149]

[0150] Where i, j represent pixel indices, c represents category index, and Y l represents a one-hot vector transformed by the ground truth or pseudo label and P represents the predicted vector.

[0151] (3) The semantic level graph records the semantic relationship between features. By aggregating semantic neighborhood knowledge, the noise pseudo-label can be effectively corrected. Specifically, the present invention constructs an undirected graph from all the pixels of the unlabeled data in a mini-batch. in represents the vertex set composed of features generated by the feature embedding head. The variable ε (S) represents a collection of edges that measure the affinity between pairs of pixel-level features. is a feature matrix, each row corresponds to a node feature. An intuitive way to construct edges is to connect all pixels, but this method has high computational complexity and some original features may be covered by incorrect neighbors. Therefore, the present invention only retains k nearest neighbors and uses the affinity matrix To model this relationship:

[0152]

[0153] in is the feature vector corresponding to pixel i, γ is the scaling parameter, which is generally set to 1. yes is the set of k nearest neighbors, n = H × W is the number of pixels in each image, The resulting affinity matrix is ​​then normalized:

[0154]

[0155] in yes The diagonal matrix is

[0156] The Class-Level Graph (CLG) records the consistency information of the predicted categories, that is, the edges between nodes represent pairs of pixels belonging to the same class. By aggregating the similarities of the predicted category labels, the relationship between feature nodes belonging to the same category can be strengthened. In addition, this aggregation can make the node features context-aware and obtain more accurate feature representation. Similar to SLG, specifically, the present invention constructs an undirected graph from all unlabeled pixels in a mini-batch in represents the set of vertices consisting of the label predictions output by the segmentation head. (C) Represents a collection of edges used to measure the affinity between predicted nodes. is a confidence matrix, each row of which corresponds to a predicted probability vector. The present invention uses another affinity matrix To model the relationship between label confidence scores:

[0157]

[0158] in, is the predicted probability vector corresponding to pixel i. Pixels with pseudo labels that do not belong to the same class are not connected, and each pixel is connected to itself with a weight of 1. Then the obtained similarity matrix is ​​normalized:

[0159]

[0160] in yes The diagonal matrix is

[0161] (4) The mutual evolution process of the multi-level graph of the present invention is as follows. In each semi-supervised iteration, the present invention needs to perform K graph convolution operations on the constructed graph. In the convolution operation of the kth graph, the present invention first updates the label-level graph node The propagation rules are as follows:

[0162]

[0163] in is a The parameters form a nonlinear function, and [·,·] is a joint function. The area with low prediction confidence is more likely to be noisy, so we should pay more attention to the low confidence area. According to FlexMatch, the present invention uses a class-specific dynamic threshold to separate high confidence labels from low confidence labels. The dynamic threshold of a specific class is expressed as follows:

[0164]

[0165] Where σ is a fixed threshold, represents the learning effect of the current category c in the current small batch t, ​​p i is the prediction vector output by the original segmentation head. is a function that returns 1 when its input is true and 0 otherwise. Based on the dynamic threshold, we update the label-level graph nodes The update rules are as follows:

[0166]

[0167] Where α is a hyperparameter that controls the proportion of nodes before and after label propagation. After updating the nodes, the present invention constructs the formula matrix to update the edge connection W through the affinity matrix of the category-level graph (k) .

[0168] Label level graph After updating, use Reconstructing semantic level graphs from information in Node. By W (k)Aggregating the correct feature neighbors, the update process is as follows:

[0169]

[0170] in is a After updating the node, the edge connection A is updated according to the affinity matrix of the semantic level graph. (k) .

[0171] (5) The present invention needs to train the semantic level graph branch and the category level graph branch separately. In order to train the semantic level graph, the present invention proposes a contrastive learning scheme combining intra-image and inter-image comparison. The loss formula is as follows:

[0172]

[0173] Where λ is a weight factor that controls the balance between the two loss terms. if and Have the same tags, is 1, otherwise it is 0. Represents the features in the k-th SLG. The waves on the vector represent the strongly data augmented version. Represents a prototype. It is the pseudo label of the weakly enhanced image at pixel i predicted by the network, which determines the positive prototype τ is a temperature, we usually set it to 0.1. The first term of the semantic-level graph loss works in a single image, intuitively, it forces the network to learn highly confident semantic affinities. The second term evaluates the difference between node features and prototypes to encourage features to move to the correct cluster center. It combines similar features of all data and therefore has knowledge across images. We use feature averaging to calculate the prototype, which can be used as the feature center of each class. The prototype is calculated as follows:

[0174]

[0175] in is the feature in the kth SLG, y i is the true label of the labeled image at position i. However, when a few examples are used to represent the category center, deviations will occur. Therefore, for unlabeled images, the present invention uses pseudo labels Instead of yi, the prototype is calculated. In order to solve the problem that the prototype representation may be unbalanced when using class imbalanced data during training, the present invention estimates the prototype as an exponential moving average of small batch clusters, so that the prototype can be slowly moved during the training process until the correct cluster center. Specifically, in each iteration, the prototype is estimated as follows:

[0176]

[0177] where β is a hyperparameter that controls the prototype update speed, and we set it to 0.99.

[0178] In order to train the category-level graph, this paper proposes a dynamic weighted cross entropy loss to further reduce the influence of noise labels. The loss formula is as follows:

[0179]

[0180] in is the pseudo label generated by weakly enhanced data, and Pk is the kth CLG confidence matrix, that is, Pseudo Labels It is obtained by aggregating the confidence matrices of K CLGs to obtain a more accurate output. The dynamic weight is expressed as follows:

[0181]

[0182] The present invention uses the confidence of the strongly enhanced data as the dynamic weight instead of the confidence of the weakly enhanced data.

[0183] In summary, after combining SLG loss and CLG loss, the overall loss of unlabeled data in each mini-batch is as follows:

[0184]

[0185] Among them, λ SLG is the weight of SLG loss, λ CLG is the weight of CLG loss. The present invention uses loss for each graph convolution layer to accelerate training.

[0186] (6) Use gradient descent to update the student network parameters, and use exponential moving average to update the teacher network parameters. The gradient descent formula is:

[0187]

[0188] in β is the learning rate. The teacher network update formula is as follows:

[0189] ξ(t+1)←αξ(t)+(1-α)θ(t+1)

[0190] Where α is a fixed hyperparameter. Repeat steps (2) to (4) until convergence.

[0191] In this embodiment, the reliability of the semi-supervised semantic segmentation method based on multi-level label correction is verified by using known network data, and is compared with the existing semi-supervised semantic segmentation method, as shown below.

[0192] (1) Comparison with the baseline using only labeled data

[0193] In this example, DeepLabv2 is used to build the teacher-student network. They have the same structure. All experiments in this section use the Stochastic Gradient Descent (SGD) optimizer and use the polynomial decay strategy. Completed. Cityscapes and PASAL VOC 2012 are trained for 100 and 150 epochs, with batch sizes of 2 and 10, respectively. Unlabeled loss balance coefficient λ SLG and λ CLG are set to 0.1 and 1 respectively. According to FlexMatch, we set δ to 0.95. The label propagation control hyperparameter α is set to 0.8. The balance parameter λ of the SLG contrast loss is set to 0.5. The number of cycles K of the multi-level graph is set to 2. It should be noted that the above parameters are mainly exemplary and do not mean that the present invention can only be implemented using the above parameters; for example, based on the same technical concept, those skilled in the art may fine-tune the above parameters (for example, ±10%, or +1, -1), which will affect the training effect, but will not have a fundamental change on the training effect.

[0194] The numbers in the brackets in the last row of Tables 1 and 2 below are the comparison results of the semi-supervised semantic segmentation algorithm described in this example and the baseline using only labeled supervision on two different datasets and different labeled data divisions. The Cityscapes and PASCAL VOC 2012 datasets have 2975 and 10582 images respectively. The numbers in brackets in the table represent the number of labeled data, the tabular data represents the semantic segmentation evaluation index (mloU), and the brackets thereafter represent the improvement value compared to the baseline. It can be seen from Tables 1 and 2 that the method described in this embodiment surpasses the baseline using only labeled supervision on two different datasets.

[0195] Figure 3 The display effect of this embodiment compared with the baseline on PASCAL VOC 2012 is shown, and it can be found that the accuracy of semantic segmentation of this embodiment is closer to the ground truth.

[0196] Figure 4 The comparison results of the training method of this embodiment and the training process of the supervised baseline are shown. It can be found that under the same number of iterations, the evaluation index of the training method provided by this embodiment is significantly higher than that of the supervised baseline, which means that the training method provided by the present invention significantly improves the training efficiency of the semantic segmentation model.

[0197] (2) Comparison with the fully supervised baseline

[0198] In Tables 1 and 2 below, FS is represented as the fully supervised baseline, which can show the comparison results of the semi-supervised semantic segmentation algorithm described in this example and the fully supervised baseline under different labeled data divisions on two different datasets. The fully supervised baseline represents the results that can be achieved by treating all data as labeled data. As can be seen from Tables 1 and 2, the method described in this embodiment can achieve the same effect as the fully supervised method using only a small portion of labeled data on two different datasets.

[0199] (3) Comparison with existing semi-supervised semantic segmentation methods

[0200] Tables 1 and 2 below show the comparison results of the semi-supervised semantic segmentation algorithm described in this example and the existing semi-supervised semantic segmentation method on two different data sets and different labeled data divisions. Among them, MLC is the method described in this embodiment. It can be seen from Tables 1 and 2 that the recognition method described in this embodiment effectively improves the performance of the semi-supervised segmentation model.

[0201] Table 1

[0202]

[0203] Table 2

[0204]

[0205]

[0206] In addition, for the impact of the number of graph convolutions in one iteration on the training results, see Figure 5 As shown in the figure, the evaluation index of graph convolution increases significantly compared with no graph convolution (number of times is 0). The optimal number of graph convolutions is 2 times, but based on experimental results, 1-3 times of graph convolution can achieve significant improvement compared with no graph convolution. At the same time, this does not mean that the number of graph convolutions is limited to this range. This is only the optimal range for this embodiment. When some basic conditions and specific settings change, the optimal number of graph convolutions can also be appropriately adjusted based on experimental results.

[0207] This embodiment also provides an electronic device, including: one or more processors; and a memory, wherein the memory is used to store one or more executable instructions; the one or more processors are configured to perform the steps of the identification method described in this embodiment by executing the one or more executable instructions.

[0208] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0209] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium on which are loaded computer-readable program instructions for causing a processor to implement various aspects of the present invention. The computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0210] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-supervised semantic segmentation model training method based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, characterized in that: include: Construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the student network has the same structure as the teacher network, the labeled data set includes labeled data and its corresponding labels, and the unlabeled data set includes unlabeled data; Using a labeled data set to perform labeled training on the student network; Generate pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network; Using a semantic level graph to construct a semantic affinity matrix of pixel feature pairs in the intermediate features, using a category level graph to construct a classification consistency matrix in the pseudo labels, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively; Interchanging structural information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels; The updated intermediate features and the corrected pseudo-labels are used to update the parameters of the student network and the teacher network, and unlabeled training is performed; and the labeled training and unlabeled training processes are iterated until the end condition is reached to obtain a semantic segmentation model.

2. The semi-supervised semantic segmentation model training method according to claim 1, characterized in that: The training of the student network is performed by cross entropy training; The labeled loss function for training the student network is expressed as: in Indicates label loss, X l Represents the labeled data, Y l represents a hot vector obtained by the label transformation, f θ Denotes that the parameter corresponding to the student network is θ, and H represents the cross entropy loss function, which is defined as: Where i, j represents the pixel index, c represents the category index, and Y l represents a one-hot vector transformed by the true label, and P represents the prediction vector of the student network; The teacher network is input after weak image enhancement, and the student network is input after strong image enhancement.

3. The semi-supervised semantic segmentation model training method according to claim 1, characterized in that: The process of constructing the semantic level graph specifically includes: Retain a first number of nearest neighbor pixels of any pixel, and construct a first undirected graph of all pixels in the unlabeled data of the same batch using a first semantic affinity matrix; Normalizing the first semantic affinity matrix to obtain a first normalized matrix; The process of constructing the category-level graph specifically includes: Using a second semantic affinity matrix to represent the relationship between label confidence scores of all pixels in the unlabeled data of the same batch, constructing a second undirected graph; The second semantic affinity matrix is ​​normalized to obtain a second normalized matrix.

4. The semi-supervised semantic segmentation model training method according to claim 3, characterized in that: The construction process of the semantic level graph is expressed as: in, represents the first semantic affinity matrix, represents the feature vector corresponding to pixel i, represents the feature vector corresponding to pixel j, γ represents the scaling parameter, express The set of k nearest neighbors of Wherein, A represents the first normalized matrix, express The angle matrix is The first undirected graph in, represents the vertex set consisting of generated features; ε (S) represents a set of edges, measuring the affinity between pixel-level feature pairs, corresponding to the first normalized matrix; is the feature matrix; The construction process of the category-level graph is expressed as: in, represents the second semantic affinity matrix, represents the predicted probability vector corresponding to pixel i, represents the predicted probability vector corresponding to pixel j, express The corresponding maximum probability category; Wherein, W represents the second normalized matrix, express The diagonal matrix of The second undirected graph in represents the set of vertices consisting of the output label predictions; ε (C) represents a set of edges, used to measure the affinity between prediction nodes, corresponding to the second normalized matrix; is the confidence matrix, each row corresponds to the predicted probability vector.

5. The semi-supervised semantic segmentation model training method according to claim 4, characterized in that: The gradual correction process of the semantic level graph and the category level graph specifically includes: In one iteration, the semantic level graph is subjected to a first graph convolution operation several times, and the category level graph is subjected to a second graph convolution operation several times; In the kth convolution operation of the second graph, the first node of the category-level graph is updated, and the propagation rule is expressed as: in, represents the first node, represents a parameter of the first graph convolution operation Composed of nonlinear functions, [·,·] is a joint function, represents the predicted probability vector corresponding to pixel i in the k-1th second graph convolution operation, represents the first normalized matrix in the k-th second graph convolution operation, Represents the predicted probability vector corresponding to pixel j in the k-1th second graph convolution operation; In the kth convolution operation of the first graph, the second node of the semantic level graph is updated, and the update rule is expressed as: in, represents the second node in the kth first graph convolution operation, represents the second node in the k-1th convolution operation of the first graph, α is a hyperparameter that controls the proportion of nodes before and after label propagation, and η (t,c) represents the dynamic threshold, σ is the fixed threshold, represents the learning effect of the current category c in the current small batch t, ​​p i is the prediction vector output by the original segmentation head, is a function that returns 1 if its input is true, and 0 otherwise; After updating the second node, the edge A of the semantic level graph is updated by constructing a formula using the first semantic affinity matrix of the semantic level graph. (k) And the edge W of the category-level graph is updated by constructing a formula using the second semantic affinity matrix of the category-level graph (k) .

6. The semi-supervised semantic segmentation model training method according to claim 5, characterized in that: The loss function of the unlabeled training includes a semantic level graph loss that reflects the similarity of the intermediate features provided by the teacher network and the student network, and a category level graph loss that reflects the similarity of the provided pseudo labels; The calculation method of the semantic level graph loss is expressed as: in, represents the semantic level graph loss, λ represents the weight factor, if and Have the same tags, is 1, otherwise it is 0; represents a strongly data-enhanced version of the feature corresponding to pixel i in the k-th semantic level map, Represents a strongly data-enhanced version of the feature corresponding to pixel j in the k-th semantic level map; Represents a collection of prototypes, where Represents a prototype; It is the pseudo label of the weakly enhanced image at pixel i predicted by the network, which determines the positive prototype τ represents the temperature constant; The calculation method of the category-level graph loss is expressed as: in, represents the category-level graph loss, is a pseudo label generated by weakly enhanced data, is the confidence matrix of the k-th category-level graph, i.e. Pseudo Labels It is obtained by aggregating the confidence matrices of K category-level graphs. Represents dynamic weight, and the calculation method is expressed as:

7. The semi-supervised semantic segmentation model training method according to claim 6, characterized in that: The prototype is calculated using feature averaging, and the resulting prototype is used as the feature center of each class; The computation of the prototype is expressed as: in is the feature in the k-th semantic level graph, is the pseudo label of the unlabeled data at position i; The prototype is estimated as the exponential moving average of the mini-batch clustering. Specifically, in each round of iteration, the estimate of the prototype is expressed as: where β is a hyperparameter that controls the speed at which the prototype is updated.

8. The semi-supervised semantic segmentation model training method according to claim 7, characterized in that: The unlabeled loss function of the unlabeled training is the weighted addition of the semantic level graph loss and the category level graph loss; The total loss function of iterative training is the sum of the unlabeled loss function and the labeled loss function; In multiple rounds of iterations, the student network is updated by gradient descent, and the parameters of the teacher network come from the exponential moving average of the corresponding parameters of the student network; The gradient descent process is expressed as: The parameter calculation method of the teacher network is expressed as: ξ(t+1)←αξ(t)+(1-α)θ(t+1) in, represents the total loss function, β is the learning rate, ξ represents the parameters of the teacher network, θ represents the parameters of the student network, α represents the hyperparameter, and 0<α<1.

9. A method for image semantic segmentation, characterized in that: include: Providing a semantic segmentation model obtained by the semi-supervised semantic segmentation model training method according to any one of claims 1 to 8; The semantic segmentation model is used to perform semantic segmentation on the image to be segmented.

10. A semi-supervised semantic segmentation model training system based on multi-level label correction, wherein the semantic segmentation model is used to implement image semantic segmentation, characterized in that: include: A basic module, used to construct a student network and a teacher network, and obtain a labeled data set and an unlabeled data set; the structure of the student network is the same as that of the teacher network, the labeled data set contains labeled data and its corresponding labels, and the unlabeled data set contains unlabeled data; A labeled training module, used for performing labeled training on the student network using a labeled data set; A module for generating pseudo labels and intermediate features for the unlabeled dataset using the student network and the teacher network; A matrix construction module, for constructing a semantic affinity matrix of pixel feature pairs in the intermediate features using a semantic level graph, constructing a classification consistency matrix in the pseudo labels using a category level graph, and extracting corresponding structural information from the semantic affinity matrix and the classification consistency matrix respectively; A structure exchange module, used for exchanging the structure information corresponding to the semantic level graph and the category level graph, so as to gradually correct the semantic level graph and the category level graph, update the intermediate features and correct the pseudo labels; The iterative update module is used to update the parameters of the student network and the teacher network using the updated intermediate features and the corrected pseudo labels to perform unlabeled training; and iteratively perform the process of labeled training and unlabeled training until the end condition is reached to obtain a semantic segmentation model.

11. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the semi-supervised semantic segmentation model training method described in any one of claims 1 to 8 are executed.

Images