Image semantic segmentation method and device

Abstract

The application discloses an image semantic segmentation method and device, and the method comprises the following steps: acquiring a coarse-labeled image set; inputting the coarse-labeled image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarse-labeled image set; inputting the coarse-labeled image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unlabeled pixel in the coarse-labeled image set and a first loss function value corresponding to each pixel in the coarse-labeled image set; determining a second loss function value corresponding to each pixel in the coarse-labeled image set according to a reweighted weight matrix corresponding to each coarse-labeled image in the coarse-labeled image set and the first loss function value corresponding to each pixel in the coarse-labeled image set; and inputting an image to be predicted into a third semantic segmentation model to obtain a semantic segmentation result corresponding to the image to be predicted. The application suppresses the influence of false labeling on the third semantic segmentation model, and improves the efficiency and accuracy of image semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image semantic segmentation method and device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Existing semantic segmentation technologies mostly focus on either fine or weak labeling. While numerous technical achievements have demonstrated promising performance for fine labeling, the cost of labeling is prohibitive for large-scale, complex scenarios like device anomaly detection, making it difficult to scale. Limited by the inherent constraints of deep learning methods, if the dataset is too small, the model's generalization cannot be guaranteed, while the large number of labels reduces the efficiency of image semantic segmentation. While weak supervision can address some of these issues through very inexpensive labeling, the model's prediction accuracy is low and it is fundamentally unsuitable for complex tasks involving irregular, multi-category objects. Summary of the Invention

[0004] An embodiment of the present invention provides a method for image semantic segmentation to improve the efficiency and accuracy of image semantic segmentation. The method includes:

[0005] Get a coarsely annotated image set;

[0006] Inputting the coarsely annotated image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; wherein the first semantic segmentation model is obtained by training a machine learning model using the coarsely annotated image set;

[0007] Training the first semantic segmentation model using the coarsely annotated image set and the first predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set to obtain a second semantic segmentation model;

[0008] Inputting the coarsely annotated image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set;

[0009] Determine a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set according to the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set;

[0010] Determine a second loss function value corresponding to each pixel in the coarsely annotated image set according to a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set;

[0011] The second semantic segmentation model is trained using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model;

[0012] Obtain the image to be predicted;

[0013] The image to be predicted is input into the third semantic segmentation model to obtain a semantic segmentation result corresponding to the image to be predicted.

[0014] An embodiment of the present invention further provides a coarsely annotated image semantic segmentation device for improving the efficiency and accuracy of image semantic segmentation, the device comprising:

[0015] A training module is used to obtain a coarsely annotated image set; input the coarsely annotated image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; wherein the first semantic segmentation model is obtained by training a machine learning model using the coarsely annotated image set; the first semantic segmentation model is trained using the coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain a second semantic segmentation model; input the coarsely annotated image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; determine a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set based on the coarsely annotated image set and the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; determine a second loss function value corresponding to each pixel in the coarsely annotated image set based on the reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set; train the second semantic segmentation model using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model;

[0016] The recognition module is used to obtain the image to be predicted; input the image to be predicted into the third semantic segmentation model to obtain the semantic segmentation result corresponding to the image to be predicted.

[0017] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image semantic segmentation method when executing the computer program.

[0018] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned image semantic segmentation method is implemented.

[0019] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned image semantic segmentation method.

[0020] In an embodiment of the present invention, a coarsely annotated image set is obtained; the coarsely annotated image set is input into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; the first semantic segmentation model is trained using the coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain a second semantic segmentation model; the coarsely annotated image set is input into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set is determined based on the coarsely annotated image set and the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set are determined based on the reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set. The method determines the second loss function value corresponding to each pixel in the coarsely annotated image set; trains the second semantic segmentation model using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model; obtains the image to be predicted; inputs the image to be predicted into the third semantic segmentation model to obtain the semantic segmentation result corresponding to the image to be predicted. Compared with the prior art, the method uses the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to train the second semantic segmentation model to obtain the third semantic segmentation model, and weights the first loss function value corresponding to each pixel in the coarsely annotated image set by a reweighted weight matrix, thereby suppressing the influence of incorrect labeling on the third semantic segmentation model, and using self-supervision to avoid the disadvantage of unsupervised unlabeled pixels, thereby improving the performance of the third semantic segmentation model and improving the efficiency and accuracy of image semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A flow chart of the image semantic segmentation method provided by the present invention;

[0023] Figure 2 A flow chart of the image semantic segmentation method provided by the present invention;

[0024] Figure 3 A flow chart of the image semantic segmentation method provided by the present invention;

[0025] Figure 4 A flow chart of the image semantic segmentation method provided by the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of the image semantic segmentation device provided by the present invention. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] Figure 1 A flowchart of an image semantic segmentation method according to an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0029] Step 101: Obtain a coarsely annotated image set.

[0030] Fine labeling requires assigning a representative category label to every pixel in an image, while coarse labeling only requires roughly outlining the polygonal outline of the object. Unlike weak labeling, which uses image-level annotation or scribble annotation, polygonal outlines minimize labeling costs while preserving the original shape of the object, making them more suitable for semantic segmentation tasks involving complex scenes and objects.

[0031] Step 102: Input the coarsely annotated image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set.

[0032] It should be noted that the first semantic segmentation model is obtained by training the machine learning model using a coarsely annotated image set.

[0033] In one possible implementation, the machine learning model uses DeepLabv3+.

[0034] Step 103 : Training the first semantic segmentation model using the coarsely annotated image set and the first predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set to obtain a second semantic segmentation model.

[0035] Step 104 : Input the coarsely annotated image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set.

[0036] In one possible implementation, the semantic segmentation model uses cross entropy as the loss function.

[0037] Step 105 : determining a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set according to the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set.

[0038] Step 106 : determining a second loss function value corresponding to each pixel in the coarsely annotated image set according to the reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set.

[0039] In the embodiment of the present invention, in order to repair the erroneous supervisory signal, the first loss function value corresponding to each pixel in the coarsely annotated image set is weighted by a reweighted weight matrix.

[0040] Step 107 : Training the second semantic segmentation model using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model.

[0041] In the embodiment of the present invention, the second loss function value is back-propagated, and after multiple iterative training, a third semantic segmentation model is obtained.

[0042] Step 108: Obtain the image to be predicted.

[0043] Step 109: input the image to be predicted into the third semantic segmentation model to obtain a semantic segmentation result corresponding to the image to be predicted.

[0044] In an embodiment of the present invention, a first semantic segmentation model is trained using a coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain a second semantic segmentation model. Unlabeled pixels are processed using a self-supervised approach and gradient values ​​are backpropagated. Specifically, the first predicted semantic category predicted by the neural network at the unlabeled pixels is used as a pseudo-label, which is then used for supervision.

[0045] The above scheme weights the first loss function value corresponding to each pixel in the coarsely annotated image set through a reweighted weight matrix, thereby suppressing the impact of incorrect labeling on the third semantic segmentation model. The use of self-supervision avoids the disadvantage of unlabeled pixels having no supervision, improves the performance of the third semantic segmentation model, and improves the efficiency and accuracy of image semantic segmentation.

[0046] In step 105 of the embodiment of the present invention, a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set is determined based on the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set. The steps are as follows: Figure 2 As shown, the details are as follows:

[0047] Step 201 : constructing a plurality of undirected graphs according to the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set.

[0048] It should be noted that each undirected graph in the multiple undirected graphs corresponds one-to-one to each coarsely annotated image in the coarsely annotated image set; the vertices of the undirected graphs correspond one-to-one to each pixel in the coarsely annotated image; and the edges of the undirected graphs are used to indicate the positional relationship between each pixel in the coarsely annotated image.

[0049] Step 202 : determining a confidence matrix corresponding to each coarsely annotated image according to the undirected graph corresponding to each coarsely annotated image.

[0050] It should be noted that each element in the confidence matrix corresponds one-to-one to each pixel in the coarsely annotated image, and is used to indicate the credibility of the semantic category corresponding to each annotated pixel in the coarsely annotated image and the credibility of the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image.

[0051] Step 203 : determining a reweighted weight matrix corresponding to each coarsely annotated image according to the confidence matrix corresponding to each coarsely annotated image.

[0052] In this embodiment of the present invention, the output of the second semantic segmentation model is first defined as:

[0053] logits(A i，1：C )=model(X i )

[0054] Where: C is the number of categories, A i,1:C Represents the feature vector of pixel i, X i Represents a vector of RGB values ​​of pixel i in the coarsely labeled image.

[0055] In order to better calculate the formal relationship between adjacent pixels in the image, the embodiment of the present invention converts logits into an undirected graph And explore the spatial feature relationship based on the graph. More formally, there is a graph

[0056]

[0057] Where: Is the set of all vertices. Each feature point i in logits is converted into a vertex in the graph, thus forming a vertex set on the graph It is the set of eigenvectors belonging to the vertex, which is each eigenvector A on the original logits i,1:C A collection of . is the set of all edges. is the adjacency matrix, which indicates the existence of edges between vertices.

[0058] Next, we construct edges on the undirected graph. Because spatially adjacent pixels in the coarsely labeled image are correlated, we establish edges between locally adjacent pixels. That is, in the constructed undirected graph, if the pixel represented by vertex u is spatially adjacent to the pixel represented by vertex v—that is, they are adjacent in all four directions: up, down, left, and right—then an undirected edge will be established between vertices u and v.

[0059] is the set of all edges. If there is an undirected edge between vertices u and v, it can be expressed as

[0060]

[0061] For vertices u and v, there exists

[0062]

[0063] The above scheme constructs an undirected graph to formally represent the relationship between each pixel in logits.

[0064] In a possible implementation, when converting logits into an undirected graph, four adjacent pixels around the current pixel are used to build edges, namely, in the four directions of up, down, left, and right.

[0065] Furthermore, based on the undirected graph, the corresponding confidence matrix is ​​calculated. The steps are as follows: Figure 3 As shown, the details are as follows:

[0066] Step 301: Calculate the correlation coefficient between the current vertex and the adjacent vertices.

[0067] In this embodiment of the present invention, the maximum element at position i of the current vertex's eigenvector and the maximum element at position j of the adjacent vertex's eigenvector are first calculated. The elements at positions i and j of the current vertex and the adjacent vertex are then extracted to form new vectors of length 2. The dot product of the two vectors is used as the correlation coefficient between the current vertex and the adjacent vertex.

[0068] Step 302: Calculate the dot product between the current vertex and all adjacent vertices, take the average and normalize to obtain the confidence.

[0069] The above scheme calculates the corresponding confidence matrix based on an undirected graph, thereby improving the efficiency of image semantic segmentation.

[0070] In this embodiment of the present invention, the reweighting weight is inversely proportional to the confidence level. For high-confidence samples, regardless of whether the labeling is correct, the loss function value should be suppressed. For low-confidence samples, regardless of whether the labeling is correct, the loss function value should be increased. Therefore, the closer the relationship between a vertex and its adjacent vertices, the larger its correlation coefficient, and the greater the confidence level, the smaller the reweighting weight should be. Otherwise, the opposite relationship will be observed.

[0071] The specific calculation formula of the reweighted weight matrix is ​​as follows:

[0072]

[0073] Where Φ represents the reweighted weight matrix, represents the confidence matrix.

[0074] In the embodiment of the present invention, before the coarsely annotated image set is input into the first semantic segmentation model, the steps are as follows: Figure 4 As shown, the details are as follows:

[0075] Step 401: Use the coarsely annotated image set as sample data to construct a training set and a test set.

[0076] Step 402: Use the training set to train the machine learning model to obtain a first semantic segmentation model.

[0077] Step 403: Test the first semantic segmentation model using the test set.

[0078] The above scheme uses the coarsely annotated image set as sample data to construct a training set and a test set, and uses the training set to train the machine learning model to obtain the first semantic segmentation model, thereby improving the effectiveness of the first semantic segmentation model.

[0079] The present invention also provides an image semantic segmentation device, as described in the following embodiments. Figure 5 As shown, the device includes:

[0080] The training module 501 is used to obtain a coarsely annotated image set; input the coarsely annotated image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; wherein the first semantic segmentation model is obtained by training a machine learning model using the coarsely annotated image set; the first semantic segmentation model is trained using the coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain a second semantic segmentation model; input the coarsely annotated image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; determine a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set based on the coarsely annotated image set and the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; determine a second loss function value corresponding to each pixel in the coarsely annotated image set based on the reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set; train the second semantic segmentation model using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model;

[0081] The recognition module 502 is used to obtain the image to be predicted; input the image to be predicted into the third semantic segmentation model to obtain the semantic segmentation result corresponding to the image to be predicted.

[0082] In the embodiment of the present invention, the training module 501 is specifically used to:

[0083] Constructing a plurality of undirected graphs based on the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set; each undirected graph in the plurality of undirected graphs corresponds one-to-one to each coarsely annotated image in the coarsely annotated image set; a vertex of the undirected graph corresponds one-to-one to each pixel in the coarsely annotated image; and an edge of the undirected graph is used to indicate a positional relationship between each pixel in the coarsely annotated image;

[0084] Determining a confidence matrix corresponding to each coarsely annotated image based on an undirected graph corresponding to each coarsely annotated image; each element in the confidence matrix corresponds one-to-one to each pixel in the coarsely annotated image, and is used to indicate the credibility of the semantic category corresponding to each annotated pixel in the coarsely annotated image and the credibility of the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image;

[0085] The re-weighted weight matrix corresponding to each coarsely annotated image is determined according to the confidence matrix corresponding to each coarsely annotated image.

[0086] In the embodiment of the present invention, the training module 501 is specifically used to:

[0087] The specific calculation formula of the reweighted weight matrix is ​​as follows:

[0088]

[0089] Where Φ represents the reweighted weight matrix, represents the confidence matrix.

[0090] In the embodiment of the present invention, the training module 501 is further configured to:

[0091] Before inputting the coarsely annotated image set into the first semantic segmentation model, the coarsely annotated image set is used as sample data to construct a training set and a test set;

[0092] Training the machine learning model using the training set to obtain the first semantic segmentation model;

[0093] The first semantic segmentation model is tested using a test set.

[0094] Since the principle of solving the problem by the device is similar to that of the image semantic segmentation method, the implementation of the device can refer to the implementation of the image semantic segmentation method, and the repeated parts will not be repeated.

[0095] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image semantic segmentation method when executing the computer program.

[0096] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned image semantic segmentation method is implemented.

[0097] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned image semantic segmentation method.

[0098] In an embodiment of the present invention, a coarsely annotated image set is obtained; the coarsely annotated image set is input into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; the first semantic segmentation model is trained using the coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain a second semantic segmentation model; the coarsely annotated image set is input into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set is determined based on the coarsely annotated image set and the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set are determined based on the reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set. The method determines the second loss function value corresponding to each pixel in the coarsely annotated image set; trains the second semantic segmentation model using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model; obtains the image to be predicted; inputs the image to be predicted into the third semantic segmentation model to obtain the semantic segmentation result corresponding to the image to be predicted. Compared with the prior art, the method uses the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to train the second semantic segmentation model to obtain the third semantic segmentation model, and weights the first loss function value corresponding to each pixel in the coarsely annotated image set by a reweighted weight matrix, thereby suppressing the influence of incorrect labeling on the third semantic segmentation model, and using self-supervision to avoid the disadvantage of unsupervised unlabeled pixels, thereby improving the performance of the third semantic segmentation model and improving the efficiency and accuracy of image semantic segmentation.

[0099] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0103] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image semantic segmentation, characterized in that: include: Get a coarsely annotated image set; Inputting the coarsely annotated image set into a first semantic segmentation model to obtain a first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; wherein the first semantic segmentation model is obtained by training a machine learning model using the coarsely annotated image set; Training the first semantic segmentation model using the coarsely annotated image set and the first predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set to obtain a second semantic segmentation model; Inputting the coarsely annotated image set into a second semantic segmentation model to obtain a second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; Determine a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set according to the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set; Determine a second loss function value corresponding to each pixel in the coarsely annotated image set according to a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set and a first loss function value corresponding to each pixel in the coarsely annotated image set; The second semantic segmentation model is trained using the coarsely annotated image set and the second loss function value corresponding to each pixel in the coarsely annotated image set to obtain a third semantic segmentation model; Obtain the image to be predicted; Inputting the image to be predicted into the third semantic segmentation model to obtain a semantic segmentation result corresponding to the image to be predicted; The method of determining a reweighted weight matrix corresponding to each coarsely annotated image in the coarsely annotated image set according to the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set comprises: Constructing a plurality of undirected graphs based on the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set; each undirected graph in the plurality of undirected graphs corresponds one-to-one to each coarsely annotated image in the coarsely annotated image set; a vertex of the undirected graph corresponds one-to-one to each pixel in the coarsely annotated image; and an edge of the undirected graph is used to indicate a positional relationship between each pixel in the coarsely annotated image; Determining a confidence matrix corresponding to each coarsely annotated image based on an undirected graph corresponding to each coarsely annotated image; each element in the confidence matrix corresponds one-to-one to each pixel in the coarsely annotated image, and is used to indicate the credibility of the semantic category corresponding to each annotated pixel in the coarsely annotated image and the credibility of the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image; Determine a reweighted weight matrix corresponding to each coarsely annotated image according to the confidence matrix corresponding to each coarsely annotated image; Determining the confidence matrix corresponding to each coarsely annotated image according to the undirected graph corresponding to each coarsely annotated image includes: Calculate the correlation coefficient between the current vertex and the adjacent vertex; Calculate the dot product between the current vertex and all adjacent vertices, take the average and normalize to get the confidence score; The specific calculation formula of the reweighted weight matrix is ​​as follows: Φ=1-C 2 Where Φ represents the reweighted weight matrix and C represents the confidence matrix.

2. The image semantic segmentation method according to claim 1, wherein: Before inputting the coarsely annotated image set into the first semantic segmentation model, the method further includes: Use the coarsely annotated image set as sample data to construct the training set and test set; Training the machine learning model using the training set to obtain the first semantic segmentation model; The first semantic segmentation model is tested using a test set.

3. An image semantic segmentation device, characterized in that: include: A training module, used to obtain a set of coarsely annotated images; The coarsely annotated image set is input into the first semantic segmentation model to obtain the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; wherein the first semantic segmentation model is obtained by training the machine learning model using the coarsely annotated image set; the first semantic segmentation model is trained using the coarsely annotated image set and the first predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set to obtain the second semantic segmentation model; the coarsely annotated image set is input into the second semantic segmentation model to obtain the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set and the first loss function value corresponding to each pixel in the coarsely annotated image set; the reweighted weight matrix corresponding to each coarse annotated image in the coarsely annotated image set is determined according to the coarsely annotated image set and the second predicted semantic category corresponding to each unlabeled pixel in the coarsely annotated image set; the second loss function value corresponding to each pixel in the coarsely annotated image set is determined according to the reweighted weight matrix corresponding to each coarse annotated image in the coarse annotated image set and the first loss function value corresponding to each pixel in the coarse annotated image set; the second semantic segmentation model is trained using the coarse annotated image set and the second loss function value corresponding to each pixel in the coarse annotated image set to obtain the third semantic segmentation model; The recognition module is used to obtain the image to be predicted; input the image to be predicted into the third semantic segmentation model to obtain the semantic segmentation result corresponding to the image to be predicted; The training module is specifically used for: Constructing a plurality of undirected graphs based on the coarsely annotated image set and the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image set; each undirected graph in the plurality of undirected graphs corresponds one-to-one to each coarsely annotated image in the coarsely annotated image set; a vertex of the undirected graph corresponds one-to-one to each pixel in the coarsely annotated image; and an edge of the undirected graph is used to indicate a positional relationship between each pixel in the coarsely annotated image; Determining a confidence matrix corresponding to each coarsely annotated image based on an undirected graph corresponding to each coarsely annotated image; each element in the confidence matrix corresponds one-to-one to each pixel in the coarsely annotated image, and is used to indicate the credibility of the semantic category corresponding to each annotated pixel in the coarsely annotated image and the credibility of the second predicted semantic category corresponding to each unannotated pixel in the coarsely annotated image; Determine a reweighted weight matrix corresponding to each coarsely annotated image according to the confidence matrix corresponding to each coarsely annotated image; The training module is specifically used for: Calculate the correlation coefficient between the current vertex and the adjacent vertex; Calculate the dot product between the current vertex and all adjacent vertices, take the average and normalize to get the confidence score; The training module is specifically used for: The specific calculation formula of the reweighted weight matrix is ​​as follows: Φ=1-C 2 Where Φ represents the reweighted weight matrix and C represents the confidence matrix.

4. The image semantic segmentation device according to claim 3, wherein: The training module is also used to: Before inputting the coarsely annotated image set into the first semantic segmentation model, the coarsely annotated image set is used as sample data to construct a training set and a test set; Training the machine learning model using the training set to obtain the first semantic segmentation model; The first semantic segmentation model is tested using a test set.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images