Image semantic segmentation model training, segmentation method, device and storage medium

Abstract

The present disclosure provides an image semantic segmentation model training, an image semantic segmentation method, an apparatus, and a storage medium, and relates to the field of computer technology, wherein the method includes: using a discriminator model to perform judgment processing on a semantic segmentation image generated by a semantic segmentation model; constructing a loss function corresponding to the discriminator model, including: a target domain loss function generated based on a target domain image; the target domain loss function includes at least one of a first semantic loss function generated based on the semantic consistency of an image block, a second semantic loss function generated based on the semantic consistency of a cluster, and a third semantic loss function generated based on a logical construction of an image space. The method, apparatus, and storage medium disclosed herein constrain the semantic segmentation model inference results of the model on the target domain image in the form of a regularization term during the training process, perform cross-domain migration on the image semantic segmentation model, and improve the efficiency and accuracy of image semantic segmentation model training.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image semantic segmentation model training, an image semantic segmentation method, a device and a storage medium. Background Art

[0002] The goal of image semantic segmentation is to identify the semantic category of each pixel in the image. A deep semantic segmentation model can be built using a neural network. The training of a deep semantic segmentation model requires a large amount of training data with pixel-level annotations. However, obtaining this training data is very expensive and slow. Currently, computer-generated images are used in model training. However, there is a large difference between computer-generated images and real images. This difference causes the semantic segmentation model trained with synthetic images to perform poorly on real images, which is a phenomenon called "domain mismatch". Summary of the invention

[0003] In view of this, a technical problem to be solved by the present invention is to provide an image semantic segmentation model training, an image semantic segmentation method, an apparatus and a storage medium.

[0004] According to one aspect of the present disclosure, a method for training an image semantic segmentation model is provided, comprising: constructing an image semantic segmentation model for semantically segmenting an image; wherein the image semantic segmentation model is a generative adversarial network model, comprising a semantic segmentation model and a discriminator model; training the semantic segmentation model based on a source domain image; using the discriminator model to perform decision processing on a semantic segmentation image generated by the semantic segmentation model; constructing a loss function corresponding to the discriminator model; wherein the loss function comprises: a target domain loss function generated based on the target domain image; the target domain loss function comprises: at least one of a first semantic loss function generated based on the semantic consistency of an image block, a second semantic loss function generated based on the semantic consistency of a clustering cluster, and a third semantic loss function generated based on a logical construction of an image space; and adjusting the semantic segmentation model and the discriminator model according to the loss function.

[0005] Optionally, the target domain image is segmented into multiple superpixels using a superpixel segmentation algorithm; wherein the image block corresponds to the superpixel; a first prediction probability that a pixel in the superpixel is predicted to be a first category is obtained; wherein the first category is the category corresponding to this superpixel; and the first semantic loss function is generated according to the first prediction probability and a rule for regularizing the superpixel for semantic consistency.

[0006] Optionally, the first semantic loss function is:

[0007]

[0008] Among them, x t is the target domain image, For the first category, is the jth pixel of the ith superpixel, is the first predicted probability, λ pc is the threshold, I (·) is an indicative function, indicating whether the j-th pixel of the i-th superpixel belongs to the first category, i is the number of the superpixel, and j is the number of the pixel.

[0009] Optionally, a residual convolutional network is used to extract a feature vector of the superpixel; based on the feature vector, the superpixel is clustered using a clustering algorithm to obtain a plurality of clusters; a second prediction probability that a pixel in the superpixel is predicted to be a second category is obtained; wherein the second category is a category corresponding to the cluster to which the superpixel belongs; and the second semantic loss function is generated according to the second prediction probability and a rule for regularizing the clusters for semantic consistency.

[0010] Optionally, the second semantic loss function is:

[0011]

[0012] Among them, x t is the target domain image, For the second category, is the i-th superpixel S i The jth pixel in S i represents the i-th superpixel, superpixel S i Belongs to the kth cluster C k ; cc is the threshold value, is the second predicted probability; (·) is an indicator function, taking values ​​of 1 or 0.

[0013] Optionally, the semantic segmentation prediction results are divided into multiple columns according to the boundaries of the superpixels; a category sequence is formed according to the categories of each superpixel in each column; the category sequence is processed using a trained encoder model to generate sequence features; the sequence features are reconstructed using a trained decoder model, and a third prediction probability that the superpixel is predicted to be a third category is determined based on the learned spatial logical construction information; wherein the third category is the category corresponding to this superpixel and output during the reconstruction of the sequence features; a fourth prediction probability that a pixel in the superpixel is predicted to be a third category is obtained; and the third semantic loss function is generated based on the third prediction probability and the fourth prediction probability and the rule for regularizing the superpixel.

[0014] Optionally, the third semantic loss function is:

[0015]

[0016] Among them, S i is the i-th superpixel, is the jth pixel in the i-th superpixel, is the fourth predicted probability, is the third predicted probability, λ sl is the probability threshold, and I(·) is the indicative function.

[0017] Optionally, the encoder model and the decoder model are constructed based on an LSTM network model.

[0018] Optionally, the loss function further includes: a source domain loss function generated based on the source domain image, and a mathematical expectation function for the source domain image and the target domain image.

[0019] Optionally, the semantic segmentation model and the discriminator model are constructed using a fully convolutional neural network; the source domain image includes: a synthetic image; and the target domain image includes: a real image.

[0020] According to a second aspect of the present disclosure, a method for image semantic segmentation is provided, comprising: obtaining a trained image semantic segmentation model; wherein the image semantic segmentation model is a generative adversarial network model, comprising a semantic segmentation model and a discriminator model; performing semantic segmentation processing on an image using the semantic segmentation model; wherein the generative adversarial network model is trained by the training method described above.

[0021] According to a third aspect of the present disclosure, there is provided an image semantic segmentation model training device, comprising: a model building module, for building an image semantic segmentation model for semantically segmenting an image; wherein the image semantic segmentation model is a generative adversarial network model, comprising a semantic segmentation model and a discriminator model; a model training module, for training the semantic segmentation model based on a source domain image; an image discrimination module, for using the discriminator model to perform judgment processing on a semantic segmentation image generated by the semantic segmentation model; a loss function building module, for building a loss function corresponding to the discriminator model; wherein the loss function comprises: a target domain loss function generated based on the target domain image; the target domain loss function comprises: at least one of a first semantic loss function generated based on the semantic consistency of an image block, a second semantic loss function generated based on the semantic consistency of a clustering cluster, and a third semantic loss function generated based on a logical construction of an image space; and a model adjustment module, for adjusting the semantic segmentation model and the discriminator model according to the loss function.

[0022] According to a fourth aspect of the present disclosure, an image semantic segmentation device is provided, comprising: a model acquisition module, used to acquire a trained image semantic segmentation model; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; an image processing module, used to perform semantic segmentation processing on an image using the semantic segmentation model; wherein the generative adversarial network model is trained by the training method described above.

[0023] According to a fifth aspect of the present disclosure, there is provided an image semantic segmentation model training device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method as described above based on instructions stored in the memory.

[0024] According to a sixth aspect of the present disclosure, there is provided an image semantic segmentation apparatus, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method as described above based on instructions stored in the memory.

[0025] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the instructions are executed by a processor to perform the above-mentioned image semantic segmentation model training method, and / or to perform the above-mentioned image semantic segmentation method.

[0026] The image semantic segmentation model training, image semantic segmentation method, device and storage medium disclosed in the present invention obtain general rules in semantic segmentation task scenarios that do not require label information in the target domain, set three constraints for evaluating the quality of image semantic segmentation based on semantic consistency within image blocks, semantic consistency within clusters and rationality of spatial logic, and constrain the semantic segmentation model inference results of the model on the target domain image in the form of regularization terms during the training process, perform cross-domain migration of the image semantic segmentation model, achieve correct semantic segmentation of images in the target domain, and improve the efficiency and accuracy of image semantic segmentation model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 It is a schematic diagram of image semantic segmentation in the prior art;

[0029] Figure 2A flowchart of an embodiment of an image semantic segmentation model training method according to the present disclosure;

[0030] Figure 3 A training schematic diagram of generating a first semantic loss function in one embodiment of the image semantic segmentation model training method disclosed herein;

[0031] Figure 4 A schematic diagram of evaluating segmentation quality based on semantic consistency of image blocks in one embodiment of the image semantic segmentation model training method disclosed herein;

[0032] Figure 5 A training schematic diagram of generating a second semantic loss function in one embodiment of the image semantic segmentation model training method disclosed herein;

[0033] Figure 6 A schematic diagram of evaluating segmentation quality based on semantic consistency of clustering clusters in one embodiment of the image semantic segmentation model training method disclosed herein;

[0034] Figure 7 A training schematic diagram of generating a third semantic loss function in one embodiment of the image semantic segmentation model training method disclosed herein;

[0035] Fig. 8A A schematic diagram of evaluating segmentation quality based on image space logic construction in one embodiment of the image semantic segmentation model training method disclosed herein;

[0036] Figure 8B A schematic diagram of using an LSTM codec to learn spatial logic in prediction results in one embodiment of the image semantic segmentation model training method disclosed herein;

[0037] Fig. 9 A schematic diagram of a framework application based on an adversarial learning mechanism in one embodiment of the image semantic segmentation model training method disclosed herein;

[0038] Fig.10 A flowchart of an embodiment of the image semantic segmentation method according to the present disclosure;

[0039] Fig.11 A schematic diagram of modules of an embodiment of an image semantic segmentation model training device according to the present disclosure;

[0040] Fig.12 A schematic diagram of a module for constructing a loss function module in one embodiment of an image semantic segmentation model training device according to the present disclosure;

[0041] Fig.13A schematic diagram of modules of another embodiment of the image semantic segmentation model training device according to the present disclosure;

[0042] Fig.14 is a flow chart of an embodiment of an image semantic segmentation device according to the present disclosure;

[0043] Fig.15 The figure is a flowchart of another embodiment of the image semantic segmentation device according to the present disclosure. DETAILED DESCRIPTION

[0044] The present disclosure is described more fully below with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are described. The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure. The technical solutions of the present disclosure are described in many aspects below in conjunction with various figures and embodiments.

[0045] The terms "first", "second", etc. in the following text are only used to distinguish between the two and have no other special meanings.

[0046] In the existing technology, deep semantic segmentation models based on supervised learning require a large amount of training data with pixel-level annotations, which is very expensive and slow to obtain. For example, among the 5,000 images with high-quality pixel-level annotations contained in the Cityscapes dataset, it takes an average of more than 1.5 hours for a worker to fully annotate each image.

[0047] Fully Convolutional Networks (FCN) have made breakthrough progress in the spatially dense classification task of image semantic segmentation. Based on the fully convolutional neural network, using low-cost computer-generated data to train image semantic segmentation models has also become one of the research hotspots. Limited by existing technology, computer-generated images are still very different from real-world images. Synthetic images can be easily distinguished from real images by naked eye observation. This difference causes the semantic segmentation model trained with synthetic images to perform poorly on real images, which is called "domain mismatch".

[0048] like Figure 1As shown in the figure, the FCN trained on the source domain data can obtain very accurate semantic segmentation results on the synthetic images, but when the FCN is applied to the target domain images, the FCN reasoning results show a lot of errors. The inventor believes that the main reason for the domain mismatch of the image semantic segmentation model is that the model is overfitted on the synthetic image data in the source domain during the training process. Given the unsupervised setting, there is no available annotation information in the target domain, and the training samples in the target domain cannot be used to fine-tune the model.

[0049] Figure 2 FIG. 1 is a flow chart of an embodiment of the image semantic segmentation model training method according to the present disclosure. Figure 2 As shown:

[0050] Step 201: construct an image semantic segmentation model for performing semantic segmentation on an image.

[0051] In one embodiment, the image semantic segmentation model is a generative adversarial network model (GAN), including a semantic segmentation model and a discriminator model. The semantic segmentation model and the discriminator model can be constructed using a fully convolutional neural network. The source domain image includes a synthetic image, etc., and the target domain image includes a real image, etc.

[0052] Generative adversarial network is a deep learning model that produces good output through the game learning between semantic segmentation model and discriminator model. The semantic segmentation model is a generator model (Discriminative Model), which is used for image semantic segmentation; the discriminator model is used to judge the authenticity of the semantic segmentation image output by the semantic segmentation model during training, and is trained based on the idea of ​​"game theory".

[0053] Step 202: training a semantic segmentation model based on the source domain image.

[0054] In one embodiment, the source domain image is a synthetic image synthesized by computer technology, and the target domain image is a real image in the real world. The source domain image is selected as a training set to train the semantic segmentation model. After training, the source domain image and the target domain image are input into the semantic segmentation model to generate a semantic segmentation image corresponding to the source domain image and the target domain image.

[0055] Step 203: Use the discriminator model to perform decision processing on the semantic segmentation image generated by the semantic segmentation model.

[0056] In one embodiment, the discriminator model is equivalent to a binary classifier, which receives the semantic segmentation image corresponding to the source domain image and the target domain image output by the semantic segmentation model, and determines whether the semantic segmentation image is a true semantic segmentation image, rather than the semantic segmentation image output by the semantic segmentation model. It can generate the probability that the semantic segmentation image is a true semantic segmentation image, etc.

[0057] The goal of the semantic segmentation model is to generate real semantic segmentation images corresponding to the source domain images and the target domain images as much as possible to deceive the discriminator model D; the goal of the discriminator model D is to recognize the semantic segmentation images generated by the semantic segmentation model as much as possible. The semantic segmentation model and the discriminator model D constitute a dynamic "game process".

[0058] Step 204: construct a loss function corresponding to the discriminator model.

[0059] In one embodiment, the loss function includes a target domain loss function generated based on a target domain image; the target domain loss function includes at least one of a first semantic loss function generated based on the semantic consistency of an image block, a second semantic loss function generated based on the semantic consistency of a clustering cluster, and a third semantic loss function generated based on a logical construction of an image space.

[0060] Step 205: adjust the semantic segmentation model and the discriminator model according to the loss function.

[0061] In one embodiment, the parameters of the semantic segmentation model and the discriminator model can be adjusted based on the loss function to achieve the training goal. The generative adversarial network model is trained based on the loss function, and the semantic segmentation model and the discriminator model can be adjusted by existing iterative training methods.

[0062] The image semantic segmentation model training method in the above embodiment alleviates the problem of model overfitting source domain data by obtaining universal rules in semantic segmentation task scenarios that do not require label information in the target domain; three constraints are set to evaluate the quality of image semantic segmentation, namely, semantic consistency within image blocks, semantic consistency within clusters, and rationality of spatial logic; semantic consistency within image blocks and semantic consistency within clusters guide semantic segmentation based on the consistency of semantic categories between pixels within image blocks and between image blocks belonging to the same cluster, respectively, and the rationality of spatial logic associates label predictions with contexts based on the spatial relationship between image blocks; by setting constraints, a Regularizer of Prediction Transfer (RPT) method is proposed for cross-domain migration of image semantic segmentation models that are trained using synthetic data and perform reasoning on real images.

[0063] In one embodiment, the target domain loss function can take effect as a regularization term during the model training process, migrating the model that is overfitted to the source domain image to the target domain, thereby achieving the goal of correct semantic segmentation of the target domain image, and improving the semantic segmentation performance of the image semantic segmentation model trained by synthetic data on real data.

[0064] The regularization constraints are called inference transfer regularization, which are regularization based on semantic consistency within image blocks, regularization based on semantic consistency within clusters, and regularization based on spatial logic. They are used to guide the migration of image semantic segmentation models trained in the source domain to the target domain.

[0065] Figure 3 FIG. 1 is a training diagram for generating a first semantic loss function in one embodiment of the image semantic segmentation model training method disclosed in the present invention, as shown in FIG. Figure 3 As shown:

[0066] Step 301: segment the target domain image into multiple superpixels using a superpixel segmentation algorithm.

[0067] In one embodiment, the image block corresponds to the superpixel. For example, the image block corresponds to the superpixel one-to-one, that is, one superpixel is one image block.

[0068] Step 302: Obtain a first prediction probability that a pixel in a superpixel is predicted to be a first category. The first category is the category corresponding to the superpixel. For example, if the superpixel belongs to a vehicle in the target domain image, the first category is a vehicle.

[0069] Step 303: Generate a first semantic loss function according to the first prediction probability and a rule for regularizing the semantic consistency of superpixels.

[0070] In one embodiment, the goal based on semantic consistency within an image block is to hope that pixels within the same image block are predicted to be of the same semantic category. Superpixels can be used to divide image blocks to ensure that pixels within the same image block have highly similar visual characteristics. The superpixel segmentation algorithm can be a variety of existing superpixel segmentation algorithms, such as the SLIC (simple linear iterative cluster) algorithm. The SLIC algorithm adopts the idea of ​​k-means clustering and can efficiently generate superpixels based on the pixel information of the image.

[0071] like Figure 4 As shown, for a target domain image x t , the image can be divided into N superpixels {S i |i=1,…,N}, where any superpixel S i Includes M iadjacent pixels with similar appearance, i.e. Assume that at least the majority of pixels in a superpixel should be predicted to be of the same first class The first category It is defined as the semantic category predicted most by all pixels in this superpixel.

[0072] Since the SLIC algorithm only considers the visual appearance similarity of pixels in an image, in real images in the target domain, a superpixel often contains multiple sub-regions with different semantic categories. If we simply force all regions in a superpixel to be predicted as the dominant category, i.e., the first category Therefore, when performing regularization based on semantic consistency within an image block, some pixels that meet the conditions are removed.

[0073] For example, for a superpixel S i , where the pixels According to its first category The prediction scores on are divided into two groups:

[0074] 1. This condition indicates that the pixel The predicted score on the first category is small, and it has a relatively high probability of being predicted as other categories. For such pixels, they need to be removed from regularization; is the pixel inferred by the current network model (discriminator model) Belongs to the first category The probability of the first category The most predicted semantic category for all pixels in this superpixel.

[0075] 2. This condition indicates that the pixel For the predicted The probability of is large, and regularization can better guide the prediction of such pixels;

[0076] According to the first prediction probability and the rule of regularizing the semantic consistency of superpixels, the first semantic loss function is generated as follows:

[0077]

[0078] Among them, x t is the target domain image, For the first category, is the jth pixel of the i-th image block, is the first predicted probability, λ pc is the threshold, I (·)is an indicative function, indicating whether the jth pixel of the i-th image block belongs to the first category, i is the number of the image block, j is the number of the pixel, and I (·) is an indicator function, taking values ​​of 1 or 0.

[0079] Figure 5 FIG. 1 is a training diagram for generating a second semantic loss function in one embodiment of the image semantic segmentation model training method disclosed in the present invention, as shown in FIG. Figure 5 As shown:

[0080] Step 501, extracting the feature vector of the superpixel using a residual convolutional network.

[0081] Step 502: Based on the feature vector, a clustering algorithm is used to cluster the superpixels to obtain a plurality of clusters.

[0082] Step 503: Obtain a second prediction probability that a pixel in the superpixel is predicted to be of a second category. The second category is a category corresponding to the cluster to which the superpixel belongs.

[0083] Step 504: Generate a second semantic loss function according to the second prediction probability and a rule for regularizing the semantic consistency of the clusters.

[0084] In one embodiment, in addition to regularization based on semantic consistency within an image block, the consistency of the predicted category can be strengthened between image blocks belonging to the same cluster. Cluster-based regularization hopes that the category of superpixels with similar visual attributes is predicted as the main category of the cluster to which it belongs, i.e., the second category.

[0085] like Figure 6 As shown, first, all superpixels in the target domain image are divided into several clusters. To achieve clustering of superpixels, the existing residual convolutional network can be used to extract the feature vector of the superpixel. For example, the residual convolutional network is ResNet-101, and the standard ResNet-101 pre-trained with the ImageNet dataset is used to extract the feature expression for each superpixel, and further average the response map of each superpixel in the res5c layer in the spatial dimension, and the final 2048-dimensional vector is used as the feature vector of the superpixel. In the feature space of the superpixel, the k-means algorithm is used to cluster all superpixels from the target domain image into k = 2048 clusters. For each cluster, its dominant category, that is, the second category It is determined by the majority of superpixel categories in the cluster.

[0086] Similar to the SLIC algorithm, k-means clustering of superpixels cannot guarantee that each superpixel is correctly classified into the corresponding cluster, so similar filtering rules are used. Assume that the superpixel S where the pixel is located i Belongs to cluster C k ,but Indicates the probability that the semantic category of the pixel is consistent with the dominant category of the current cluster. Pixels that are greatly different from the dominant category of the cluster can be excluded from this regularization loss. is the pixel inferred by the current network model Belongs to the second category The probability of the second category is the semantic category determined by the majority of superpixel categories in the cluster.

[0087] According to the second prediction probability, the second semantic loss function is generated by the rule of regularization processing for semantic consistency of clustering clusters:

[0088]

[0089] Among them, x t is the target domain image, For the second category, is the i-th image block (superpixel) S i The jth pixel in S i represents the i-th superpixel, superpixel S i Belongs to the kth cluster C k ; cc is the threshold value.

[0090] is the second predicted probability, Indicates that the pixel The semantic category of the cluster and the dominant category of the current cluster Probability of agreement; I (·) is an indicator function, taking values ​​of 1 or 0.

[0091] Figure 7 FIG. 1 is a training diagram for generating a third semantic loss function in one embodiment of the image semantic segmentation model training method disclosed in the present invention, as shown in FIG. Figure 7 As shown:

[0092] Step 701, divide the semantic segmentation prediction results into multiple columns according to the boundaries of superpixels.

[0093] Step 702: construct a category sequence according to the category of each superpixel in each column.

[0094] Step 703: Use the trained encoder model to process the category sequence to generate sequence features.

[0095] Step 704: reconstruct the sequence features using the trained decoder model, and determine the third prediction probability that the superpixel is predicted to be a third category based on the learned spatial logical construction information. The third category is the category corresponding to the superpixel and output during the reconstruction of the sequence features.

[0096] Step 705, obtaining a fourth prediction probability that a pixel in the superpixel is predicted to be of a third category;

[0097] Step 706: Generate a third semantic loss function according to the third prediction probability and the fourth prediction probability and a rule for regularizing superpixels.

[0098] In one embodiment, for the semantic segmentation task, there is a correlation between the semantic labels of pixels at different positions in the image. When directly observing the failed semantic segmentation results, many "irrational" places in the segmentation results are found based on the understanding of the spatial logical construction rules of real-world physics. Fig. 8A As shown in the figure, pixels of the "sky" category are usually located above the "buildings" and "roads" in the image, rather than the other way around. If such illogical errors occur in the prediction results, it is hoped that the corresponding regularization constraints can be used to help the model learn the correct predictions. It can be considered that the logical relationship between different semantic categories in space is shared by the source domain and the target domain. Therefore, regularization based on spatial logic construction can be used to transfer the spatial logic knowledge learned by the model from the source domain to the target domain.

[0099] The existing LSTM (Long Short-Term Memory) network model can be used to build the encoder model and decoder model, and the existing LSTM encoder-decoder learning method can be used to learn the spatial logic in the prediction results. A model based on the existing LSTM encoder-decoder framework is used to learn the relationship between superpixels in the vertical direction of the image. The structure of the model is as follows: Figure 8B As shown, the overall goal of the model is to use the contextual information in the sequence to infer the semantic categories to which the segments covered by the mask in the sequence belong, and the probability obtained during the inference can be used to evaluate the credibility of the original semantic categories of the segments covered by the mask from a spatial logical perspective.

[0100] For a sequence sequence It is composed of the prediction results of T superpixels located in the same column of the prediction result graph, where represents the one-hot vector of the t-th prediction in the sequence, whose dimension is the C semantic categories in semantic segmentation plus an additional category for marking the mask position.

[0101] Given a prediction sequence Mark a segment with the same continuous prediction category as a masked state (replace its original inferred semantic category with the category used to mark the mask position as described above), and obtain the corresponding masked prediction sequence When the original prediction sequence Obtained through mask processing After that, it is fed into the LSTM encoder, which is used to embed the sequence into a sequence feature expression.

[0102] Then, the LSTM decoder connected to the end of the encoder is used to reconstruct the feature expression. During the reconstruction process, the semantic category and its probability corresponding to the mask position can be obtained from the perspective of spatial logic. The semantic category of the masked position in is replaced by the “masked category”, which will cause the decoder to try to reconstruct the entire original sequence When , it is necessary to infer the category of the masked position based on the semantic categories of other positions in the sequence. The inference result includes the probabilities of all possible semantic categories of the masked position.

[0103] The model can use the label information in the source domain and optimize based on the cross entropy loss. Since this model needs to be trained with label information, and label information only exists in the source domain, the model can be trained with label information in the source domain; when training the model, a corresponding loss function needs to be used, and "cross entropy loss" is selected here. Using the trained LSTM encoder-decoder model, the rationality of the target domain prediction results can be effectively evaluated from a spatial logic perspective. For the target domain image x t , first divide its semantic segmentation prediction results into several columns according to the boundaries of superpixels, and form a sequence of the categories of each superpixel in each column. I , its spatial logic rationality This is the category output during sequence reconstruction probability. It is the probability of the superpixel belonging to a category obtained by LSTM encoder and decoder inference. The process of reconstructing the sequence is to predict the category of each element in the sequence, obtain the probability of each element belonging to each semantic category, and take the semantic category with the highest probability as the reconstruction result of the element. The probability of an element belonging to each semantic category indicates that the element belongs to the category probability.

[0104] If the probability is less than the threshold λ sl , the predicted category in this superpixel is The so-called "penalty" means adding these pixels that meet the requirements to the loss function, and guiding the segmentation model to tend not to predict these pixels as categories by minimizing the loss function.

[0105] The third semantic loss function based on the regularization of spatial logic construction is:

[0106]

[0107] Among them, S i is the i-th superpixel, is the jth pixel in the i-th superpixel, Pixels inferred by the current network model Belongs to the third category The probability of The superpixel inferred by the LSTM encoder-decoder belongs to the third category The probability of sl is the probability threshold, and I(·) is the indicative function.

[0108] In one embodiment, the loss function further includes a source domain loss function generated based on the source domain image and a mathematical expectation function for the source domain image and the target domain image. Fig. 9 As shown in Figure 1, an unsupervised domain adaptation framework based on adversarial learning mechanism is used. The principle of this framework is to “fool” the domain discriminator D(·) by using the feature representations of source and target domain images to guide the model’s learning for semantic segmentation tasks.

[0109] For a given source domain training dataset Target domain dataset The loss function of adversarial learning is constructed as:

[0110]

[0111] Among them, x t : a target domain image; Target domain image set; x s : a source domain image; The source domain image set; D(·): The probability that the discriminator infers that the input image comes from the target domain; E represents the mathematical expectation of the entire image dataset.

[0112] The domain discriminator D(·) tries to distinguish the feature expressions of the source domain and the target domain images to minimize the loss function At the same time, the fully convolutional neural network used to learn feature representation also tries to deceive the domain discriminator as much as possible by adjusting itself. The domain discriminator D(·) is a classifier used to determine the domain to which the image from which the input feature comes belongs. During the training process, the parameters in the domain discriminator D are adjusted by the gradient descent algorithm to minimize its loss function This loss function represents whether the discriminator can correctly distinguish whether the image (or its features) is from the source domain or the target domain.

[0113] In this framework, since the computer-generated images in the source domain have corresponding pixel-level semantic annotation information, the loss function of the entire framework is based on the pixel-level classification loss on the source domain.

[0114] By comparing the difference between the semantic segmentation results and labels obtained by the current network inference, the corresponding loss function is calculated; its meaning is to characterize the quality of the current network semantic segmentation. The smaller the loss, the higher the quality.

[0115] Since there is no available annotation information in the target domain, the three regularizers included in RPT are used to construct the loss function on the target domain:

[0116]

[0117] in, and They refer to the loss functions generated according to image block consistency, cluster consistency and spatial logical construction respectively.

[0118] The loss function of the entire framework is:

[0119]

[0120] The loss function of formula (1-6) is the loss function of the entire system. It means that the loss function is minimized by adjusting the discriminator D during the optimization process. Formula (1-6) starts with It means that by adjusting the parameters in FCN, the sum of the three losses is minimized. This term has a negative sign in front of it, and the adversarial learning edge is minimized by adjusting D At the same time, by adjusting FCN to minimize (i.e. maximize ).

[0121] In one embodiment, the feasibility of the image semantic segmentation model training method disclosed in the present invention is verified by performing two cross-domain semantic segmentation model migration tasks, GTA5→Cityscapes and SYNTHIA→Cityscapes, based on two computer-generated semantic segmentation datasets, GTA5 and SYNTHIA, and a real street scene image semantic segmentation data dataset Cityscapes. The semantic segmentation quality of the image semantic segmentation model after migration on the real image is calculated to measure the performance of the model, using the commonly used quality evaluation indicators for semantic segmentation (such as the intersection over union (IoU) of each category and the average intersection over union (meanIoU, mIoU) of all categories). The test results show that the performance of this method has reached the world's leading level.

[0122] Fig.10 FIG. 1 is a flow chart of an embodiment of the image semantic segmentation method according to the present disclosure. Fig.10 As shown:

[0123] Step 1001, obtaining a trained image semantic segmentation model; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model.

[0124] Step 1002, use the semantic segmentation model to perform semantic segmentation processing on the image; wherein the generative adversarial network model is trained by the training method in any of the above embodiments. The image can be a real image in the target domain, etc.

[0125] In one embodiment, Fig.11 As shown, the present disclosure provides an image semantic segmentation model training device 110, including: a model construction module 111, a model training module 112, an image discrimination module 113, a loss function construction module 114 and a model adjustment module 115.

[0126] The model building module 111 builds an image semantic segmentation model for semantic segmentation of images; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; the model training module 112 trains the semantic segmentation model based on the source domain image; the image discrimination module 113 uses the discriminator model to perform judgment processing on the semantic segmentation image generated by the semantic segmentation model.

[0127] The loss function construction module 114 constructs a loss function corresponding to the discriminator model; wherein the loss function includes: a target domain loss function generated based on a target domain image; the target domain loss function includes: at least one of a first semantic loss function generated based on the semantic consistency of an image block, a second semantic loss function generated based on the semantic consistency of a clustering cluster, and a third semantic loss function generated based on a logical construction of an image space; the model adjustment module 115 adjusts the semantic segmentation model and the discriminator model according to the loss function.

[0128] In one embodiment, Fig.12 As shown, the loss function construction module 114 includes a first loss function unit 1141, a second loss function unit 1142 and a third loss function unit 1143. The first loss function unit 1141 uses a superpixel segmentation algorithm to segment the target domain image into multiple superpixels, and the image blocks correspond to the superpixels; the first loss function unit 1141 obtains a first prediction probability that a pixel in the superpixel is predicted to be a first category, and the first category is the category corresponding to the superpixel; the first loss function unit 1141 generates a first semantic loss function according to the first prediction probability and a rule for regularizing the semantic consistency of the superpixel.

[0129] The second loss function unit 1142 utilizes a residual convolutional network to extract a feature vector of a superpixel, and based on the feature vector, uses a clustering algorithm to cluster the superpixel to obtain a plurality of cluster clusters; the second loss function unit 1142 obtains a second prediction probability that a pixel in the superpixel is predicted to be a second category, and the second category is a category corresponding to the cluster cluster to which the superpixel belongs; the second loss function unit 1142 generates a second semantic loss function according to the second prediction probability and the rule of regularization processing for semantically consistent cluster clusters.

[0130] The third loss function unit 1143 divides the semantic segmentation prediction results into multiple columns according to the boundaries of superpixels; constructs a category sequence according to the categories of each superpixel in each column; uses the trained encoder model to process the category sequence to generate sequence features; the third loss function unit 1143 reconstructs the sequence features using the trained decoder model, and determines the third prediction probability that the pixel in the superpixel is predicted to be a third category based on the learned spatial logical construction information; wherein the third category is the category corresponding to this superpixel and output during the reconstruction of the sequence features; the third loss function unit 1143 generates a third semantic loss function according to the third prediction probability and the rule for regularizing the superpixel.

[0131] In one embodiment, Fig.13 FIG. 2 is a module diagram of another embodiment of the image semantic segmentation model training device according to the present disclosure. Fig.13As shown, the apparatus may include a memory 131, a processor 132, a communication interface 133, and a bus 134. The memory 131 is used to store instructions, the processor 132 is coupled to the memory 131, and the processor 132 is configured to execute the above-mentioned image semantic segmentation model training method based on the instructions stored in the memory 131.

[0132] The memory 131 may be a high-speed RAM memory, a non-volatile memory, etc. The memory 131 may also be a memory array. The memory 131 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules. The processor 72 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the image semantic segmentation model training method disclosed in the present invention.

[0133] In one embodiment, Fig.14 As shown, the present disclosure provides an image semantic segmentation device 140, including: a model acquisition module 141 and an image processing module 142. The model acquisition module 141 acquires a trained image semantic segmentation model, which is a generative adversarial network model, including a semantic segmentation model and a discriminator model. The image processing module 142 uses the semantic segmentation model to perform semantic segmentation processing on the image, and the generative adversarial network model is trained by the training method in any of the above embodiments.

[0134] In one embodiment, Fig.15 FIG. 1 is a module diagram of another embodiment of the image semantic segmentation device according to the present disclosure. Fig.15 As shown, the apparatus may include a memory 151, a processor 152, a communication interface 153 and a bus 154. The memory 151 is used to store instructions, the processor 152 is coupled to the memory 151, and the processor 152 is configured to execute the above-mentioned image semantic segmentation method based on the instructions stored in the memory 151.

[0135] The memory 151 may be a high-speed RAM memory, a non-volatile memory, etc. The memory 151 may also be a memory array. The memory 151 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules. The processor 152 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the image semantic segmentation method disclosed in the present invention.

[0136] In one embodiment, the present disclosure provides a computer-readable storage medium, which stores computer instructions. When the instructions are executed by a processor, an image semantic segmentation model training method as in any of the above embodiments is implemented, and / or an image semantic segmentation method as in any of the above embodiments.

[0137] The image semantic segmentation model training, image semantic segmentation method, device and storage medium provided in the above embodiments obtain the rules that are universal in the semantic segmentation task scenario and do not require label information in the target domain. Three constraints are set to evaluate the quality of image semantic segmentation based on semantic consistency within image blocks, semantic consistency within clustering clusters and rationality of spatial logic. The semantic segmentation model inference results of the model on the target domain image are constrained in the form of regularization terms during the training process. The image semantic segmentation model trained with synthetic data and inferred on real images is migrated across domains to achieve correct semantic segmentation of images in the target domain, thereby improving the efficiency and accuracy of image semantic segmentation model training.

[0138] The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0139] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A method for training an image semantic segmentation model, comprising: Constructing an image semantic segmentation model for semantic segmentation of images; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; Training the semantic segmentation model based on source domain images; Using the discriminator model to perform decision processing on the semantic segmentation image generated by the semantic segmentation model; Constructing a loss function corresponding to the discriminator model; The loss function includes: a target domain loss function generated based on the target domain image; segmenting the target domain image into multiple superpixels using a superpixel segmentation algorithm; clustering the superpixels using a clustering algorithm to obtain multiple clusters; the target domain loss function includes: a first semantic loss function generated based on the semantic consistency of the superpixels, a second semantic loss function generated based on the semantic consistency of the clusters, and a third semantic loss function generated based on the spatial logical construction rule of the target domain image; The semantic segmentation model and the discriminator model are adjusted according to the loss function.

2. The method of claim 1, further comprising: Obtaining a first prediction probability that a pixel in the superpixel is predicted to be of a first category; wherein the first category is a category corresponding to the superpixel; The first semantic loss function is generated according to the first prediction probability and a rule for regularizing the superpixel for semantic consistency.

3. The method of claim 2, wherein: The first semantic loss function is: Among them, x t is the target domain image, For the first category, is the jth pixel of the ith superpixel, is the first predicted probability, λ pc is the threshold, I (·) is an indicative function, indicating whether the jth pixel of the i-th superpixel belongs to the first category, i is the number of the superpixel and j is the number of the pixel.

4. The method of claim 2, further comprising: Extracting a feature vector of the superpixel using a residual convolutional network; Based on the feature vector, clustering the superpixel using a clustering algorithm to obtain the multiple clusters; Obtaining a second prediction probability that a pixel in the superpixel is predicted to be of a second category; wherein the second category is a category corresponding to the cluster to which the superpixel belongs; The second semantic loss function is generated according to the second prediction probability and a rule for regularizing the semantic consistency of the clustering clusters.

5. The method of claim 4, wherein: The second semantic loss function is: Among them, x t is the target domain image, For the second category, is the i-th superpixel S i The jth pixel in S i represents the i-th superpixel, superpixel S i Belongs to the kth cluster C k ; cc is the threshold value, is the second predicted probability; (·) is an indicator function, taking values ​​of 1 or 0.

6. The method of claim 2, wherein: Dividing the semantic segmentation prediction result into multiple columns according to the boundaries of the superpixel; Construct a category sequence according to the category of each superpixel in each column; Using the trained encoder model to process the category sequence to generate sequence features; Reconstructing the sequence features using the trained decoder model, and determining a third prediction probability that the superpixel is predicted to be a third category based on the learned spatial logical construction information; wherein the third category is a category corresponding to the superpixel and output during the reconstruction of the sequence features; Obtaining a fourth prediction probability that a pixel in the superpixel is predicted to be a third category; performing regularization processing on the superpixel according to the third prediction probability and the fourth prediction probability The rule generates the third semantic loss function.

7. The method of claim 6, wherein: The third semantic loss function is: Among them, S i is the i-th superpixel, is the jth pixel in the i-th superpixel, is the third category; is the fourth predicted probability, is the third predicted probability, λ sl is the probability threshold, I(·) is the indicative function, and its value is 0 or 1.

8. The method of claim 6, wherein: The encoder model and the decoder model are constructed based on the LSTM network model.

9. The method of claim 1, wherein: The loss function further includes: a source domain loss function generated based on the source domain image, and a mathematical expectation function for the source domain image and the target domain image.

10. The method according to any one of claims 1 to 9, wherein: Using a fully convolutional neural network to construct the semantic segmentation model and the discriminator model; The source domain image includes: a synthetic image; the target domain image includes: a real image.

11. A method for image semantic segmentation, comprising: Obtain a trained image semantic segmentation model; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; Perform semantic segmentation processing on the image using the semantic segmentation model; Wherein, the generative adversarial network model is trained by the training method described in any one of claims 1 to 10.

12. An image semantic segmentation model training device, comprising: A model building module, used to build an image semantic segmentation model for semantic segmentation of images; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; A model training module, used for training the semantic segmentation model based on the source domain image; An image discrimination module, used for using the discriminator model to perform discrimination processing on the semantic segmentation image generated by the semantic segmentation model; A loss function construction module, used to construct a loss function corresponding to the discriminator model; wherein the loss function includes: a target domain loss function generated based on the target domain image; using a superpixel segmentation algorithm to segment the target domain image into multiple superpixels; using a clustering algorithm to cluster the superpixels to obtain multiple clusters; the target domain loss function includes: a first semantic loss function generated based on the semantic consistency of the superpixels, a second semantic loss function generated based on the semantic consistency of the clusters, and a third semantic loss function generated based on the spatial logical construction rule of the target domain image; A model adjustment module is used to adjust the semantic segmentation model and the discriminator model according to the loss function.

13. An image semantic segmentation device, comprising: A model acquisition module is used to acquire a trained image semantic segmentation model; wherein the image semantic segmentation model is a generative adversarial network model, including a semantic segmentation model and a discriminator model; An image processing module is used to perform semantic segmentation processing on an image using the semantic segmentation model; wherein the generative adversarial network model is trained by the training method described in any one of claims 1 to 10.

14. An image semantic segmentation model training device, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the method according to any one of claims 1 to 10 based on instructions stored in the memory.

15. An image semantic segmentation device, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the method according to any one of claims 11 based on instructions stored in the memory. 16 . A computer-readable storage medium, wherein the computer-readable storage medium non-transitorily stores computer instructions, wherein the instructions are used by a processor to execute the method according to claim 1 , and / or the method according to claim 11 .

Images