Image semantic segmentation method based on dual category-level adversarial network
Through the domain adaptation method based on dual-class-level adversarial network, the semantic inconsistency and category missession problems of image semantic segmentation during domain adaptation are solved, and the efficient semantic segmentation and generalization performance of label-free data sets are improved.
Patent Information
- Application Number
- CN202210172502.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-24
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-02-24
AI Technical Summary
Existing image semantic segmentation methods are prone to semantic inconsistency or category missegment problems when domain adaptation is performed, especially when there is visual difference between the source domain and the target domain, it is difficult to generalize to the target domain dataset.
The domain adaptive segmentation method based on dual-category-level adversarial network is adopted. By constructing an inter-domain class-level adversarial network, the adversarial loss function is calculated, and the target domain is divided using information entropy value, and the intra-domain adaptive training is carried out to improve the generalization performance of the model.
It effectively improves the adaptability level between different data sets, realizes semantic segmentation of label-free data sets, and improves the generalization performance and segmentation accuracy of the algorithm.
Smart Images

Figure CN114612658B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning and relates to an image semantic segmentation method based on a dual category-level adversarial network. Background Art
[0002] Image semantic segmentation refers to classifying each pixel in an image and obtaining a segmentation result containing semantic information through integration, that is, segmenting the image according to the semantics of the image, where semantics refers to the content or information details of the image. Semantic segmentation is the basis of image understanding and is widely used in medical image analysis, autonomous driving, geographic information systems and other fields. The initial image semantic segmentation methods were mainly based on machine learning methods such as RDF and probabilistic graph models, and the results of semantic segmentation were generally the categories of objects in the image. The emergence of fully convolutional neural networks (FCNs) has enabled deep learning technology to be widely used in image semantic segmentation, and semantic segmentation can now identify individuals contained in an image. In recent years, image semantic segmentation technology based on deep learning has made great progress. For example, the DeepLab series of models combines deep convolutional neural networks (DCNNs) and probabilistic graph models. Deeplab-v2 refers to multi-resolution information of different feature layers and adopts a atrous spatial pyramid pooling structure (ASPP) to effectively learn relevant information of targets of different scales and further improve the segmentation accuracy of the target; DeepLab-v3+ uses an arbitrarily controllable encoder-decoder to improve the resolution of features and restore the original pixel information, and uses a dilated convolution method to balance the semantic segmentation accuracy and efficiency.
[0003] Image semantic segmentation generally uses supervised learning methods. Training samples require pixel-level annotation information. The cost of manual annotation is high. One solution is to use a training set composed of synthetic images. Synthetic image datasets are usually called source domains, and real scene datasets are called target domains. There are often large visual differences between the source domain and the target domain, namely domain shift, such as texture, lighting, and perspective differences. These differences will cause the deep learning model trained by the source domain to have unsatisfactory segmentation accuracy for the target domain, and it is easy to overfit and difficult to generalize to the target domain dataset. Domain adaptation is one of the important technologies of transfer learning in the field of computer vision. It mainly solves how to use source domain data to train a model that adapts to target domain data when there is a deviation in data distribution between the source domain and the target domain. Among them, the source domain contains samples with annotation information, while the target domain has no or only a few data with annotation information. Most of the existing domain adaptation methods use original images or high-level abstract feature maps for cross-domain training, which often results in semantic inconsistency or category misclassification problems. Summary of the invention
[0004] Purpose of the invention: This patent proposes a domain adaptive segmentation method based on a dual category-level adversarial network. By improving the level of adaptation between different data sets, it effectively realizes the semantic segmentation of unlabeled data sets and improves the generalization performance of the algorithm.
[0005] Technical solution: The present invention proposes an image semantic segmentation method based on a dual category-level adversarial network, the method comprising the following steps:
[0006] Step 1: Obtain an image dataset. Obtain a synthetic image dataset GTA5 (video game dataset) and a virtual city dataset SYNTHIA for the source domain. Obtain a real image dataset Cityscapes (city landscape dataset) for the target domain. The dataset consists of a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the generalization ability of the model.
[0007] Step 2: construct an inter-domain category-level adversarial network, which consists of an inter-domain generator and an inter-domain discriminator, wherein the inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The source domain and target domain images in step 1 are input into the inter-domain category-level adversarial network for collaborative training, and the difference between the source domain and the target domain is calculated.
[0008] Step 3: Calculate the adversarial loss function of the network and train the adaptive network between the two domains: After the input image is operated in step 2, two prediction tensors are obtained through two mutually exclusive classifiers. The distance difference between the two tensors is calculated to adjust the adversarial loss size, and the adversarial loss function is used to train the category-level adversarial network between the source domain and the target domain.
[0009] Step 4: Divide the target domain image, use the network trained in step 3 to output the target prediction map, calculate the information entropy value of the prediction map, sort it according to the entropy value, and use the simple ratio λ E The target domain is divided into two parts: set 1 and set 2. The entropy value of set 1 is smaller and the proportion of set 1 to the total target domain is λ E ;
[0010] Step 5: Adaptive training in the target domain. Set 1 in step 4 is regarded as the source domain, its predicted image is used as the pseudo label, and set 2 is regarded as the target domain. The category-level adversarial network is used for adaptive training in the domain.
[0011] Step 6: Train the overall model and obtain the segmentation results. Input the test set images in Cityscapes into the network trained in step 5 to achieve semantic segmentation of the test set images.
[0012] Wherein, the step of obtaining the image data set in step 1 comprises the following steps:
[0013] Step 1: The source domain datasets GTA5 and SYNTHIA are both synthetic virtual data, and the datasets contain image information and pixel-by-pixel semantic information as segmentation labels;
[0014] Step 2: The target domain dataset Cityscapes is real data of real scenes. The dataset contains a training set of 2975 images and a test set of 500 images. The target domain data does not contain any label information.
[0015] In step 2, an inter-domain category-level adversarial network is constructed through the following steps:
[0016] Step 1: Figure 2 As shown, the inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The pre-trained framework of the residual network ResNet-101 on the ImageNet dataset is used as the backbone of the inter-domain generator network, that is, the first half of the framework is the feature extractor, and then proceed to step 2;
[0017] Step 2: Copy two copies of the last classification module in the framework and arrange them in parallel after the feature extractor as two mutually exclusive classifiers, and proceed to step 3;
[0018] Step 3: Input the source domain and target domain images into the network for joint training and calculate the segmentation loss:
[0019]
[0020] Among them, sample X s ∈R H×W×3 is an image from the source domain, Y s is its corresponding pixel label, C is the number of categories for semantic segmentation, and P s (h,w,c) is the discrete distribution of C-type semantics on pixels (h, w); Y s (h,w,c) is the correct labeling probability of the C-type semantics on the pixel (h, w). If the pixel (h, w) belongs to the C class, Y s (h,w,c) =1, otherwise Y s (h,w,c) =0.
[0021] Among them, in step 3, calculating the adversarial loss function of the network and training the adaptive network between the two domains includes the following steps:
[0022] Step 1: During the training process, two mutually exclusive classifiers perform pixel-by-pixel semantic classification on the input image. The two classifiers C 1 and C 2 With different parameters, minimizing the cosine similarity of two classifiers to enhance the weight divergence of their convolutional layers has the following weight difference loss:
[0023]
[0024] in, and C 1 and C 2 The vector obtained by flattening and concatenating the weights of the convolution filters;
[0025] Step 2: Input the image into the mutually exclusive classifier and output two prediction tensors P (1) and P (2) , calculate the distance difference between the two prediction tensors, and use the distance difference as the weight of the adversarial loss, so that each pixel on the segmentation map can have a different weight to adjust the adversarial loss, and enter step three;
[0026] Step 3: Use adaptive adversarial loss function for training:
[0027]
[0028] Among them, P (1) and P (2) C1 and C 2 The prediction tensor of M(·,·) is the cosine distance, and the parameter λ local Control the adaptive weight of the adversarial loss, ε is a parameter, G 1 is the inter-domain generator, D 1 is the inter-domain discriminator, X s is the source domain image, X t is the target domain image.
[0029] Wherein, in step 4, dividing the target domain image includes the following steps:
[0030] Step 1: Input the target domain image into the category-level adversarial network trained in the above step, and obtain the prediction image of the target domain image;
[0031] Step 2: Calculate the information entropy value of each pixel in the target domain prediction image:
[0032]
[0033] Among them, the inter-domain generator G 1 Take the target image X t As input, generate pixel-level segmentation prediction value P t (h,w,c) , and perform entropy mapping calculation on it;
[0034] Step 3: Based on step 2, find the entropy mapping value of each image and arrange the values from small to large:
[0035]
[0036] Among them, R(X t ) is the entropy graph I t The average value of the inter-domain adaptation network model is as follows: 1 The generated predicted segmentation map calculates the entropy value of the semantic probability of each pixel being C class And find the average value R(X t ), sort the target domain entropy map in order from low to high, and then make a one-to-one correspondence between the target domain entropy map and the original target domain image;
[0037] Step 4: Introduce hyperparameter λ E As the ratio of the target domain set 1 to the total target domain, the target domain is divided into two parts, that is, the average entropy map of set 1 is low, which means the segmentation effect is good, while the average entropy map of set 2 is high, which means the segmentation effect is poor, and further training is needed for the images of set 2. Let X te and X th denote the target images assigned to set 1 and set 2 respectively, λ EThe definition is as follows:
[0038]
[0039] Among them, |X te | is the cardinality of set one, |X t | is the cardinality of the entire target domain.
[0040] Among them, in step 5, the adaptive training in the target domain includes the following steps:
[0041] Step 1: The target domain set 1 divided in the above step is regarded as the source domain dataset, the prediction map of set 1 is used as pseudo-label for auxiliary training, and set 2 is regarded as the target domain dataset. At the same time, set 1 and set 2 are used to train the second inter-domain category-level adversarial network model.
[0042] Among them, in step six, training the overall model and obtaining the segmentation results includes the following steps:
[0043] Step 1: Set the overall loss function L for inter-domain adaptation and intra-domain adaptation as follows:
[0044]
[0045] The target model is learned according to the following formula:
[0046]
[0047] Since a dual-category adversarial network is used, it includes two parameters: inter-domain and intra-domain, which are represented by i=1 and i=2 respectively, that is, L 1 is the overall loss function for inter-domain adaptation, L 2 is the overall loss function of the domain adaptation, G 1 is the inter-domain generator, G 2 is the in-domain generator, D 1 is the inter-domain discriminator, D 2 is the intra-domain discriminator, They are the inter-domain adaptive segmentation loss, weight difference loss and adversarial loss functions, respectively. They are the domain-adaptive segmentation loss, weight difference loss, and adversarial loss functions, respectively. are the hyperparameters for controlling weight difference loss and adversarial loss in inter-domain adaptation, respectively. They are the hyperparameters controlling weight difference loss and adversarial loss in intra-domain adaptation, respectively;
[0048] Step 2: First, perform inter-domain adaptive training on the model to optimize the G 1 and D 1, Secondly, based on R(X t ) sort the target images and use G 1 Generate pseudo labels for the target domain and train the adaptive model in the domain to optimize G 2 and D 2 ;
[0049] Step 3: Adapt G to cross-domain 1 After the model was trained for 70,000 iterations, G 1 Generate segmentation maps and entropy maps of 2975 images in the Cityscapes training set, using R(X t ) to obtain the segmentation difficulty coefficient of all target domain images, R(X t ) value is large, indicating that segmentation is difficult, and vice versa, indicating that segmentation is easy. The sorting algorithm is used to sort all the images R(X t ) values are sorted from small to large, and combined with λ E Divide the target domain into two subsets, namely R(X t ) value is small and the proportion of set 1 to the total target domain is λ E , R(X t ) value is large and the proportion of set 2 to the total target domain is 1-λ E ;
[0050] Step 4: Train G with the SGD optimizer 1 and G 2 , the momentum is 0.9, and the initial learning rate is set to 2.5×10 -4 , weight decay is set to 10 -4 , train D using the Adam optimizer 1 and D 2 , the initial learning rate is 10 -4 , and remain unchanged during training. During training, a 512×1024-sized image is used, and the prediction image is upsampled by 2 times to evaluate the segmentation results;
[0051] Step 5: The experiment uses the detection evaluation function:
[0052]
[0053] The semantic segmentation performance is evaluated for each category, where TP, FP, and FN are the number of true, false positive, and false negative pixels, respectively, and k is the number of categories. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is an overall framework diagram of an image semantic segmentation method based on a dual category-level adversarial network described in the present invention;
[0055] Figure 2It is a schematic diagram of the structure of the category-level adversarial network in the present invention. Specific implementation methods
[0056] See also Figure 1 As shown, the present invention provides an image semantic segmentation method based on a dual category-level adversarial network, the method comprising the following steps:
[0057] Step 1: Obtain an image dataset. Obtain a synthetic image dataset GTA5 (video game dataset) and a virtual city dataset SYNTHIA for the source domain. Obtain a real image dataset Cityscapes (city landscape dataset) for the target domain. The dataset consists of a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the generalization ability of the model.
[0058] Step 2: construct an inter-domain category-level adversarial network, which consists of an inter-domain generator and an inter-domain discriminator, wherein the inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The source domain and target domain images in step 1 are input into the inter-domain category-level adversarial network for collaborative training, and the difference between the source domain and the target domain is calculated.
[0059] Step 3: Calculate the adversarial loss function of the network and train the adaptive network between the two domains: After the input image is operated in step 2, two prediction tensors are obtained through two mutually exclusive classifiers. The distance difference between the two tensors is calculated to adjust the adversarial loss size, and the adversarial loss function is used to train the category-level adversarial network between the source domain and the target domain.
[0060] Step 4: Divide the target domain image, use the network trained in step 3 to output the target prediction map, calculate the information entropy value of the prediction map, sort it according to the entropy value, and use the simple ratio λ E The target domain is divided into two parts: set 1 and set 2. The entropy value of set 1 is smaller and the proportion of set 1 to the total target domain is λ E ;
[0061] Step 5: Adaptive training in the target domain. Set 1 in step 4 is regarded as the source domain, its predicted image is used as the pseudo label, and set 2 is regarded as the target domain. The category-level adversarial network is used for adaptive training in the domain.
[0062] Step 6: Train the overall model and obtain the segmentation results. Input the test set images in Cityscapes into the network trained in step 5 to achieve semantic segmentation of the test set images.
[0063] Wherein, the step of obtaining the image data set in step 1 comprises the following steps:
[0064] Step 1: The source domain datasets GTA5 and SYNTHIA are both synthetic virtual data, and the datasets contain image information and pixel-by-pixel semantic information as segmentation labels;
[0065] Step 2: The target domain dataset Cityscapes is real data of real scenes. The dataset contains a training set of 2975 images and a test set of 500 images. The target domain data does not contain any label information.
[0066] In step 2, an inter-domain category-level adversarial network is constructed through the following steps:
[0067] Step 1: Figure 2 As shown, the inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The pre-trained framework of the residual network ResNet-101 on the ImageNet dataset is used as the backbone of the inter-domain generator network, that is, the first half of the framework is the feature extractor, and then proceed to step 2;
[0068] Step 2: Copy two copies of the last classification module in the framework and arrange them in parallel after the feature extractor as two mutually exclusive classifiers, and proceed to step 3;
[0069] Step 3: Input the source domain and target domain images into the network for joint training and calculate the segmentation loss:
[0070]
[0071] Among them, sample X s ∈R H×W×3 is an image from the source domain, Y s is its corresponding pixel label, C is the number of categories for semantic segmentation, and P s (h,w,c) is the discrete distribution of C-type semantics on pixels (h, w); Y s (h,w,c) is the correct labeling probability of the C-type semantics on the pixel (h, w). If the pixel (h, w) belongs to the C class, Y s (h,w,c) =1, otherwise Y s (h,w,c) =0.
[0072] Among them, in step 3, calculating the adversarial loss function of the network and training the adaptive network between the two domains includes the following steps:
[0073] Step 1: During the training process, two mutually exclusive classifiers perform pixel-by-pixel semantic classification on the input image. The two classifiers C 1 and C 2With different parameters, minimizing the cosine similarity of two classifiers to enhance the weight divergence of their convolutional layers has the following weight difference loss:
[0074]
[0075] in, and C 1 and C 2 The vector obtained by flattening and concatenating the weights of the convolution filters;
[0076] Step 2: Input the image into the mutually exclusive classifier and output two prediction tensors P (1) and P (2) , calculate the distance difference between the two prediction tensors, and use the distance difference as the weight of the adversarial loss, so that each pixel on the segmentation map can have a different weight to adjust the adversarial loss, and enter step three;
[0077] Step 3: Use adaptive adversarial loss function for training:
[0078]
[0079] Among them, P (1) and P (2) C 1 and C 2 The prediction tensor of M(·,·) is the cosine distance, and the parameter λ local Control the adaptive weight of the adversarial loss, ε is a parameter, G 1 is the inter-domain generator, D 1 is the inter-domain discriminator, X s is the source domain image, X t is the target domain image.
[0080] Wherein, in step 4, dividing the target domain image includes the following steps:
[0081] Step 1: Input the target domain image into the category-level adversarial network trained in the above step, and obtain the prediction image of the target domain image;
[0082] Step 2: Calculate the information entropy value of each pixel in the target domain prediction image:
[0083]
[0084] Among them, the inter-domain generator G 1 Take the target image X t As input, generate pixel-level segmentation prediction value P t (h,w,c) , and perform entropy mapping calculation on it;
[0085] Step 3: Based on step 2, find the entropy mapping value of each image and arrange the values from small to large:
[0086]
[0087] Among them, R(X t ) is the entropy graph I t The average value of the inter-domain adaptation network model is as follows: 1 The generated predicted segmentation map calculates the entropy value of the semantic probability of each pixel being C class And find the average value R(X t ), sort the target domain entropy map in order from low to high, and then make a one-to-one correspondence between the target domain entropy map and the original target domain image;
[0088] Step 4: Introduce hyperparameter λ E As the ratio of the target domain set 1 to the total target domain, the target domain is divided into two parts, that is, the average entropy map of set 1 is low, which means the segmentation effect is good, while the average entropy map of set 2 is high, which means the segmentation effect is poor, and further training is needed for the images of set 2. Let X te and X th denote the target images assigned to set 1 and set 2 respectively, λ E The definition is as follows:
[0089]
[0090] Among them, |X te | is the cardinality of set one, |X t | is the cardinality of the entire target domain.
[0091] Among them, in step 5, the adaptive training in the target domain includes the following steps:
[0092] Step 1: The target domain set 1 divided in the above step is regarded as the source domain dataset, the prediction map of set 1 is used as pseudo-label for auxiliary training, and set 2 is regarded as the target domain dataset. At the same time, set 1 and set 2 are used to train the second inter-domain category-level adversarial network model.
[0093] Among them, in step six, training the overall model and obtaining the segmentation results includes the following steps:
[0094] Step 1: Set the overall loss function L for inter-domain adaptation and intra-domain adaptation as follows:
[0095]
[0096] The target model is learned according to the following formula:
[0097]
[0098] Since a dual-category adversarial network is used, it includes two parameters: inter-domain and intra-domain, which are represented by i=1 and i=2 respectively, that is, L 1 is the overall loss function for inter-domain adaptation, L 2 is the overall loss function of the domain adaptation, G 1 is the inter-domain generator, G 2 is the in-domain generator, D 1 is the inter-domain discriminator, D 2 is the intra-domain discriminator, They are the inter-domain adaptive segmentation loss, weight difference loss and adversarial loss functions, respectively. They are the domain-adaptive segmentation loss, weight difference loss, and adversarial loss functions, respectively. are the hyperparameters for controlling weight difference loss and adversarial loss in inter-domain adaptation, respectively. They are the hyperparameters controlling weight difference loss and adversarial loss in intra-domain adaptation, respectively;
[0099] Step 2: First, perform inter-domain adaptive training on the model to optimize the G 1 and D 1 , Secondly, based on R(X t ) sort the target images and use G 1 Generate pseudo labels for the target domain and train the adaptive model in the domain to optimize G 2 and D 2 ;
[0100] Step 3: Adapt G to cross-domain 1 After the model was trained for 70,000 iterations, G 1 Generate segmentation maps and entropy maps of 2975 images in the Cityscapes training set, using R(X t ) to obtain the segmentation difficulty coefficient of all target domain images, R(X t ) value is large, indicating that segmentation is difficult, and vice versa, indicating that segmentation is easy. The sorting algorithm is used to sort all the images R(X t ) values are sorted from small to large, and combined with λ E Divide the target domain into two subsets, namely R(X t ) value is small and the proportion of set 1 to the total target domain is λ E , R(X t ) value is large and the proportion of set 2 to the total target domain is 1-λ E ;
[0101] Step 4: Train G with the SGD optimizer1 and G 2 , the momentum is 0.9, and the initial learning rate is set to 2.5×10 -4 , weight decay is set to 10 -4 , train D using the Adam optimizer 1 and D 2 , the initial learning rate is 10 -4 , and remain unchanged during training. During training, a 512×1024-sized image is used, and the prediction image is upsampled by 2 times to evaluate the segmentation results;
[0102] Step 5: The experiment uses the detection evaluation function:
[0103]
[0104] The semantic segmentation performance is evaluated for each category, where TP, FP, and FN are the number of true, false positive, and false negative pixels, respectively, and k is the number of categories.
[0105] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in the relevant technical field, are also included in the patent protection scope of the present invention.
Claims
1. A method for image semantic segmentation based on a dual category-level adversarial network, characterized in that: The method comprises the following steps: Step 1: Obtain an image dataset. Obtain a synthetic image dataset GTA5 (video game dataset) and a virtual city dataset SYNTHIA for the source domain. Obtain a real image dataset Cityscapes (city landscape dataset) for the target domain. The dataset consists of a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the generalization ability of the model. Step 2: construct an inter-domain category-level adversarial network, which consists of an inter-domain generator and an inter-domain discriminator, wherein the inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The source domain and target domain images in step 1 are input into the inter-domain category-level adversarial network for collaborative training, and the difference between the source domain and the target domain is calculated. Step 3: Calculate the adversarial loss function of the network and train the adaptive network between the two domains: After the input image is operated in step 2, two prediction tensors are obtained through two mutually exclusive classifiers. The distance difference between the two tensors is calculated to adjust the adversarial loss size, and the adversarial loss function is used to train the category-level adversarial network between the source domain and the target domain. Step 4: Divide the target domain image, use the network trained in step 3 to output the target prediction map, calculate the information entropy value of the prediction map, sort it according to the entropy value, and use the ratio λ E The target domain is divided into two parts: set 1 and set 2. The entropy of set 1 is smaller than that of set 2, and the proportion of set 1 to the total target domain is λ E ; Step 5: Adaptive training in the target domain. Set 1 in step 4 is regarded as the source domain, its predicted image is used as the pseudo label, and set 2 is regarded as the target domain. The category-level adversarial network is used for adaptive training in the domain. Step 6: Train the overall model and obtain the segmentation results. Input the test set images in Cityscapes into the network trained in step 5 to achieve semantic segmentation of the test set images. In step 4, dividing the target domain image includes the following steps: Step 1: Input the target domain image into the category-level adversarial network trained in the above step, and obtain the prediction image of the target domain image; Step 2: Calculate the information entropy value of each pixel in the target domain prediction image: Among them, the inter-domain generator G1 takes the target image X t As input, generate pixel-level segmentation prediction value P t (h,w,c) , and perform entropy mapping calculation on it; Step 3: Based on step 2, find the entropy mapping value of each image and arrange the values from small to large: Among them, R(X t ) is the entropy graph I t The average value of H and W is the height and width of the image, h and w are the height and width of each pixel. When the inter-domain adaptation network model reaches the optimal value, the entropy value I of the semantic probability of each pixel being class C is calculated according to the predicted segmentation map generated by the inter-domain generator G1. t (h,w,c) , and find the average value R(X t ), sort the target domain entropy map in order from low to high, and then make a one-to-one correspondence between the target domain entropy map and the original target domain image; Step 4: Introduce hyperparameter λ E As the ratio of target domain set 1 to the total target domain, the target domain is divided into two parts, namely set 1 and set 2. Moreover, the average entropy map of set 2 is higher than the average entropy map of set 1. It is necessary to further train the images of set 2. Let X te and X th denote the target images assigned to set 1 and set 2 respectively, λ E The definition is as follows: Among them, |X te | is the cardinality of set one, |X t | is the cardinality of the entire target domain; In step six, training the overall model and obtaining the segmentation results includes the following steps: Step 1: Set the overall loss function L for inter-domain adaptation and intra-domain adaptation as follows: The target model is learned according to the following formula: Since a dual category-level adversarial network is used, it includes two parameters, inter-domain and intra-domain, which are represented by i=1 and i=2 respectively. That is, L1 is the overall loss function of inter-domain adaptation, L2 is the overall loss function of intra-domain adaptation, G1 is the inter-domain generator, G2 is the intra-domain generator, D1 is the inter-domain discriminator, and D2 is the intra-domain discriminator. They are the inter-domain adaptive segmentation loss, weight difference loss and adversarial loss functions, respectively. They are the adaptive segmentation loss, weight difference loss and adversarial loss functions in the domain, λ1 weight ,λ1 adv are the hyperparameters for controlling weight difference loss and adversarial loss in inter-domain adaptation, λ2 weight ,λ2 adv They are the hyperparameters controlling weight difference loss and adversarial loss in intra-domain adaptation, respectively; Step 2: First, the model is trained adaptively across domains to optimize the G1 and D1 of the model. t ) Sort the target images, generate pseudo labels for the target domain using G1, and train the in-domain adaptive model to optimize G2 and D2; Step 3: After training the cross-domain adaptation G1 model for 70,000 iterations, G1 was used to generate segmentation maps and entropy maps of 2975 images in the Cityscapes training set. t ) to obtain the segmentation difficulty coefficients of all target domain images, and use the sorting algorithm to sort the R(X t ) values are sorted from small to large, and combined with λ E The target domain is divided into two subsets, namely, set 1 and set 2, and R(X t ) value is greater than R(X t ) value, the proportion of set 1 to the total target domain is λ E , the proportion of set 2 to the total target domain is 1-λ E ; Step 4: Train G1 and G2 with SGD optimizer, momentum is 0.9, and initial learning rate is set to 2.5×10 -4 , weight decay is set to 10 -4 , use Adam optimizer to train D1 and D2, with an initial learning rate of 10 -4 , and remain unchanged during training. During training, a 512×1024-sized image is used, and the prediction image is upsampled by 2 times to evaluate the segmentation results; Step 5: The experiment uses the detection evaluation function: The semantic segmentation performance is evaluated for each category, where TP, FP, and FN are the number of true, false positive, and false negative pixels, respectively, and k is the number of categories.
2. The image semantic segmentation method based on a dual category-level adversarial network according to claim 1, characterized in that: The step of obtaining the image dataset described in step 1 includes the following steps: Step 1: The synthetic image datasets GTA5 and SYNTHIA used in the source domain are both synthetic virtual data. The datasets contain image information and pixel-by-pixel semantic information as segmentation labels; Step 2: The target domain dataset Cityscapes is real data of real scenes. The dataset contains a training set of 2975 images and a test set of 500 images. The target domain data does not contain any label information.
3. The image semantic segmentation method based on a dual category-level adversarial network according to claim 1, characterized in that: In step 2, the inter-domain category-level adversarial network is constructed through the following steps: Step 1: The inter-domain generator consists of a feature extractor and two mutually exclusive classifiers. The pre-trained framework of the residual network ResNet-101 on the ImageNet dataset is used as the backbone of the inter-domain generator network, that is, the first half of the framework is the feature extractor, and then proceed to step 2; Step 2: Copy two copies of the last classification module in the framework and arrange them in parallel after the feature extractor as two mutually exclusive classifiers, and proceed to step 3; Step 3: Input the source domain and target domain images into the network for joint training and calculate the segmentation loss: Among them, sample X s ∈R H×W×3 is an image from the source domain, Y s is its corresponding pixel label, C is the number of categories for semantic segmentation, and P s (h,w,c) is the discrete distribution of C-type semantics on pixels (h, w); Y s (h,w,c) is the correct labeling probability of the semantics of category C on pixel (h, w). If the pixel (h, w) belongs to category C, Y s (h,w,c) =1, otherwise Y s (h,w,c) =0.
4. The image semantic segmentation method based on a dual category-level adversarial network according to claim 1, characterized in that: In step 3, calculating the adversarial loss function of the network and training the adaptive network between the two domains includes the following steps: Step 1: During the training process, two mutually exclusive classifiers perform pixel-by-pixel semantic classification on the input image. The two classifiers C1 and C2 have different parameters. The weight divergence of their convolutional layers is enhanced by minimizing the cosine similarity of the two classifiers. The weight difference loss is as follows: in, and is the vector obtained by flattening and concatenating the weights of the convolution filters of C1 and C2 respectively; Step 2: Input the image into the mutually exclusive classifier and output two prediction tensors P (1) and P (2) , calculate the distance difference between the two prediction tensors, and use the distance difference as the weight of the adversarial loss, so that each pixel on the segmentation map can have a different weight to adjust the adversarial loss, and enter step three; Step 3: Use adaptive adversarial loss function for training: Among them, P (1) and P (2) are the prediction tensors of C1 and C2 respectively, M(·,·) is the cosine distance, and the parameter λ local The adaptive weight that controls the adversarial loss, ε is a parameter, G1 is the inter-domain generator, D1 is the inter-domain discriminator, X s is the source domain image, X t is the target domain image.
5. The image semantic segmentation method based on dual category-level adversarial network according to claim 1, characterized in that: In step 5, the adaptive training in the target domain includes the following steps: Step 1: The target domain set 1 divided in the above step is regarded as the source domain dataset, the prediction map of set 1 is used as the pseudo label for auxiliary training, and set 2 is regarded as the target domain dataset. At the same time, set 1 and set 2 are used to train the second inter-domain category-level adversarial network model.
