A few-shot semantic segmentation method and system based on domain-invariant semantic class prototypes

Abstract

The present invention discloses a domain-invariant semantic class prototype-based few-shot semantic segmentation method and system. The method first extracts features of target domain images and source domain images separately through deep learning; then, through masked average pooling, the average feature representation of a certain class in the target domain image is obtained as the target class prototype; secondly, the source domain image is segmented by searching for the target class prototype closest to each pixel in the source domain image, and a semantic feature network is trained by labeling the source domain image; finally, the domain-invariant class prototype is extracted using the trained semantic feature extraction network, and the target domain image is segmented through prototype-based semantic segmentation, i.e., calculating the most similar class prototype for each pixel. The semantic segmentation migration method of the present invention has the outstanding advantages of simple training and strong versatility.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a few-sample semantic segmentation method and system based on domain-invariant semantic class prototypes. Background Art

[0002] Semantic segmentation aims to assign semantic labels (such as cars, pedestrians, etc.) to each pixel. It is one of the main research difficulties and hot spots in the field of computer vision. It is widely used in fields such as robot visual perception and unmanned driving. Driven by deep learning, the accuracy of semantic segmentation has been greatly improved in recent years, but deep learning is highly dependent on manually labeled samples. Faced with the scarcity of labeled samples in practical applications, how to achieve high-precision semantic segmentation has become a difficult problem that needs to be broken through. Currently, a more popular method is to use a source domain (such as virtual data) where annotations can be easily obtained to train the semantic segmentation model, and then migrate it to the target domain (real scene) to achieve semantic segmentation. However, due to the distribution difference between the source domain and the target domain, the segmentation accuracy drops significantly when the model is migrated. The current solutions to avoid this problem mainly include self-training-based methods and adversarial training-based methods. Among them, the self-training scheme uses the model trained in the source domain to generate pseudo labels for the target domain images, and then uses the pseudo labels to realize the training of target domain image segmentation; the adversarial training scheme uses adversarial training to make the semantic segmentation feature extraction network unable to distinguish the domain attributes of the input; however, both of the above schemes involve relatively complex and tedious training processes. For example, the self-training method requires multiple iterative training, and the adversarial loss in adversarial training is difficult to converge and is sensitive to hyperparameters.

[0003] Due to the large scale of the source domain dataset in semantic segmentation model migration, how to improve the migration efficiency of the migration method, that is, to reduce the cost of network model training, is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0004] The present invention proposes a few-shot semantic segmentation method and system based on domain-invariant semantic class prototypes, which learns and extracts domain-invariant prototype representations of semantic classes from a small number of target domain annotated samples and a large number of source domain annotated samples, thereby achieving cross-domain adaptive semantic segmentation.

[0005] The technical solutions provided by the present invention are as follows:

[0006] On the one hand, a domain-invariant semantic class prototype-based few-shot semantic segmentation method comprises the following steps:

[0007] Step 1: Obtain source domain and target domain sample datasets;

[0008] Select all source domain images for semantic labeling to obtain a source domain sample dataset, and select no more than one-tenth of the target domain images from the collected target domain images for semantic labeling to obtain a target domain sample dataset;

[0009] After the image is annotated with semantic labels, a semantic label image is obtained, and the pixel values ​​in the image are the semantic label values;

[0010] Step 2: Use the semantic feature extraction network to extract the semantic feature matrix;

[0011] Step 3: Use the target domain images in the target domain sample dataset to obtain the semantic class domain invariant feature prototype matrix;

[0012] The semantic class domain invariant feature prototype matrix is ​​obtained by assigning values ​​to the target domain images in the target domain sample data set based on the semantic feature extraction network, and the corresponding semantic label images, and then performing binary mask calculation on each semantic class according to the semantic label index;

[0013] Step 4: Using the data obtained in steps 2 and 3, input the prototype-based semantic segmentation model to perform prototype-based semantic segmentation;

[0014] Step 5: Train the semantic feature extraction network;

[0015] The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network.

[0016] The semantic segmentation loss is calculated using the semantic segmentation results obtained by the prototype-based semantic segmentation model;

[0017] The semantic segmentation loss is calculated based on the semantic segmentation result obtained by the prototype semantic segmentation model;

[0018] The semantic segmentation loss is a loss that uses the semantic segmentation results and the semantic label image G of the source domain image in the source domain and target domain sample datasets. s , the semantic label image G of the target domain image T The semantic segmentation result is obtained by performing prototype-based semantic segmentation on the source domain image and the target domain image in the source domain and the target domain sample data set;

[0019] Step 6: Traverse the source domain images and target domain images in the source domain and target domain sample datasets, iterate steps 2 to 5, and iterate for the specified number of times;

[0020] Step 7: Use the iteratively trained semantic feature extraction network to perform steps 2-3 on the target domain images in the target domain sample dataset to obtain the final semantic class domain invariant feature prototype matrix, and input it into the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

[0021] Source domain images are images with low manual labeling costs or automatically acquired labels (such as virtual data). Target domain images are collected images to be segmented and require high manual labeling costs (such as real images from different applications). We manually annotate a small number of target domain images with semantic labels and design a domain-invariant "class prototype" representation extraction method. This solves the problem that existing semantic segmentation model transfer is only suitable for target domain segmentation and greatly improves transfer efficiency.

[0022] When performing semantic segmentation on the image to be segmented, the updated semantic feature extraction network is first used to extract semantic features from the image to be segmented, and the semantic class domain invariant feature prototype matrix of the final target domain image is determined, and then input into the prototype-based semantic segmentation model for semantic segmentation.

[0023] Furthermore, the extraction of semantic feature matrix using semantic feature extraction network refers to:

[0024] Step 2-1: Random sampling of images;

[0025] Randomly sample traversal images and corresponding label sample pairs from the source domain image sample dataset and the target domain image sample dataset (I S ,G S )、(I T ,G T ), G S and G T Represents the semantic label images of the source domain image and the target domain image respectively; I S represents the source domain image, I T Represents the target domain image;

[0026] Step 2-2: Image feature extraction;

[0027] The semantic feature extraction network PSPNet is constructed using the same deep image classification network to extract semantic features of the source domain image and the target domain image respectively, and the semantic feature matrix of the source domain image and the semantic feature matrix of the target domain image are obtained.

[0028] Furthermore, the semantic class domain invariant feature prototype matrix of the target domain image in the target domain sample dataset is obtained based on the semantic feature matrix of the target domain image in the target domain sample dataset obtained by the semantic feature extraction network, and the corresponding semantic label image, and is obtained by performing binary mask calculation on each semantic class according to the semantic label index assignment, which specifically includes the following steps:

[0029] Step 3-1: Extract the label set contained in the target domain label image;

[0030] Find the semantic classes that appear in the semantic label image of the target domain image, and put the pixel semantic label values ​​that appear into the semantic label set C;

[0031] Step 3-2: Label replacement of source domain image and target domain image;

[0032] In the semantic label images of the source domain image and the target domain image, find the index of the semantic label value corresponding to each pixel in the semantic label set C, and then replace the semantic label value of the searched pixel with the corresponding index value to obtain the new semantic label images of the source domain and target domain images; if the semantic label of the searched pixel is not in the semantic label set C, replace the pixel label value with the invalid label value 255;

[0033] Step 3-3: Calculate the binary mask of each semantic class of the target domain image;

[0034] For each semantic class c in the semantic label set C, set the labels in the semantic label image of the target domain image that are not equal to the semantic class c to 0, and set the labels that are equal to the semantic class c to 1, and obtain the binary mask matrix of the target domain image in each semantic class c;

[0035] Step 3-4: Mask average pooling;

[0036] Multiply the binary mask matrix of the target domain image in each semantic class c with the semantic feature matrix of the target domain image to obtain the semantic feature subset of all pixels of semantic class c, and perform average pooling to obtain the semantic class domain invariant feature prototype p c , and finally obtain the semantic class domain invariant feature prototype matrix of all pixels under all semantic types.

[0037] Furthermore, the specific process of semantic segmentation of the prototype-based semantic segmentation model is as follows:

[0038] Step 4-1: For the semantically segmented image, perform prototype-based cosine similarity calculation;

[0039] For the semantic feature matrix of the semantically segmented image, the cosine similarity score between the pixel semantic feature and the corresponding invariant feature prototype of each semantic class domain is calculated pixel by pixel:

[0040]

[0041] Among them, ξ(p c ,f h,w ) represents f h,w is the eigenvector at the coordinate (h,w) of the semantic feature matrix space and pc The cosine similarity score of h, w is the spatial coordinate index of a pixel in the semantic feature matrix, f h,w is the eigenvector at the coordinate (h,w) of the semantic feature matrix space, p c is the domain-invariant feature prototype vector of semantic class c, ∈ is a small constant;

[0042] ∈ is usually set to 0.0001 to avoid the situation where the denominator is 0;

[0043] Step 4-2: The similarity score in step 4-1 is converted into the probability of each class to which the pixel belongs through the softmax function, and semantic segmentation is performed on each pixel based on the maximum probability:

[0044]

[0045] in, is the probability that a pixel (h, w) in the source domain image or the target domain image belongs to the semantic class c, N is the size of the semantic label set C, and exp is the exponential function.

[0046] Furthermore, the semantic segmentation loss is obtained by calculating the cross entropy loss pixel by pixel and calculating the average value of the cross entropy loss of all pixels;

[0047] Based on the probability of each semantic class to which the pixel output in step 4-2 belongs, and the semantic label images of the new source domain and target domain images output in step 3-2, the cross entropy loss is calculated pixel by pixel and the average is taken as the final cross entropy loss;

[0048] The cross entropy loss for each pixel is calculated as:

[0049] in, It is the probability that the pixel (h, w) in the segmentation result of the image to be segmented belongs to its semantic class true label g according to the corresponding semantic label image.

[0050] Furthermore, the semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network.

[0051] The calculation is as follows:

[0052] θ:=θ-αdθ

[0053] Among them, θ is the semantic feature extraction network parameter, dθ is the gradient, and α is the gradient descent rate.

[0054] Furthermore, the deep image classification network is VGG16 or ResNet.

[0055] On the other hand, a few-shot semantic segmentation system based on domain-invariant semantic class prototypes includes:

[0056] Obtaining source domain and target domain sample data set unit: used to select all source domain images for label sample annotation to obtain source domain sample data set, and select a small number of target domain images for pixel-level annotation to obtain target domain sample data set;

[0057] Semantic feature matrix acquisition unit: uses the semantic feature extraction network to extract semantic features from the source domain and target domain images in the source domain and target domain sample data sets at the same time, and obtains the source domain image semantic feature matrix and the target domain image semantic feature matrix;

[0058] Domain-invariant prototype matrix acquisition unit: Based on the semantic feature matrix and semantic label image of the target domain image obtained by the semantic feature matrix acquisition unit, the semantic label index is assigned and binary mask calculation is performed according to each semantic class to obtain the domain-invariant feature prototype matrix of the target domain image in the target domain sample dataset;

[0059] Semantic feature extraction network training unit: The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network.

[0060] The semantic segmentation loss is calculated based on the semantic segmentation result obtained by the prototype semantic segmentation model;

[0061] Iteration unit: used to traverse the source domain images and target domain images in the source domain and target domain sample data sets, iteratively execute the semantic feature matrix acquisition unit, domain invariant prototype matrix acquisition unit and semantic feature extraction network training unit, and iterate to a specified number of times;

[0062] Semantic segmentation unit: The target domain images in the target domain sample dataset are extracted using the semantic feature extraction network that has been iteratively trained by the iterative unit to obtain the target domain semantic class domain invariant feature prototype matrix, and then input it into the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

[0063] In another aspect, an electronic device includes:

[0064] one or more processors;

[0065] a memory storing one or more computer programs;

[0066] The processor calls the computer program to implement the above-mentioned domain-invariant semantic class prototype-based few-sample semantic segmentation method.

[0067] In another aspect, a readable storage medium stores a computer program, wherein the computer program is invoked by a processor to implement:

[0068] The above is a few-shot semantic segmentation method based on domain-invariant semantic class prototypes.

[0069] Beneficial effects

[0070] The technical solution of the present invention proposes a few-sample semantic segmentation method and system based on domain-invariant semantic class prototypes, a semantic segmentation model migration method based on domain-invariant "class prototype" under few samples, firstly extracting the features of the target domain image and the source domain image respectively through deep learning; then obtaining the average feature representation of a certain class of the target domain image as the target class prototype through mask average pooling; secondly, searching for the target class prototype closest to the pixel on the source domain image pixel by pixel to realize source domain image segmentation, and training the semantic feature extraction network through source domain image labeling; finally, domain-invariant class prototype extraction is realized through the trained semantic feature extraction network, and the target domain image is segmented through prototype-based semantic segmentation, that is, calculating the most similar class prototype of each pixel; the semantic segmentation migration method of the present invention has the outstanding advantages of simple training and good versatility.

[0071] The technical solution provided by the present invention can realize the multi-domain generalization of the semantic segmentation model with only one training.

[0072] Compared with existing semantic segmentation migration methods, the advantages of this invention are:

[0073] 1) We propose a domain-invariant “class prototype” representation extraction method, which solves the problem that existing semantic segmentation model migration is only suitable for target domain segmentation, greatly improving the migration efficiency;

[0074] 2) A novel cross-domain migration framework for single-stage semantic segmentation models is proposed. Compared with self-training methods, it does not require the model to be pre-trained in the source domain and self-trained for multiple iterations. Compared with adversarial training methods, it avoids the problem of difficult training convergence caused by using adversarial loss.

[0075] 3) A more general few-shot semantic segmentation model is proposed. Existing few-shot semantic segmentation models typically consider images containing only a small number of objects, making them difficult to apply to images containing an arbitrary number of objects. The few-shot semantic segmentation method proposed in this paper can be applied to any image. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a schematic diagram of the execution flow of the method described in the technical solution of the present invention;

[0077] Figure 2 This is an overall block diagram of the method described in the technical solution of the present invention;

[0078] Figure 3Schematic diagram of the innovation of the method described in the technical solution of the present invention and the existing method, wherein (a) is the self-training method, (b) is the adversarial training method, and (c) is the method described in the technical solution of the present invention. DETAILED DESCRIPTION

[0079] The present invention will be further described in detail below with reference to the accompanying drawings.

[0080] The source domain involved in the embodiment of the present invention may be a virtual traffic scene dataset GTA5, and the target domain dataset may be a city road dataset Cityscapes.

[0081] A few-shot semantic segmentation method based on domain-invariant semantic class prototypes, such as Figure 1-Figure 3 The specific steps are as follows:

[0082] Step 1: Prepare source and target domain data.

[0083] Select GTA5 source domain labeled samples to obtain the source domain sample dataset D S , select a small number of target domain Cityscapes image annotation samples to obtain a small number of target domain sample dataset D T ;

[0084] Step 2: Extract source domain and target domain image features based on the Siamese network.

[0085] Using the same deep feature extraction network, we act on the source domain image I S And the target domain image I T , get the source domain image semantic feature matrix F S And the target domain image semantic feature matrix F T ;

[0086] Step 2-1: Random sampling of images;

[0087] Randomly sample image and label sample pairs from the source domain sample dataset and the few-sample target domain sample dataset (I S ,G S )、(I T ,G T );

[0088] Step 2-2: Image feature extraction;

[0089] Use VGG16 or ResNet network to build semantic feature extraction network PSPNet and apply it to I S ,I T , and get the semantic feature F S ,F T .

[0090] Step 3: Based on the target domain image semantic features F obtained in step 2 T , using the target domain images in the target domain sample dataset to obtain the semantic class domain invariant feature prototype matrix, the specific steps are as follows:

[0091] The acquisition of the semantic class domain invariant feature prototype matrix includes the following steps:

[0092] Step 3-1: Extract the label set contained in the target domain label image;

[0093] Find the semantic classes that appear in the semantic label image of the target domain image, and put the pixel semantic label values ​​that appear into the semantic label set C;

[0094] Step 3-2: Label replacement of source domain image and target domain image;

[0095] In the semantic label images of the source domain image and the target domain image, find the index of the semantic label value corresponding to each pixel in the semantic label set C, and then replace the semantic label value of the searched pixel with the corresponding index value to obtain the new semantic label images of the source domain and target domain images; if the semantic label of the searched pixel is not in the semantic label set C, replace the pixel label value with the invalid label value 255;

[0096] Step 3-3: Calculate the binary mask of each semantic class of the target domain image;

[0097] For each semantic class c in the semantic label set C, set the labels in the semantic label image of the target domain image that are not equal to the semantic class c to 0, and set the labels that are equal to the semantic class c to 1, and obtain the binary mask matrix of the target domain image in each semantic class c;

[0098] Step 3-4: Mask average pooling;

[0099] Multiply the binary mask matrix of the target domain image in each semantic class c with the semantic feature matrix of the target domain image to obtain the semantic feature subset of all pixels of semantic class c, and perform average pooling to obtain the semantic class domain invariant feature prototype p c , and finally obtain the semantic class domain invariant feature prototype matrix of all pixels under all semantic types.

[0100] Step 4: Using the data obtained in steps 2 and 3, input the prototype-based semantic segmentation model to perform prototype-based semantic segmentation;

[0101] Step 4-1: Semantic segmentation of source domain images based on prototype similarity;

[0102] Given the source domain image and target domain image in step 2, output the semantic feature F S , F T, calculate the cosine similarity score of pixel features and the feature prototypes output in step 3-3 pixel by pixel. The cosine similarity is calculated as follows:

[0103]

[0104] where f h,w is the feature vector of the pixel with spatial coordinates (h, w), p c is the feature prototype vector of semantic type c, ∈ is a small constant;

[0105] Step 4-2: The similarity score is converted into the probability of each class to which the pixel belongs through the softmax function, which is calculated as follows:

[0106]

[0107] is the probability that pixel (h, w) belongs to class c, and N is the size of set C in step 3-1;

[0108] Step 4-3: cross entropy loss calculation;

[0109] Given step 4-1 outputs the semantic segmentation result, and step 3-2 outputs the source domain and target domain label images G' S , G' T , calculate the cross entropy loss pixel by pixel and take the average as the final cross entropy loss. The cross entropy loss of each pixel is calculated as follows.

[0110]

[0111] in, is the probability that the pixel (h, w) in the segmentation result of the image to be segmented belongs to its semantic class true label g according to the corresponding semantic label image;

[0112] Step 5: Train the semantic feature extraction network;

[0113] The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network.

[0114] The semantic segmentation loss is calculated using the semantic segmentation results obtained by the prototype-based semantic segmentation model;

[0115] The semantic segmentation loss is a loss that uses the semantic segmentation results and the semantic label image G of the source domain image in the source domain and target domain sample datasets. s , the semantic label image G of the target domain image TThe semantic segmentation result is obtained by performing prototype-based semantic segmentation on the source domain image and the target domain image in the source domain and the target domain sample data set;

[0116] The parameter gradient of the semantic feature extraction network is calculated using the semantic segmentation loss as follows:

[0117] θ:=θ-αdθ

[0118] Among them, θ is the parameter of the semantic feature extraction network PSPNet, dθ is the gradient, and α is the gradient descent rate.

[0119] Step 6: Traverse the source domain images and target domain images in the source domain and target domain sample datasets, iterate steps 2 to 5, and iterate for the specified number of times;

[0120] Step 7: Use the iteratively trained semantic feature extraction network to extract the target domain images in the target domain sample dataset and execute steps 2-3 to obtain the final semantic class domain invariant feature prototype matrix. Then input the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

[0121] Step 7-1: Extract semantic features of target domain support samples;

[0122] Given a few-shot target domain dataset D T , for each image Use the iteratively trained semantic feature extraction network to extract its semantic features

[0123] Step 7-2: Calculation of target domain semantic class prototype features;

[0124] Use steps 3-3 and 3-4 to Extract the semantic feature prototypes, and finally, for each semantic class c, Obtaining semantic feature prototypes Take the average and get the final semantic class domain invariant feature prototype of this class.

[0125] Step 7-3: Prototype-based semantic segmentation.

[0126] For the image to be semantically segmented, prototype-based semantic segmentation is implemented using steps 4-1 and 4-2 according to the semantic class domain invariant feature prototype obtained in step 7-2.

[0127] At this point, the domain adaptation method of the semantic segmentation model based on the domain-invariant "class prototype" under few samples has been completed.

[0128] This embodiment further provides a few-shot semantic segmentation system based on domain-invariant semantic class prototypes, comprising:

[0129] Obtaining source domain and target domain sample data set unit: used to select all source domain images for label sample annotation to obtain source domain sample data set, and select a small number of target domain images for pixel-level annotation to obtain target domain sample data set;

[0130] Semantic feature matrix acquisition unit: uses the semantic feature extraction network to extract semantic features from the source domain and target domain images in the source domain and target domain sample data sets at the same time, and obtains the source domain image semantic feature matrix and the target domain image semantic feature matrix;

[0131] Domain-invariant prototype matrix acquisition unit: Based on the semantic feature matrix and semantic label image of the target domain image obtained by the semantic feature matrix acquisition unit, it assigns values ​​according to the semantic label index and performs binary mask calculation according to each semantic class to obtain the semantic class domain-invariant feature prototype matrix of the target domain image;

[0132] The semantic class domain invariant feature prototype matrix is ​​obtained by assigning values ​​to the target domain images in the target domain sample data set based on the semantic feature extraction network, and the corresponding semantic label images, and then performing binary mask calculation on each semantic class according to the semantic label index;

[0133] Semantic feature extraction network training unit: The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network.

[0134] The semantic segmentation loss is calculated based on the semantic segmentation result obtained by the prototype semantic segmentation model;

[0135] Iteration unit: used to traverse the source domain images and target domain images in the source domain and target domain sample data sets, iteratively execute the semantic feature matrix acquisition unit, domain invariant prototype matrix acquisition unit and semantic feature extraction network training unit, and iterate to a specified number of times;

[0136] Semantic segmentation unit: The target domain images in the target domain sample dataset are extracted using the semantic feature extraction network that has been iteratively trained by the iterative unit to obtain the target domain semantic class domain invariant feature prototype matrix, and then input it into the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

[0137] Please refer to the above-mentioned methods for the specific implementation process of each module, and will not be elaborated here. It should be understood that the above-mentioned division of functional modules is merely a division of logical functions. In actual implementation, other division methods can be used. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented. At the same time, the above-mentioned integrated units can be implemented in the form of hardware or software functional units.

[0138] This embodiment provides an electronic device, including:

[0139] one or more processors;

[0140] a memory storing one or more computer programs;

[0141] The processor calls the computer program to implement the above-mentioned domain-invariant semantic class prototype-based few-sample semantic segmentation method.

[0142] This embodiment further provides a readable storage medium storing a computer program, wherein the computer program is invoked by a processor to implement:

[0143] The above is a few-shot semantic segmentation method based on domain-invariant semantic class prototypes.

[0144] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the controller described in any of the aforementioned embodiments, such as a hard disk or memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0145] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0147] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A domain-invariant semantic class prototype-based few-shot semantic segmentation method, characterized by: The following steps are involved: Step 1: Obtain source domain and target domain sample datasets; All source domain images are selected for semantic labeling to obtain a source domain sample dataset; no more than one tenth of the target domain images are selected for semantic labeling from the collected target domain images to obtain a target domain sample dataset; Step 2: Use the semantic feature extraction network to extract the semantic feature matrix; Step 3: Use the target domain images in the target domain sample dataset to obtain the semantic class domain invariant feature prototype matrix; The semantic class domain invariant feature prototype matrix is ​​obtained by assigning values ​​to the target domain images in the target domain sample data set based on the semantic feature extraction network, and the corresponding semantic label images, and then performing binary mask calculation on each semantic class according to the semantic label index; Step 4: Use the data obtained in steps 2 and 3 to input the prototype-based semantic segmentation model to perform prototype-based semantic segmentation; Step 5: Train the semantic feature extraction network; The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network. The semantic segmentation loss is calculated using the semantic segmentation results obtained by the prototype-based semantic segmentation model; Step 6: Traverse the source domain images and target domain images in the source domain and target domain sample datasets, iterate steps 2 to 5, and iterate for the specified number of times; Step 7: Use the iteratively trained semantic feature extraction network to perform steps 2-3 on the target domain images in the target domain sample dataset to obtain the final semantic class domain invariant feature prototype matrix, and input it into the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

2. The method according to claim 1, characterized in that The use of a semantic feature extraction network to extract a semantic feature matrix refers to: Step 2-1: Random sampling of images; Randomly sample traversal images and corresponding label sample pairs from the source domain image sample dataset and the target domain image sample dataset (I S ,G S )、(I T ,G T ), G S and G T Represents the semantic label images of the source domain image and the target domain image respectively; I S represents the source domain image, I T Represents the target domain image; Step 2-2: Image feature extraction; The semantic feature extraction network PSPNet is constructed using the same deep image classification network to extract semantic features of the source domain image and the target domain image respectively, and the semantic feature matrix of the source domain image and the semantic feature matrix of the target domain image are obtained.

3. The method according to claim 1, characterized in that The semantic class domain invariant feature prototype matrix is ​​obtained by assigning values ​​to the target domain images in the target domain sample data set based on the semantic feature extraction network, and the corresponding semantic label images, and then performing binary mask calculation on each semantic class according to the semantic label index; The specific steps include: Step 3-1: Extract the label set contained in the target domain label image; Find the semantic classes that appear in the semantic label image of the target domain image, and put the semantic label values ​​of the pixels that appear into the semantic label set C; Step 3-2: Label replacement of source domain image and target domain image; In the semantic label images of the source domain image and the target domain image, find the index of the semantic label value corresponding to each pixel in the semantic label set C, and then replace the semantic label value of the searched pixel with the corresponding index value to obtain the new semantic label images of the source domain and target domain images; if the semantic label of the searched pixel is not in the semantic label set C, replace the semantic label value of the pixel with the invalid label value 255; Step 3-3: Calculate the binary mask of each semantic class of the target domain image; For each semantic class c in the semantic label set C, set the labels in the semantic label image of the target domain image that are not equal to the semantic class c to 0, and set the labels that are equal to the semantic class c to 1, and obtain the binary mask matrix of the target domain image in each semantic class c; Step 3-4: Mask average pooling; Multiply the binary mask matrix of the target domain image in each semantic class c with the semantic feature matrix of the target domain image to obtain the semantic feature subset of all pixels of semantic class c, and perform average pooling to obtain the semantic class domain invariant feature prototype p c , and finally obtain the semantic class domain invariant feature prototype matrix of all pixels under all semantic types.

4. The method according to claim 1, characterized in that The specific process of semantic segmentation of the prototype-based semantic segmentation model is as follows: Step 4-1: For the semantically segmented image, perform prototype-based cosine similarity calculation; For the semantic feature matrix of the semantically segmented image, the cosine similarity score between the pixel semantic feature and the corresponding invariant feature prototype of each semantic class domain is calculated pixel by pixel: Among them, ξ(p c ,f h,w ) represents f h,w With p c The cosine similarity score of h, w is the spatial coordinate index of a pixel in the semantic feature matrix, f h,w is the eigenvector at the coordinate (h,w) of the semantic feature matrix space, p c is the domain-invariant feature prototype vector of semantic class c, ∈ is a small constant; Step 4-2: The similarity score in step 4-1 is converted into the probability of each class to which the pixel belongs through the softmax function, and semantic segmentation is performed on each pixel based on the maximum probability: in, is the probability that a pixel (h, w) in the source domain image or the target domain image belongs to the semantic class c, N is the size of the semantic label set C, and exp is the exponential function.

5. The method according to claim 4, characterized in that The semantic segmentation loss is obtained by calculating the cross entropy loss pixel by pixel and then calculating the average value of the cross entropy loss of all pixels; Based on the probability of each semantic class to which the pixel output in step 4-2 belongs, and the semantic label images of the new source domain and target domain images output in step 3-2, the cross entropy loss is calculated pixel by pixel and the average is taken as the final cross entropy loss; The cross entropy loss for each pixel is calculated as: in, It is the probability that the pixel (h, w) in the segmentation result of the image to be segmented belongs to its semantic class true label g according to the corresponding semantic label image.

6. The method according to claim 5, characterized in that The parameter gradient of the semantic feature extraction network is calculated using the semantic segmentation loss as follows: θ:=θ-αdθ Among them, θ is the semantic feature extraction network parameter, dθ is the gradient, and α is the gradient descent rate.

7. The method according to claim 2, characterized in that The deep image classification network is VGG16 or ResNet.

8. A domain-invariant semantic class prototype-based few-shot semantic segmentation system, characterized by: Include: Obtain source domain and target domain sample data set unit: used to select all source domain images for label sample annotation to obtain source domain sample data set; select no more than one tenth of the target domain images for pixel-level annotation to obtain target domain sample data set; Semantic feature matrix acquisition unit: uses the semantic feature extraction network to simultaneously extract semantic features from the source domain and target domain images in the source domain and target domain sample data sets to obtain the source domain image semantic feature matrix and the target domain image semantic feature matrix; Domain-invariant prototype matrix acquisition unit: Based on the semantic feature matrix and semantic label image of the target domain image obtained by the semantic feature matrix acquisition unit, the semantic label index is assigned and binary mask calculation is performed according to each semantic class to obtain the domain-invariant feature prototype matrix of the target domain image in the target domain sample dataset; The semantic class domain invariant feature prototype matrix is ​​obtained by assigning values ​​to the target domain images in the target domain sample data set based on the semantic feature extraction network, and the corresponding semantic label images, and then performing binary mask calculation on each semantic class according to the semantic label index; Semantic feature extraction network training unit: The source domain image and the target domain image are used as the input data of the semantic feature extraction network, and the semantic feature matrix of the source domain image and the target domain image is used as the output data of the semantic feature extraction network. The semantic segmentation loss is used to calculate the parameter gradient of the semantic feature extraction network to train the semantic feature extraction network. The semantic segmentation loss is calculated based on the semantic segmentation result obtained by the prototype semantic segmentation model; Iteration unit: used to traverse the source domain images and target domain images in the source domain and target domain sample data sets, iteratively execute the semantic feature matrix acquisition unit, domain invariant prototype matrix acquisition unit and semantic feature extraction network training unit, and iterate to a specified number of times; Semantic segmentation unit: The target domain image in the target domain sample dataset is extracted using the semantic feature extraction network that has been iteratively trained by the iterative unit to obtain the final semantic class domain invariant feature prototype matrix, and then input it into the prototype-based semantic segmentation model to perform semantic segmentation on the image to be segmented.

9. An electronic device, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: A few-shot semantic segmentation method based on domain-invariant semantic class prototypes as described in claim 1.

10. A readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: A few-shot semantic segmentation method based on domain-invariant semantic class prototypes as described in claim 1.

Images