A zero-shot semantic segmentation method and device based on meaningful learning

Abstract

The application discloses a zero-shot semantic segmentation method and device based on meaningful learning. The method comprises the following steps: S1: pre-training a segmentation model and a generation model by using large-scale known class data with labeled information; S2: obtaining generated visual representation for unknown classes by using the pre-trained generation model and semantic encoding of the unknown classes; S3: randomly selecting samples from visible classes to extract visual representation, and constructing conjugate correlation loss by combining the visual representation with the visual representation of the unknown classes generated in step S2; and S4: performing twice prediction on the generated visual representation of the unknown classes by using a fast classifier and a slow classifier respectively, and calculating corresponding loss functions to realize zero-shot image segmentation. The application pays more attention to the correlation between visible classes and invisible classes, naturally promotes continuous construction of knowledge, and can make the image segmentation effect more superior.

Description

TECHNICAL FIELD

[0001] The application relates to a zero-shot semantic segmentation method based on meaningful learning and also relates to a corresponding zero-shot semantic segmentation device and belongs to the technical field of computer vision. BACKGROUND

[0002] Zero-shot learning is one of the commonly used recognition methods in the field of computer vision. By using the zero-shot learning technology, data categories that have never been seen can be recognized, that is, the trained classifier can not only recognize the data categories in the training set, but also distinguish data from categories that have never been seen. In recent years, zero-shot learning technology has attracted widespread attention, mainly because it can alleviate the lack of labeled samples under the condition of clear semantic boundary and relax the harsh conditions of large-scale data, so as to quickly utilize existing knowledge and generalize to unobserved categories or tasks.

[0003] At present, the mainstream zero-shot learning method mainly includes two kinds, one is a projection-based method, and the other is a generation-based method. The projection-based method is to learn a projection function to project semantic embedding and visual representation into a common hidden space. In the test stage, the projection function is used to project the semantics of the unobserved class to obtain the visual parameters of the unobserved class, so as to perform discrimination. However, the projection-based method usually performs poorly on the general zero-shot learning task which is more suitable for practical applications, because there is no projection constraint of unobserved classes, and the projection of unobserved classes is inevitably affected by the bias of visible classes. On the other hand, the generation-based method mainly uses a generative adversarial network to generate the features of unobserved classes, and uses the labeled features to train a classifier, thereby converting zero-shot learning into a fully supervised task. Although the generation-based method can achieve good results on the zero-shot learning task, most zero-shot learning is for classification tasks, and it is still difficult to achieve good results on dense tasks such as image segmentation.

[0004] In the Chinese invention patent with the patent number ZL 202110093474.5, a zero-shot image semantic segmentation method is disclosed, which is divided into two parts: a class-independent pre-background image segmentation module and a zero-shot target classification module. Among them, the class-independent pre-background image segmentation adopts a two-stage image segmentation framework based on Mask-RCNN, and assists the internal and external edge discriminator, and the edge self-supervision module improves the accuracy of the image pre-background segmentation. The zero-shot target classification module is based on the CADA-VAE algorithm, and assists the Deep Inversion reverse generation visual feature to reduce the domain distance between the visual feature and the semantic feature, and improves the accuracy of the zero-shot target classification. By using this method, after training on known classes, better image segmentation performance can be obtained on unknown class targets, greatly reducing the demand for samples and complex manual labeling, and greatly improving the performance of image semantic segmentation tasks in the scene without samples or with fewer samples. SUMMARY

[0005] The primary technical problem to be solved by the present application is to provide a zero-shot semantic segmentation method based on meaningful learning.

[0006] Another technical problem to be solved by the present application is to provide a zero-shot semantic segmentation device based on meaningful learning.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] According to the first aspect of the embodiment of the present application, a zero-shot semantic segmentation method based on meaningful learning is provided, comprising the following steps:

[0009] S1: pre-training the segmentation model and the generation model by using large-scale known class data with labeled information;

[0010] S2: obtaining the generated visual representation for the unknown class by using the pre-trained generation model and the semantic encoding of the unknown class;

[0011] S3: randomly selecting samples from the visible class to extract visual representation, and constructing a conjugate correlation loss with the visual representation of the unknown class generated in step S2;

[0012] S4: performing two predictions on the generated visual representation of the unknown class through the fast classifier and the slow classifier respectively, and calculating the corresponding loss function to realize zero-shot image segmentation.

[0013] Preferably, in step S1, the semantic image segmentation model used is DeepLab V2, and the discriminator therein is Fast classifier And the slow classifier is realized by two 1x1 convolution layers respectively, and the cognitive controller is realized by a convolution layer.

[0014] wherein preferably, in the step S1, a true semantic graph of the nth visible class data is randomly sampled a label word vector graph is constructed and a randomly sampled noise graph Z n is spliced and input to a generator to obtain a generated visual representation and guide the generated visual representation and the real visual representation to approach each other, thereby establishing a label-visual semantic mapping, wherein is a feature extractor, and n is a positive integer.

[0015] wherein preferably, the generated visual representation and the real visual representation are input to the fast classifier and the discriminator shares the parameters of the first layer with the fast classifier .

[0016] wherein preferably, in the step S2, a semantic encoding matrix is randomly sampled as the semantic encoding of the unknown class according to the semantic encoding of the visible class and the invisible class, and the corresponding label word vector graph is given and input to the pre-trained generator to obtain a generated visual representation for the unknown class.

[0017] wherein preferably, the step S3 specifically comprises the following sub-steps:

[0018] S31, a picture is randomly sampled from the visible class;

[0019] S32, the semantic space correlation between the real vector graph in the semantic space and the random vector is calculated;

[0020] S33, the visual semantic correlation between the real visual representation and the generated visual representation in the visual space is calculated;

[0021] S34, the conjugate correlation loss is calculated according to the semantic space correlation and the visual semantic correlation.

[0022] wherein preferably, the semantic space correlation is the correlation relationship between the ith patch of the real vector graph randomly generated in the semantic space and the jth patch of the real label word vector graph .

[0023] wherein preferably, the visual semantic correlation is the visual semantic correlation between the real visual representation​ the jth patch of and generating visual representations the ith patch of between the visual representations.

[0024] wherein preferably, the step S4 specifically comprises the following sub-steps:

[0025] S41, predicting all visual semantics by a fast classifier;

[0026] S42, evaluating the prediction of the fast classifier by a cognitive controller, and estimating the compatibility between the prediction and the concept of the visible class;

[0027] S43, further predicting the visual semantic representation uncertain by the cognitive controller by a slow classifier;

[0028] S44, calculating the loss function of the weighted classification and the loss function of the cognitive controller.

[0029] According to a second aspect of the embodiments of the present application, a zero-shot semantic segmentation device based on meaningful learning is provided, comprising a processor and a memory, the processor reading a computer program in the memory, for performing the following operations:

[0030] S1: pre-training a segmentation model and a generation model by using large-scale known class data with labeled information;

[0031] S2: obtaining a generated visual representation for an unknown class by using the pre-trained generation model and the semantic encoding of the unknown class;

[0032] S3: randomly selecting a sample from the visible class to extract a visual representation, and constructing a conjugate correlation loss between the visual representation and the visual representation of the unknown class generated in step S2;

[0033] S4: performing twice prediction on the generated visual representation of the unknown class by a fast classifier and a slow classifier respectively, and calculating the corresponding loss function, to realize zero-shot image segmentation.

[0034] Compared with the prior art, the present application has the following technical effects: new concepts can be well learned under the condition of no visual sample, and by constructing a conjugate correlation loss between the visual representation of the visible class and the visual representation of the invisible class, the conflict between the new concept and the existing concept is handled, a fused concept mode is formed, the new concept (invisible class) is associated with the existing concept (visible class) for rapid learning. Compared with the previous zero-shot segmentation method based on projection and generation, the present application pays more attention to the correlation between the visible class and the invisible class, naturally promotes the continuous construction of knowledge, and can make the image segmentation effect more superior. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 A flowchart of the zero-shot semantic segmentation method based on meaningful learning provided by the embodiment of the present application is shown in FIG. 1.

[0036] Figure 2 A flowchart of constructing the conjugate correlation loss between the visual representation of the visible class and the visual representation of the invisible class in the embodiment of the present application is shown in FIG. 2.

[0037] Figure 3 A flowchart of predicting the generated visual representation of the unknown class and calculating the loss function in the embodiment of the present application is shown in FIG. 3.

[0038] FIG. 4(a) is a schematic diagram of an input picture for a matting experiment in the embodiment of the present application.

[0039] FIG. 4(b) is a schematic diagram of the result of matting by SPNet in the embodiment of the present application.

[0040] FIG. 4(c) is a schematic diagram of the result of matting by ZS3Net in the embodiment of the present application.

[0041] FIG. 4(d) is a schematic diagram of the result of matting by CaGNet in the embodiment of the present application.

[0042] FIG. 4(e) is a schematic diagram of the result of matting by the method of the present application in the embodiment of the present application.

[0043] FIG. 4(f) is a schematic diagram of the label of the result of matting by the method of the present application in the embodiment of the present application.

[0044] Figure 5 A structural schematic diagram of the zero-shot semantic segmentation device based on meaningful learning provided by the embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0045] The technical content of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0046] First of all, it needs to be pointed out that in the field of computer vision, the processing object of the present application is mainly image, so it can also be called zero-shot semantic image segmentation method and zero-shot semantic image segmentation device. In order to express simply, they are simply called zero-shot semantic segmentation method and zero-shot semantic segmentation device in the present application.

[0047] Figure 1 A flowchart of the zero-shot semantic segmentation method based on meaningful learning in the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the zero-shot semantic segmentation method at least includes the following steps:

[0048] S1: pre-training the segmentation model and the generation model using large-scale known-class data with annotation information.

[0049] In an embodiment of the present application, the semantic image segmentation model used is DeepLab V2 (further details can be found in the paper DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, at https: / / arxiv.org / abs / 1606.00915). DeepLab V2 mainly consists of two parts, one is the region segmentation network, and the other is the CRF probability graph model for boundary optimization. In addition, the discriminator Fast classifier and slow classifier are implemented using two 1x1 convolutional layers respectively, and share the parameters of the first layer, and the cognitive controller is implemented using a convolutional layer.

[0050] On this basis, for all training data of the visible class (in different embodiments of the present application, specifically, pictures of each class), the true value semantic graph of the nth visible class data is randomly sampled The label word vector graph is constructed and is spliced with the randomly sampled noise graph Z n and is input to the generator to obtain the generated visual representation and guide the generated visual representation to be close to the real visual representation , thereby establishing a label-visual semantic mapping, where is a feature extractor, and n is a positive integer. In addition, the generated visual representation and the real visual representation are sent to the fast classifier and the classification loss is calculated, and the discriminator shares the parameters of the first layer with the fast classifier to distinguish the generated visual representation and the real visual representation as much as possible.

[0051] The generator and the discriminator form a generative adversarial network. The generator generates visual representations that look real but contain noise to deceive the discriminator the discriminator It is needed to determine whether the visual representation is real or generated. Through the above iterative training, the pre-training of the generator feature extractor discriminator and fast classifier is finally completed.

[0052] Through the above pre-training step, an excellent performance of the generation model can be constructed, so as to better learn the real probability distribution of the invisible class (i.e. unknown class, the same below), generate synthetic samples as close as possible to the real invisible class samples, thereby reducing the data imbalance between the visible class and the invisible class, and improving the image classification accuracy.

[0053] S2: using the pre-trained generation model and the semantic encoding of the unknown class, obtaining the generated visual representation for the unknown class.

[0054] In an embodiment of the present application, the input sample can be represented as where H, W and C represent the height, width and channel number of the image respectively. On this basis, according to the semantic encoding of the visible class and the invisible class, a semantic encoding matrix is randomly sampled as the semantic encoding of the unknown class, and the corresponding label vector graph is given and input into the pre-trained generator to obtain the generated visual representation for the unknown class

[0055] S3: randomly selecting a sample from the visible class to extract a visual representation, and constructing a conjugate correlation loss with the visual representation of the unknown class generated in step S2.

[0056] As shown in the figure, in an embodiment of the present application, the above step S3 specifically includes the following sub-steps S31-S34: Figure 2

[0057] S31, randomly sampling a picture from the visible class;

[0058] Specifically, a picture is randomly sampled from the visible class training set, assuming it is the nth picture, and the corresponding real visual representation generated visual representation and label vector graph where n is a positive integer.

[0059] S32, calculating the semantic space correlation between the real vector graph and the random vector in the semantic space;

[0060] In the field of computer vision, semantics refers to the content of an image. The corresponding semantic space correlation is specifically: the real vector graph randomly generated in the semantic space ​the i-th patch of the real visual representation and the real label word vector graph the j-th patch of the generated visual representation The association relationship between the i-th patch of the real visual representation

[0061]

[0062] S33, calculate the visual semantic association of the real visual representation and the generated visual representation in the visual space;

[0063] In the field of computer vision, visual semantics refers to the underlying semantic features commonly understood, such as color, texture, and shape. The corresponding visual semantic association is specifically: the association relationship between the i-th patch of the real visual representation the j-th patch of the generated visual representation The association relationship between the i-th patch of the real visual representation

[0064] S34, calculate the conjugate association loss according to the semantic space association and the visual semantic association.

[0065] In an embodiment of the present application, the conjugate association loss l CON is calculated by the following formula:

[0066]

[0067]

[0068] wherein, α i,j is a pixel-level adaptive edge for fully guaranteeing the visual semantic diversity of different pixel positions. By calculating the conjugate association loss of the semantic space association and the visual semantic association, the invisible class and the visible class can be linked to establish the correlation between the invisible class and the visible class. Wherein, the smaller the conjugate association loss l CON , the stronger the correlation between the invisible class and the visible class.

[0069] S4: the generated visual representation of the unknown class is predicted by the fast classifier and the slow classifier respectively, and the corresponding loss function is calculated to realize zero-shot image segmentation.

[0070] As Figure 3 shown, in an embodiment of the present application, the above step S4 specifically includes sub-steps S41-S44:

[0071] S41, predict all visual semantics by the fast classifier;

[0072] The prediction process expression of the fast classifier is: ​​

[0073]

[0074] wherein, is generated by the generator.

[0075] S42, evaluating the prediction of the fast classifier by the cognitive controller, and estimating the compatibility between the prediction and the concept of the visible class;

[0076] The evaluation expression of the cognitive controller is:

[0077]

[0078] wherein, p m,i ∈[0,1] 2 is a two-dimensional vector, respectively representing the matching degree and the mismatching degree of the existing concept.

[0079] S43, further predicting the visual semantic representation uncertain by the cognitive controller by the slow classifier;

[0080] The secondary prediction expression of the slow classifier is:

[0081]

[0082] wherein, represents the Kronecker symbol function, if p m,i [0]<p m,i [1] is true, the value is 1, otherwise the value is 0.

[0083] S44, calculating the loss function of the weighted classification and the loss function of the cognitive controller.

[0084] In an embodiment of the present application, the binary cross-entropy is used to guide the learning of the cognitive controller, and the loss function l CON of the cognitive controller is calculated.

[0085]

[0086] wherein, The value of the loss function is calculated by the true value label of the pixel point, and the calculation formula is:

[0087]

[0088] Since part of the parameters of the slow classifier are initialized by the known concept, in order to limit the punishment degree of the visible class and the invisible class, a balance factor is introduced in the loss, so that more attention is paid to the invisible class when training the slow classifier. In an embodiment of the present application, the balance factor is formalized as:

[0089]

[0090] Therefore, the loss function l of the weighted classification CLS is calculated as follows:

[0091]

[0092] It should be noted that the smaller the loss function l of the weighted classification CLS , the better the classification effect of the slow classifier.

[0093] The above steps perform twice prediction on the generated visual representation through the fast classifier and the slow classifier, and the loss function l of the weighted classification and the loss function of the cognitive controller are calculated to be as small as possible, so that the classification of the unknown class of the generated visual representation can be finally realized more accurately.

[0094] In an embodiment of the present application, the above steps are applied to the matting processing in computer vision. In this embodiment, the object segmentation selects the data set PASCAL-VOC. In the specific implementation, the above steps S1-S4 are iteratively executed, the feature extractor adopts the initial learning rate 2.5e -4 of the SGD optimizer, and the optimizers of the generator and the discriminator are two Adam optimizers with initial learning rates 2e -4 and 2.5e -4 respectively. The weight decay is set to 5e -4 , and the batch size is set to 8. For the hyperparameters, λ=10, β1=0.5, and β1=0.5 are set by cross-validation. In the test, the pixels of the input image are first adjusted to 513X513. In the experiment, two different label embedding models are used, word2vec trained on Google News (dimension d=600) and fasttext trained on Common Crawl (dimension d=300), and the two embeddings are spliced as the final word vector according to the previous work. For the categories with multiple words, the embedding representation of each word is directly averaged.

[0095] As shown in Fig. 4(a), the present application can complete the matting processing of the input image through the semantic encoding of different categories provided by the present application and the segmentation model and the generation model pre-trained by the present application. As shown in Fig. 4(b), after the input image is processed by other methods, the real label cannot be well recognized, but the image segmentation effect of the zero-shot semantic segmentation method provided by the present application is better, and the image content can be accurately recognized and the correct label can be given. Figure 4(b) to Figure 4(f)

[0096] Table 1: The coincidence degree between the segmentation results and the real labels of the matting processing by different methods ​

[0097] Method SPNet ZS3 CaGNet The present invention mIoU 0.5467 0.6143 0.6423 0.6702

[0098] In combination with Table 1, wherein mIoU is an index for calculating the coincidence degree between the predicted segmentation result and the real label. As can be seen from Table 1, the coincidence degree of the zero-shot semantic segmentation method provided by the present application is relatively highest. It can be seen that, whether it is the actual matting effect or the coincidence degree with the real segmentation result, the zero-shot semantic segmentation method provided by the present application can obtain a significant effect better than the prior art.

[0099] On the basis of the above-mentioned zero-shot semantic segmentation method, the present application further provides a zero-shot semantic segmentation device based on meaningful learning. As shown in the Figure 5 The zero-shot semantic segmentation device includes one or more processors 51 and a memory 52. The memory 52 is coupled to the processor 51 and is used to store one or more programs. When the one or more programs are executed by the one or more processors 51, the one or more processors 51 implement the zero-shot semantic segmentation method in the above-mentioned embodiments.

[0100] The processor 51 is used to control the overall operation of the zero-shot semantic segmentation device to complete all or part of the steps of the zero-shot semantic segmentation method. The processor 51 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable logic gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory 52 is used to store various types of data to support the operation of the zero-shot semantic segmentation device. These data can include, for example, instructions for operating any application or method on the zero-shot semantic segmentation device, and application-related data. The memory 52 can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, etc.

[0101] In summary, the zero-shot semantic segmentation method and device based on meaningful learning provided by the present application firstly pre-train the segmentation model and the generation model by using large-scale known class data with annotation information, and obtain the generated visual representation for the unknown class by the pre-trained generation model and the semantic encoding of the unknown class; then the focus is on constructing the conjugate correlation loss between the visual representation of the visible class and the visual representation of the invisible class, learning the correlation between the visible class and the invisible class, and finally performing two predictions on the generated visual representation of the unknown class by the fast classifier and the slow classifier respectively, and calculating the corresponding loss function to realize zero-shot image segmentation.

[0102] Compared with the prior art, the present application has the following technical effects: it can learn new concepts well without visual samples, and by constructing the conjugate correlation loss between the visual representation of the visible class and the visual representation of the invisible class, it can handle the conflict between the new concept and the existing concept, form a fused concept mode, and associate the new concept (invisible class) with the existing concept (visible class) for rapid learning. Compared with the previous zero-shot segmentation method based on projection and generation, the present application pays more attention to the correlation between the visible class and the invisible class, naturally promotes the continuous construction of knowledge, and can make the image segmentation effect more superior.

[0103] The zero-shot semantic segmentation method and device based on meaningful learning provided by the present application are described in detail above. Any obvious modification made by a person skilled in the art without departing from the essential content of the present application will constitute an infringement of the patent right of the present application and will bear the corresponding legal responsibility.

Claims

1. A zero-shot semantic segmentation method based on meaningful learning, characterized in that... Includes the following steps: S1: Pre-train the segmentation model and the generative model using large-scale known category data with labeled information; S2: Utilize a pre-trained generative model and the semantic encoding of the unknown class to obtain a generated visual representation of the unknown class; S3: Randomly select samples from the visible class to extract visual representations, and construct a conjugate association loss with the visual representations of the unknown class generated in step S2; S4: The generated visual representation of the unknown class is predicted twice by a fast classifier and a slow classifier respectively; firstly, all visual semantics are predicted by the fast classifier, the prediction of the fast classifier is evaluated by the cognitive controller, and the compatibility between the prediction and the concept of the visible class is estimated, and then the visual semantic representation uncertain by the cognitive controller is further predicted by the slow classifier. Calculate the loss function for weighted classification and the loss function for the cognitive controller to achieve zero-shot image segmentation.

2. The zero-shot semantic segmentation method as described in claim 1, characterized in that: In step S1, the semantic image segmentation model used is DeepLab V2, in which the discriminator... Fast classifier Slow classifier The cognitive controller is implemented using two 1×1 convolutional layers, while the cognitive controller is implemented using one convolutional layer.

3. The zero-shot semantic segmentation method as described in claim 2, characterized in that: In step S1, the truth semantic graph of the nth visible class data is randomly sampled. Construct a tag word vector graph and compared with the noise map of random sampling Concatenate the data and input it into the generator. To obtain the generated visual representation And guide the generation of visual representations and real visual representation By being similar in appearance, a semantic mapping between labels and visual representations is established, where... It is a feature extractor, where n is a positive integer.

4. The zero-shot semantic segmentation method as described in claim 3, characterized in that: The generated visual representation and the true visual representation The data is fed into the fast classifier. And calculate the classification loss, the discriminator With the fast classifier Share the parameters of the first layer.

5. The zero-shot semantic segmentation method as described in claim 1, characterized in that: In step S2, based on the semantic encoding of the visible and invisible classes, a semantic encoding matrix is ​​randomly sampled as the semantic encoding of the unknown class, and a corresponding tag word vector graph is given. And feed it into the pre-trained generator. To obtain the generated visual representation for unknown classes.

6. The zero-shot semantic segmentation method as described in claim 1, characterized in that... Step S3 specifically includes the following sub-steps: S31, randomly sample an image from the visible class; S32, calculate the semantic space association between the real vector graph and random vectors in the semantic space; S33, Visual semantic association between real visual representations and generated visual representations in computational visual space; S34, calculate the conjugate association loss based on semantic space association and visual semantic association.

7. The zero-shot semantic segmentation method as described in claim 6, characterized in that: The semantic space association is a real vector graph randomly generated in the semantic space. The patch and the actual label word vector graph The patch The relationships between them.

8. The zero-shot semantic segmentation method as described in claim 6, characterized in that: The visual semantic association is the true visual representation in visual space. The j-th patch and generating visual representations The i-th patch The relationships between them.

9. A zero-shot semantic segmentation device based on meaningful learning, characterized in that... It includes a processor and a memory, wherein the processor reads a computer program from the memory for executing the zero-sample semantic segmentation method according to any one of claims 1 to 8.

Images