Image and semantic mask labeling method and device

Abstract

The embodiments of this specification provide a method and device for image and semantic mask labeling. The method includes: obtaining a scene description and a category prompt word; inputting the scene description and the category prompt word into a diffusion model to generate an image related to the scene description and a semantic mask label corresponding to the image, as well as a scoring result for evaluating the quality of the image and the semantic mask label; screening out images and semantic mask labels corresponding to the images whose scoring results are greater than a scoring result threshold. In this way, by calculating the scoring results for evaluating the quality of the image and the semantic mask label in the process of generating images and semantic mask labels by the scaling model, the scoring results are used to screen high-quality images and semantic mask labels, thereby improving the performance and generalization ability of the training model.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly to a method and device for labeling images and semantic masks thereof. Background Art

[0002] In machine learning tasks related to image processing (such as image segmentation, object detection, etc.), the performance and generalization ability of image processing models usually depend on the images and their corresponding semantic labels.

[0003] To train a model with high performance and generalization ability, you need a large number of high-quality images and their semantic mask labels. An image (or image data) refers to visual data composed of pixels. Each pixel contains information about each point in the image, usually including information such as color and brightness. Images can be color (RGB images) or grayscale (single-channel images). In machine learning, image data is provided as input to the model for training or prediction.

[0004] With the emergence of stable diffusion technology, it is now possible to generate more images and their semantic mask labels based on fewer images and their semantic mask labels using generative diffusion algorithms (such as DiffuMask, DatasetDM, DatasetDiffusion, etc.), without the need to manually write or collect a sufficient number of images and their semantic mask labels.

[0005] However, although the generative diffusion algorithm can automatically generate a sufficient number of images and their semantic mask labels, the quality of these images and their semantic mask labels is uneven. If low-quality images and their semantic mask labels are directly used to train the model, the performance and generalization ability of the model will be affected. Summary of the invention

[0006] In view of this, one or more embodiments of the present specification provide a method and device for labeling an image and its semantic mask, an electronic device, and a storage medium to solve the problems existing in the related art.

[0007] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0008] According to a first aspect of an embodiment of this specification, a method for labeling an image and a semantic mask thereof is provided, the method comprising:

[0009] Obtain scene descriptions and category cue words;

[0010] Inputting the scene description and the category prompt words into a diffusion model, generating an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label;

[0011] Filter out images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images.

[0012] Optionally, the step of inputting the scene description and the category prompt words into a diffusion model to generate an image related to the scene description and a semantic mask label corresponding to the image includes:

[0013] The scene description and the category prompt words are input into a diffusion model, random noise is denoised according to the scene description and the category prompt words, and an image and a semantic mask label corresponding to the scene description are generated.

[0014] Optionally, before performing denoising processing on the random noise according to the scene description and the category prompt words, the method further includes: replacing the random noise with prior noise;

[0015] The denoising process of the random noise according to the scene description and the category prompt words includes:

[0016] The prior noise is denoised according to the scene description and the category prompt words.

[0017] Optionally, the scoring result is obtained based on a mask evaluation index and a semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image area; and the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label;

[0018] Generating the scoring result for evaluating the quality of the image and the semantic mask label includes:

[0019] Calculating the mask evaluation index and calculating the semantic evaluation index;

[0020] The mask evaluation index is added to the semantic evaluation index to obtain the scoring result.

[0021] Optionally, the calculating the mask evaluation index includes:

[0022] Acquire denoising results of several time steps in the process of denoising random noise according to the scene description and the category prompt words; wherein the denoising results include intermediate images after denoising at each time step and masks corresponding to the intermediate images;

[0023] Extract mask features of the mask at each time step;

[0024] The variance between the mask features at each time step is calculated and the calculated variance is used as the final mask evaluation metric.

[0025] Optionally, calculating a semantic evaluation index of the mask in the second data set includes:

[0026] Synchronously saving the multi-scale features of the diffusion model in the generated image;

[0027] Extracting a first feature vector corresponding to the category of the category prompt word from the multi-scale features;

[0028] The statistical mean of the first feature vector and each category is measured, and the average of the measurement results with each category is used as the final semantic evaluation indicator.

[0029] Optionally, the statistical mean of each category is obtained by:

[0030] Acquire a sample set; wherein the sample set includes sample pairs of a plurality of scene description samples and category prompt word samples;

[0031] In the process of inputting the sample set into the diffusion model and generating a sample image corresponding to the sample set and a sample semantic mask label corresponding to the sample image, the multi-scale feature samples of the diffusion model are simultaneously saved;

[0032] Extracting a sample feature vector of a category corresponding to each category prompt word sample in the sample set from the multi-scale feature samples;

[0033] The sample feature vectors of each category are counted and the mean is calculated to obtain the statistical mean of each category.

[0034] Optionally, the diffusion model is trained in the following manner:

[0035] Obtain a training set consisting of real images and their real semantic mask label pairs;

[0036] Inputting the training set into a controllable network-based diffusion model to generate training images and training semantic mask label pairs thereof;

[0037] The diffusion model is iteratively trained based on the difference between the real image and its real semantic mask label pair and the generated training image and its training semantic mask label pair.

[0038] Optionally, after filtering out images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images, the method further includes:

[0039] The images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images are used to train the semantic segmentation model, and the trained semantic segmentation model is used to segment the real image.

[0040] According to a second aspect of an embodiment of this specification, a device for labeling an image and a semantic mask thereof is provided, the device comprising:

[0041] Acquisition unit, obtains scene description and category prompt words;

[0042] A generating unit, inputting the scene description and the category prompt words into a diffusion model, generating an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label;

[0043] The screening unit screens out images whose scoring results are greater than a scoring result threshold and semantic mask labels corresponding to the images.

[0044] Optionally, the generating unit includes:

[0045] The denoising subunit inputs the scene description and the category prompt words into the diffusion model, performs denoising processing on the random noise according to the scene description and the category prompt words, and generates an image and a semantic mask label corresponding to the scene description.

[0046] Optionally, after the denoising subunit denoises the random noise according to the scene description and the category prompt words, the denoising subunit further includes: replacing the random noise with prior noise;

[0047] Correspondingly, when the denoising subunit performs denoising processing on random noise according to the scene description and the category prompt words, it is converted into performing denoising processing on prior noise according to the scene description and the category prompt words.

[0048] Optionally, the scoring result is obtained based on a mask evaluation index and a semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image; and the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label;

[0049] The generating unit, when generating the scoring result for evaluating the quality of the image and the semantic mask label, includes:

[0050] A first calculation subunit calculates the mask evaluation index;

[0051] A second calculation subunit calculates the semantic evaluation index;

[0052] The third calculation subunit adds the mask evaluation index and the semantic evaluation index to obtain the scoring result.

[0053] Optionally, the first computing subunit is further used to obtain denoising results of several time steps in the process of denoising random noise according to the scene description and category prompt words; wherein the denoising results include the intermediate image after denoising at each time step and the mask corresponding to the intermediate image; extract the mask features of the mask at each time step; calculate the variance between the mask features of each time step, and use the calculated variance as the final mask evaluation indicator.

[0054] Optionally, the second computing subunit is further used to synchronously save the multi-scale features of the diffusion model in the generated image; extract a first feature vector corresponding to the category of the category prompt word from the multi-scale features; measure the statistical mean of the first feature vector and each category, and take the average of the measurement results with each category as the final semantic evaluation indicator.

[0055] Optionally, the mean representation of each category is obtained by the statistical subunit:

[0056] The statistical subunit is further used to obtain a sample set; wherein the sample set includes sample pairs of several scene description samples and category prompt word samples; in the process of inputting the sample set into the diffusion model to generate a sample image corresponding to the sample set and a sample semantic mask label corresponding to the sample image, the multi-scale feature samples of the diffusion model are synchronously saved; sample feature vectors of the category corresponding to each category prompt word sample in the sample set are extracted from the multi-scale feature samples; the sample feature vectors of each category are counted and the mean is calculated to obtain the statistical mean of each category.

[0057] Optionally, the diffusion model is obtained through the following training units:

[0058] The training unit obtains a training set consisting of real images and their real semantic mask label pairs; inputs the training set into a diffusion model based on a controllable network to generate training images and their training semantic mask label pairs; and iteratively trains the diffusion model based on the difference between the real images and their real semantic mask label pairs and the generated training images and their training semantic mask label pairs.

[0059] Optionally, after the screening unit, it also includes:

[0060] The segmentation unit uses the image whose scoring result is greater than the scoring result threshold and the semantic mask label corresponding to the image to train a semantic segmentation model, and the trained semantic segmentation model is used to segment the real image.

[0061] According to a third aspect of the embodiments of this specification, there is provided an electronic device, including a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0062] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0063] According to a fourth aspect of the embodiments of this specification, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-readable instructions, and the machine-readable instructions implement the above method when called and executed by a processor.

[0064] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0065] In the process of generating images and semantic mask labels by the scaling model, scoring results for evaluating the quality of the images and semantic mask labels are calculated, and the scoring results are used to screen high-quality images and semantic mask labels, thereby improving the performance and generalization ability of the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of the architecture of an image segmentation network provided as an exemplary embodiment of this specification;

[0067] Figure 2 A schematic diagram of image segmentation provided as an exemplary embodiment of this specification;

[0068] Figure 3 A schematic diagram of improving a diffusion model provided as an exemplary embodiment of this specification;

[0069] Figure 4 A flowchart of an image and a semantic mask labeling method thereof provided as an exemplary embodiment of this specification;

[0070] Figure 5 A schematic diagram of mask differences at different time steps provided as an exemplary embodiment of this specification;

[0071] Figure 6 A schematic diagram of calculating evaluation indicators provided for an exemplary embodiment of this specification;

[0072] Figure 7 A schematic diagram of the structure of an electronic device where an image and semantic mask labeling device are provided in an exemplary embodiment of this specification;

[0073] Figure 8 A block diagram of an image and a semantic mask labeling device thereof provided by an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0075] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0076] In the real world, we often encounter categories that we have never encountered in the model learning stage. For example, during the model training process, we will learn about several configured categories to learn the relationship between features in the image and each category. In this way, the image segmentation model can only recognize a limited number of categories for the input image, and cannot recognize categories that have never appeared before.

[0077] In this regard, this manual provides Figure 1 The schematic diagram of the architecture of an image segmentation network shown in FIG. The image segmentation network may refer to an open set semantic segmentation model.

[0078] The open set semantics is designed to handle unknown or unseen categories and is applicable to open scenes in the real world. Open set semantic segmentation usually requires the model to have better generalization capabilities so that it can recognize and label unknown categories. When the initial image is input, the category prompt word (which can be the category used in the model training or the category not used in the model training) is also input. The text encoding features of the category prompt word are projected to the image space area, and the spatial distance between the text encoding features and the image encoding features is calculated to obtain the pre-annotated result that conforms to the category corresponding to the category prompt word, thereby completing the segmentation of the image.

[0079] Taking autonomous driving based on vision as an example, it is necessary to identify and segment the real-time images collected by the vehicle, such as Figure 2 Schematic diagram of image segmentation in , with the help of the input category prompt words, the driving image can be segmented into the desired building area, vehicle area and ground area.

[0080] Based on the semantic segmentation model, if you want to train a model with high performance and generalization ability, you need a large number of high-quality images and their semantic mask labels. Among them, images (or image data) refer to visual data composed of pixels. Each pixel contains information about each point in the image, usually including information such as color and brightness. Images can be color (RGB images) or grayscale (single-channel images). In machine learning, image data is provided as input to the model for training or prediction. In order to make image segmentation more accurate, the category prompt words related to the semantic mask label can be input into the model along with the image.

[0081] With the emergence of generative diffusion technology, it is now possible to use generative diffusion algorithms (such as DiffuMask, DatasetDM, Dataset Diffusion, etc.) to generate more images and their semantic mask labels based on fewer images and semantic mask labels, without the need to manually write or collect a sufficient number of images and their semantic mask labels.

[0082] The basic idea of ​​diffusion models is to systematically and slowly destroy the structure in the data distribution through an iterative forward diffusion process. Then, a backward diffusion process is learned that restores the structure in the data, resulting in a highly flexible and tractable model.

[0083] In the forward diffusion process, random noise is continuously added to the initial image until the initial image becomes pure noise; in the backward diffusion process, the added random noise is gradually denoised, the noise is learned and the initial image is restored. In the backward diffusion process, a neural network is usually used for denoising, and the neural network can be a U-Net neural network.

[0084] Through the above learning, the model can be enabled to have the ability to generate new images, so that with the help of the diffusion model, new images and their semantic mask labels can be automatically generated based on the input initial image.

[0085] However, although the generative diffusion algorithm can automatically generate a sufficient number of images and their semantic mask labels, the quality of these images and their semantic mask labels is uneven. If low-quality images and their semantic mask labels are directly used to train the model, the performance and generalization ability of the model will be affected.

[0086] like Figure 3 As shown in this specification, the process of how the diffusion model generates images and their semantic mask labels is not intervened. Instead, a quality screening step is added after the image and its semantic mask label are generated. The quality screening step is used to screen high-quality images and their semantic mask labels, thereby ensuring the quality of the images and their semantic mask labels used to train the scaling model.

[0087] Please refer to the following Figure 4 A flowchart of a method for labeling an image and its semantic mask is shown, and the method may include the following steps:

[0088] Step 210, obtaining scene description and category prompt words.

[0089] Step 220, input the scene description and category prompt words into a diffusion model, generate an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label.

[0090] Step 230, filtering out a third data set from the second data set based on a scoring result threshold; wherein the third data set includes images whose scoring results are greater than the scoring result threshold and masks corresponding to the images.

[0091] The following describes the training process of the diffusion model:

[0092] Obtain a training set consisting of real images and their real semantic mask label pairs;

[0093] Inputting the training set into a controllable network-based diffusion model to generate training images and training semantic mask label pairs thereof;

[0094] The diffusion model is iteratively trained based on the difference between the real image and its real semantic mask label pair and the generated training image and its training semantic mask label pair.

[0095] After the diffusion model is trained, the trained diffusion model can be used to generate high-quality images and their semantic mask labels. Specifically, the prepared scene description and category prompt words are input into the diffusion model, thereby obtaining an image related to the scene description and a semantic mask label corresponding to the image generated by the diffusion model, as well as a scoring result for evaluating the quality of the image and the semantic mask label.

[0096] During the scaling model inference process, a random noise is randomly generated, and then the random noise is denoised according to the scene description and category prompt words to generate an image and a semantic mask label corresponding to the scene description.

[0097] In this embodiment, the diffusion model needs to use the U-Net neural network to gradually denoise the random noise over several time steps during the image generation process. As the time steps are iterated, the image and its mask gradually become clearer. In this process, the core areas of interest are basically the same because they are controlled by the same category prompt words. However, due to the influence of model errors and random noise, the masks obtained at each time step are different.

[0098] A mask is a mark or label used to indicate a specific area in an image. A mask is usually a binary array or matrix of the same size as the image pixels, in which each element indicates whether the corresponding pixel in the image belongs to a specific area or object. For example, in medical images, a mask may be used to mark the location of a tumor; in autonomous driving, a mask can mark the location of pedestrians or vehicles.

[0099] The semantic mask label is a label that semantically describes the mask. For example, in autonomous driving, the mask marks the location of pedestrians, so the semantic mask label of the mask can be the label "pedestrian".

[0100] exist Figure 5 Schematic diagram of the mask difference at different time steps shown, Figure 5 The middle part is divided into an exemplary example to give the category hint word is face. In the process of denoising random noise by diffusion model, the mask diagram at the 1st time step, the 251st time step, the 501st time step and the 751st time step (i.e. Figure 5 ). At the first time step, the mask includes the entire head area, including not only the face but also the hair and background; at the 251st time step, the mask can remove the background, the 501st time step can remove the hair, and at the 751st time step, a mask that only includes the face can be obtained. It can be seen that with the iteration of time steps, the generated image and its semantic mask label will gradually become clear.

[0101] Exemplarily, before denoising the random noise according to the scene description and the category prompt words, the method further includes: replacing the random noise with prior noise;

[0102] The denoising process of the random noise according to the scene description and the category prompt words includes:

[0103] The prior noise is denoised according to the scene description and the category prompt words.

[0104] In this embodiment, since the prior noise can improve the robustness of the model, the random noise randomly generated by the diffusion model is replaced with the prior noise, so that the image generated by the diffusion model and its semantic mask label can be more ideal.

[0105] After introducing how the scaling model generates an image related to the scene description and a semantic mask label corresponding to the image, the following describes how the scaling model calculates a scoring result for evaluating the quality of the image and the semantic mask label.

[0106] In an exemplary embodiment, a mask evaluation index (Smask) and a semantic evaluation index (Sclass) are introduced in this specification, and the scoring result can be obtained based on the mask evaluation index and the semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image; and the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label;

[0107] Accordingly, generating the scoring result for evaluating the quality of the image and the semantic mask label may include:

[0108] Calculating the mask evaluation index and calculating the semantic evaluation index;

[0109] The mask evaluation index is added to the semantic evaluation index to obtain the scoring result.

[0110] This specification introduces a mask evaluation index and a semantic evaluation index to comprehensively evaluate the quality of the image and the semantic mask label from the degree of alignment between the mask and the image and the accuracy of the mask and the semantic mask label, respectively, so that the scoring result is more comprehensive and accurate.

[0111] In an exemplary embodiment, the calculating the mask evaluation index may include:

[0112] Acquire denoising results of several time steps in the process of denoising random noise according to the scene description and the category prompt words; wherein the denoising results include intermediate images after denoising at each time step and masks corresponding to the intermediate images;

[0113] Extract mask features of the mask at each time step;

[0114] The variance between the mask features at each time step is calculated and the calculated variance is used as the final mask evaluation metric.

[0115] In the process of generating an image with the help of the diffusion model, this embodiment needs to use a U-Net neural network to gradually denoise the random noise over several time steps, thereby obtaining masks of several time steps.

[0116] Specifically, at each time step, the attention features of the denoised intermediate image are extracted, the category cue word is projected into the image space of the denoised intermediate image, and a mask that matches the category cue word is obtained; wherein the mask is composed of pixels in the denoised intermediate image that match the category cue word.

[0117] like Figure 6As shown, the random noise 61 generated by the diffusion model and the scene description and category prompt words 62 of the input scaling model are input into the U-Net neural network 63 of the diffusion model. The U-Net neural network 63 uses the scene description and category prompt words 62 to denoise the random noise 61 to generate an image and its semantic mask label 64.

[0118] In the U-Net neural network denoising process, it is necessary to extract the attention features of the denoised intermediate image. Figure 6 As shown, the attention feature may include a self-attention feature (Self Attention) and a cross-attention feature (Cross Attention). Further, the category prompt word is projected into the image space of the intermediate image, and a mask that conforms to the category prompt word is obtained.

[0119] Among them, the denoising process is divided into several iterative time steps, such as Figure 6 Shown are k time steps including T1, T2, T3, ..., Tk and a mask 65 for each time step.

[0120] After obtaining the masks of each time step and extracting the mask features of these masks, the variance between the mask features of each time step can be calculated.

[0121] Exemplarily, the calculation of the variance requires calculating the feature mean of the mask features at each time step, calculating the absolute value distance between the mask features at each time step and the feature mean, accumulating the absolute value distance between the mask features at each time step and the feature mean, and obtaining the variance between the mask features at each time step. The variance can be used as the final mask evaluation indicator.

[0122] Please continue to refer to Figure 6 As shown in , the mask evaluation metric , ;

[0123] Where k represents the total number of time steps, i is an integer between (0, K], Represents the mask feature of the i-th time step With the characteristic mean The absolute value distance.

[0124] Through the above example, by calculating the mask evaluation index between masks at different time steps, the stability of the mask and the image can be characterized, which can be used to evaluate the quality of the generated image and its semantic mask label.

[0125] After introducing the calculation example of the mask evaluation index, the calculation example of the semantic evaluation index is introduced below.

[0126] In an exemplary embodiment, the computational semantic evaluation index may include:

[0127] Synchronously saving the multi-scale features of the diffusion model in the generated image;

[0128] Extracting a first feature vector corresponding to the category of the category prompt word from the multi-scale features;

[0129] The statistical mean of the first feature vector and each category is measured, and the average of the measurement results with each category is used as the final semantic evaluation indicator.

[0130] Please continue to refer to Figure 6 As shown in , the semantic evaluation index is shown in the following formula:

[0131]

[0132] Among them, C represents the total number of all categories in the feature center library, The first eigenvector representing the i-th category; represents the statistical mean of the i-th category.

[0133] Since the generated image can have multiple semantic mask labels, the above formula sets .

[0134] In this example, for a single image, image features of different categories may fluctuate significantly. For example, an image may include image features with large differences such as people, vehicles, and buildings. However, since the model training goal is certain, from the perspective of the entire sample set, the semantic mask labels of different categories have certain similarities. In simple terms, if the purpose of model training is for autonomous driving, the images in the dataset will usually include categories with a higher probability of occurrence, such as pedestrians, vehicles, buildings, and roads, while image features of categories such as airplanes and ships have a lower probability of occurrence.

[0135] For example, the statistical mean of each of the above categories can be obtained in the following way:

[0136] Acquire a sample set; wherein the sample set includes sample pairs of a plurality of scene description samples and category prompt word samples;

[0137] In the process of inputting the sample set into the diffusion model and generating a sample image corresponding to the sample set and a sample semantic mask label corresponding to the sample image, the multi-scale feature samples of the diffusion model are simultaneously saved;

[0138] Extracting a sample feature vector of a category corresponding to each category prompt word sample in the sample set from the multi-scale feature samples;

[0139] The sample feature vectors of each category are counted and the mean is calculated to obtain the statistical mean of each category.

[0140] This specification constructs a feature center library by calculating the statistical mean corresponding to each category in the historical image generated by the sample set; and by measuring the statistical mean corresponding to each category in the feature center library with the first feature vector (such as calculating the absolute value distance), the average value of the measurement result with each category is obtained. The average of the measurement result can characterize the similarity between the generated semantic mask label and each category in the feature center library. The higher the average value of the measurement result, the more similar the semantics are, and thus the more credible the generated image and its semantic mask label are.

[0141] Through the above examples, the accuracy of the mask and the semantic mask label can be characterized by calculating the average value of the measurement results, which can then be used to evaluate the quality of the generated image and its semantic mask label.

[0142] It should also be noted that for the images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images screened out in step 230, they can be added to the sample set, and the mean representation of each category in the feature center library can be updated by Fk=(1-a)*Fk'+a*fk; wherein Fk is the mean representation of category k after the update, Fk' is the mean representation of category k before the update, fk is the first feature vector of category k in the image, and a is the preset weight parameter.

[0143] After the diffusion model generates an image related to the scene description, a semantic mask label corresponding to the image, and a corresponding scoring result, the scoring result can be compared with the scoring result threshold. If the scoring result is greater than the scoring result threshold, it means that the degree of alignment between the mask and the image and the accuracy of the mask and the semantic mask label are high, and thus it can be considered that the quality of the image and its semantic mask label generated by the diffusion model is high. On the contrary, if the scoring result is less than the scoring result threshold, it can be considered that the quality of the generated image and its semantic mask label is low.

[0144] The scoring result threshold may be an empirical value (e.g., 70 points), and the scoring result threshold may be flexibly configured according to actual business needs. If you want a more accurate segmentation effect, you may increase the scoring result threshold; otherwise, you may lower the scoring result threshold.

[0145] In an exemplary embodiment, after the above step 23, the following may also be included:

[0146] The images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images are used to train the semantic segmentation model, and the trained semantic segmentation model is used to segment the real image.

[0147] In this specification, with the help of the above-mentioned method embodiment of the image and its semantic mask label, more high-quality images and their semantic mask labels are generated. These high-quality images and their semantic mask labels are used to retrain the semantic segmentation model, or in the model fine-tuning stage of the original training process of the semantic segmentation model, these high-quality images and their semantic mask labels are used to fine-tune the model parameters; ultimately, a semantic segmentation model with better performance and more accurate image segmentation can be trained.

[0148] In an exemplary embodiment of the present specification, a device capable of implementing the above method is also provided.

[0149] Figure 7 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 7 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 509 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0150] Please refer to Figure 8 In a software implementation, a device for labeling an image and its semantic mask is provided, the device comprising:

[0151] An acquisition unit 810 acquires a scene description and a category prompt word;

[0152] A generating unit 820 inputs the scene description and the category prompt words into a diffusion model, generates an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label;

[0153] The screening unit 830 screens out images whose scoring results are greater than a scoring result threshold and semantic mask labels corresponding to the images.

[0154] Optionally, the generating unit 820 includes:

[0155] The denoising subunit inputs the scene description and the category prompt words into the diffusion model, performs denoising processing on the random noise according to the scene description and the category prompt words, and generates an image and a semantic mask label corresponding to the scene description.

[0156] Optionally, after the denoising subunit denoises the random noise according to the scene description and the category prompt words, the denoising subunit further includes: replacing the random noise with prior noise;

[0157] Correspondingly, when the denoising subunit performs denoising processing on random noise according to the scene description and the category prompt words, it is converted into performing denoising processing on prior noise according to the scene description and the category prompt words.

[0158] Optionally, the scoring result is obtained based on a mask evaluation index and a semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image; and the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label;

[0159] The generating unit 820, when generating the scoring result for evaluating the quality of the image and the semantic mask label, includes:

[0160] A first calculation subunit calculates the mask evaluation index;

[0161] A second calculation subunit calculates the semantic evaluation index;

[0162] The third calculation subunit adds the mask evaluation index and the semantic evaluation index to obtain the scoring result.

[0163] Optionally, the first computing subunit is further used to obtain denoising results of several time steps in the process of denoising random noise according to the scene description and category prompt words; wherein the denoising results include the intermediate image after denoising at each time step and the mask corresponding to the intermediate image; extract the mask features of the mask at each time step; calculate the variance between the mask features of each time step, and use the calculated variance as the final mask evaluation indicator.

[0164] Optionally, the second computing subunit is further used to synchronously save the multi-scale features of the diffusion model in the generated image; extract a first feature vector corresponding to the category of the category prompt word from the multi-scale features; measure the statistical mean of the first feature vector and each category, and take the average of the measurement results with each category as the final semantic evaluation indicator.

[0165] Optionally, the mean representation of each category is obtained by the statistical subunit:

[0166] The statistical subunit is further used to obtain a sample set; wherein the sample set includes sample pairs of several scene description samples and category prompt word samples; in the process of inputting the sample set into the diffusion model to generate a sample image corresponding to the sample set and a sample semantic mask label corresponding to the sample image, the multi-scale feature samples of the diffusion model are synchronously saved; sample feature vectors of the category corresponding to each category prompt word sample in the sample set are extracted from the multi-scale feature samples; the sample feature vectors of each category are counted and the mean is calculated to obtain the statistical mean of each category.

[0167] Optionally, the diffusion model is obtained through the following training units:

[0168] The training unit obtains a training set consisting of real images and their real semantic mask label pairs; inputs the training set into a diffusion model based on a controllable network to generate training images and their training semantic mask label pairs; and iteratively trains the diffusion model based on the difference between the real images and their real semantic mask label pairs and the generated training images and their training semantic mask label pairs.

[0169] Optionally, after the screening unit 830, the following method is further included:

[0170] The segmentation unit uses the image whose scoring result is greater than the scoring result threshold and the semantic mask label corresponding to the image to train a semantic segmentation model, and the trained semantic segmentation model is used to segment the real image.

[0171] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned image and its semantic mask labeling method. For relevant parts, please refer to the partial description of the method implementation method, which will not be repeated here.

[0172] The device implementation described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0173] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0174] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0176] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0177] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0178] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0179] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0180] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0181] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for generating an image and its semantic mask label, characterized in that: The method comprises: Obtain scene descriptions and category cue words; Inputting the scene description and the category prompt words into a diffusion model, generating an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label; Filter out images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images; The scoring result is obtained based on the mask evaluation index and the semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image; the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label; the generating of the scoring result for evaluating the quality of the image and the semantic mask label includes: calculating the mask evaluation index and calculating the semantic evaluation index; adding the mask evaluation index and the semantic evaluation index to obtain the scoring result; The calculation of the mask evaluation index includes: obtaining denoising results of several time steps in the process of denoising random noise according to the scene description and the category prompt word; wherein the denoising result includes an intermediate image after denoising at each time step and a mask corresponding to the intermediate image; extracting mask features of the mask at each time step; calculating the variance between the mask features at each time step, and using the calculated variance as the final mask evaluation index; The calculation of the semantic evaluation index includes: synchronously saving the multi-scale features of the diffusion model in the generated image; extracting a first feature vector corresponding to the category of the category prompt word from the multi-scale features; measuring the first feature vector with the statistical mean of each category to obtain the measurement value of each category, and taking the average of the measurement values ​​of each category as the final semantic evaluation index.

2. The method according to claim 1, characterized in that The step of inputting the scene description and the category prompt words into a diffusion model to generate an image related to the scene description and a semantic mask label corresponding to the image includes: The scene description and the category prompt words are input into a diffusion model, random noise is denoised according to the scene description and the category prompt words, and an image and a semantic mask label corresponding to the scene description are generated.

3. The method according to claim 1, characterized in that Before the random noise is denoised according to the scene description and the category prompt words, the method further includes: replacing the random noise with prior noise; The denoising process of the random noise according to the scene description and the category prompt words includes: The prior noise is denoised according to the scene description and the category prompt words.

4. The method according to claim 1, characterized in that: The statistical mean of each category is obtained by: Acquire a sample set; wherein the sample set includes sample pairs of a plurality of scene description samples and category prompt word samples; In the process of inputting the sample set into the diffusion model and generating a sample image corresponding to the sample set and a sample semantic mask label corresponding to the sample image, the multi-scale feature samples of the diffusion model are simultaneously saved; Extracting a sample feature vector of a category corresponding to each category prompt word sample in the sample set from the multi-scale feature samples; The sample feature vectors of each category are counted and the mean is calculated to obtain the statistical mean of each category.

5. The method according to claim 1, characterized in that The diffusion model is trained in the following way: Obtain a training set consisting of real images and their real semantic mask label pairs; Inputting the training set into a controllable network-based diffusion model to generate training images and training semantic mask label pairs thereof; The diffusion model is iteratively trained based on the difference between the real image and its real semantic mask label pair and the generated training image and its training semantic mask label pair.

6. The method according to claim 1, characterized in that After filtering out images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images, it also includes: The images whose scoring results are greater than the scoring result threshold and the semantic mask labels corresponding to the images are used to train the semantic segmentation model, and the trained semantic segmentation model is used to segment the real image.

7. An image and its semantic mask labeling device, characterized in that: The device comprises: Acquisition unit, obtains scene description and category prompt words; A generating unit, inputting the scene description and the category prompt words into a diffusion model, generating an image related to the scene description and a semantic mask label corresponding to the image, and a scoring result for evaluating the quality of the image and the semantic mask label; A screening unit, screening out images whose scoring results are greater than a scoring result threshold and semantic mask labels corresponding to the images; The scoring result is obtained based on the mask evaluation index and the semantic evaluation index; wherein the mask evaluation index is used to evaluate the degree of alignment between the mask and the image; the semantic evaluation index is used to evaluate the accuracy of the mask and the semantic mask label; the generating of the scoring result for evaluating the quality of the image and the semantic mask label includes: calculating the mask evaluation index and calculating the semantic evaluation index; adding the mask evaluation index and the semantic evaluation index to obtain the scoring result; The calculation of the mask evaluation index includes: obtaining denoising results of several time steps in the process of denoising random noise according to the scene description and the category prompt word; wherein the denoising result includes an intermediate image after denoising at each time step and a mask corresponding to the intermediate image; extracting mask features of the mask at each time step; calculating the variance between the mask features at each time step, and using the calculated variance as the final mask evaluation index; The calculation of the semantic evaluation index includes: synchronously saving the multi-scale features of the diffusion model in the generated image; extracting a first feature vector corresponding to the category of the category prompt word from the multi-scale features; measuring the first feature vector with the statistical mean of each category to obtain the measurement value of each category, and taking the average of the measurement values ​​of each category as the final semantic evaluation index.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A machine-readable storage medium, characterized in that: Machine-readable instructions are stored thereon, and when the instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images