Semantic segmentation data set generation method based on image label joint diffusion

Abstract

The invention discloses a semantic segmentation data set generation method and device based on image tag joint diffusion, a medium and equipment. The method comprises the following steps: acquiring a real semantic segmentation data set; constructing a label variation auto-encoder and a trained label variation auto-encoder; constructing a de-noising diffusion probability model based on the text, projecting the RGB image, the semantic tag image and the text description to the same potential space through the de-noising diffusion probability model based on the text to obtain a first potential representation, and restoring the potential representation of the image and the potential representation of the tag through a joint diffusion and de-noising process; according to the method and the device, through joint distribution of modeling images and labels in a diffusion model, collaborative generation is carried out in the same potential representation space, and the semantic segmentation data set can be obtained through the joint distribution of the modeling images and the labels in the same potential representation space. And a large-scale, multi-category and multi-scene high-quality semantic segmentation data set is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a semantic segmentation dataset generation method and device based on image label joint diffusion, a medium and equipment. BACKGROUND

[0002] Semantic segmentation is one of the core tasks of computer vision, aiming to assign a semantic label to each pixel in an image, which has key applications in autonomous driving, medical image analysis, robot navigation, etc. Although deep learning models have made significant progress in this task, their performance is highly dependent on large-scale high-quality labeled data. However, image data collection and pixel-level annotation are costly and time-consuming, especially in complex scenes (such as multi-target interaction, small object dense distribution), the difficulty of manual annotation is further intensified. This bottleneck seriously restricts the promotion and application of semantic segmentation models in real scenes.

[0003] To reduce the cost of data collection, more and more work has begun to focus on training data generation, trying to generate synthetic samples through generative models to replace or supplement real data. However, existing generation methods either need to predict pseudo-labels after image generation, such as Prior Art One, or generate images conditioned on human-labeled labels, such as Prior Art Two. However, the former relies on the similarity matching of text-image models, and it is difficult to avoid problems such as class confusion and boundary ambiguity caused by the abstractness of text representation. The latter mostly uses label-guided soft constraints, which are prone to boundary drift, detail misplacement, and even the phenomenon of misgenerating other contents in the target region during the generation process, resulting in mismatch between generated images and labels. SUMMARY

[0004] The main purpose of the present application is to provide a semantic segmentation dataset generation method and device based on image label joint diffusion, a medium and equipment, which aims to directly model the joint distribution of semantic labels and images, so that the diffusion model generates high-quality images and pixel-level accurate semantic labels simultaneously during the generation process.

[0005] To achieve the above object, the application provides a semantic segmentation dataset generation method based on image label joint diffusion, comprising: obtaining a real semantic segmentation dataset, the dataset containing RGB images and their corresponding semantic segmentation masks; constructing a label variational autoencoder, and using the label variational autoencoder to process input binary encoded semantic label images and output decoded semantic label images, and training the label variational autoencoder with the cross-entropy loss of the input and output to obtain a trained label variational autoencoder; constructing a text-based denoising diffusion probability model based on an image variational autoencoder, an image encoder, a text encoder, the trained label variational autoencoder and a UViT network, projecting the RGB images, semantic label images and text descriptions to the same latent space to obtain first latent representations through text-based denoising diffusion probability model coding, and then through joint diffusion and denoising processes to restore the latent representations of the images and the latent representations of the labels; decoding the generated label latent representations, correcting the label correction values of the pixels in the small target regions or noise regions in the label latent representations using a boundary mode-based label optimization algorithm to obtain corrected label latent representations, and obtaining a synthetic semantic segmentation dataset based on the latent representations of the images and the corrected label latent representations.

[0006] Optionally, the method further comprises: constructing the label variational autoencoder based on a Label VAE; inputting the binary encoded semantic label images into the label encoder of the Label VAE to obtain latent features; decoding the latent features using the label decoder of the Label VAE to obtain hidden features; taking the cross-entropy loss of the binary encoded semantic label images and the hidden features as an objective function to train the label variational autoencoder, and obtaining the trained label variational autoencoder.

[0007] Optionally, the method further comprises: processing the hidden features through a maximum value index function to obtain a reconstructed label image.

[0008] Optionally, the model adopted by the image variational autoencoder is an Image VAE, the model adopted by the image encoder is a CLIP Image Encoder, and the model adopted by the text encoder is a CLIP Text Encoder.

[0009] Optionally, the text-based denoising diffusion probability model encoding projects the RGB image, semantic label image, and text description into the same latent space to obtain a latent representation, including: processing the RGB image using a multimodal model to obtain a text description; processing the RGB image using a CLIP Image Encoder to obtain image latent features; processing the text description using a CLIP Text Encoder to obtain text latent features; determining the image latent representation based on the text latent features and image latent features; processing the semantic label image using a label variational autoencoder to obtain a label latent representation; and obtaining a first latent representation by projecting it into the same latent space based on the label latent representation and the image latent representation.

[0010] Optionally, the process of recovering the latent representation of the image and the latent representation of the label through joint diffusion and denoising includes: determining a second latent representation representing the original RGB image and the original semantic label image based on the latent representation; processing the second latent representation based on a preset denoising expression to obtain a joint denoising distribution of the third latent representation and the second latent representation; and constructing a joint denoising distribution based on the third latent representation at the current time, the text latent features, and the third latent representation at the previous time.

[0011] Optionally, after processing the third latent representation and textual latent features at the current time based on the denoising network, the difference between the first and second latent representations is obtained to get the first multiplier, and the second multiplier is determined according to the noise time step; after multiplying the first and second multipliers, the denoised mean of the joint denoising distribution is obtained.

[0012] Optionally, the model used to train the text-based denoising diffusion probability model is a denoising network, and the mean squared error loss is determined based on the output of the denoising network and the output of the text-based denoising diffusion probability model.

[0013] Optionally, a distribution threshold for the label distribution of pixels in the target region in the latent label representation is set. The label optimization algorithm based on the boundary mode corrects the label correction value of pixels in small target regions or noise regions in the latent label representation, including: determining a small target region or noise region if the number of pixels in the target region in the latent label representation is less than a preset threshold; determining the label correction value of pixels in the small target region or noise region based on the mode of pixel label values ​​in the small target region or noise region and based on the comparison result of the mode of pixel label values ​​with preset boundary pixel values.

[0014] Furthermore, to achieve the above objectives, this application also provides a method for generating a semantic segmentation dataset based on joint diffusion of image labels, comprising: a dataset acquisition module for acquiring a real semantic segmentation dataset, wherein the dataset includes RGB images and their corresponding semantic segmentation masks; an encoder training module for constructing a label variational autoencoder and using the label variational autoencoder to process the input binary encoded semantic label image and the cross-loss entropy of the output decoded semantic label image to train the label variational autoencoder, thereby obtaining the trained label variational autoencoder; and a latent data representation module for representing the latent data based on the image variational autoencoder, image encoder, text encoder, and trained label variational autoencoder. A text-based denoising diffusion probability model is constructed using an autoencoder and a UViT network. The RGB image, semantic label image, and text description are projected into the same latent space through encoding using this model to obtain the first latent representation. Then, a joint diffusion and denoising process is used to recover the latent representations of the image and the labels. A synthetic semantic segmentation module decodes the generated label latent representations and uses a boundary mode-based label optimization algorithm to correct the label correction values ​​of pixels in small target regions or noise regions within the label latent representation, resulting in a corrected label latent representation. Based on the latent representations of the image and the corrected label latent representations, a synthetic semantic segmentation dataset is obtained.

[0015] To achieve the above objectives, this application also provides an electronic device, the electronic device comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the semantic segmentation dataset generation method based on image label joint diffusion provided in any of the foregoing embodiments.

[0016] This application proposes a method, apparatus, medium, and device for generating a semantic segmentation dataset based on joint diffusion of image labels. By obtaining a real semantic segmentation dataset, the dataset includes RGB images and their corresponding semantic segmentation masks. Construct a label variational autoencoder, and use the label variational autoencoder to process the input binary encoded semantic label image and the cross loss entropy of the output decoded semantic label image to train the label variational autoencoder, thus obtaining the trained label variational autoencoder. A text-based denoising diffusion probability model is constructed based on an image variational autoencoder, an image encoder, a text encoder, a trained label variational autoencoder, and a UViT network. The RGB image, the semantic label image, and the text description are projected into the same latent space through the text-based denoising diffusion probability model to obtain the first latent representation. Then, the latent representation of the image and the latent representation of the label are recovered through a joint diffusion and denoising process. The generated latent label representation is decoded, and a label optimization algorithm based on boundary mode is used to correct the label correction values ​​of pixels in small target regions or noise regions in the latent label representation, resulting in a corrected latent label representation. Based on the latent representation of the image and the corrected latent label representation, a synthetic semantic segmentation dataset is obtained. This application models the joint distribution of images and labels in a diffusion model, enabling them to be co-generated in the same latent representation space. This achieves the generation of high-quality semantic segmentation datasets for large-scale, multi-class, and multi-scene applications while ensuring the semantic consistency, category boundary accuracy, and spatial structure diversity of the generated images and labels. This provides efficient, accurate, and controllable data support for downstream dense prediction tasks and effectively enhances the training performance and generalization ability of downstream models in complex scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the semantic segmentation dataset generation method based on joint diffusion of image labels provided in this application; Figure 2 This is a schematic diagram of an embodiment of the semantic segmentation dataset generation method based on joint diffusion of image labels in this application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0020] The main solution of this application is as follows: To overcome the shortcomings of the prior art, this application provides a semantic segmentation dataset generation method based on joint diffusion of image labels. Unlike previous methods that first generate images or labels and then process them, this application directly models the joint distribution of semantic labels and images, enabling the diffusion model to simultaneously generate high-quality images and pixel-accurate semantic labels during the generation process. To achieve this goal, this application first uses a variational autoencoder to learn the latent distribution of real labels and encodes them into a latent space. Then, this application employs a diffusion model to jointly diffuse and denoise text, images, and labels in the latent space, enabling them to be co-generated within the same latent representation space, thus ensuring the structural and semantic consistency of the final generated images and labels. However, due to the incomplete denoising of the diffusion model, this application further designs a label optimization strategy. For label regions with small areas, the mode value of their edge pixels is analyzed and used to adjust the label value of the target region, thereby enhancing label consistency and reducing the impact of noise. Finally, this application improves the performance of downstream semantic segmentation models by generating high-quality semantic segmentation data.

[0021] Reference Figure 1 , Figure 1 The flowchart illustrates a semantic segmentation dataset generation method based on joint diffusion of image labels provided in the first embodiment of this application. This method can be executed by a processor, which can be located in a terminal or server. The method may include: S10. Obtain a real semantic segmentation dataset, which includes RGB images and their corresponding semantic segmentation masks; Construct a label variational autoencoder, and use the label variational autoencoder to process the input binary encoded semantic label image and the cross loss entropy of the output decoded semantic label image to train the label variational autoencoder, thus obtaining the trained label variational autoencoder. In one embodiment of this application, the step of constructing a label variational autoencoder and using the label variational autoencoder to process the input binary encoded semantic label image and the cross-loss entropy of the output decoded semantic label image to train the label variational autoencoder, thereby obtaining the trained label variational autoencoder, includes: A label variational autoencoder was constructed based on Label VAE. After the semantically labeled image is encoded into binary code, it is input into the label encoder in Label VAE to obtain latent features; Latent features are obtained by decoding latent features using the Label VAE label decoder; Using the cross-entropy loss function of the binary encoded semantic label image and latent features as the objective function, a label variational autoencoder is trained to obtain the trained label variational autoencoder.

[0022] It should be noted that for a real semantic segmentation dataset The goal of this application is to learn a text-based... Image mask joint generation model Generate a synthetic dataset This makes it similar to the real dataset in terms of category distribution, target structure, and visual characteristics. It maintains consistency while providing more diverse scenarios to enhance the generalization ability of semantic segmentation models. Among these... To generate the parameters of the model, and This is a semantic segmentation mask for RGB images and their corresponding images. The final processor uses... and Train a semantic segmentation model to verify the effectiveness of the method.

[0023] Before proceeding, a label variational autoencoder needs to be constructed and trained.

[0024] Specifically, the processor first trains a variational autoencoder to encode the semantic labels, giving them a compact representation in the latent space. For a semantically labeled image... ,in For the number of categories, and These represent the height and width of the image, respectively. To improve class discrimination and reduce computational overhead, the processor uses binary encoding as the input representation of the variational autoencoder. Specifically, the processor assigns the class of each pixel... Convert to The binary representation of each channel, namely:

[0025] Thus, the encoded mask representation is obtained. .

[0026] Since this variational autoencoder is used only as a compression tool and not for generative modeling, this application does not impose standard normal distribution constraints on its latent variables, i.e., no KL divergence regularization is required, and it is trained solely through cross-entropy loss.

[0027] in Represents pixels The true category distribution at that location, It is the predicted probability obtained from the decoder output of the label variational autoencoder.

[0028] The process of reconstructing semantic label images using trained label variational autoencoders to obtain reconstructed label images may include: The latent features are processed by the maximum index function to obtain the reconstructed label image.

[0029] Specifically, after the processor has trained the variational autoencoder, for the encoder... Encoded latent features Reconstructed labels It can be represented as:

[0030] in, This is the trained label decoder.

[0031] It should be noted that latent variables refer to the output of the image after it is input into the label variational autoencoder.

[0032] After obtaining the trained label variational autoencoder, the processor can combine the label variational autoencoder with the existing diffusion model unidiffuser to obtain a new diffusion model, which is the text-based denoising diffusion probability model in step S20.

[0033] S20. Construct a text-based denoising diffusion probability model based on the image variational autoencoder, image encoder, text encoder, trained label variational autoencoder and UViT network. After encoding the RGB image, semantic label image and text description into the same latent space to obtain the first latent representation, recover the latent representation of the image and the latent representation of the label through the joint diffusion and denoising process. In one embodiment of this application, the image variational autoencoder uses the Image VAE model, the image encoder uses the CLIP Image Encoder model, and the text encoder uses the CLIP Text Encoder model.

[0034] In one embodiment of this application, the process of using a text-based denoising diffusion probability model encoding to project an RGB image, a semantic label image, and a text description into the same latent space to obtain a first latent representation may include the following: The text-based denoising diffusion probability model encoding projects the RGB image, semantically labeled image, and text description into the same latent space to obtain a latent representation, including: RGB images are processed using a multimodal model to obtain text descriptions; The CLIP Image Encoder is used to process RGB images to obtain image latent features, and the CLIP Text Encoder is used to process text descriptions to obtain text latent features. Based on the text latent features and image latent features, the latent representation of the image is determined. A label variational autoencoder is used to process semantic label images to obtain latent label representations; Based on the label latent representation and the image latent representation, the first latent representation is obtained by projecting it onto the same latent space.

[0035] Specifically, the processor can use a text-based denoising diffusion probability model to jointly model the image and the label. For example, given an RGB image... and its corresponding tags The processor can first use a multimodal model, in this application the BLIP2 model, to generate descriptions for the image. Then the processor can use the CLIP text encoder. Image encoder Image variational autoencoder And the label variational autoencoder trained in the previous step RGB image and its corresponding tags Encode into the latent space:

[0036]

[0037]

[0038] In one embodiment of this application, the process of recovering the latent representation of the image and the latent representation of the label through joint diffusion and denoising may include the following: A second latent representation is determined based on the latent representation to characterize the original RGB image and the original semantic label image; The second latent representation is processed based on a preset noise-adding expression to obtain the joint noise-adding distribution of the third latent representation and the second latent representation; A joint denoising distribution is constructed based on the third latent representation at the current time step, the textual latent features, and the third latent representation at the previous time step.

[0039] After optimizing the text-based denoising diffusion probability model, the processor can then... and To perform joint diffusion, directly model its joint distribution. Based on the standard forward process of the diffusion model, the processor can define the joint noise distribution as follows:

[0040] Wherein, the second potential is represented as , , Used to control time steps Noise scheduling. The third potential representation is... .

[0041] In this embodiment, during the denoising process, the processor aims to determine the denoising parameters from the noisy latent variables based on text conditions. Simultaneously restore the original image and label latent representation. Therefore, the processor directly models the joint denoising distribution:

[0042] in, Modeling the relationship between images and labels for denoised mean: In one embodiment of this application, after processing the third latent representation and textual latent features at the current time based on the denoising network, the difference between the first and second latent representations is obtained, and the second multiplier is determined based on the noise time step. The denoised mean of the joint denoised distribution is obtained by multiplying the first multiplier and the second multiplier.

[0043] For example, the expression for the denoised mean in this application can be:

[0044] in, This is a denoising network used to remove noise added during the forward prediction stage.

[0045] The model used to train the text-based denoising diffusion probability model is a denoising network, and the mean squared error loss is determined based on the output of the denoising network and the output of the text-based denoising diffusion probability model.

[0046] In this application, the diffusion model is optimized using standard mean squared error loss:

[0047] in, This represents the noise from the sampling. Exemplary, a specific embodiment of this application is given below: For a given dataset Each sample includes a natural image. Its corresponding category label and the corresponding text description ,in, and These represent the height and width of the image, respectively. For all data, the processor resizes the image and mask to 512. 512, and applied random horizontal flipping as data augmentation to enhance the model's generalization ability.

[0048] This application employs a lightweight variational autoencoder to encode labels, which consists of an encoder and a decoder. The encoder converts the input mask into a compressed latent representation. This process begins with an initial convolutional layer that has 3D padding. A 3-kernel approach is applied to the input mask, increasing its depth to match the first output block, which by default consists of 128 output channels. Following the initial convolution, SiLU activation is used to introduce non-linearity. Next, the encoder has three downsampling blocks that progressively reduce spatial resolution while increasing feature map depth. Each downsampling block comprises a 3-kernel approach. 3 convolutions with padding to preserve spatial dimensions, followed by another 3. Three convolutions with a stride of 2 reduce the spatial resolution by half. SiLU activation is applied after each convolutional layer. The number of channels increases in blocks, starting with 128, gradually increasing to 256, 512, and finally 512 channels, allowing the model to capture increasingly abstract features. The encoder ends with a group normalization layer and a final convolution that maps the feature maps to the latent space. The latent space is represented by a set of latent variables, with each mask having 4 latent channels and 2 latent variables by default. The encoder outputs the latent distribution, which is used for mask reconstruction in the decoder.

[0049] The decoder reconstructs the original mask from the latent space representation. It begins with a convolutional layer that projects the latent variables into a higher-dimensional feature space. This layer uses 3... The decoder uses a 3-kernel and padding approach, with the input channel count set to the number of latent channels and the output channel count set to the number of intermediate channels. The decoder contains multiple upsampling blocks that progressively upsample the feature map to the original mask resolution using transposed convolutions. Each upsampling block consists of a deconvolutional layer to double the spatial resolution, followed by layer normalization and SiLU activation for normalization and non-linearity. These layers efficiently reconstruct the mask by increasing the resolution of the feature map. The decoder then uses a group of normalization layers and a final 3-kernel approach. The third convolutional layer ends, generating the reconstructed mask.

[0050] This application uses the AdamW optimizer to train the variational autoencoder model, with a weight decay of 0.05 and a maximum gradient norm of 3, and applies cross-entropy loss for training. In other words, training the model involves solving the following optimization problem:

[0051]

[0052] in, The cross-entropy loss is defined above. The number of training samples in each batch. These are the parameters of the variational autoencoder.

[0053] After obtaining the parameters of the variational autoencoder, this application fixes these parameters and trains the denoising network part of the diffusion model. This application uses the AdamW optimizer with a fixed learning rate of 5e-5, a weight decay of 1e-5, and a maximum gradient norm of 1 to train the denoising network. That is, the meta-learning pre-training of the model is to solve the following optimization problem:

[0054]

[0055] in, The number of training samples in each batch. These are the parameters for the denoising network. Finally, this application uses the descriptions of the training images from each dataset as text prompts, uses the DPM multi-step scheduler to perform 50 sampling steps on the trained diffusion model, and finally uses... The generated mask is optimized to ultimately generate a large amount of semantic segmentation data.

[0056] It should be noted that the output features are decoded by the decoders of the image variational autoencoder and the label variational autoencoder to obtain the corresponding output image and label.

[0057] S30. Decode the generated latent label representation, and use a label optimization algorithm based on boundary mode to correct the label correction values ​​of pixels in small target regions or noise regions in the latent label representation to obtain the corrected latent label representation. Based on the latent representation of the image and the corrected latent label representation, obtain the synthetic semantic segmentation dataset.

[0058] In one embodiment of this application, a distribution threshold is set for the label distribution of pixels in the target region in the latent label representation. The label optimization algorithm based on the boundary mode, which corrects the mean of the label correction values ​​of pixels in small target regions or noisy regions in the latent label representation, may include: If the number of pixels in the target region in the latent representation of the label is less than a preset threshold, then a small target region or a noise region is determined. Based on the mode of pixel label values ​​in small target areas or noise areas, and according to the comparison result between the mode of pixel label values ​​and the preset boundary pixel values, the label correction value of the pixels in the small target areas or noise areas is determined.

[0059] For example, imperfections in the above denoising process may introduce speckled noise or label inconsistencies, especially in small target regions and near object boundaries. This noise can cause local labels to deviate from the true category, thus affecting the quality of the segmentation results. This application proposes a label optimization strategy based on the mode of boundaries. By analyzing the label distribution of pixels at the boundary of the target region and utilizing the most common label category (i.e., the mode), the label values ​​of the target region are corrected, thereby enhancing label consistency and reducing the impact of noise.

[0060] Specifically, the processor can set the target region. Let be a sub-region in the image, and ,in, Indicates the area The number of pixels in This threshold is selected based on the features of the dataset, typically corresponding to small targets or noisy regions in the image. These regions, due to their small size and sparse pixel count, are easily affected by noise and require special label optimization processing. To improve label accuracy, the processor can define the boundary pixel set of this region as... And calculate the mode of these boundary pixel label values. As a basis for adjusting the target area label:

[0061] in, Represents pixels The label value at that location, Let be the indicator function, and count the frequency of each category label in the boundary pixels. Calculate the mode. Representing the most common category among boundary pixels, the processor uses it as a region. Correction values ​​for inner pixel labels:

[0062] In another embodiment of this application, the text-to-image diffusion model may employ the Unidiffuser model.

[0063] In summary, this application models the joint distribution of images and labels within a diffusion model, enabling their collaborative generation within the same latent representation space. This achieves the generation of high-quality semantic segmentation datasets across large scales, multiple categories, and various scenarios, while ensuring semantic consistency, category boundary accuracy, and spatial structure diversity between the generated images and labels. This provides efficient, accurate, and controllable data support for downstream intensive prediction tasks, effectively enhancing the training performance and generalization ability of downstream models in complex scenarios.

[0064] Based on the above method embodiments, this application also provides a method for generating a semantic segmentation dataset based on joint diffusion of image labels, comprising: a dataset acquisition module for acquiring a real semantic segmentation dataset, the dataset containing RGB images and their corresponding semantic segmentation masks; an encoder training module for constructing a label variational autoencoder, and using the label variational autoencoder to process the input binary encoded semantic label image and the cross-loss entropy of the output decoded semantic label image to train the label variational autoencoder, thereby obtaining the trained label variational autoencoder; and a latent data representation module for representing the latent data based on the image variational autoencoder, image encoder, text encoder, and trained label variational autoencoder. A text-based denoising diffusion probability model is constructed using an autoencoder and a UViT network. The RGB image, semantic label image, and text description are projected into the same latent space through encoding using this model to obtain the first latent representation. Then, a joint diffusion and denoising process is used to recover the latent representations of the image and the labels. A synthetic semantic segmentation module decodes the generated label latent representations and uses a boundary mode-based label optimization algorithm to correct the label correction values ​​of pixels in small target regions or noise regions within the label latent representation, resulting in the corrected label latent representation. Based on the latent representations of the image and the corrected label latent representations, a synthetic semantic segmentation dataset is obtained.

[0065] Based on the above method embodiments, this application also provides an electronic device, the electronic device comprising: At least one processor, memory, and input / output unit; The memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute the aforementioned semantic segmentation dataset generation method based on joint diffusion of image labels.

[0066] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for generating semantic segmentation datasets based on joint diffusion of image labels, characterized in that, include: Obtain a real semantic segmentation dataset, which contains RGB images and their corresponding semantic segmentation masks; Construct a label variational autoencoder, and use the label variational autoencoder to process the input binary encoded semantic label image and the cross loss entropy of the output decoded semantic label image to train the label variational autoencoder, thus obtaining the trained label variational autoencoder. A text-based denoising diffusion probability model is constructed based on an image variational autoencoder, an image encoder, a text encoder, a trained label variational autoencoder, and a UViT network. The RGB image, the semantic label image, and the text description are projected into the same latent space through the text-based denoising diffusion probability model to obtain the first latent representation. Then, the latent representation of the image and the latent representation of the label are recovered through a joint diffusion and denoising process. The generated latent label representation is decoded, and a label optimization algorithm based on boundary mode is used to correct the label correction values ​​of pixels in small target regions or noisy regions in the latent label representation, so as to obtain the corrected latent label representation. Based on the latent representation of the image and the corrected latent label representation, a synthetic semantic segmentation dataset is obtained.

2. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 1, characterized in that, The construction of the label variational autoencoder, and the use of the cross-loss entropy of the input binary encoded semantic label image and the output decoded semantic label image to train the label variational autoencoder, to obtain the trained label variational autoencoder, includes: Construct a label variational autoencoder based on Label VAE; After the semantically labeled image is encoded into binary code, it is input into the label encoder in Label VAE to obtain latent features; Latent features are obtained by decoding latent features using the Label VAE label decoder; Using the cross-entropy loss function of the binary encoded semantic label image and latent features as the objective function, a label variational autoencoder is trained to obtain the trained label variational autoencoder.

3. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 1, characterized in that, The image variational autoencoder uses the Image VAE model, the image encoder uses the CLIP ImageEncoder model, and the text encoder uses the CLIP Text Encoder model.

4. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 3, characterized in that, The text-based denoising diffusion probability model encoding projects the RGB image, semantically labeled image, and text description into the same latent space to obtain a latent representation, including: RGB images are processed using a multimodal model to obtain text descriptions; The CLIP Image Encoder is used to process RGB images to obtain image latent features, and the CLIP Text Encoder is used to process text descriptions to obtain text latent features. Based on the text latent features and image latent features, the latent representation of the image is determined. A label variational autoencoder is used to process semantic label images to obtain latent label representations; Based on the label latent representation and the image latent representation, the first latent representation is obtained by projecting it onto the same latent space.

5. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 1, characterized in that, The process of recovering the latent representation of the image and the latent representation of the label through joint diffusion and denoising includes: A second latent representation is determined based on the latent representation to characterize the original RGB image and the original semantic label image; The second latent representation is processed based on a preset noise-adding expression to obtain the joint noise-adding distribution of the third latent representation and the second latent representation; A joint denoising distribution is constructed based on the third latent representation at the current time step, the textual latent features, and the third latent representation at the previous time step.

6. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 5, characterized in that, After processing the third latent representation and text latent features at the current time step using a denoising network, the difference between the denoising network and the third latent representation is calculated to obtain the first multiplier. The second multiplier is then determined based on the noise time step. The denoised mean of the joint denoised distribution is obtained by multiplying the first multiplier and the second multiplier.

7. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 1, characterized in that, The model used to train the text-based denoising diffusion probability model is a denoising network, and the mean squared error loss is determined based on the output of the denoising network and the output of the text-based denoising diffusion probability model.

8. The semantic segmentation dataset generation method based on joint diffusion of image labels as described in claim 1, characterized in that, Setting a distribution threshold for the label distribution of pixels in the target region within the latent label representation, the label optimization algorithm based on the boundary mode corrects the label correction values ​​of pixels in small target regions or noisy regions within the latent label representation, including: If the number of pixels in the target region in the latent representation of the label is less than a preset threshold, then a small target region or a noise region is determined. Based on the mode of pixel label values ​​in small target areas or noise areas, and according to the comparison result between the mode of pixel label values ​​and the preset boundary pixel values, the label correction value of the pixels in the small target areas or noise areas is determined.

9. A method for generating semantic segmentation datasets based on joint diffusion of image labels, characterized in that, include: The dataset acquisition module is used to acquire a real semantic segmentation dataset, which includes RGB images and their corresponding semantic segmentation masks. The encoder training module is used to construct a label variational autoencoder and use the label variational autoencoder to process the input binary encoded semantic label image and the cross loss entropy of the output decoded semantic label image to train the label variational autoencoder, thus obtaining the trained label variational autoencoder. The latent data representation module is used to construct a text-based denoising diffusion probability model based on the image variational autoencoder, image encoder, text encoder, trained label variational autoencoder and UViT network. After the RGB image, semantic label image and text description are projected into the same latent space through the text-based denoising diffusion probability model to obtain the first latent representation, the latent representation of the image and the latent representation of the label are recovered through the joint diffusion and denoising process. The synthetic semantic segmentation module is used to decode the generated latent label representation. It uses a boundary mode-based label optimization algorithm to correct the label correction values ​​of pixels in small target regions or noise regions in the latent label representation, thus obtaining the corrected latent label representation. Based on the latent representation of the image and the corrected latent label representation, the synthetic semantic segmentation dataset is obtained.

10. An electronic device, characterized in that, The electronic device includes: At least one processor, memory, and input / output unit; The memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute the semantic segmentation dataset generation method based on joint diffusion of image labels according to any one of claims 1 to 7.

Images