Composition type scene modeling method and device based on diffusion model taking object as center

By introducing diffusion model and multiple denoising processes into object-centric representation learning model, the problem of difficult mask modeling in complex scenarios is solved, and a higher quality scene image segmentation effect is achieved.

CN118967738AActive Publication Date: 2024-11-15FUDAN UNIVERSITY

Patent Information

Application Number
CN202411028457.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

Existing object-centric representation learning models are difficult to explicitly model object masks when generating complex scene images, limiting the generalization ability of the model, and the diffusion model performs poorly when processing complex natural data sets.

Method used

Using an object-centered diffusion model, the image encoding-noise module, the object representation encoder module and the hybrid diffusion image decoder module are used to generate more accurate object and background segmentation results through the image encoding-noise module, the object representation encoder module and the hybrid diffusion image decoder module, combined with random noise sampling and multiple denoising processes. .

Benefits of technology

It improves the accuracy and quality of scene image segmentation and enhances the model's performance ability in complex scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967738A_ABST
    Figure CN118967738A_ABST
Patent Text Reader

Abstract

The invention provides a constitutive scene modeling method and device based on a diffusion model taking an object as a center, and the method is characterized in that the method comprises the following steps: S1, obtaining original noise; s2, inputting the scene image and the original noise into an image segmentation model together to obtain a preliminary object segmentation result and a background segmentation result; s3, up-sampling is carried out on the preliminary object segmentation result and the preliminary background segmentation result to obtain segmentation results, and the image segmentation model comprises an image coding-noise adding module used for coding and noise adding of a scene image in sequence to obtain a noise-added image potential representation; the object representation encoder module is used for obtaining object representation according to the scene image; and the mixed diffusion image decoder module is used for obtaining a preliminary object segmentation result and a preliminary background segmentation result according to the object representation and the noise-added image potential representation. In a word, the method can perform image segmentation on the scene image more accurately.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a compositional scene modeling method and device based on an object-centric diffusion model. Background Art

[0002] Humans use visual observation in a combined way to understand their living environment, obtain information about individual objects through vision, and then understand the whole world based on this information. As an important research direction, Object-Centric Learning (OCL) aims to enable machines to learn the feature representation of objects in a combined way like humans. This ability is crucial for the study of causal inference and out-of-distribution generation. Since the OCL method usually learns object representations through unsupervised reconstruction, it does not require extremely costly manual annotation and can be applied to various downstream computer vision tasks.

[0003] Early OCL methods typically adopt a pixel-mixing decoder that generates object masks and appearances separately, and then uses the masks as mixing weights to combine the appearances for image reconstruction. Since such simple decoders have difficulties in processing complex scenes, recent OCL methods have focused on exploring more powerful decoder structures to enhance the model's ability to process complex natural datasets. However, these models do not explicitly model object masks during the generation process, making it difficult to generate images of individual objects based on the representation, limiting the model's generalization ability. Therefore, an intuitive idea is to combine the hybrid model with an advanced diffusion model to enable the model to imagine or reconstruct complex scene images like humans.

[0004] Diffusion models are a new generative model for generating images through a denoising process. So far, diffusion models have shown impressive capabilities in image generation. Among these methods, Latent Diffusion Models, or LDM, first transform the image into a latent space and train a diffusion model on the latent space. Recent OCL methods, such as SlotDiffusion and LSD, introduce LDM into the field of object-centric representation learning and achieve excellent performance, demonstrating the great potential of diffusion models in object-centric representation learning.

[0005] However, the existing object-centric representation learning still has a lot of room for improvement. Summary of the invention

[0006] The present invention is made to solve the above-mentioned problems, and aims to provide a composition scene modeling method and device based on an object-centered diffusion model.

[0007] The present invention provides a compositional scene modeling method based on an object-centered diffusion model, which is used to obtain a segmentation result of a scene image and has the following characteristics, comprising the following steps: step S1, obtaining original noise by random sampling; step S2, inputting the scene image and the original noise into a trained image segmentation model together to obtain a preliminary object segmentation result and a background segmentation result; step S3, upsampling the preliminary object segmentation result and the preliminary background segmentation result respectively, and obtaining the corresponding final object segmentation result and the final background segmentation result as the segmentation result respectively, wherein the image segmentation model comprises: an image encoding-noising module, which is used to encode and add noise to the scene image in sequence to obtain a potential representation of the noisy image; an object representation encoder module, which is used to obtain an object representation based on the scene image; and a mixed diffusion image decoder module, which is used to obtain a preliminary object segmentation result and a preliminary background segmentation result based on the object representation and the potential representation of the noisy image.

[0008] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the image encoding-noising module includes: an image encoder unit, used to encode the scene image to obtain a potential representation of the image; and a noisy unit, used to add noise to the potential representation of the image to obtain a potential representation of the noisy image.

[0009] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the object representation encoder module includes: a feature extraction unit, used to convert the scene image into a feature map; an instance coloring and folding process unit, used to obtain the object representation according to the feature map.

[0010] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the instance coloring and folding process unit includes: an inference mask generation subunit, which is used to cluster the pixels of the feature map to obtain the inference masks corresponding to each class; an object representation generation subunit, which is used to combine the inference masks with the feature map as weights for each inference mask, and calculate the corresponding weighted average feature map as the object representation.

[0011] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the hybrid diffusion image decoder module is used to perform multiple denoising on the potential representation of the noisy image to obtain a preliminary object segmentation result and a preliminary background segmentation result, and the hybrid diffusion image decoder module includes: a calculation unit, which is used to calculate the corresponding object mask and the denoised image potential prediction corresponding to each object representation according to the object representation and the image potential representation corresponding to the denoising each time; a normalization unit, which is used to perform normalization operation on all object masks each time denoising to obtain a corresponding normalized mask; a summation unit, which is used to sum the potential predictions of each denoised image with the normalized mask as the weight each time denoising to obtain an image potential representation prediction, and obtain the image potential representation for the next denoising according to the image potential representation prediction, and the image potential representation corresponding to the first denoising is the noisy image potential representation.

[0012] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the training process of the image segmentation model includes the following steps: step T1, taking multiple existing pictures as training pictures to construct a training data set; step T2, selecting training pictures from the training data set, processing them respectively through the image encoder unit and the object representation encoder module, and obtaining the training image potential representation and the training inference mask respectively; step T3, sampling time step t and corresponding noise, adding noise to the training image potential representation, and obtaining the training noisy image potential representation; step T4, processing the training noisy image potential representation through the hybrid diffusion image decoder module , and obtain the generated mask and the intermediate denoised image potential representation; step T5, calculate the loss function according to the training image potential representation, the training inference mask, the generated mask and the intermediate denoised image potential representation, and obtain the loss result; step T6, back-propagate and update the parameters of the image segmentation model according to the loss result; step T7, repeat steps T2 to T6 until the training completion condition is met, and the trained image segmentation model is obtained, the generated mask is the normalized mask obtained after the training noisy image potential representation is used as the image potential representation for denoising, and the intermediate denoised image potential representation is the image potential representation prediction obtained after the training noisy image potential representation is used as the image potential representation for denoising.

[0013] The compositional scene modeling method based on the object-centered diffusion model provided by the present invention may also have the following characteristics: wherein, in step T3, the potential representation z of the training noisy image corresponding to the time step t t The calculation expression is: Where z0 is the potential representation of the training image, ∈ t is the noise corresponding to time step t, β tis the variance sequence β that increases with time t 1:t The tth value in .

[0014] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, there may also be such a feature: wherein the calculation expression of the loss function is: Where η is a hyperparameter, To calculate m through the KL function inf and KL divergence between two class distributions, m inf Infer masks for training, is to generate the mask, z0 is the potential representation of the training image, g θ (z t ,t,S) is the latent representation of the intermediate denoised image.

[0015] The present invention also provides a compositional scene modeling device based on an object-centered diffusion model, which is used to obtain the segmentation result of a scene image and has the following characteristics: a noise generation unit, which is used to obtain the original noise by random sampling; a preliminary segmentation unit, which stores a trained image segmentation model and is used to input the scene image and the original noise into the trained image segmentation model to obtain a preliminary object segmentation result and a background segmentation result; an upsampling unit, which is used to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively to obtain the corresponding final object segmentation result and the final background segmentation result as the segmentation result, wherein the image segmentation model includes: an image encoding-noise module, which is used to encode and add noise to the scene image in sequence to obtain a potential representation of the noisy image; an object representation encoder module, which is used to obtain an object representation based on the scene image; and a mixed diffusion image decoder module, which is used to obtain a preliminary object segmentation result and a preliminary background segmentation result based on the object representation and the potential representation of the noisy image.

[0016] Functions and Effects of the Invention

[0017] According to the compositional scene modeling method and device based on the object-centered diffusion model involved in the present invention, a hybrid diffusion image decoder module is used to perform multiple denoising on the potential representation of the noisy image through the object representation to obtain the final image potential representation, thereby obtaining a higher quality image potential representation prediction than that obtained by single denoising, and then the normalized mask corresponding to the image potential representation can also have a better scene segmentation result. Therefore, the compositional scene modeling method and device based on the object-centered diffusion model of the present invention can more accurately perform image segmentation on the scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of a scene image segmentation device in an embodiment of the present invention;

[0019] Figure 2 is a block diagram of an image segmentation model 200 in an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a process for training an image segmentation model in an embodiment of the present invention;

[0021] Figure 4 It is a flowchart of a compositional scene modeling method based on an object-centered diffusion model in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the compositional scene modeling method and device based on the object-centric diffusion model of the present invention.

[0023] This embodiment provides a compositional scene modeling device based on an object-centered diffusion model, hereinafter referred to as a scene image segmentation device, which is used to obtain a segmentation result of a scene image.

[0024] In this embodiment, the scene image is an image of a scene at a certain viewing angle, and the scene image includes at least one object and a background.

[0025] Figure 1 It is a block diagram of a scene image segmentation device in an embodiment of the present invention.

[0026] like Figure 1 As shown, the scene image segmentation device 100 includes a noise generating unit 10 , a preliminary segmentation unit 20 and an up-sampling unit 30 .

[0027] The noise generating unit 10 is used to obtain the original noise by random sampling. In this embodiment, the noise generating unit 10 obtains the original noise by sampling from a standard Gaussian distribution.

[0028] The preliminary segmentation unit 20 stores a trained image segmentation model and is used to input the scene image and the original noise into the trained image segmentation model to obtain preliminary object segmentation results and background segmentation results.

[0029] Figure 2 is a block diagram of the image segmentation model 200 in an embodiment of the present invention.

[0030] like Figure 2 As shown, the image segmentation model 200 includes an image encoding-noising module 40 , an object representation encoder module 50 and a hybrid diffusion image decoder module 60 .

[0031] The image coding-noising module 40 is used to sequentially encode and add noise to the scene image to obtain a potential representation of the noisy image, where the added noise is the original noise. The image coding-noising module 40 includes an image encoder unit 401 and a noise adding unit 402.

[0032] The image encoder unit 401 is used to encode the scene image to obtain a potential representation of the image.

[0033] In this embodiment, the image encoder unit 401 is a pre-trained VQ-VAE encoder. The image potential representation contains information of the entire scene image, that is, one image potential representation corresponds to one scene image.

[0034] The noise adding unit 402 is used to add noise to the image potential representation to obtain a noisy image potential representation.

[0035] The object representation encoder module 50 is used to obtain the object representation according to the scene image. The object representation encoder module 50 includes a feature extraction unit 501 and an instance coloring and folding process unit 502.

[0036] The feature extraction unit 501 is used to convert the scene image into a feature map. In this embodiment, the feature extraction unit 501 is a UNet backbone network, which extracts information from the scene image at different scales to construct a feature map.

[0037] The example coloring stick process unit 502 is used to obtain the object representation according to the feature map. The example coloring stick process unit 502 includes an inference mask generation subunit 5021 and an object representation generation subunit 5022.

[0038] The inference mask generation subunit 5021 is used to cluster the pixels of the feature map to obtain the inference mask corresponding to each class. In this embodiment, the number of classes is the total number of objects in the scene image plus 1, that is, the total number of all objects and backgrounds, and the number of classes is set as a hyperparameter.

[0039] The object representation generation subunit 5022 is used to calculate the corresponding weighted average feature map as the object representation for each inferred mask, using the inferred mask as a weight combined with the feature map. In this embodiment, each object representation corresponds to an object or background in the scene image, that is, the background and foreground objects are regarded as equal, and each object representation contains information about the corresponding object, so the set formed by all object representations can contain information about the entire scene image.

[0040] Compared with the existing slot attention encoder, the example coloring stick process unit 502 of this embodiment adopts a more direct way to extract object representation. The existing slot attention encoder extracts the representation of each object through multiple competitions between object representations, which will cause the representations of each object to be entangled to a certain extent. The example coloring stick process of the example coloring stick process unit 502 of this embodiment explicitly models the object mask, and then directly uses the object mask as the weight to calculate the weighted average feature map as the object representation, which can effectively reduce the entanglement between object representations.

[0041] The hybrid diffusion image decoder module 60 is used to obtain a preliminary object segmentation result and a preliminary background segmentation result according to the object representation and the noisy image potential representation.

[0042] The mixed diffusion image decoder module 60 is used to perform multiple denoising on the potential representation of the noisy image to obtain a preliminary object segmentation result and a preliminary background segmentation result. The mixed diffusion image decoder module 60 includes a calculation unit 601, a normalization unit 602 and a summing unit 603. In this embodiment, the mixed diffusion image decoder module 60 uses all object representations as denoising conditions.

[0043] The calculation unit 601 is used to calculate the corresponding object mask and the denoised image potential prediction corresponding to each object representation according to the object representation and the image potential representation corresponding to the denoising each time denoising.

[0044] The normalization unit 602 is used to perform a normalization operation on all object masks each time denoising to obtain a corresponding normalized mask. The normalized mask obtained by the last denoising in this embodiment is the preliminary object segmentation result and the preliminary background segmentation result.

[0045] The summing unit 603 is used to sum the potential predictions of each denoised image using the normalized mask as a weight during each denoising operation to obtain a potential representation prediction of the image.

[0046] The image potential representation for the next denoising is predicted based on the image potential representation. The image potential representation corresponding to the first denoising is the noisy image potential representation.

[0047] The design of the latent conditional diffusion model of the hybrid diffusion image decoder module 60 in this embodiment mainly refers to the standard implicit diffusion model. K The conditional distribution of the latent representation z0 under the condition of} can be described as p(z0|S)=∫p(z 0:T |S)dz 1:T , where the joint distribution p(z 0:T |S) is modeled as a Markov chain:

[0048] p(z 0:T |S)=p(z T )∏ t=T,...,1 p(z t-1 |z t ,S),

[0049] Where p(z T )=N(0,I),p(z t-1 |z t ,S)=N(μ θ (z t ,t,S),β t I).

[0050] In this embodiment, the image potential representation z t In the denoising process, according to the image potential representation z t Calculate the image potential representation prediction Then predict based on the image potential representation Calculate μ θ (z t ,t,S), and then p(z t-1 |z t ,S)=N(μ θ (z t ,t,S),β t I) Sampling, get the potential image representation z for the next denoising t-1 .

[0051] μ θ (z t ,t,S) is:

[0052]

[0053] In this embodiment, the calculation unit 601, the normalization unit 602 and the summation unit 603 constitute a UNet network shared between all object representations, which is used to fully extract the information of each object representation. A cross-attention mechanism is used between the individual object representations and the image feature maps at different levels of the UNet. Although this design will ignore image information to some extent, the goal is to predict the unnoised potential representation rather than the noise, the importance of image information is relatively weak, and the object representations therefore contain more information, thereby improving their own quality.

[0054] Figure 3 It is a schematic diagram of the process of training an image segmentation model in an embodiment of the present invention.

[0055] like Figure 3 As shown, the training process of the image segmentation model 200 includes the following steps:

[0056] Step T1: Use multiple existing pictures as training pictures to construct a training data set.

[0057] In step T2, training pictures are selected from the training data set, and processed by the image encoder unit 401 and the object representation encoder module 50, respectively, to obtain the training image potential representation and the training inference mask.

[0058] Step T3, sampling time step t and corresponding noise, adding noise to the potential representation of the training image, and obtaining the potential representation of the training noisy image. In this embodiment, the noise is standard Gaussian noise, and the mixed diffusion image decoder module 60 performs T times of denoising on the potential representation of the noisy image, then t∈{1,...,T}.

[0059] Among them, the potential representation z of the training noisy image corresponding to time step t t The calculation expression is:

[0060]

[0061]

[0062] α t =1-β t ,

[0063] Where z0 is the potential representation of the training image, ∈ t is the noise corresponding to time step t, β t is the variance sequence β that increases with time t 1:t The tth value in .

[0064] Step T4, the training noisy image latent representation is processed by the hybrid diffusion image decoder module 60 to obtain a generated mask and an intermediate denoised image latent representation.

[0065] The generated mask is a normalized mask obtained by denoising the training noisy image potential representation as the image potential representation. The intermediate denoised image potential representation is an image potential representation prediction obtained by denoising the training noisy image potential representation as the image potential representation.

[0066] Step T5, calculating the loss function according to the training image potential representation, the training inference mask, the generated mask and the intermediate denoised image potential representation to obtain the loss result.

[0067] Among them, in step T5, the calculation expression of the loss function is:

[0068]

[0069] L diff =||z0-g θ (z t,t,S)|| 2 ,

[0070] Where η is a hyperparameter, To calculate m through the KL function inf and KL divergence between two class distributions, m inf Infer masks for training, is to generate the mask, z0 is the potential representation of the training image, g θ (z t ,t,S) is the latent representation of the intermediate denoised image.

[0071] In this embodiment, nograd is understood as: generating a mask The value of is assigned to a variable c, and the variable c is a constant and has nothing to do with any network parameters. Therefore, the final loss function is L = L diff +ηKL(m inf ||c), that is, the parameters of the model will only be affected by L during back propagation diff and m inf The impact of these two items.

[0072] In this embodiment, in order to ensure that the training inference mask m inf and generate the mask With the same spatial dimensions, the mask will be generated Upsample to training inference mask m inf The size of , thus calculating the loss function.

[0073] Step T6, back-propagating and updating the parameters of the image segmentation model 200 according to the loss result.

[0074] Step T7, repeat steps T2 to T6 until the training completion condition is met, and a trained image segmentation model 200 is obtained.

[0075] The up-sampling unit 30 is used to up-sample the preliminary object segmentation result and the preliminary background segmentation result respectively, and obtain the corresponding final object segmentation result and the final background segmentation result as the segmentation results respectively. In this embodiment, the up-sampling is bilinear interpolation up-sampling.

[0076] The following describes the process of using the scene image segmentation device 100 to perform a compositional scene modeling method based on an object-centered diffusion model in conjunction with the accompanying drawings.

[0077] Figure 4 It is a flowchart of a compositional scene modeling method based on an object-centered diffusion model in an embodiment of the present invention.

[0078] like Figure 4As shown, the compositional scene modeling method based on the object-centered diffusion model includes the following steps:

[0079] Step S1 : using the noise generating unit 10 to obtain original noise by random sampling.

[0080] Step S2: using the preliminary segmentation unit 20 to input the scene image and the original noise into the trained image segmentation model 200 to obtain preliminary object segmentation results and background segmentation results.

[0081] Step S3, using the upsampling unit 30 to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain the corresponding final object segmentation result and final background segmentation result as the segmentation results respectively.

[0082] In this embodiment, on the commonly used CLEVRTEX dataset with a resolution of 128, the performance of the compositional scene modeling method based on the object-centered diffusion model, that is, the present method and the existing LSD method, is tested. The segmentation performance indicators ARI-FG and mIOU of the present method are 77.9 and 40.5 respectively, while the ARI-FG and mIOU of the LSD method are 64.2 and 29.0 respectively. It can be seen that the present method is better than the existing LSD method.

[0083] Functions and Effects of the Embodiments

[0084] According to the compositional scene modeling method and device based on the object-centered diffusion model involved in this embodiment, the hybrid diffusion image decoder module is used to perform multiple denoising on the potential representation of the noisy image through the object representation to obtain the final image potential representation, thereby obtaining a higher quality image potential representation prediction than that obtained by single denoising, and thus the normalized mask corresponding to the image potential representation can also have a better scene segmentation result. In short, this method can more accurately perform image segmentation on scene images.

[0085] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A compositional scene modeling method based on an object-centered diffusion model to obtain the segmentation result of a scene image. It is characterized in that The following steps are involved: Step S1, obtaining original noise by random sampling; Step S2, inputting the scene image and the original noise into a trained image segmentation model to obtain a preliminary object segmentation result and a background segmentation result; Step S3, upsampling the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain the corresponding final object segmentation result and final background segmentation result as the segmentation results respectively, Wherein, the image segmentation model includes: An image encoding-noising module, used for sequentially encoding and adding noise to the scene image to obtain a potential representation of the noisy image; An object representation encoder module, configured to obtain an object representation based on the scene image; A hybrid diffusion image decoder module is used to obtain the preliminary object segmentation result and the preliminary background segmentation result according to the object representation and the potential representation of the noisy image.

2. The compositional scene modeling method based on the object-centered diffusion model according to claim 1 , Features: Wherein, the image encoding-noise adding module comprises: An image encoder unit, configured to encode the scene image to obtain a potential image representation; The denoising unit is used to add noise to the image potential representation to obtain the noisy image potential representation.

3. The compositional scene modeling method based on the object-centered diffusion model according to claim 2 , Features: Wherein, the object representation encoder module comprises: A feature extraction unit, used for converting the scene image into a feature map; The instance coloring and folding stick process unit is used to obtain the object representation according to the feature map.

4. The compositional scene modeling method based on the object-centered diffusion model according to claim 2 , Features: Wherein, the example colored stick folding process unit comprises: An inference mask generation subunit, used to cluster the pixels of the feature map to obtain an inference mask corresponding to each class; The object representation generating subunit is used to calculate the corresponding weighted average feature map as the object representation by combining the inferred mask with the feature map as a weight for each of the inferred masks.

5. The compositional scene modeling method based on the object-centric diffusion model according to claim 4 , Features: The hybrid diffusion image decoder module is used to perform multiple denoising on the potential representation of the noisy image to obtain the preliminary object segmentation result and the preliminary background segmentation result. The hybrid diffusion image decoder module comprises: A calculation unit, configured to calculate, during each denoising operation, a corresponding object mask and a denoised image potential prediction corresponding to each of the object representations according to the object representation and the image potential representation corresponding to the denoising operation; A normalization unit, used for performing a normalization operation on all the object masks during each denoising operation to obtain a corresponding normalized mask; A summing unit is used to sum each of the denoised image potential predictions with the normalized mask as a weight to obtain an image potential representation prediction during each denoising operation. According to the image potential representation, the potential representation of the image to be denoised next time is predicted, The potential representation of the image corresponding to the first denoising is the potential representation of the noisy image.

6. The compositional scene modeling method based on the object-centric diffusion model according to claim 5 , Features: The training process of the image segmentation model includes the following steps: Step T1, using multiple existing pictures as training pictures to construct a training data set; Step T2, selecting a training picture from the training data set, and processing them through the image encoder unit and the object representation encoder module, respectively, to obtain a training image potential representation and a training inference mask; Step T3, sampling time step t and corresponding noise, adding noise to the potential representation of the training image, and obtaining a potential representation of the training noisy image; Step T4, processing the training noisy image potential representation via the hybrid diffusion image decoder module to obtain a generated mask and an intermediate denoised image potential representation; Step T5, calculating a loss function according to the training image potential representation, the training inference mask, the generated mask and the intermediate denoised image potential representation to obtain a loss result; Step T6, back-propagating and updating the parameters of the image segmentation model according to the loss result; Step T7, repeating steps T2 to T6 until the training completion condition is met, thereby obtaining the trained image segmentation model. The generated mask is the normalized mask obtained by performing a denoising operation on the potential representation of the training noisy image as the potential representation of the image. The intermediate denoised image potential representation is the image potential representation prediction obtained after the training denoised image potential representation is used as the image potential representation for one denoising operation.

7. The compositional scene modeling method based on the object-centered diffusion model according to claim 6, characterized in that: in, In step T3, the potential representation z of the training noisy image corresponding to time step t is t The calculation expression is: α t =1-β t , Where z 0 is the potential representation of the training image, ε t is the noise corresponding to time step t, β t is the variance sequence β that increases with time t 1:t The tth value in .

8. The compositional scene modeling method based on the object-centered diffusion model according to claim 6, characterized in that: in, The calculation expression of the loss function is: L diff =||z 0 -g θ (z t ,t,S)|| 2 , Where η is a hyperparameter, To calculate m through the KL function inf and KL divergence between two class distributions, m inf inferring a mask for said training, Generate a mask for the 0 is the potential representation of the training image, g θ (z t ,t,S) is the potential representation of the intermediate denoised image.

9. A compositional scene modeling device based on an object-centered diffusion model, used to obtain a segmentation result of a scene image, It is characterized in that include: A noise generating unit, used for obtaining original noise by random sampling; A preliminary segmentation unit storing a trained image segmentation model, for inputting the scene image and the original noise together into the trained image segmentation model to obtain preliminary object segmentation results and background segmentation results; an upsampling unit, used to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain the corresponding final object segmentation result and final background segmentation result as the segmentation result, Wherein, the image segmentation model includes: An image encoding-noising module, used for sequentially encoding and adding noise to the scene image to obtain a potential representation of the noisy image; An object representation encoder module, configured to obtain an object representation based on the scene image; A hybrid diffusion image decoder module is used to obtain the preliminary object segmentation result and the preliminary background segmentation result according to the object representation and the potential representation of the noisy image.

Citation Information

Patent Citations

  • Method and device for determining body part semantic graph, model training method and pedestrian re-identification method

    CN112836611A

  • Image processing method and electronic equipment

    CN117670871A

  • Estimating 3D scene representations of images

    US20240046516A1

Cited By

  • Video sequence prediction method and system based on object segmentation guidance

    CN120932161A