Object-based composition modeling method and device based on object-centric diffusion model

Abstract

The application provides a kind of based on object center diffusion model's component formula scene modeling method and device, with such characteristics, comprising the following steps: step S1, obtains original noise;Step S2, scene image and original noise are input into image segmentation model together, obtain preliminary object segmentation result and background segmentation result;Step S3, preliminary object segmentation result and preliminary background segmentation result are respectively up-sampled, obtain segmentation result, wherein, image segmentation model includes: image coding-noise adding module, for sequentially coding and noise adding to scene image, obtain noisy image latent representation;Object representation encoder module is used to obtain object representation according to scene image;Mixed diffusion image decoder module is used to obtain preliminary object segmentation result and preliminary background segmentation result according to object representation and noisy image latent representation.In short, the method can more accurately carry out image segmentation to scene image.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning technology, and in particular relates to a compositional scene modeling method and device based on an object-centered diffusion model. Background Art

[0002] Humans use vision to observe and understand their environment in a combinatorial manner, acquiring information about individual objects and then using this information to understand the world as a whole. Object-Centric Learning (OCL), a key research direction, aims to enable machines to learn feature representations of objects in a combinatorial manner, similar to humans. This capability is crucial for research in causal inference and out-of-distribution generation. Because OCL methods typically learn object representations through unsupervised reconstruction, they do not require costly manual annotation and can be applied to a variety of downstream computer vision tasks.

[0003] Early OCL methods typically employed a pixel-mixing decoder that generated object masks and appearances separately, and then used the masks as mixing weights to combine the appearances for image reconstruction. Because such simple decoders have difficulty processing complex scenes, recent OCL methods have focused on exploring more powerful decoder structures to enhance the model's ability to process complex natural datasets. However, these models do not explicitly model object masks during generation, making it difficult to generate images of individual objects based on the representations, limiting the model's generalization capabilities. Therefore, an intuitive idea is to combine the hybrid model with an advanced diffusion model to enable the model to imagine or reconstruct complex scene images like humans.

[0004] Diffusion models are a new generative model for generating images through a denoising process. To date, diffusion models have demonstrated impressive capabilities in image generation. Among these methods, Latent Diffusion Models (LDMs) first transform images into a latent space and then train a diffusion model on this latent space. Recent OCL methods, such as SlotDiffusion and LSD, have introduced LDMs to the field of object-centric representation learning and achieved excellent performance, demonstrating the great potential of diffusion models in object-centric representation learning.

[0005] However, the existing object-centric representation learning still has a lot of room for improvement. Summary of the Invention

[0006] The present invention is made to solve the above problems, and its purpose is to provide a compositional scene modeling method and device based on an object-centered diffusion model.

[0007] The present invention provides a compositional scene modeling method based on an object-centered diffusion model, which is used to obtain a segmentation result of a scene image and has the following characteristics: step S1, obtaining original noise by random sampling; step S2, inputting the scene image and the original noise into a trained image segmentation model to obtain a preliminary object segmentation result and a background segmentation result; step S3, upsampling the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain corresponding final object segmentation results and final background segmentation results as segmentation results, respectively, wherein the image segmentation model includes: an image encoding-noising module, which is used to encode and add noise to the scene image in sequence to obtain a potential representation of the noisy image; an object representation encoder module, which is used to obtain an object representation based on the scene image; and a hybrid diffusion image decoder module, which is used to obtain a preliminary object segmentation result and a preliminary background segmentation result based on the object representation and the potential representation of the noisy image.

[0008] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the image encoding-noising module includes: an image encoder unit, which is used to encode the scene image to obtain a potential image representation; and a noisy unit, which is used to add noise to the potential image representation to obtain a noisy image potential representation.

[0009] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the object representation encoder module includes: a feature extraction unit for converting the scene image into a feature map; an instance coloring and folding process unit for obtaining the object representation based on the feature map.

[0010] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the instance coloring and folding process unit includes: an inference mask generation subunit, which is used to cluster the pixels of the feature map to obtain the inference mask corresponding to each class; an object representation generation subunit, which is used to combine the inference mask with the feature map as a weight for each inference mask, and calculate the corresponding weighted average feature map as the object representation.

[0011] In the compositional scene modeling method based on the object-centered diffusion model provided by the present invention, it can also have the following characteristics: wherein, the hybrid diffusion image decoder module is used to perform multiple denoising on the noisy image potential representation to obtain preliminary object segmentation results and preliminary background segmentation results, and the hybrid diffusion image decoder module includes: a calculation unit, which is used to calculate the corresponding object mask and the denoised image potential prediction corresponding to each object representation according to the object representation and the image potential representation corresponding to the denoising each time; a normalization unit, which is used to perform normalization operation on all object masks each time denoising to obtain the corresponding normalized mask; a summation unit, which is used to sum the potential predictions of each denoised image with the normalized mask as the weight each time denoising to obtain the image potential representation prediction, and obtain the image potential representation for the next denoising based on the image potential representation prediction, and the image potential representation corresponding to the first denoising is the noisy image potential representation.

[0012] The compositional scene modeling method based on the object-centered diffusion model provided by the present invention may also have the following features: wherein, the training process of the image segmentation model includes the following steps: step T1, taking multiple existing pictures as training pictures to construct a training data set; step T2, selecting training pictures from the training data set, processing them through the image encoder unit and the object representation encoder module respectively, and obtaining the training image potential representation and the training inference mask respectively; step T3, sampling the time step t and the corresponding noise, adding noise to the training image potential representation, and obtaining the training noisy image potential representation; step T4, processing the training noisy image potential representation through the hybrid diffusion image decoder module , obtaining the generated mask and the intermediate denoised image potential representation; step T5, calculating the loss function according to the training image potential representation, the training inference mask, the generated mask and the intermediate denoised image potential representation, and obtaining the loss result; step T6, backpropagating and updating the parameters of the image segmentation model according to the loss result; step T7, repeating steps T2 to T6 until the training completion condition is met, and then obtaining the trained image segmentation model, the generated mask is the normalized mask obtained after denoising the training noisy image potential representation as the image potential representation, and the intermediate denoised image potential representation is the image potential representation prediction obtained after denoising the training noisy image potential representation as the image potential representation.

[0013] The compositional scene modeling method based on the object-centered diffusion model provided by the present invention may also have the following characteristics: wherein, in step T3, the potential representation z of the training noisy image corresponding to the time step t is t The calculation expression is: Where z0 is the potential representation of the training image, ∈ t is the noise corresponding to time step t, β tis the variance sequence β that increases with time t 1:t The tth value in .

[0014] The compositional scene modeling method based on the object-centered diffusion model provided by the present invention may also have the following characteristics: wherein the calculation expression of the loss function is: Where η is a hyperparameter, To calculate m through KL function inf and KL divergence of the two class distributions, m inf Infer masks for training, To generate the mask, z0 is the potential representation of the training image, g θ (z t ,t,S) is the latent representation of the intermediate denoised image.

[0015] The present invention also provides a compositional scene modeling device based on an object-centered diffusion model, which is used to obtain the segmentation result of a scene image and has the following characteristics: a noise generation unit, which is used to obtain original noise through random sampling; a preliminary segmentation unit, which stores a trained image segmentation model and is used to input the scene image and the original noise into the trained image segmentation model to obtain preliminary object segmentation results and background segmentation results; an upsampling unit, which is used to upsample the preliminary object segmentation results and the preliminary background segmentation results respectively, and obtain the corresponding final object segmentation results and final background segmentation results as segmentation results, respectively, wherein the image segmentation model includes: an image encoding-noise addition module, which is used to encode and add noise to the scene image in sequence to obtain a potential representation of the noisy image; an object representation encoder module, which is used to obtain an object representation based on the scene image; and a mixed diffusion image decoder module, which is used to obtain a preliminary object segmentation result and a preliminary background segmentation result based on the object representation and the potential representation of the noisy image.

[0016] Functions and effects of the invention

[0017] According to the compositional scene modeling method and apparatus based on an object-centered diffusion model of the present invention, a hybrid diffusion image decoder module is used to perform multiple denoising operations on the noisy image latent representation using the object representation to obtain a final image latent representation. This results in a higher-quality image latent representation prediction than that obtained by a single denoising operation. Furthermore, the normalized mask corresponding to the image latent representation also produces better scene segmentation results. Therefore, the compositional scene modeling method and apparatus based on an object-centered diffusion model of the present invention can more accurately segment scene images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of a scene image segmentation device according to an embodiment of the present invention;

[0019] Figure 2 is a block diagram of an image segmentation model 200 in an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of the process of training an image segmentation model in an embodiment of the present invention;

[0021] Figure 4 It is a flowchart of a compositional scene modeling method based on an object-centered diffusion model in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and drawings specifically illustrate the compositional scene modeling method and device based on the object-centered diffusion model of the present invention.

[0023] This embodiment provides a compositional scene modeling device based on an object-centered diffusion model, hereinafter referred to as a scene image segmentation device, for obtaining a segmentation result of a scene image.

[0024] In this embodiment, the scene image is an image of a scene at a certain viewing angle, and the scene image includes at least one object and a background.

[0025] Figure 1 4 is a block diagram of a scene image segmentation device in an embodiment of the present invention.

[0026] like Figure 1 As shown, the scene image segmentation device 100 includes a noise generating unit 10 , a preliminary segmentation unit 20 and an upsampling unit 30 .

[0027] The noise generating unit 10 is used to obtain the original noise by random sampling. In this embodiment, the noise generating unit 10 obtains the original noise by sampling from a standard Gaussian distribution.

[0028] The preliminary segmentation unit 20 stores a trained image segmentation model and is used to input the scene image and original noise into the trained image segmentation model to obtain preliminary object segmentation results and background segmentation results.

[0029] Figure 2 FIG. 2 is a block diagram of an image segmentation model 200 according to an embodiment of the present invention.

[0030] like Figure 2 As shown, the image segmentation model 200 includes an image encoding-noising module 40 , an object representation encoder module 50 , and a hybrid diffusion image decoder module 60 .

[0031] The image encoding and denoising module 40 is used to sequentially encode and add noise to the scene image to obtain a potential representation of the noisy image, where the added noise is the original noise. The image encoding and denoising module 40 includes an image encoder unit 401 and a denoising unit 402.

[0032] The image encoder unit 401 is used to encode the scene image to obtain a potential representation of the image.

[0033] In this embodiment, the image encoder unit 401 is a pre-trained VQ-VAE encoder. The image latent representation contains information about the entire scene image, that is, one image latent representation corresponds to one scene image.

[0034] The noise adding unit 402 is used to add noise to the image potential representation to obtain a noisy image potential representation.

[0035] The object representation encoder module 50 is used to obtain an object representation according to a scene image. The object representation encoder module 50 includes a feature extraction unit 501 and an instance coloring and folding process unit 502.

[0036] The feature extraction unit 501 is used to convert the scene image into a feature map. In this embodiment, the feature extraction unit 501 is a UNet backbone network that extracts information from the scene image at different scales to construct a feature map.

[0037] The instance coloring stick process unit 502 is used to obtain an object representation based on the feature map. The instance coloring stick process unit 502 includes an inferred mask generation subunit 5021 and an object representation generation subunit 5022.

[0038] The inferred mask generation subunit 5021 is used to cluster the pixels of the feature map to obtain the inferred masks corresponding to each class. In this embodiment, the number of classes is the total number of objects in the scene image plus 1, that is, the total number of all objects and background. The number of classes is set as a hyperparameter.

[0039] The object representation generation subunit 5022 is configured to combine each inferred mask with the feature map using the inferred mask as a weight, and calculate the corresponding weighted average feature map as the object representation. In this embodiment, each object representation corresponds to an object or background in the scene image. That is, background and foreground objects are treated as equal. Each object representation contains information about the corresponding object, and the set of all object representations can contain information about the entire scene image.

[0040] Compared to existing slot attention encoders, the example coloring stick process unit 502 of this embodiment uses a more direct approach to extract object representations. Existing slot attention encoders extract representations of each object through multiple competitions between object representations, which can, to a certain extent, lead to entanglement between the representations of each object. In contrast, the example coloring stick process of the example coloring stick process unit 502 of this embodiment explicitly models the object mask and then directly uses the object mask as a weight to calculate a weighted average feature map as the object representation. This effectively reduces entanglement between object representations.

[0041] The hybrid diffusion image decoder module 60 is used to obtain a preliminary object segmentation result and a preliminary background segmentation result according to the object representation and the noisy image potential representation.

[0042] The hybrid diffuse image decoder module 60 is used to perform multiple denoising operations on the latent representation of the noisy image to obtain preliminary object segmentation results and preliminary background segmentation results. The hybrid diffuse image decoder module 60 includes a calculation unit 601, a normalization unit 602, and a summation unit 603. In this embodiment, the hybrid diffuse image decoder module 60 uses all object representations as denoising conditions.

[0043] The calculation unit 601 is used to calculate the corresponding object mask and the denoised image potential prediction corresponding to each object representation according to the object representation and the image potential representation corresponding to the denoising each time denoising.

[0044] The normalization unit 602 is used to perform a normalization operation on all object masks during each denoising operation to obtain corresponding normalized masks. In this embodiment, the normalized masks obtained after the last denoising operation are the preliminary object segmentation results and the preliminary background segmentation results.

[0045] The summing unit 603 is used to sum the potential predictions of each denoised image using the normalized mask as a weight during each denoising operation to obtain a potential representation prediction of the image.

[0046] The image potential representation for the next denoising is predicted based on the image potential representation. The image potential representation corresponding to the first denoising is the noisy image potential representation.

[0047] The design of the latent conditional diffusion model of the hybrid diffusion image decoder module 60 in this embodiment mainly refers to the standard implicit diffusion model. K The conditional distribution of the potential representation z0 under the condition can be described as p(z0|S)=∫p(z 0:T |S)dz 1:T , where the joint distribution p(z 0:T |S) is modeled as a Markov chain:

[0048] p(z 0:T |S)=p(z T )∏ t＝T,...,1 p(z t-1 |z t ,S),

[0049] Where p(z T )=N(0,I),p(z t-1 |z t ,S)=N(μ θ (z t ,t,S),β t I).

[0050] In this embodiment, the image potential representation z t In the denoising process, according to the image potential representation z t Calculate the image potential representation prediction Then predict based on the image potential representation Calculate μ θ (z t ,t,S), then p(z t-1 |z t ,S)=N(μ θ (z t ,t,S),β t I) Sampling, get the potential representation z of the image for the next denoising t-1 .

[0051] μ θ (z t ,t,S) is:

[0052]

[0053] In this embodiment, the computation unit 601, the normalization unit 602, and the summation unit 603 form a UNet network shared among all object representations to fully extract the information of each object representation. A cross-attention mechanism is used between the individual object representations and the image feature maps at different levels of the UNet. Although this design will ignore image information to some extent, the goal is to predict the unnoised latent representation rather than the noise. The importance of image information is relatively weak, and the object representations therefore contain more information, thereby improving their own quality.

[0054] Figure 3 3 is a flow chart of training an image segmentation model in an embodiment of the present invention.

[0055] like Figure 3 As shown, the training process of the image segmentation model 200 includes the following steps:

[0056] In step T1, a plurality of existing images are used as training images to construct a training data set.

[0057] In step T2, training pictures are selected from the training data set and processed by the image encoder unit 401 and the object representation encoder module 50 to obtain the training image potential representation and the training inference mask respectively.

[0058] In step T3, the training image latent representation is noised at time step t and the corresponding noise, to obtain the noisy training image latent representation. In this embodiment, the noise is standard Gaussian noise. The mixed diffusion image decoder module 60 performs T denoising on the noisy image latent representation, where t∈{1,...,T}.

[0059] Among them, the potential representation z of the training noisy image corresponding to time step t is t The calculation expression is:

[0060]

[0061]

[0062] α t =1-β t ,

[0063] Where z0 is the potential representation of the training image, ∈ t is the noise corresponding to time step t, β t is the variance sequence β that increases with time t 1:t The tth value in .

[0064] In step T4, the training noisy image latent representation is processed by the hybrid diffused image decoder module 60 to obtain a generated mask and an intermediate denoised image latent representation.

[0065] The generated mask is the normalized mask obtained by denoising the training noisy image latent representation as the image latent representation. The intermediate denoised image latent representation is the image latent representation prediction obtained by denoising the training noisy image latent representation as the image latent representation.

[0066] In step T5, a loss function is calculated based on the training image potential representation, the training inference mask, the generated mask, and the intermediate denoised image potential representation to obtain a loss result.

[0067] Among them, in step T5, the calculation expression of the loss function is:

[0068]

[0069] L diff =||z0-g θ (z t,t,S)|| 2 ,

[0070] Where η is a hyperparameter, To calculate m through KL function inf and KL divergence of the two class distributions, m inf Infer masks for training, To generate the mask, z0 is the potential representation of the training image, g θ (z t ,t,S) is the latent representation of the intermediate denoised image.

[0071] In this embodiment, nograd is understood as: generating a mask The value of is assigned to a variable c, and the variable c is a constant and has nothing to do with any network parameters. Therefore, the final loss function is L = L diff +ηKL(m inf ||c), that is, the parameters of the model will only be affected by L during back propagation. diff and m inf The impact of these two items.

[0072] In this embodiment, in order to ensure the training inference mask m inf and generate a mask With the same spatial dimensions, the mask will be generated Upsample to training inference mask m inf The size of , thus calculating the loss function.

[0073] Step T6: back-propagate and update the parameters of the image segmentation model 200 according to the loss result.

[0074] Step T7, repeat steps T2 to T6 until the training completion condition is met, and a trained image segmentation model 200 is obtained.

[0075] The upsampling unit 30 is used to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively, and obtain the corresponding final object segmentation result and final background segmentation result as the segmentation results. In this embodiment, the upsampling is bilinear interpolation upsampling.

[0076] The following describes the process of using the scene image segmentation device 100 to perform a compositional scene modeling method based on an object-centered diffusion model with reference to the accompanying drawings.

[0077] Figure 4 It is a flowchart of a compositional scene modeling method based on an object-centered diffusion model in an embodiment of the present invention.

[0078] like Figure 4As shown, the compositional scene modeling method based on the object-centered diffusion model includes the following steps:

[0079] In step S1 , the noise generating unit 10 is used to obtain original noise through random sampling.

[0080] In step S2, the scene image and the original noise are input into the trained image segmentation model 200 using the preliminary segmentation unit 20 to obtain preliminary object segmentation results and background segmentation results.

[0081] In step S3 , the upsampling unit 30 is used to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain the corresponding final object segmentation result and final background segmentation result as segmentation results.

[0082] In this example, a performance test was conducted on the commonly used CLEVRTEX dataset with a resolution of 128 pixels. The present method, a compositional scene modeling method based on an object-centered diffusion model, and the existing LSD method were compared. The segmentation performance indicators ARI-FG and mIOU for the present method were 77.9 and 40.5, respectively, while those for the LSD method were 64.2 and 29.0, respectively. This shows that the present method outperforms the existing LSD method.

[0083] Functions and Effects of the Embodiments

[0084] According to the compositional scene modeling method and apparatus based on the object-centric diffusion model of this embodiment, a hybrid diffusion image decoder module is used to perform multiple denoising operations on the noisy image latent representation using the object representation to obtain the final image latent representation. This results in a higher-quality image latent representation prediction than that obtained by a single denoising operation. Consequently, the normalized mask corresponding to the image latent representation also produces better scene segmentation results. In short, this method can more accurately segment scene images.

[0085] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A compositional scene modeling method based on an object-centered diffusion model for obtaining a segmentation result of a scene image, characterized in that: The following steps are involved: Step S1, obtaining original noise by random sampling; Step S2, inputting the scene image and the original noise together into a trained image segmentation model to obtain preliminary object segmentation results and preliminary background segmentation results; Step S3, upsampling the preliminary object segmentation result and the preliminary background segmentation result respectively to obtain the corresponding final object segmentation result and final background segmentation result as the segmentation results, Wherein, the image segmentation model includes: An image encoding and denoising module, configured to sequentially encode and add noise to the scene image to obtain a potential representation of the noisy image; An object representation encoder module is used to obtain an object representation based on the scene image, The object representation encoder module includes: A feature extraction unit, configured to convert the scene image into a feature map; An instance coloring and folding process unit is used to obtain the object representation according to the feature map, The example coloring stick process unit includes: An inference mask generation subunit, configured to cluster the pixels of the feature map to obtain an inference mask corresponding to each class; An object representation generating subunit is configured to calculate a corresponding weighted average feature map as the object representation by combining the inferred mask with the feature map using the inferred mask as a weight; A hybrid diffusion image decoder module is configured to obtain the preliminary object segmentation result and the preliminary background segmentation result based on the object representation and the noisy image potential representation.

2. The compositional scene modeling method based on the object-centered diffusion model according to claim 1, Its characteristics are: The image encoding and denoising module includes: an image encoder unit, configured to encode the scene image to obtain a potential image representation; The denoising unit is configured to add noise to the image potential representation to obtain the noisy image potential representation.

3. The compositional scene modeling method based on the object-centered diffusion model according to claim 2, Its characteristics are: The hybrid diffusion image decoder module is used to perform multiple denoising on the potential representation of the noisy image to obtain the preliminary object segmentation result and the preliminary background segmentation result. The hybrid diffusion image decoder module includes: a computing unit configured to, during each denoising operation, calculate, based on the object representation and the image potential representation corresponding to the denoising operation, a corresponding object mask and a denoised image potential prediction corresponding to each object representation; a normalization unit, configured to perform a normalization operation on all the object masks during each denoising operation to obtain corresponding normalized masks; A summing unit is used to sum each of the denoised image potential predictions with the normalized mask as a weight during each denoising operation to obtain an image potential representation prediction, The potential representation of the image for the next denoising is predicted based on the potential representation of the image. The potential representation of the image corresponding to the first denoising is the potential representation of the noisy image.

4. The compositional scene modeling method based on the object-centered diffusion model according to claim 3, Its characteristics are: The training process of the image segmentation model includes the following steps: Step T1: Use multiple existing images as training images to construct a training dataset; Step T2: processing the training images selected from the training dataset through the image encoder unit and the object representation encoder module to obtain a training image latent representation and a training inference mask respectively; Step T3, sampling time step t and corresponding noise, adding noise to the potential representation of the training image, and obtaining a potential representation of the training noisy image; Step T4, processing the training noisy image latent representation through the hybrid diffused image decoder module to obtain a generated mask and an intermediate denoised image latent representation; Step T5, calculating a loss function based on the training image potential representation, the training inference mask, the generated mask, and the intermediate denoised image potential representation to obtain a loss result; Step T6, back-propagating and updating the parameters of the image segmentation model according to the loss result; Step T7, repeating steps T2 to T6 until the training completion condition is reached, thereby obtaining the trained image segmentation model. The generated mask is the normalized mask obtained by performing a denoising operation on the potential representation of the training noisy image as the potential representation of the image. The intermediate denoised image potential representation is the image potential representation prediction obtained after the training denoised image potential representation is used as the image potential representation for one denoising operation.

5. The compositional scene modeling method based on the object-centered diffusion model according to claim 4, characterized in that: in, In step T3, the potential representation of the training noise image corresponding to time step t is The calculation expression is: ， ， ， In the formula is the potential representation of the training image, is the noise corresponding to time step t, is the variance sequence that increases with time t The values.

6. The compositional scene modeling method based on the object-centered diffusion model according to claim 4, characterized in that: in, The calculation expression of the loss function is: ， ， In the formula is a hyperparameter, To calculate by KL function and KL divergence of two class distributions, inferring a mask for the training, Generate a mask for the is the potential representation of the training image, is the latent representation of the intermediate denoised image.

7. A compositional scene modeling device based on an object-centered diffusion model, for obtaining a segmentation result of a scene image, characterized in that: include: A noise generating unit, configured to obtain original noise through random sampling; a preliminary segmentation unit storing a trained image segmentation model, configured to input the scene image and the original noise together into the trained image segmentation model to obtain preliminary object segmentation results and preliminary background segmentation results; an upsampling unit, configured to upsample the preliminary object segmentation result and the preliminary background segmentation result respectively, to obtain the corresponding final object segmentation result and final background segmentation result as the segmentation results, Wherein, the image segmentation model includes: An image encoding and denoising module, configured to sequentially encode and add noise to the scene image to obtain a potential representation of the noisy image; An object representation encoder module is used to obtain an object representation based on the scene image, The object representation encoder module includes: A feature extraction unit, configured to convert the scene image into a feature map; An instance coloring and folding process unit is used to obtain the object representation according to the feature map, The example coloring stick process unit includes: An inference mask generation subunit, configured to cluster the pixels of the feature map to obtain an inference mask corresponding to each class; An object representation generating subunit is configured to calculate a corresponding weighted average feature map as the object representation by combining the inferred mask with the feature map using the inferred mask as a weight; A hybrid diffusion image decoder module is configured to obtain the preliminary object segmentation result and the preliminary background segmentation result based on the object representation and the noisy image potential representation.

Images