A desktop layer object generation method based on a conditional diffusion model

Abstract

The application discloses a kind of based on condition diffusion model's desktop layer object generation method, the method is first obtained desktop scene data, including different view pictures and furniture point cloud data, and is preprocessed.Secondly, the different view pictures in training data set after pre-processing are used, and corresponding semantic label is extracted by pre-training model desktop style semantic feature;Using the furniture point cloud data in training data set after pre-processing, extract geometric feature and constraint feature, obtain surface point cloud and boundary point cloud.Then diffusion model is constructed using U network, integrate desktop style semantic feature, surface point cloud and boundary point cloud, obtain the model of generating desktop layer.Finally, using the diffusion model obtained by training, select different scene, infer the desktop arrangement under corresponding scene, realize desktop layer object layout generation.The application can accurately generate corresponding object arrangement based on different desktop scene.

Description

TECHNICAL FIELD

[0001] The present application relates to three-dimensional reconstruction of computer vision, in particular to a method for generating object placement of desktop layer furniture. BACKGROUND

[0002] The desktop object generation task is a subtask of indoor three-dimensional scene layout generation task, which is of great significance in the field of computer vision. It not only involves the position, orientation and size of the desktop object, but also includes the object category and relative relationship. This task is particularly critical in application scenarios such as game development, virtual reality and augmented reality, and can enhance the realism of virtual environments. However, in indoor three-dimensional scene synthesis datasets, the placement of objects on the desktop layer furniture is often ignored.

[0003] In recent years, generative artificial intelligence technology has developed rapidly, and various representative generation paradigms have emerged, including sequence generation methods based on autoregressive, adversarial generation methods based on generative adversarial networks (GANs), and probabilistic generation methods based on diffusion models. These methods have shown excellent performance in image synthesis, scene generation and other tasks, but their application research in the field of desktop object generation is relatively scarce. SUMMARY

[0004] To solve the above problems, the present application proposes a desktop object generation method (DESK-GEN) based on a conditional diffusion model. The goal of this algorithm is to generate a desktop object configuration that meets functional requirements and has a reasonable layout by inputting two-dimensional image data and three-dimensional point cloud data containing desktop furniture. Specifically, the DESK-GEN method proposed in the present application first extracts the material, shape and style of the two-dimensional image data of the desktop furniture through the desktop style feature, generating a semantic condition vector. At the same time, the desktop geometry feature extraction module is used to process the point cloud and construct a geometric condition vector containing spatial distribution and size features. In the diffusion model generation stage, the semantic condition vector and the geometric condition vector of the spatial distribution and size features are jointly encoded by the desktop object generation module to drive the denoising network to predict the category distribution and pose parameters of the object. Finally, an object set that meets functional consistency and physical rationality is output. This method not only ensures the uniformity of the desktop scene style, but also significantly improves the spatial compliance and detail expressiveness of the object layout.

[0005] Several key parts of the present application are described:

[0006] 1. Adjust the training and validation data set, obtain the desktop scene data, including pictures of different perspectives and furniture point cloud data, and preprocess to achieve a unified standard, the data set used in the method is desktop scene data in residential, office and commercial three types of environment, the dimensions include but are not limited to: dense point cloud data of different furniture scenes, high-definition pictures of different perspectives, object categories, annotation of three-dimensional bounding box, and multi-level semantic system such as material type, function attribute and style label and other information. Different data sets will have different dimensions, and before training and reasoning, they need to be unified into this consistent standard;

[0007] 2. Desktop style extraction: using the pictures of different perspectives in the preprocessed training data set and the corresponding semantic labels, the desktop style semantic features are extracted through the pre-trained model to guide the uniformity of object category generation;

[0008] 3. Desktop geometric style extraction: using the furniture point cloud data in the preprocessed training data set, the geometric features and constraint features are extracted through the desktop layer furniture point cloud data, specifically surface point cloud Xsurface and boundary point cloud Xcontour, which are used to control the generated object placement;

[0009] 4. Desktop object generation: integrating desktop style semantic features, surface point cloud Xsurface and boundary point cloud Xcontour, a diffusion model (U-Net) is constructed by using a U-shaped network to train the desktop layer model.

[0010] 5. Prediction and reasoning of desktop generation: using the diffusion model trained in the previous step, different scenes are selected, such as desks, coffee tables, bedside cabinets, etc., to generate corresponding scene desktop arrangements, realizing the generation of desktop object layout that meets the physical rules and style requirements.

[0011] The present application has the following advantages and innovations:

[0012] 1. The preprocessing method of training and verification data unifies the standard and classification, and provides a feasible and efficient method for the processing of different scenes in the later period.

[0013] 2. A complete and effective desktop graphics generation method, specifically including: desktop style extraction based on a pre-trained model, desktop geometric feature extraction based on an autoencoder, and finally denoising layout generation based on a U-shaped network (U_Net). It can accurately generate corresponding object arrangements based on different desktop scenes. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The method flowchart of the present application;

[0015] Figure 2Generate a U-shaped network (U_Net) for the denoising diffusion network structure of desktop layer objects;

[0016] Figure 3 A schematic diagram of the desktop style feature extraction module;

[0017] Figure 4 For desktop point cloud data processing;

[0018] Figure 5 Example of desktop point cloud processing results;

[0019] Figure 6 This is a schematic diagram of the results generated by the present invention. Detailed Implementation

[0020] like Figure 1 As shown, a method for generating desktop layer objects based on a conditional diffusion model specifically includes the following steps:

[0021] Step 1: Preprocessing of training and validation datasets:

[0022] The specific preprocessing method is as follows: unify the classification criteria of the dataset, filter the data, and specify the classification range and coordinate units.

[0023] Point cloud geometric parameters are normalized. To address the problem of model size discreteness, scale normalization is performed based on the diagonal length of the object's 3D bounding box. The model vertex coordinates are scaled to the unit bounding box space to eliminate size deviations caused by differences in modeling scale.

[0024] Orientation unification is achieved by defining the main orientation axis (such as the positive Z-axis of a monitor screen) based on the object's function. The local coordinate system of the model is aligned with the global coordinate system through rotation matrix transformation to ensure that the object's pose parameters are consistent with the semantics of the real world during the generation process.

[0025] Data augmentation is achieved by applying geometric perturbations to the desktop point cloud, including random rotation transformations (θ∈[−15°, 15°]), translation perturbations (Δx,Δy∈[−0.2m, 0.2m]), and scaling (s ∈ [0.9, 1.1]), to improve the model's robustness to spatial transformations.

[0026] Step 2: Desktop Style Feature Extraction:

[0027] This step takes a pre-trained model (including but not limited to BLIP-2) as the base model to extract the style category information of the desktop and convert it into a conditional vector that can be used for the diffusion model. The style of the desktop layer furniture can be defined as features related to the application scenario or functional category, such as coffee tables, desks, etc. These categories not only reflect the diversity of desktop scenarios, but also provide style classification and semantic guidance for the generation process. Different categories of desktop layer furniture have significant differences in material, shape, and placement habits, so in the desktop layer object generation task, accurate extraction and encoding of these category information is crucial to ensure consistency in style.

[0028] As shown in Figure 3 The desktop style semantic feature extraction function is composed of an image encoder, a visual-linguistic translation (Q-Former), and a large language model (LLM) decoder, which can extract rich desktop style information through visual-linguistic modeling. First, the input two-dimensional image of the desktop layer furniture is extracted by a pre-trained image encoder to extract the underlying visual features. This encoder can effectively capture the shape, texture, material, and scene layout information of the object. Then, these visual features are input into the Q-Former for cross-modal processing. The Q-Former uses a Transformer structure to learn high-level visual-semantic alignment relationships through self-attention mechanisms, allowing different desktop styles to be mapped to a unified embedding space. In this process, the Q-Former generates a set of learnable embedding sequences that not only contain desktop categories (such as desks, coffee tables, kitchen tables, etc.), but also implicitly encode environmental features such as item placement patterns, color styles, and functional attributes of the desktop.

[0029] The system realizes cross-modal generation through a "visual encoding → language decoding" cascade architecture, which specifically manifests as: the Q-Former first compresses pixel-level image information into compact visual semantic representations; these visual representations are input into the LLM as conditions; the LLM generates the final style description text (T) based on the visual semantic context.

[0030] This feature can be used to control the subsequent diffusion model, allowing it to generate personalized according to the desktop style.

[0031] In addition, the learnable embedding mechanism of this function allows the model to adapt to different desktop environments without the need for manual definition of desktop categories, improving the flexibility of style expression.

[0032] Step three: desktop geometric style extraction:

[0033] Input desktop scene point cloud data X ∈ R N×3(Where N is the number of point clouds, and three-dimensional coordinates contain x, y, and z axis spatial information) The semantic label provided by the desktop scene dataset is used to filter instance objects belonging to objects on the desktop. Based on the object instance name index, the bounding box parameter range is calculated.

[0034] In order to extract the desktop layer from the point cloud data, it is necessary to first determine the height range of the desktop furniture. The specific steps are as follows:

[0035] 1. Desktop height reference calculation: First, obtain all coordinate values of the desktop point cloud in the vertical direction (Z axis), denoted as set Z. Take the minimum value in Z as the reference datum, and add a preset thickness compensation value δ (the value of δ ranges from 0.05 to 0.1 meters) to obtain the final support surface reference height hbase.

[0036] 2. Surface point cloud extraction: According to the reference height hbase and δ, filter out the point cloud data with a height in the interval [hbase−δ, hbase]. This part of the point cloud is the surface point cloud Xsurface of the desktop.

[0037] 3. Boundary point cloud extraction: In order to further extract the boundary contour of the desktop, the following methods are used:

[0038] Horizontal plane projection: Project the surface point cloud Xsurface onto the horizontal plane (ignore the Z coordinate) to obtain a two-dimensional point set.

[0039] Convex hull calculation: Perform convex hull calculation on the projected two-dimensional point set to find the smallest convex polygon that can contain all points, and the vertex set is denoted as V.

[0040] Three-dimensional boundary reconstruction: Map the two-dimensional coordinates of the convex hull vertex V back to the three-dimensional space, i.e. set the Z coordinate of each vertex vj to the reference height hbase, to obtain the three-dimensional boundary point cloud Xcontour.

[0041] Through the above steps, the surface point cloud Xsurface and the boundary point cloud Xcontour of the desktop can be separated from the original point cloud data.

[0042] Data standardization stage adopts affine transformation to eliminate scale difference: The extracted surface point cloud (Xsurface) and boundary point cloud (Xcontour) are respectively centered X = X − µ, where µ is the point cloud centroid coordinate, and the centered point cloud is scaled to standardize its distribution range:

[0043] Calculate the maximum Euclidean distance of all point cloud coordinates (i.e. the distance of the farthest point from the origin).

[0044] Dividing each point cloud by the maximum distance ensures that all points fall within the unit sphere (i.e., the distance to the origin does not exceed 1).

[0045] This process makes different sizes of tabletop point clouds comparable in feature space, laying the foundation for subsequent feature decoupling learning of the VAE encoder, and ultimately outputting the corresponding standardized point cloud.

[0046] The standardized point cloud is then input into the feature encoding module. The encoder network is composed of multiple layers of graph convolution (GraphConv) modules. The decoder of this point cloud reconstruction model uses an inverse graph convolution layer to reconstruct the point cloud coordinates X (belonging to the N x 3-dimensional real number space).

[0047] The reconstruction loss function uses the chamfer distance, which involves point-by-point distance calculations between the original point cloud X and the estimated value X. Specifically, it first calculates the sum of the squares of the shortest distances from each point in one set of point clouds to the other set, and then adds the averages of the two sets of results.

[0048] To maintain the regularity of the latent space, the model introduces a KL divergence loss term. Its calculation method is to sum all latent dimensions: the calculation of each dimension includes the log variance, mean square, and variance adjustment term, and finally takes one-half as the result.

[0049] The total loss function is the weighted sum of the chamfer distance loss and the KL divergence loss, where the loss weight coefficient β adopts an annealing strategy - it is set to 0.01 at the beginning of training and gradually increases linearly to 1.0 as the training process progresses, to balance the reconstruction accuracy and the strength of distribution regularization.

[0050] The network finally outputs two features: surface point cloud features and boundary point cloud features, which will be used as conditional inputs for the subsequent diffusion model denoising process. The final generation effect can be referred to Figure 5 The results are shown in the first figure, which shows the original state of the dataset, the second figure shows the tabletop layer area, and the third figure shows the tabletop point cloud extraction results, with the upper half being the surface point cloud style and the lower half being the boundary point cloud style.

[0051] Step four, tabletop object generation

[0052] This module achieves the generation of tabletop layer object layouts that meet physical rules and style requirements by fusing tabletop style features, tabletop point cloud geometric features, and boundary contour constraint features. In the following, each type of conditional feature will be analyzed, and the conditional fusion and generation control in the denoising process will be described in detail.

[0053] 4.1 Conditional feature analysis

[0054] This algorithm guides the generation process through a multi-condition control mechanism, and the synergistic effect of the three types of conditional features ensures that the generated layout meets the geometric constraints and semantic adaptability.

[0055] Desktop style semantic feature: Based on the pre-trained model, the semantic analysis of the input two-dimensional image is carried out, and the 256-dimensional feature vector representing the functional attributes of the desktop layer furniture (such as "desk" corresponding to the preference of stationery objects, and "dining table" associated with the distribution of tableware objects) is generated through visual-linguistic alignment. The feature vector is mapped into a conditional embedding vector through a fully connected network, and interacts with the diffusion network feature through a channel attention mechanism to guide the generation of object categories and the functional consistency of desktop types.

[0056] Desktop point cloud geometry feature: In Figure 4 , the 32-dimensional latent vector extracted by the VAE encoder from the surface point cloud encodes the geometric attributes of the desktop support surface. The geometric feature serves as a control condition for the diffusion model, making the generated object layout conform to the spatial constraints, while these features provide a global representation of the desktop surface shape, enabling the generation model to better match the desktop morphology when laying out, improving rationality and physical consistency.

[0057] Boundary contour constraint feature: The 32-dimensional feature vector generated by VAE encoding the boundary point cloud represents the three-dimensional spatial constraints of the desktop boundary, helping the desktop layer object to perceive the boundary range of the desktop, so that the object can be placed reasonably within the effective area, avoiding its appearance outside the desktop or overlapping outside the desktop edge. This condition effectively improves the placement accuracy of desktop objects, making the generated results more consistent with the desktop morphology, avoiding unreasonable layout due to the lack of boundary information.

[0058] The three types of conditions form a collaborative control system of "semantic-geometry-boundary": the style semantic feature controls the functional rationality of the object category, the point cloud geometry feature optimizes the contact relationship between the object and the support surface, and the boundary contour feature imposes spatial hard constraints. This multi-condition feature fusion mechanism makes the generation process comply with functional logic, physical rules, and spatial boundaries at the same time, providing core driving conditions for high realistic generation goals.

[0059] 4.2 Condition feature fusion and generation control

[0060] This module aims to solve the problems of style consistency and layout rationality of object layout in complex desktop scenes, and proposes a hierarchical condition injection generation control mechanism. Through multi-condition feature fusion and attention guidance strategy, the controllability of the diffusion model generation process is optimized. The specific process is as follows:

[0061] Firstly, the three types of condition features need to be preprocessed. The 256-dimensional style semantic features (representing the type of desktop function) extracted by the pre-trained model BLIP-2, the 32-dimensional geometric features (describing the geometric shape of the support surface) encoded by the surface point cloud VAE, and the 32-dimensional boundary features (defining the spatial range of the effective area) encoded by the boundary point cloud VAE are concatenated to form a 320-dimensional joint condition vector. This vector is compressed to 128 dimensions through a two-layer fully connected network to obtain the condition code C ∈ R 128 .

[0062] Subsequently, in the architecture design of the diffusion network U-shaped network (U-Net) shown in Figure 2 , a cross-attention layer is embedded after each residual block to establish a dynamic interaction mechanism between the noise latent code and the condition code. Specifically, the query vector Q ∈ R H×W×d is derived from the spatial feature mapping of the current noise latent code (where H, W are the sizes of the feature map, and d = 128 is the feature dimension), and the key-value pair K, V ∈ R 128×d is generated by the condition code C through a learnable linear projection matrix Wk, Wv∈ R 128×128 .

[0063] The calculation process of the attention weight follows the standard scaled dot-product attention formula:

[0064]

[0065] where the temperature coefficient (d = 128 here, the actual calculation value is 11.31) is used to stabilize the gradient propagation and avoid the Softmax function from entering the saturation region.

[0066] Figure 6 The generated result diagram of the present application shows that the method can maintain the creativity of the generation process while strictly following the physical space constraints and functional logic, and finally output the expected high-quality layout.

Claims

1. A method for generating desktop layer objects based on a conditional diffusion model, characterized in that, Includes the following processes: Acquire desktop scene data, including images from different perspectives and furniture point cloud data, and perform preprocessing; Using images from different perspectives in the preprocessed training dataset, along with their corresponding semantic labels, desktop style semantic features are extracted through a pre-trained model. Using furniture point cloud data from the preprocessed training dataset, geometric and constraint features are extracted from the desktop layer furniture point cloud data to obtain surface point cloud and boundary point cloud; The specific implementation process for extracting geometric and constraint features is as follows: Obtain all coordinate values ​​of the desktop point cloud in the vertical direction, denoted as set Z; take the minimum value in Z as the reference base, and add a preset thickness compensation value δ to obtain the final support surface reference height hbase; Based on the baseline height hbase and δ, the point cloud data with heights within the interval [hbase-δ, hbase] are filtered out. This part of the point cloud is the surface point cloud Xsurface of the desktop. Project the surface point cloud Xsurface onto the horizontal plane to obtain a two-dimensional point set. Perform convex hull calculation on the projected two-dimensional point set to find the smallest convex polygon that can contain all points. Its vertex set is denoted as V. Remap the two-dimensional coordinates of the convex hull vertex V back to three-dimensional space, that is, set the Z coordinate of each vertex vj as the reference height hbase to obtain the three-dimensional boundary point cloud Xcontour. In the data standardization stage, affine transformation is used to eliminate scale differences: the extracted surface point cloud Xsurface and boundary point cloud Xcontour are centered respectively, and the centered point clouds are scaled to standardize their distribution range: the maximum Euclidean distance of all point cloud coordinates is calculated; each point cloud coordinate is divided by the maximum Euclidean distance, and the corresponding standardized point cloud is finally output. The standardized point cloud is then input into the feature encoding module; the encoder network consists of a multi-layer graph convolutional GraphConv module; the decoder uses inverse graph convolutional layers to reconstruct the point cloud coordinates to obtain the estimated value X. The reconstruction loss function adopts chamfer distance loss, and its calculation process includes point-by-point distance calculation between the original point cloud X and the estimated value X. Specifically, it is implemented as follows: first, calculate the sum of squares of the shortest distances from each point in the two sets of point clouds to the other set of point clouds, and then average the results of the two sets and add them together. The KL divergence loss term is introduced, which is calculated by summing all potential dimensions: the calculation for each dimension includes logarithmic variance, mean squared sum and variance adjustment term, and finally half of the result is taken; The total loss function is a weighted sum of the chamfer distance loss and the KL divergence loss, where the weight coefficients of the loss adopt an annealing strategy to balance the reconstruction accuracy and the strength of distribution regularization. A diffusion model is constructed using a U-shaped network, integrating desktop style semantic features, surface point clouds, and boundary point clouds to obtain a model for generating the desktop layer; Using the diffusion model trained in the previous step, different scenarios are selected, and desktop decorations in the corresponding scenarios are generated to achieve desktop layer object generation.

2. The desktop layer object generation method based on the conditional diffusion model according to claim 1, characterized in that, The specific implementation process for extracting desktop style semantic features is as follows: A pre-trained model is used as the base model to extract style category information of the desktop and convert it into a conditional vector that can be used for the diffusion model; the style of the desktop furniture is defined as a feature related to the application scenario or functional category. The desktop style semantic feature extraction function consists of an image encoder, a visual-language translation Q-Former, and a large language model LLM decoder, which extracts desktop style information through visual-language modeling.

3. The desktop layer object generation method based on the conditional diffusion model according to claim 2, characterized in that, The desktop style semantic feature extraction function is implemented as follows: First, the input 2D image of the desktop furniture is processed by a pre-trained image encoder to extract the underlying visual features. This encoder can capture the shape, texture, material and scene layout information of the object. Subsequently, the visual features are input into Q-Former for cross-modal processing. Q-Former adopts a Transformer structure and learns visual-semantic alignment relationships through a self-attention mechanism, so that different desktop styles can be mapped to a unified embedding space. In this process, Q-Former generates a set of learnable embedding sequences, which not only contain desktop categories but also implicitly encode environmental features. The embedded sequence is used as a conditional input to the LLM, which generates the final style description text, i.e., desktop style semantic features, based on the visual semantic context.

4. The desktop layer object generation method based on the conditional diffusion model according to claim 3, characterized in that, The specific implementation process of the model for generating the desktop layer is as follows: The generation process is guided by a multi-condition control mechanism, and the three types of condition features work together to ensure that the generated layout meets geometric constraints and semantic adaptability. A hierarchical condition injection generation control mechanism is proposed, which optimizes the controllability of the diffusion model generation process by fusing three types of conditional features and using an attention-guided strategy.

5. The desktop layer object generation method based on the conditional diffusion model according to claim 4, characterized in that, The three types of conditional features are: desktop style semantic features, desktop point cloud geometric features, and boundary contour constraint features.

6. The desktop layer object generation method based on the conditional diffusion model according to claim 5, characterized in that, The hierarchical conditional injection generation control mechanism optimizes the controllability of the diffusion model generation process through multi-conditional feature fusion and attention-guided strategies; the specific process is as follows: First, the three types of conditional features are preprocessed. The style semantic features extracted by the pre-trained model, the geometric features encoded by VAE of surface point cloud, and the boundary features encoded by VAE of boundary point cloud are concatenated to form a joint conditional vector. This vector is compressed through a fully connected network to obtain the conditional code C. Subsequently, in the architecture design of the diffusion network U-shaped network, a cross-attention layer is embedded after each residual block to establish a dynamic interaction mechanism between the noise latent code and the conditional code. Specifically, the query vector Q comes from the spatial feature mapping of the current noise latent code, and the key-value pairs K and V are generated by the conditional code C through a learnable linear projection matrix. The calculation process of the attention weight follows the scaling dot product attention formula.

Images