3D Scene Automatic Generation Method and System Based on Scene Semantics and Geometric Constraints

Abstract

The present application discloses a method and system for automatically generating three-dimensional scenes based on scene semantics and geometric constraints, which belongs to the field of deep learning technology, including: The present invention proposes an innovative three-dimensional indoor scene generation method based on a hybrid discrete-continuous diffusion model architecture, namely a cross-modal KanMiDiffusion algorithm. First, the MiDiffusion model is introduced, and on this basis, a cross-modal KanMiDiffusion algorithm is proposed by integrating the Bert text encoder and the Dinov2 visual encoder, and introducing the Kolmogorov-Arnold network to optimize the geometric feature mapping. The present invention elaborates on the network architecture of the algorithm, including the integration of the visual-text pre-training model and the design of the multimodal feature interaction module. Through simulation experiments, it is verified that the method proposed in the present invention can improve the semantic and geometric accuracy of scene generation.

Description

Technical Field

[0001] The present application belongs to the field of neural network technology, and specifically relates to a method and system for automatically generating three-dimensional scenes based on scene semantics and geometric constraints. Background Art

[0002] Realistic 3D scene synthesis technology has important research value in the fields of computer vision, robotics, and virtual reality. By constructing high-fidelity virtual environments, this technology can not only provide training data for complex traffic scenarios for autonomous driving systems, but also provide a diverse indoor environment testing platform for service robot development. The MiDiffusion model has pioneered a new paradigm for hybrid discrete-continuous diffusion models. By introducing two-dimensional floor plans and structured object layout representations, the MiDiffusion model innovatively implements a hybrid domain diffusion process across discrete semantics (such as furniture categories) and continuous geometry (such as location coordinates). However, existing technologies have the following problems:

[0003] First, existing MiDiffusion models only rely on two-dimensional planes as spatial priors and lack explicit modeling of three-dimensional spatial structures.

[0004] Second, the interaction mechanism between discrete semantics and continuous geometry in the existing MiDiffusion model is not perfect, and its ability to understand semantics and extract geometric structures is insufficient.

[0005] The above problems lead to low semantic and geometric accuracy of scene generation. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a method and system for automatically generating three-dimensional scenes based on scene semantics and geometric constraints, which can solve the problem of low semantic and geometric accuracy of scene generation in the prior art.

[0007] In order to solve the above technical problems, this application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints, the method comprising:

[0009] Build a preset cross-modal KanMiDiffusion denoising network model, and perform text encoding processing on the latent variables according to the DeBERTa text encoder inside the KanMiDiffusion denoising network model to obtain text encoding features. The cross-modal KanMiDiffusion denoising network model includes: a first Kolmogorov-Arnold network, a second Kolmogorov-Arnold network, a third Kolmogorov-Arnold network, a Transformer decoder, a DeBERTa text encoder, and a Dinov2 visual encoder;

[0010] According to the first Kolmogorov-Arnold network, the geometric structure of the preset indoor object is encoded to obtain geometric coding features;

[0011] Add the geometric encoding features and the text encoding features to generate the fusion features of geometry and text;

[0012] The preset indoor floor plan is visually encoded according to the Dinov2 visual encoder and then feature-stitched with the preset position encoding features to obtain the fusion features of vision and position;

[0013] The fusion features of geometry and text, and the fusion features of vision and position are processed according to the Transformer decoder to obtain the output of the Transformer decoder;

[0014] The output of the Transformer decoder is processed according to the second and third Kolmogorov-Arnold networks to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features;

[0015] Generate 3D scenes based on the categorical distribution of semantic labels and the Gaussian mean of geometric features.

[0016] As an optional implementation of the first aspect of the present application, the Kolmogorov-Arnold network includes multiple KAN layers, and the mathematical expression is:

[0017] ;

[0018] in, represents the first layer, represents the first layer, represents the first layer of the KAN network, represents the 0th layer of the KAN network, represents the input vector, Represents the input vector The corresponding output result;

[0019] Among them, each KAN network has The dimension input and The dimension output, Indicates multiplication processing, Depend on Learnable activation functions The mathematical expression is:

[0020] ;

[0021] Kolmogorov-Arnold network from Layer to The mathematical expression of the layer is:

[0022] ;

[0023] in, Indicates the The output vector of the layer, Indicates the The output vector of the layer, represents the first Layer Matrix.

[0024] As an optional implementation of the first aspect of the present application, the DeBERTa text encoder includes: multiple layers of parallel-connected Transformer encoder layers, word embedding layers, and word vector representation layers, each layer of the Transformer encoder layer includes multiple serially connected Transformer encoders, wherein the word embedding layer is used to embed each position information, sentence embedding, and word embedding as input vectors, which are respectively input into the multiple layers of parallel-connected Transformer encoder layers for encoding processing to generate word vectors corresponding to each Transformer encoder layer.

[0025] As an optional implementation of the first aspect of the present application, the Dinov2 visual encoder uses ViT as the basic architecture to divide the image into multiple small blocks to obtain an image block sequence to capture long-distance dependencies in the image. The Dinov2 visual encoder includes a multi-layer Transformer module to generate semantically rich high-level feature representations.

[0026] As an optional implementation of the first aspect of the present application, a Transformer decoder includes multiple Transformer blocks, each Transformer block is sequentially connected to a first adaptive normalization layer, a multi-head self-attention layer, a second adaptive normalization layer, a multi-head cross attention layer, and a feedforward neural network layer. The output of the Transformer decoder is obtained by connecting the output of each self-attention head in the multi-head self-attention layer and performing linear projection calculation. The mathematical expression of the output of the Transformer decoder is:

[0027] ;

[0028] in, Represents the input vector The corresponding output of the Transformer decoder, are learnable parameters, Indicates feature splicing processing, Represents the weighted sum of N value vectors.

[0029] In a second aspect, an embodiment of the present application provides a system for automatically generating a three-dimensional scene based on scene semantics and geometric constraints, the system comprising:

[0030] DeBERTa text encoder module, used to perform text encoding processing on latent variables to obtain text encoding features;

[0031] The first Kolmogorov-Arnold module is used to encode the geometric structure of the preset indoor object to obtain geometric coding features;

[0032] A fusion feature generation module is used to add the geometric coding features and the text coding features to generate a fusion feature of geometry and text;

[0033] The Dinov2 visual encoder module is used to perform visual encoding processing on the preset indoor floor plan and then perform feature splicing with the preset position encoding features to obtain the fusion features of vision and position;

[0034] The Transformer encoder module is used to process the fusion features of geometry and text, and the fusion features of vision and position to obtain the output of the Transformer decoder;

[0035] A semantic label and geometric feature generation module is used to process the output of the Transformer decoder according to the second Kolmogorov-Arnold network and the third Kolmogorov-Arnold network to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features;

[0036] The 3D scene generation module is used to generate 3D scenes based on the classification distribution of semantic labels and the Gaussian mean of geometric features.

[0037] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0039] Compared with the existing technology, the beneficial effect of the automatic generation method of three-dimensional scenes based on scene semantics and geometric constraints provided by the present invention is that: the present invention integrates the DeBERTa text encoder and the Dinov2 visual encoder, and introduces the Kolmogorov-Arnold (KAN) network to optimize the geometric feature mapping, proposes a cross-modal KanMiDiffusion algorithm, and verifies the effectiveness of the KanMiDiffusion algorithm proposed in the present invention in improving the quality, efficiency and diversity of scene generation, can improve the semantic and geometric accuracy of scene generation, and demonstrates the powerful performance and generalization ability of the algorithm on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints provided in the first embodiment of the present application;

[0041] Figure 2 This is a Transformer model diagram provided by the first embodiment of the present application;

[0042] Figure 3 This is a diagram of a cross-modal KanMiDiffusion denoising network model provided in the first embodiment of the present application;

[0043] Figure 4 This is a ViT model diagram provided by the first embodiment of the present application;

[0044] Figure 5 This is a DeBERTa text encoder model diagram provided by the first embodiment of the present application;

[0045] Figure 6 is a multimodal feature interaction Transformer graph provided by the first embodiment of the present application;

[0046] Figure 7 This is a diagram of the multi-head attention structure provided by the first embodiment of this application;

[0047] Figure 8 A three-dimensional indoor scene image synthesized using the cross-modal KanMiDiffusion algorithm provided in the first embodiment of the present application;

[0048] Figure 9 It is a second image of a three-dimensional indoor scene synthesized using the cross-modal KanMiDiffusion algorithm provided by the first embodiment of the present application;

[0049] Figure 10 These are three images of a three-dimensional indoor scene synthesized using the cross-modal KanMiDiffusion algorithm provided in the first embodiment of the present application;

[0050] Figure 11 These are four images of a three-dimensional indoor scene synthesized using the cross-modal KanMiDiffusion algorithm provided by the first embodiment of the present application;

[0051] Figure 12 These are five images of a three-dimensional indoor scene synthesized using the cross-modal KanMiDiffusion algorithm provided in the first embodiment of the present application;

[0052] Figure 13 This is an internal structure diagram of a three-dimensional scene automatic generation system based on scene semantics and geometric constraints provided in the second embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0055] The following describes in detail the method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0056] Example 1

[0057] See also Figure 1 , which represents a flow chart of the method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints provided by the present invention, including steps S1 to S7.

[0058] Step S1: Build a preset cross-modal KanMiDiffusion denoising network model, and perform text encoding processing on the latent variables according to the DeBERTa text encoder inside the KanMiDiffusion denoising network model to obtain text encoding features, wherein the cross-modal KanMiDiffusion denoising network model includes: a first Kolmogorov-Arnold network, a second Kolmogorov-Arnold network, a third Kolmogorov-Arnold network, a Transformer decoder, a DeBERTa text encoder, and a Dinov2 visual encoder. Figure 5 Represents the DeBERTa text encoder model diagram provided by the present invention.

[0059] Specifically, the present invention first introduces the traditional MiDiffusion model, which pioneered a new paradigm of hybrid discrete-continuous diffusion model. By introducing two-dimensional floor plan and structured object layout representation, the model innovatively realizes the hybrid domain diffusion process across discrete semantics (such as furniture categories) and continuous geometry (such as location coordinates). As a hybrid discrete-continuous 3D indoor scene synthesis diffusion model, MiDiffusion assumes that each scene All in a central world frame, consisting of a two-dimensional plane and at most objects. Using a conditional vector To represent the floor plan. Each object in the scene has a semantic label , center of mass position , bounding box size , the rotation angle around the vertical axis , expressed as:

[0060] ;

[0061] in, represents the plane rotation group, Indicates the rotation angle, ranging from 0 to 360 degrees, Represents matrix transpose.

[0062] All geometric features in the scene can be mapped to . Use the 3D indoor scene representation as , Represents a given conditional input.

[0063] The last semantic tag Specify an empty tag to handle strings containing less than A scene with 1 object and defining all zero geometric properties for an "empty" object.

[0064] Although object semantics and geometric attributes are in different domains, this paper defines the destruction process in hybrid diffusion, and The mathematical expression for the noise injected into a specific area is:

[0065] ;

[0066] in, represents the mixed probability distribution of semantic labels and geometric features, Indicates that the object is given the previous step semantic label at step t Get the next step semantic label The probability distribution of Indicates that the object is given the previous step semantic label at step t Get the next step semantic label The probability distribution of .

[0067] Sampling independently during training , , and the mathematical expression of the posterior distribution is:

[0068] ;

[0069] in, represents the object semantic label, Represents the denoising process of geometric features, representing the features of given noise and Reverse the original features The probability distribution of represents the hybrid denoising process.

[0070] In the backward diffusion process, design a network The calculated probability distribution is consistent with the forward diffusion process, and the mathematical expression is:

[0071] ;

[0072] in, Represents the neural network used to fit the denoising process Get a specific probability distribution, and For fitting Probability distribution, ( and Indicates from and Reverse deduction and The joint conditional terms of Represents a given conditional input, such as a specific conditional input from which specific original features can be derived, such as the input process of a text image.

[0073] For further information, see Figure 3 , represents the cross-modal KanMiDiffusion denoising network model diagram provided by the present invention, which describes the denoising process and includes latent variables In step and predict the classification distribution , and the Gaussian mean Sampling from these distributions is used to obtain the steps latent variables .

[0074] In summary, this paper uses the text encoding DeBERTa to encode object category information. Secondly, Kolmogorov-Arnold Networks (KANs) are used to transfer geometric attributes to encode object features, and this is combined with a trainable BERT-encoded semantic embedding. Then, for each scene, the Dinov2 visual encoder is used as a plane feature extractor, as it can more easily capture floor semantics, including the floor placement area and floor boundary information. Finally, the plane features are added to the position embedding to form a conditional vector.

[0075] Step S2: According to the first Kolmogorov-Arnold network, the geometric structure of the preset indoor object is encoded to obtain a geometric encoding feature.

[0076] Specifically, Multilayer Perceptron (MLP) is a fully connected feedforward neural network, which is the basic building block of deep learning models and is often used to approximate nonlinear functions in machine learning. The MLP composed of layers can be described as a transformation matrix and activation function The mathematical expression is:

[0077] ;

[0078] in, Indicates the The linear weight function of the layer, Indicates the The linear weight function of the layer, represents the linear weight function of layer 1, represents the input vector, Represents the output of the multilayer perceptron, Indicates multiplication processing.

[0079] The Kolmogorov-Arnold Network (KAN) provided by the present invention can replace the traditional multilayer perceptron. Unlike the traditional multilayer perceptron MLP, which is based on the generalized approximation theorem, KAN is a function approximation method inspired by the Kolmogorov-Arnold representation theorem. KAN has a fully connected structure similar to MLP, but unlike MLP, which relies on fixed activation functions for each node, KAN introduces a learnable activation function on the edge, fundamentally changing the neural network structure and using a learnable one-dimensional spline function to eliminate the linear weight matrix. Similar to MLP, a The KAN layer can be described by nesting multiple KAN layers. The mathematical expression is:

[0080] ;

[0081] in, represents the first layer, represents the first layer, represents the first layer of the KAN network, represents the 0th layer of the KAN network, represents the input vector, Represents the input vector The corresponding output results.

[0082] Among them, each KAN network has The dimension input and The dimension output, Indicates multiplication processing, Depend on Learnable activation functions The mathematical expression is:

[0083] ;

[0084] Kolmogorov-Arnold network from Layer to The mathematical expression of the layer is:

[0085] ;

[0086] in, Indicates the The output vector of the layer, Indicates the The output vector of the layer, represents the first Layer Matrix.

[0087] The cross-modal KanMiDiffusion proposed in this paper uses KAN to transfer geometric attributes to encode object features and combines them with trainable semantic embeddings. Simultaneously, the output of the Tranformer decoder, a multimodal feature interaction module, is fed into two KAN networks to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features.

[0088] Step S3: Add the geometric coding features and the text coding features to generate a fusion feature of geometry and text.

[0089] Step S4: After visual encoding processing of the preset indoor floor plan according to the Dinov2 visual encoder, feature splicing is performed with the preset position coding features to obtain the fusion feature of vision and position.

[0090] Specifically, the Dinov2 visual encoder, based on the Vision Transformer (ViT) architecture, divides the image into multiple small blocks to obtain a sequence of image blocks to capture long-range dependencies in the image. The Dinov2 visual encoder includes multiple layers of Transformer modules to generate semantically rich high-level feature representations.

[0091] The ViT model introduced the Transformer architecture to the field of computer vision, demonstrating its unique advantages in tasks such as image recognition. ViT divides the image into small blocks, treating these blocks as elements in a sequence, and uses a self-attention mechanism to construct a global feature representation of the image. This process is ultimately completed through a multi-layer perceptron.

[0092] See also Figure 4 , represents the ViT model diagram provided by the present invention. The structural design of the ViT model diagram gives it multiple advantages over the traditional CNN, as follows:

[0093] 1) ViT is adaptive in feature learning, automatically extracting features from data without manually designing convolution kernels, in contrast to CNNs, which rely on pre-set convolutional layers. ViT can process input images of various sizes without cropping or scaling, whereas CNNs typically require fixed-size inputs, limiting their flexibility.

[0094] 2) ViT effectively captures the global information of the image through the self-attention mechanism, solving the problem of insufficient global information capture that CNN may have.

[0095] 3) ViT allows us to understand the key areas of the model's attention by visualizing the attention weights, which improves the interpretability of the model, whereas CNN is generally insufficient in this regard.

[0096] Step S5: Process the fusion features of geometry and text, and the fusion features of vision and position according to the Transformer decoder to obtain the output of the Transformer decoder.

[0097] For details, please refer to Figure 2 , represents the Transformer model diagram provided by the present invention. In the architecture of the model, the encoder module is on the left and the decoder module is on the right, which together constitute the core structure of the Transformer. The Transformer architecture has become an advanced model for processing sequence data in deep learning, especially in solving the problem of long-distance dependencies in sequences. This model is widely used in the field of natural language processing, including but not limited to machine translation and text summarization tasks. Unlike recurrent neural networks that rely on step-by-step processing of sequence data, the Transformer adopts a modular encoder and decoder structure.

[0098] The encoder portion of the Transformer core structure diagram consists of multiple encoder blocks, which are responsible for converting the input sequence data into a set of rich feature representations. Conversely, the decoder portion is responsible for recombining these feature representations to generate an output sequence. Taking machine translation as an example, the encoder converts the source language text into feature tokens, while the decoder generates the target language text based on these feature tokens. The performance of the Transformer is significantly affected by the model size and the quality of the training data.

[0099] For further information, see Figure 6 , represents the multimodal feature interaction Transformer graph provided by the present invention. The cross-modal KanMiDiffusion backbone adopts the time-varying transformer decoder in MiDiffusion. The multi-head attention in Transformer Block has better capture boundary constraints than the conditional one-dimensional U-Net structure. At the same time, the time step is injected through the adaptive layer normalization operator, and the conditional stream input is transmitted through multi-head cross attention.

[0100] For further information, see Figure 7 , represents the multi-head attention structure diagram provided by the present invention. The Transformer model is built based on multi-head self-attention, which enables the model to capture long-distance relationships between tokens at different positions. Specifically, is the input sequence, the input sequence lie in In the space, is the input sequence length, is the number of hidden dimensions. Figure 7 It includes normalization processing of three parallel connections, linear transformation processing of three parallel connections, dot product operation and scaling, normalized exponential function, dot product operation, splicing and residual connection processing.

[0101] Each self-attention head computes the query matrix , key matrix Sum Matrix ,pass The linear transformation is obtained, and the mathematical expression is:

[0102] ;

[0103] in represents the learnable parameters, , represents the number of hidden dimensions of a head, and the output of a self-attention head is The weighted sum of the value vectors is expressed as:

[0104] ;

[0105] in, represents the weighted sum of N value vectors, express activation function, Represents matrix transpose.

[0106] For one with The final output of the multi-head self-attention layer is obtained by connecting the output of each self-attention head and performing linear projection. The mathematical expression is:

[0107] ;

[0108] in, Represents the input vector The corresponding output of the Transformer decoder, are learnable parameters, Indicates feature splicing processing, Represents the weighted sum of N value vectors.

[0109] Step S6: Process the output of the Transformer decoder according to the second Kolmogorov-Arnold network and the third Kolmogorov-Arnold network to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features.

[0110] Step S7: Generate a three-dimensional scene based on the classification distribution of the semantic labels and the Gaussian mean of the geometric features.

[0111] Specifically, assuming that each scenario All in a central world frame, consisting of a two-dimensional plane and at most objects. Using a conditional vector To represent the floor plan. Each object in the scene has a semantic label , center of mass position , bounding box size , the rotation angle around the vertical axis , expressed as:

[0112] ;

[0113] in, represents the plane rotation group, Indicates the rotation angle, ranging from 0 to 360 degrees, Represents matrix transpose.

[0114] All geometric features in the scene can be mapped to . Use the 3D indoor scene representation as . The last semantic tag Specify an empty tag to handle strings containing less than A scene with 1 object and defining all zero geometric properties for an "empty" object.

[0115] In order to verify the feasibility of the method proposed in the present invention, the present invention conducted a simulation test:

[0116] This paper conducts in-depth research on the 3D-FRONT dataset. As a highly valuable dataset of synthetic indoor scenes, the 3D-FRONT dataset holds a key position in the field of 3D scene synthesis research. It meticulously constructs a vast and rich virtual world of indoor scenes, encompassing a staggering 18,797 rooms. These rooms are not randomly assembled but ingeniously designed, meticulously arranged with 7,302 textured 3D objects. Each room resembles a digital replica of a real interior space.

[0117] The 3D-FRONT dataset is widely used in academia and industry as an important benchmark for evaluating 3D scene synthesis methods. Its role is particularly crucial for understanding the layout and semantics of indoor environments. In indoor layout evaluation, researchers can use this dataset to compare and analyze indoor layout solutions generated by different algorithms, assessing key metrics such as layout rationality and space utilization. Furthermore, by mining the semantic information of the rooms in the dataset, it can facilitate the development of more intelligent indoor scene analysis systems. Each room in the 3D-FRONT dataset is equipped with extremely detailed layout and semantic information. Regarding layout, the room's shape, dimensions, and the spatial relationships of each object are precisely recorded. This meticulous layout information provides accurate data support for indoor space planning research. Regarding semantic information, the dataset features detailed object annotations. Each object is assigned a closely related semantic label, such as "sofa," "dining table," or "wardrobe." These labels clearly define the object's category and function, building a critical foundation for machine understanding of indoor scene semantics and providing an indispensable foundation for research on semantic-based indoor scene analysis and synthesis algorithms.

[0118] In this paper, the proposed cross-modal KanMiDiffusion 3D scene indoor generation algorithm was compiled using Python. Python, with its concise syntax, rich library resources, and strong scalability, provides strong support for the efficient implementation of the algorithm. The model was built using the PyTorch framework.

[0119] The experimental environment used an NVIDIA GeForce RTX3090 as the hardware platform. This hardware boasts powerful graphics processing capabilities and parallel computing performance, effectively accelerating the training and inference processes of deep learning models, significantly reducing experimental time. Ubuntu 18.04 was used as the software environment. Its stability and reliability, along with its excellent compatibility with various open-source software, provided a stable operating system for the smooth execution of the experiment. Python 3.8 was also used, offering an excellent balance of syntax and library support, perfectly suited to the development and execution of this algorithm. Furthermore, CUDA 11.3 provides efficient parallel computing acceleration for NVIDIA GPUs, fully leveraging the RTX3090's hardware performance. Pytorch 1.11 ensured compatibility and efficient collaboration with CUDA and other related libraries during model execution.

[0120] The Adam optimizer was used during the optimization process. Combining the advantages of the Adagrad and RMSProp algorithms, the Adam optimizer can adaptively adjust the learning rate of each parameter, effectively avoiding oscillations during parameter updates while ensuring the model's convergence speed. In terms of specific parameter settings, the learning rate was set to 0.001. This value has been debugged through multiple experiments. While ensuring that the model can effectively learn data features, it avoids problems such as excessively high learning rates causing the model to not converge, or excessively low learning rates causing the training process to be too slow. The entire model was trained 500 times. Through these 500 training sessions, the model was able to fully learn the complex features and patterns in the input data, thereby achieving effective modeling and prediction of 3D scene indoor generation tasks.

[0121] Figure 8 The figure shows a 3D indoor scene synthesized by the cross-modal KanMiDiffusion algorithm, which shows the 3D indoor scene layout reconstruction effect generated by the proposed cross-modal KanMiDiffusion algorithm. The algorithm aims to achieve high-quality conversion from a given 2D plan to a 3D indoor scene. Its core lies in fusing multimodal information to enhance the accuracy and realism of scene synthesis. Figure 8 In the figure, we can observe that the algorithm accurately reconstructs the layout of the indoor scene in 3D. The various objects in the scene, such as the sofa, chairs, and table, are spatially arranged according to the layout information in the 2D plan view. Furthermore, the algorithm is able to generate highly consistent 3D models based on the semantic categories and dimensions of the objects, as intuitively demonstrated in the figure. Even in the presence of multiple objects and complex layouts, the algorithm is able to generate a clearly structured and well-arranged 3D scene.

[0122] Figure 9 Two images of a 3D indoor scene synthesized using the cross-modal KanMiDiffusion algorithm are shown. This scene demonstrates the algorithm's effectiveness and accuracy in processing indoor environments with specific functional areas. The scene consists of several key areas: a seating area with a sofa and coffee table, a dining table, and a closet area that may be used for display or storage. The layout and placement of objects in these areas adhere to basic principles of interior space design, such as functional differentiation, smooth movement, and rational space utilization.

[0123] Figure 10 The following shows three images of 3D indoor scenes synthesized using the cross-modal KanMiDiffusion algorithm. Figure 10As you can see in the image above, the scene is primarily composed of a large black area, representing an empty room or an open space with a specific function. A small brown cube is placed in one corner of the room, representing a piece of furniture or decorative object, such as a bookshelf, side table, or other small fixture. Additionally, the scene contains several small white and gray cubes, representing other objects or decorative elements within the room.

[0124] The layout of this scene is simple yet functional. The large black area in the center provides ample space for possible furniture placement and also offers room occupants free range of movement. The brown blocks in the corners and the smaller blocks around them add details to the scene, making the entire space richer and more interesting. The application of the cross-modal KanMiDiffusion algorithm in this scene demonstrates its ability to handle minimalist interior layouts. The algorithm can accurately infer the layout of the three-dimensional space and the placement of objects based on the input two-dimensional floor plan information, maintaining the rationality and authenticity of the scene even when there are fewer objects or the space is relatively empty.

[0125] Figure 11 Four images of 3D indoor scenes synthesized using the cross-modal KanMiDiffusion algorithm are presented. A carefully designed interior space is observed, including a lounge area and a possible dining area. The lounge area is furnished with a sofa and coffee table, while the dining area has a dining table and chairs. The layout of these furniture follows the principles of interior space design, such as the division of functional areas, spatial fluidity, and symmetry in furniture placement. Furthermore, the size and proportions of the furniture and decorative elements in the scene are precisely controlled, visually enhancing the scene's realism and harmony. The algorithm also generates highly consistent 3D models based on the semantic categories and dimensional information of the furniture, as intuitively demonstrated in the images.

[0126] Figure 12Five images of 3D interior scenes synthesized using the cross-modal KanMiDiffusion algorithm are presented. The resulting layout demonstrates a high degree of rationality and harmony in spatial organization. The algorithm precisely accounts for object positioning, such as the visual and functional relationships between the sofa, coffee table, and TV cabinet. The interior scene consists of a lounge area and a dining area. The lounge area is furnished with a sofa and coffee table, while the dining area features a dining table and chairs. Furthermore, the scene includes decorative elements such as wall art and plants, which enrich the visual quality of the space and enhance its realism. The arrangement of the sofa and coffee table visually creates a comfortable resting area. The coffee table's central placement allows for easy placement, demonstrating the application of ergonomics in spatial layout. The placement of the dining table and chairs takes into account dining convenience and the openness of the space, leaving ample space around the table for movement and communication. The precise control of the size and proportions of the furniture and decorative elements in the scene enhances the scene's realism and harmony. The algorithm can also generate a highly consistent three-dimensional model based on the semantic categories and size information of the furniture, which is intuitively reflected in the figure.

[0127] The advantage of the cross-modal KanMiDiffusion algorithm lies in its ability to fuse information between different modalities, thereby achieving more refined and realistic effects in 3D scene synthesis. By incorporating multimodal data, the algorithm can not only learn the spatial relationships between objects, but also capture the detailed characteristics of different materials and textures, which is crucial for improving the visual realism of the scene.

[0128] Example 2

[0129] See also Figure 13 , which represents the internal structure of the 3D scene automatic generation system based on scene semantics and geometric constraints provided by the present invention, including:

[0130] DeBERTa text encoder module 100, used to perform text encoding processing on latent variables to obtain text encoding features;

[0131] The first Kolmogorov-Arnold module 200 is used to encode the geometric structure of the preset indoor object to obtain a geometric coding feature;

[0132] A fusion feature generation module 300 is used to add the geometric coding feature and the text coding feature to generate a fusion feature of geometry and text;

[0133] The Dinov2 visual encoder module 400 is used to perform visual encoding processing on the preset indoor floor plan and then perform feature splicing with the preset position coding features to obtain a fusion feature of vision and position;

[0134] The Transformer encoder module 500 is used to process the fusion features of geometry and text, and the fusion features of vision and position to obtain the output of the Transformer decoder;

[0135] A semantic label and geometric feature generation module 600 is configured to process the output of the Transformer decoder according to the second Kolmogorov-Arnold network and the third Kolmogorov-Arnold network to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features;

[0136] The 3D scene generation module 700 is used to generate a 3D scene based on the classification distribution of semantic labels and the Gaussian mean of geometric features.

[0137] The beneficial effects of the automatic generation system of three-dimensional scenes based on scene semantics and geometric constraints provided by the present invention are:

[0138] First, the DeBERTa text encoder module 100 is used, Transformer is used for feature extraction, and the attention mechanism is integrated to add the semantic relationship between words to better represent dynamic word vectors.

[0139] Second, using the first Kolmogorov-Arnold module 200, a learnable activation function on the edge is introduced, which fundamentally changes the neural network structure and eliminates the linear weight matrix by using a learnable one-dimensional spline function.

[0140] Third, the Transformer encoder module 500 is used to enable the model to capture long-distance relationships between tokens at different positions.

[0141] The automatic three-dimensional scene generation system based on scene semantics and geometric constraints in the embodiments of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), while the non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television, ATM, or self-service machine, etc., without specific limitations in the embodiments of the present application.

[0142] The automatic 3D scene generation system based on scene semantics and geometric constraints in the embodiments of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0143] The three-dimensional scene automatic generation system based on scene semantics and geometric constraints provided by the embodiment of the present application can achieve Figures 1 to 12 In order to avoid repetition, the various processes implemented by the three-dimensional scene automatic generation system based on scene semantics and geometric constraints in the method embodiment are not described here.

[0144] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the various processes of the above-mentioned embodiment of the method for automatic generation of three-dimensional scenes based on scene semantics and geometric constraints are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0145] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the method for automatically generating three-dimensional scenes based on scene semantics and geometric constraints are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0147] It should be noted that, in the present invention, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0149] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for automatically generating three-dimensional scenes based on scene semantics and geometric constraints, characterized in that: include: Build a preset cross-modal KanMiDiffusion denoising network model, and perform text encoding processing on the latent variables according to the DeBERTa text encoder inside the KanMiDiffusion denoising network model to obtain text encoding features, wherein the cross-modal KanMiDiffusion denoising network model includes: a first Kolmogorov-Arnold network, a second Kolmogorov-Arnold network, a third Kolmogorov-Arnold network, a Transformer decoder, a DeBERTa text encoder, and a Dinov2 visual encoder; encoding the geometric structure of the preset indoor object according to the first Kolmogorov-Arnold network to obtain a geometric coding feature; Adding the geometric coding feature to the text coding feature to generate a fusion feature of geometry and text; Performing visual encoding processing on the preset indoor plan according to the Dinov2 visual encoder and then performing feature splicing with the preset position coding features to obtain a fusion feature of vision and position; Processing the geometry and text fusion features and the vision and position fusion features according to the Transformer decoder to obtain an output of the Transformer decoder; Processing the output of the Transformer decoder according to the second Kolmogorov-Arnold network and the third Kolmogorov-Arnold network to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features; A three-dimensional scene is generated based on the categorical distribution of the semantic labels and the Gaussian mean of the geometric features.

2. The method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints according to claim 1, characterized in that: The Kolmogorov-Arnold network includes multiple KAN layers, and the mathematical expression is: ； in, represents the first layer, represents the first layer, represents the first layer of the KAN network, represents the 0th layer of the KAN network, represents the input vector, Represents the input vector The corresponding output result; Among them, each KAN network has The dimension input and The dimension output, Indicates multiplication processing, Depend on Learnable activation functions The mathematical expression is: ； The Kolmogorov-Arnold network is Layer to The mathematical expression of the layer is: ； in, Indicates the The output vector of the layer, Indicates the The output vector of the layer, represents the first Layer Matrix.

3. The method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints according to claim 1, characterized in that: The DeBERTa text encoder includes: multiple layers of parallel connected Transformer encoder layers, word embedding layers, and word vector representation layers. Each layer of Transformer encoder layer includes multiple serially connected Transformer encoders, wherein the word embedding layer is used to embed each position information, sentence embedding, and word embedding as input vectors, which are respectively input into the multiple layers of parallel connected Transformer encoder layers for encoding processing to generate word vectors corresponding to each Transformer encoder layer.

4. The method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints according to claim 1, wherein: The Dinov2 visual encoder is based on ViT architecture, which divides the image into multiple small blocks to obtain an image block sequence to capture long-distance dependencies in the image. The Dinov2 visual encoder includes multi-layer Transformer modules to generate semantically rich high-level feature representations.

5. The method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints according to claim 1, characterized in that: The Transformer decoder includes multiple Transformer blocks, each of which is sequentially connected to a first adaptive normalization layer, a multi-head self-attention layer, a second adaptive normalization layer, a multi-head cross attention layer, and a feedforward neural network layer. The output of the Transformer decoder is obtained by connecting the output of each self-attention head in the multi-head self-attention layer and performing linear projection calculation. The mathematical expression of the output of the Transformer decoder is: ； in, Represents the input vector The corresponding output of the Transformer decoder, is a learnable parameter, Indicates feature splicing processing, Represents the weighted sum of N value vectors.

6. A 3D scene automatic generation system based on scene semantics and geometric constraints, characterized by: include: DeBERTa text encoder module, used to perform text encoding processing on latent variables to obtain text encoding features; The first Kolmogorov-Arnold module is used to encode the geometric structure of the preset indoor object to obtain geometric coding features; A fusion feature generation module, configured to add the geometric coding feature and the text coding feature to generate a fusion feature of geometry and text; The Dinov2 visual encoder module is used to perform visual encoding processing on the preset indoor floor plan and then perform feature splicing with the preset position encoding features to obtain the fusion features of vision and position; A Transformer encoder module is used to process the fusion features of geometry and text and the fusion features of vision and position to obtain the output of the Transformer decoder; a semantic label and geometric feature generation module, configured to process the output of the Transformer decoder according to the second Kolmogorov-Arnold network and the third Kolmogorov-Arnold network to generate a categorical distribution of semantic labels and a Gaussian mean of geometric features; A three-dimensional scene generation module is used to generate a three-dimensional scene based on the classification distribution of the semantic tags and the Gaussian mean of the geometric features.

7. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints are implemented as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for automatically generating a three-dimensional scene based on scene semantics and geometric constraints as described in any one of claims 1 to 5 are implemented.

Images