Double-branch diffusion three-dimensional scene generation method based on multi-modal semantic graph

Abstract

The invention belongs to the technical field of three-dimensional scene modeling, and discloses a dual-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph, which comprises the following steps of: firstly, receiving multi-modal data such as sketches, texts, automatic completion instructions and scene general knowledge, extracting features and fusing the features into a unified multi-modal semantic graph; utilizing a graph neural network and an attention mechanism to enhance semantic graph features, and optimizing physical constraints through a physical engine; complementing the missing visual modality and graph structure relationship; performing quality scoring on the scene based on semantics, spatial relationships and physical constraints; and finally, respectively generating a spatial layout and a geometric shape through a double-branch diffusion model, and ensuring the coordination of the layout and the shape. The method has the advantages of multi-modal information fusion, physical rationality guarantee, high structure complementation capability, high-quality score optimization and efficient generation process, and is suitable for three-dimensional scene modeling requirements in the fields of virtual reality, augmented reality, robots and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional scene modeling, and particularly relates to a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph. BACKGROUND

[0002] With the wide application of large models (such as GPT, BERT, CLIP, etc.) in natural language processing, computer vision and multi-modal learning, especially in virtual reality, augmented reality and robotics, three-dimensional scene generation technology has become increasingly important. Especially in image generation and scene modeling, convolutional neural networks and graph neural networks play an important role in automated three-dimensional scene generation. Existing three-dimensional scene generation methods mostly rely on single modal input (such as images, sketches or texts) and use traditional computer vision techniques for object selection and spatial layout design. Although these methods have made breakthroughs in specific applications, they often have problems such as information fragmentation, insufficient physical constraints and insufficient consideration of object interaction. For example, existing multi-modal learning methods fail to fully integrate information from different modalities (such as vision and language), resulting in poor semantic relevance of the generated scene, and the generated scene may not meet the actual physical constraints, resulting in object overlap, collision or instability.

[0003] How to effectively integrate multi-modal input and generate a three-dimensional scene that is physically reasonable and semantically consistent has become a core problem in current technology development. SUMMARY

[0004] The application aims to provide a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph to solve the above technical problems.

[0005] To solve the above technical problems, the specific technical solution of the double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph is as follows: A double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph, comprising the following steps: Step 1: First, receive data from multi-modal sources, and extract and process the input multi-modal data through different processing modules; Step 2: The graph neural network enhances the features of the multi-modal semantic graph through multi-layer attention mechanisms; Step 3: Complete the missing visual modalities in the graph; at the same time, complete the missing relationships in the graph structure, and through triple relationship learning, enhance the structural representation ability of the graph in the relationship processing between different modalities; Step 4: Based on the semantic information, spatial relationships and physical constraints in the graph, score the quality of the generated three-dimensional scene; Step 5: Generate the spatial layout and geometry in the three-dimensional scene through a double-branch diffusion model.

[0006] Further, the step 1 first receives the sketch instruction, the text instruction, the automatic completion instruction and the data of the scene common sense knowledge, the sketch instruction carries out geometry information extraction and spatial relationship modeling through the graph neural network; the text instruction is converted into semantic features through the text encoder in the CLIP model, providing context information about the scene; the automatic completion instruction and the scene common sense knowledge extract supplementary information through the knowledge base and the semantic retrieval module, finally, through feature fusion and standardization processing, all modal information is unified into a multi-modal semantic graph, providing consistent feature representation for subsequent three-dimensional scene generation.

[0007] Further, the step 1 includes the following specific steps: Step 1.1: Scene combination editing instruction based on sketch Processed by the parser based on the graph neural network; natural language instruction Processed by the CLIP text encoder based on the Transformer architecture; DeepSeek automatic completion instruction Processed by a dedicated scene description encoder; scene common sense knowledge base Query and extraction through the semantic retrieval system based on Elasticsearch; Step 1.2: Node feature fusion and standardization: construct a unified feature representation for each object node; Step 1.3: Edge relationship construction and completion: build the initial relationship network between objects based on input data; Step 1.4: Multi-modal semantic graph generation: the final output multi-modal semantic graph is converted to Protocol Buffers format through a special serialization module.

[0008] Further, the step 2 updates each node according to the spatial relationship in the graph, semantic information and physical attributes, so that the nodes in the graph can obtain rich semantic information from the neighborhood; introduce physical constraints through the physical engine to optimize the node and edge structure in the graph.

[0009] Further, the step 3 includes the following specific steps: Step 3.1: Text-driven visual feature reconstruction: use the text embedding vector in the multi-modal semantic graph node to construct high-quality reconstruction features for missing or incomplete visual modalities through the VQ-VAE model; Step 3.2: Multi-modal structure expansion and dynamic enhanced graph construction: the final visual enhanced graph , which is extended to a hybrid augmented graph after Triplet-GCN completion , for subsequent diffusion model driven layout generation.

[0010] Further, the step 4 calculates the pairing relationship score between objects, evaluates the rationality of the scene layout, and scores the scene according to physical constraints; group relationship modeling and object co-occurrence analysis further help the system to evaluate the coordination and consistency of each part in the scene; the scoring system considers physical, semantic and spatial factors to provide optimization feedback for the subsequent generation process, ensuring that the final output three-dimensional scene meets the expected requirements.

[0011] Further, the double-branch diffusion model of step 5 includes a layout branch and a shape branch, the layout branch generates the position, orientation and scale of each object in the scene through the Transformer diffusion network; the shape branch generates the geometric shape of the object through the conditional variational autoencoder, and keeps consistent with the semantic category; in the diffusion process, the cross-branch consistency mechanism ensures the coordination of information between the layout branch and the shape branch; finally, the system outputs a complete three-dimensional scene graph, including the spatial layout, geometric shape and physical properties of all objects, for subsequent rendering or simulation system.

[0012] The double-branch diffusion three-dimensional scene generation method based on multi-modal semantic graph has the following advantages: Multi-modal information fusion: the present application effectively fuses text, image, sketch and physical knowledge and other multi-modal information, solves the information fragmentation problem in the prior art when processing a single mode, and makes the generated three-dimensional scene more consistent in semantics.

[0013] Physical rationality guarantee: by introducing the combination of graph neural network and physical engine, the physical constraint verification and optimization of the scene are enhanced. The generated three-dimensional scene can effectively avoid physical unreasonable phenomena such as collision and overlap between objects, and ensure its operability and real-world applicability.

[0014] Enhanced structure completion and perception ability: through the VQ-VAE and Triplet-GCN model, the missing visual and structural information in the scene can be repaired and completed, improving the integrity and accuracy of the generation process, and ensuring that the final output scene meets both visual and spatial structure requirements.

[0015] High-quality scoring and optimization: the present application adopts a multi-dimensional scoring system to comprehensively evaluate the quality, semantic consistency and physical rationality of the generated scene, ensuring that the generated three-dimensional scene meets the multiple requirements of users and application scenarios.

[0016] Efficient generation process: through the double-branch diffusion generation mechanism, the spatial layout and geometry are optimized respectively, further improving the efficiency and accuracy of generating three-dimensional scenes. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a method flowchart of an embodiment of the present application.

[0018] Figure 2 is a structural diagram of knowledge base import in an embodiment of the present application.

[0019] Figure 3 is a structural diagram of a physical constraint module in an embodiment of the present application.

[0020] Figure 4 is a structural diagram of a scoring system in an embodiment of the present application.

[0021] Figure 5 is a shape branch structural diagram of a double-branch diffusion model in an embodiment of the present application.

[0022] Figure 6 is a layout branch structural diagram of a double-branch diffusion model in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to better understand the purpose, structure and function of the present application, the following will further describe the present application in detail with reference to the drawings.

[0024] As shown in Figure 1 , a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph of the present application includes the following steps: Step 1: The system first receives data sources from multiple modalities, including sketch instructions, text instructions, automatic completion instructions, and scene common sense knowledge. The input multi-modal data is processed by different processing modules for feature extraction and processing. Specifically, the sketch instruction is processed by a graph neural network for geometric information extraction and spatial relationship modeling; the text instruction is converted into semantic features by a text encoder in the CLIP model, providing context information about the scene; the automatic completion instruction and the scene common sense knowledge are extracted by the knowledge base and the semantic retrieval module. Finally, through feature fusion and standardization processing, the system unifies all modal information into a multi-modal semantic graph, providing consistent feature representation for subsequent three-dimensional scene generation.

[0025] Step 1.1: The system first normalizes and extracts features from the four heterogeneous input data: sketch instructions, text instructions, automatic completion instructions, and scene common sense knowledge. Scene combination editing instructions based on sketches The sketch is processed by a graph neural network-based parser that employs a hierarchical attention mechanism to first identify the basic geometric elements of the sketch, then localizes individual instances through an object detection network, and finally outputs a structured layout set Natural language instructions are processed by a CLIP text encoder based on the Transformer architecture. The CLIP text encoder first tokenizes and embeds the input text, and after 12 layers of self-attention transformation, extracts the final semantic feature representation , which has strong encoding ability for scene description sentences such as furniture attributes and spatial relationships through pre-training. DeepSeek automatic completion instructions are processed by a dedicated scene description encoder. The scene description encoder uses a bidirectional LSTM structure and is pre-trained on a large indoor scene corpus to output features that are compatible with the CLIP feature dimensions. Scene common sense knowledge base is queried and extracted by a semantic retrieval system based on Elasticsearch. Given the preliminary analysis results of the current scene, the system retrieves a set of relevant spatial layout rules and then converts them into numerical features using a rule encoder . The rule encoder uses a hierarchical processing approach: first, it extracts the semantic features of each rule using BERT, then it aggregates related rules through a graph attention network, and finally it outputs a 256-dimensional edge feature vector. The input conversion formula is as follows:

[0026] In this formula, is the class label in the predefined object class set C , represents the center position coordinates of the object in the standardized scene coordinate system, represents the rotation angle of the object around the vertical axis, represents the total number of object instances in the current scene, is the fixed feature dimension, contains professional specifications such as "the main traffic path must maintain at least 80 cm clear width" and "the sleeping area and pollution area must maintain a distance of more than 2 meters."

[0027] Step 1.2: Node feature fusion and standardization.

[0028] The system constructs a unified feature representation for each object node. First, perform feature alignment and dimension unification: text features and Linear projection to 128 dimensions; geometric features Supplemental category embeddings form 64-dimensional vectors; knowledge base features Keep 256-dimensional; visual features and physical attributes are 64-dimensional and 32-dimensional, respectively. The resulting normalized node features preserve the semantic information of the original features while having better numerical properties.

[0029] Step 1.3: Edge relationship construction and completion.

[0030] The system constructs an initial relationship network between objects based on the input data, which encodes various associations between elements in the scene. Spatial relationship edges use a 3-layer MLP with 6-dimensional relative position features as input and 128-dimensional edge features as output, using residual connections and batch normalization. Semantic rule edges directly use knowledge base encoding results, filtering the most relevant 3 rules through an attention mechanism for feature fusion. Completion edges are initialized with zeros, allowing the subsequent graph network to automatically learn potential relationships. The final initial edge set is stored in a sparse matrix to support efficient batch processing. In particular, the system detects and filters redundant edges (such as mutually symmetric relationships) to maintain the simplicity of the graph structure.

[0031] Step 1.4: Multi-modal semantic graph generation.

[0032] The final output multi-modal semantic graph is converted to Protocol Buffers format by a special serialization module, fully preserving the multi-modal characteristics of the input data: node features combine semantic and geometric information, and edge features encode spatial and logical constraint relationships. This representation not only preserves all the key information of the original input but also converts it into a structured format suitable for computer processing, providing a high-quality input basis for subsequent visual enhancement, relationship reasoning, and scene generation processing stages. The following data structures are used to represent: Node set : Each node contains 256-dimensional features and object metadata; Edge set : Stored as a tensor of N × N ×128, supporting multiple relationship types; Graph attributes: Record feature indices, timestamps, version numbers, and other meta-information of the original input.

[0033] Step 2: The graph neural network enhances the multi-modal semantic graph through a multi-layer attention mechanism. The system updates each node by weighting based on spatial relationships, semantic information, and physical properties in the graph, enabling nodes to obtain rich semantic information from their neighbors. To ensure the physical plausibility of the scene, the system introduces physical constraints such as collision detection, path reachability analysis, and object stability evaluation through a physics engine, optimizing the node and edge structures in the graph. In this way, the system can correct parts that do not meet physical constraints in real time and generate a three-dimensional scene that complies with physical rules.

[0034] Step 2.1: Graph structure feature enhancement.

[0035] After receiving the user's multi-modal input and constructing the initial multi-modal semantic graph, the system introduces a graph neural network structure based on a multi-head attention mechanism to enhance the node features layer by layer. The graph neural network consists of L = 4 layers of graph attention networks, each containing K = 8 independent attention heads. For each attention head, the network first performs a linear transformation on the node features, projecting them into a shared feature space. Then, it evaluates the correlation strength between node pairs by calculating attention coefficients, which are nonlinearly processed using a LeakyReLU activation function. To ensure the rationality of weight distribution, the system performs softmax normalization on the original attention coefficients, so that the sum of the attention weights of each node for its neighbors is 1. In the feature aggregation stage, each attention head weights the sum of the features of neighboring nodes according to the calculated attention weights. This mechanism allows the network to adaptively focus on more important neighbor information for the current node. The outputs of all attention heads are fused in the last layer, usually using a concatenation operation to preserve different feature patterns learned by each attention head. To alleviate the gradient vanishing problem in deep network training, the network introduces residual connections between layers, adding the features of the previous layer to the transformed features of the current layer. In addition, the output of each layer is also subjected to layer normalization to stabilize the training process and accelerate convergence. After being processed by 4 layers of such graph attention networks, the initial node features are gradually enhanced into deep representations containing rich contextual information. These enhanced features not only encode the attributes of the nodes themselves, but also incorporate global graph structure information, enabling them to more accurately reflect the semantic relationships and spatial constraints between objects in a three-dimensional scene.

[0036] Step 2.2: Physical constraint verification and correction.

[0037] To ensure that the generated graph structure meets the physical plausibility of the three-dimensional scene, the system introduces a physics engine to analyze three types of physical constraints for candidate relationships in the graph, including collision detection, layout constraints, and path reachability evaluation. The formula is as follows:

[0038] where, represents the collision detection score evaluation function, represents the 3D bounding box of the i-th object, represents the 3D bounding box of the i-th object, represents the three-dimensional intersection-over-union between two objects; represents the layout constraint score evaluation function, is the room plan, represents the total number of object 3D bounding boxes, represents the total number of wall bounding boxes, represents the i-th wall bounding box, represents the i-th wall bounding box, represents the path reachability score evaluation function, and

[0039] Step 2.3: Dynamic graph structure optimization.

[0040] After the preliminary scoring is completed by the physical verification module, the system dynamically corrects and enhances the edge weights and node features of the graph. First, the weights of all edges in the graph are updated according to the current physical compliance score , and then the system uses a lightweight multi-layer perception to correct the semantic features of the nodes in the physical violation area. To improve the sampling resolution in physically sensitive areas, the system also introduces an adaptive sampling strategy to increase the sampling points in areas with frequent physical conflicts.

[0041] Step 2.4: Multi-scale feature fusion.

[0042] To capture spatial semantic information of different granularities in the graph structure, the system constructs a three-level hybrid feature pyramid.

[0043] First, object-level local features are extracted through standard graph neural networks. Second, regional-level functional clustering features are achieved through clustering pooling. Third, global scene features are modeled through a self-attention mechanism encoder. Finally, the three are unified encoded and standardized through a linear fusion layer.

[0044] Step 2.5: Iterative optimization convergence.

[0045] The system sets convergence conditions during graph enhancement to ensure stable termination of the optimization phase, mainly including the following three points: one is that the physical violation rate is less than 5%, two is that the feature change rate of the graph is less than 5% for two consecutive times, and three is that the maximum number of iterations =20 times. The overall optimization goal is: ​​​​

[0046] in, is the feature reconstruction loss, is the collision loss, is the reachability loss, is the static stability loss, is the sum of the three physical constraint losses, is the graph semantic consistency loss.

[0047] Step 3: The system uses the VQ-VAE model to complete missing visual modalities in the image, ensuring that the image's visual features are fully restored. Simultaneously, the Triplet-GCN (graph convolutional network) is used to complete missing relationships in the graph structure. Specifically, the system learns triplet relationships between different modalities to enhance the graph's structural representation capabilities. This module enables the system to more completely understand and express the multimodal information in the image, providing richer feature input for subsequent 3D scene generation.

[0048] Step 3.1: Text-driven visual feature reconstruction: The system uses the text embedding vectors in the multimodal semantic graph nodes to construct high-quality reconstruction features of missing or incomplete visual modalities through the VQ-VAE model. The process includes three stages: latent encoding, quantization, and decoding: The encoder transforms text features Mapped to a potential representation, then select the codebook with Recent The decoder maps the quantized vector back to the image feature space. In order to optimize the generation effect, the model maximizes the lower bound of evidence.

[0049] Step 3.2: Multimodal structure expansion and dynamic enhancement graph construction: The final visual enhancement image , which is expanded into a hybrid enhancement graph after Triplet-GCN completion , used for subsequent diffusion model-driven layout generation. Its structure contains a node feature set: , Indicates a dimensional real vector space, is the total number of objects in the image, is the feature dimension; edge feature set: ,in The edge feature dimension is also constructed simultaneously, providing physical guidance signals for the next step of the diffusion model.

[0050] Step 4: The system scores the generated three-dimensional scene based on semantic information, spatial relationships, and physical constraints in the graph. Specifically, the system calculates the pairing relationship scores between objects, evaluates the rationality of the scene layout, and scores the scene according to physical constraints. Group relationship modeling and object co-occurrence analysis further help the system assess the coordination and consistency of each part in the scene. The scoring system considers physical, semantic, and spatial factors to provide optimization feedback for the subsequent generation process, ensuring that the final output three-dimensional scene meets the expected requirements.

[0051] Step 4.1: Pairing relationship modeling: The system first constructs a pairing relationship vector with any object pair in the scene graph as the unit, which integrates object semantic features, spatial relative positions, and relationship type labels to measure the direct logical relationship between two objects.

[0052] Step 4.2: Group relationship modeling: To further capture the local clustering structure and functional partition semantics of objects, the system adopts a dual clustering strategy based on semantic labels and spatial density to group objects in the scene graph. Each group of objects is encoded as a Gaussian distribution, and the system defines an interaction matrix between groups, where is the total number of groups in the scene, and the element represents the semantic-spatial coupling strength between group and , which is calculated as follows:

[0053] where represents the mean vector of cluster , represents the mean of all points belonging to cluster , represents the covariance matrix of cluster , represents the covariance of all points belonging to cluster , represents the similarity between cluster and cluster , represents the Frobenius norm of the covariance matrix difference between cluster and cluster , which measures the difference between the two covariance matrices. is a predefined semantic template direction vector.

[0054] Step 4.3: Object co-occurrence modeling: The system builds an object co-occurrence statistical atlas to determine whether the combination pattern of the current input graph conforms to the semantic norm. For any object pair , the system uses the co-occurrence matrix and the standard co-occurrence matrix to perform Kullback-Leibler divergence matching, and the co-occurrence probability formula is defined as follows:

[0055] where represents the number of scenes in which and co-occur in the knowledge base, is the total number of reference scenes, represents Kullback-Leibler divergence ( KL divergence), which measures the difference between two distributions and .

[0056] Step 4.4: Structural similarity evaluation: The system evaluates the global consistency between the input graph structure and the reference graph structure through graph embedding comparison, and uses graph neural network to encode the graph structure and then calculates the embedding distance, while considering the co-occurrence logical deviation to form a joint scoring function formula as follows:

[0057] where represents the structural difference measure, which measures the difference between the scene features and the reference features , represents divergence, which quantifies the logical deviation of the input graph and the reference graph in node or edge distribution, represents the score related to the relationship, and are weight factors, and the empirical value is 0.5.

[0058] Step 4.5: Multi-dimensional score calculation and fusion: According to the above model, the system constructs three main score items to comprehensively evaluate the scene rationality: object relationship score , reflecting the semantic consistency of the paired relationship; layout topology score , used to evaluate whether the overall spatial distribution is balanced and coherent; and basic relationship score , which combines object co-occurrence difference and edge connectivity rate to evaluate whether there are uncommon combinations and connection missing. The final scene comprehensive score fuses the three indicators to form the final score formula as follows:

[0059] where, denotes the number of samples, denotes the relationship vector between objects in the scene and reference, respectively. and denotes the similarity score of the layout, is a tuning factor that controls the influence of the bias and density on the score, denotes the average positional bias between all nodes and the reference graph; denotes the local density standard deviation of the nodes are weight coefficients, with default values of 0.4, 0.3, and 0.3, respectively.

[0060] The Top-K scene graphs sorted by the comprehensive score are used as input for the dual-branch diffusion generation model in step five.

[0061] Step 5: The system generates the spatial layout and geometric shapes in the three-dimensional scene through a dual-branch diffusion model. The dual-branch diffusion model includes a layout branch and a shape branch. The layout branch generates the position, orientation, and scale of each object in the scene through a Transformer diffusion network, ensuring a reasonable spatial layout of the scene. The shape branch generates the geometric shape of the object through a conditional variational autoencoder, consistent with the semantic category. During the diffusion process, the system ensures information coordination between the two branches through a cross-branch consistency mechanism, avoiding mismatch between the layout and shape. Finally, the system outputs a complete three-dimensional scene graph, including the spatial layout, geometric shape, and physical properties of all objects, for subsequent rendering or simulation systems.

[0062] Step 5.1: Initial diffusion modeling: The system first initializes a geometric feature representation tensor for each node of the multi-modal semantic graph . In each branch, the training target is to learn the inverse denoising model .

[0063] Step 5.2: Layout branch diffusion generation: The layout branch is responsible for generating the spatial position , orientation , and scale parameters for each scene object. The model structure uses a Transformer diffusion network that integrates time step embedding and local graph attention information. Its loss function includes positioning error and spatial consistency regular formula as follows:

[0064] where ​loss function of layout, predicted and real positions of the i-th sample, predicted and real rotations of the i-th sample, predicted and real scales of the i-th sample, prior loss, regularization term weight. Step 5.3: Shape branch diffusion generation: lambda The shape branch focuses on modeling the local geometric properties and category semantics of the object. Each node corresponds to a shape latent variable , which represents the generation parameters of its 3D model.

[0065] The diffusion sampling target is to accurately reconstruct the object geometry and be consistent with the semantic category, and its formula is as follows:

[0066]

[0067] where, is the loss function of shape, predicted and real values of the i-th sample, is the object category predicted by the model, is the real object category label, and CE represents the cross-entropy loss. The network structure uses a conditional variational autoencoder combined with diffusion residual blocks for modeling. Step 5.4: Cross-branch consistency modeling:

[0068] To maintain the cooperative relationship between layout and shape, the system exchanges information through a cross-branch attention module in each round of diffusion. The layout features are used to modulate the shape branch generation, while a consistency loss is defined to constrain the alignment of shape and layout information, keeping the predicted shape orientation consistent with the object pose. Step 5.5: Scene fusion and output reconstruction:

[0069] Finally, the system combines the output features of the two branches to generate a complete structured three-dimensional scene graph, and its formula is as follows: where:

[0070] is the node set (object instance), is the interaction edge set (guided by physical constraints or common sense graphs), is the position set, is the orientation set, ​​​​a set of scales, a set of shape latent variables, a semantic class of the object.

[0071] The synthesized scene result will be physically corrected (such as collision detection, scale correction, etc.) by the post-processing module, and output as a three-dimensional scene file that can be used by a rendering or simulation system.

[0072] Embodiments: A multi-modal semantic graph-based double-branch diffusion three-dimensional scene generation method, comprising the following steps: Step 1: Extract and fuse the features of the input data of multiple modalities to generate a unified multi-modal semantic graph.

[0073] The input includes sketch-based scene combination editing instructions, such as this is a sketch of an indoor bedroom. In the center of the picture is a large bed, and on both sides of the bed are a bedside table and a table lamp. A long chair or low cabinet is placed against the wall on the right, and a set of sofas and a tea table are placed on the left, and a TV cabinet or desk is placed near the foreground; DeepSeek-based efficient scene combination editing instruction generation, such as plan: the living room adopts a symmetrical design, with a double sofa placed in the center, and a single chair placed on both sides, and a long tea table placed in the middle to form a reception area. The bedroom and living room are separated by a semi-transparent screen or bookshelf, the bed is placed near the window to make use of natural light, and a small wardrobe can be placed at the end of the bed to ensure storage function; User requirements, such as I want 6 plans, the space area is 50 square meters, the sofa is located in the living room part, the sofa is a right-angle sofa around the entire living room, the color is gray, the style of the sofa is a certain home type, the bed is placed in the room opposite the living room, the color of the bed is blue, and the style of the bed is a fabric bed; Scene common sense knowledge base, such as the main traffic path (such as the path from the door to the balcony) needs to maintain a clear width of at least 80 centimeters, sofas, cabinets, and other furniture should not be horizontally cut off the passage to ensure barrier-free access for people. The sleeping area (bed) and the pollution area (kitchen stove) need to be physically isolated, with a straight-line distance of not less than 2 meters, to avoid interference from oil smoke and comply with residential health standards.

[0074] As shown in Figure 2 , the implementation of the scene common sense knowledge base import is as follows: First, external multi-modal knowledge bases are imported, containing structured and unstructured data that covers common sense knowledge, physical constraints, and spatial relationship descriptions. Then, based on these data, a scene graph and a knowledge graph are constructed. The scene graph represents various objects and their relative positions in the physical space through abstraction, while the knowledge graph represents objects and their semantic relationships through nodes and edges, providing a reasoning basis for spatial modeling. Subsequently, the system integrates the scene graph and the knowledge graph to form a multi-level, cross-domain knowledge-scene graph network, and verifies and reasons through embedded common sense knowledge and rule constraints such as architectural design specifications and hygiene standards. For example, the system automatically detects whether the width of the moving line from the door to the balcony meets the minimum 80 cm net width requirement. Next, by combining the scene object library to further refine each scene element, the scene object library provides more detailed object descriptions, including their functional attributes, size specifications, and other information, enhancing the system's accuracy in scene analysis. Then, the scene graph and knowledge graph data after reasoning and verification are passed to downstream application systems such as smart home and automated design tools, ensuring accurate knowledge and information flow and providing efficient support for intelligent decision-making systems. Finally, the system feeds back the optimized scene design scheme to the user, ensuring that all spatial constraints and common sense knowledge are met, thereby achieving intelligent space design and optimization.

[0075] By fusing the entities and relationships in the graph with the multi-modal semantic graph generated by the command, the semantic ability of scene understanding is enhanced. The knowledge graph contains a large amount of entities, relationships, and attributes about common sense (e.g., the relationship between "table" and "chair"). These knowledge is crucial for understanding objects in a three-dimensional scene and their mutual relationships. In this step, the model maps each entity and relationship in the common sense knowledge graph to the same embedding space as the multi-modal semantic graph features. In this way, the common sense knowledge graph and the point cloud feature space are aligned in the same embedding space, allowing the model to reason with both pieces of information. Given the preliminary analysis results of the current scene, the system retrieves a set of relevant spatial layout rules and converts them into numerical features through a rule encoder :

[0076] where contains professional specifications such as "the main path of travel must maintain at least 80 cm net width" and "sleeping areas and pollution areas must maintain a distance of more than 2 meters."

[0077] Step 2: Use graph neural networks to enhance the features of the multi-modal semantic graph, and at the same time, optimize the scene through the physical engine for physical constraints to ensure physical rationality.

[0078] whereas Figure 3 the implementation of the physical constraint module is as follows: First, the system identifies and defines the main elements in the scene, such as rooms, furniture, doors, and windows, and determines their basic positions and roles in the space. Then, collision constraints are applied, and the system checks whether there is any overlap or collision between objects in the space. By calculating the minimum distance between objects, unnecessary overlap is avoided, ensuring that each object has enough space. Then, the system applies room layout constraints , which ensure that the placement of objects in the scene meets physical specifications, such as the distance between furniture and walls, or the width of the passageway. Subsequently, the system further introduces accessibility constraints , ensuring smooth connection between different areas, such as ensuring that the path from the room entrance to the balcony is not blocked by any object. The shortest distance of the path can be calculated using the Euclidean distance formula to ensure the smoothness of the path. At this point, the U-Net model is introduced to further process and optimize the physical constraints, adjusting the positions of objects to better meet various constraint requirements. Finally, through repeated optimization, the system generates a space layout design that meets all physical constraints, ensuring that the final design not only meets functional requirements but also satisfies the physical rules and logical constraints of space layout.

[0079] The formula is as follows:

[0080] where, represents the collision detection score evaluation function, respectively represents the 3D bounding box of the th object, represents the three-dimensional intersection ratio between two objects; represents the layout constraint score evaluation function, is the room plan, represents the total number of object 3D bounding boxes, represents the total number of wall bounding boxes, represents the th wall bounding box; represents the path accessibility score evaluation function, and positions are selected on the path, corresponding to is the 3D bounding box of .

[0081] Step 3: Complete the missing information in the graph and enhance the expression ability of the graph structure through the multi-modal perception module to make the scene more complete.

[0082] Step 4: Evaluate the semantic consistency, spatial layout rationality, and physical feasibility of the generated scene through a scoring system, providing feedback for optimization.

[0083] wherein, as shown in Figure 4 the scoring system for multi-modal semantic graphs is implemented as follows: First, a pair-wise relationship model is used for preliminary analysis, which assesses the pair-wise relationships between elements in the scene (such as objects, furniture, etc.) to determine if they conform to pre-set layout rules. Next, a group relationship model is applied to evaluate the mutual relationships between multiple objects to determine if they are reasonably grouped together. For example, a sofa and a coffee table in a living room should be placed together to form an effective functional area. Subsequently, an object co-occurrence model is applied to analyze the frequency of different objects appearing in space and their co-occurrence patterns. This helps the system determine which elements often appear simultaneously in the same space, further optimizing the layout. Next, a relationship model similarity module calculates the similarity between objects to ensure that similar objects are arranged in a suitable manner, avoiding incompatible or discordant elements from being placed together. Then, the system scores the scene layout by combining the results of multiple modules for comprehensive analysis. The basic relationship analysis and object relationship scoring modules provide detailed scoring of the layout of objects in the scene, evaluating their position and functionality in space to determine if they meet design requirements.

[0084] Step 5: Generate the spatial layout and object shapes of the three-dimensional scene through the layout branch and shape branch respectively, and finally output the complete three-dimensional scene graph.

[0085] wherein, as shown in Figure 5 the implementation of the shape branch module is as follows: First, the normalization process involves converting input data to a uniform scale or range for the accuracy and stability of subsequent processing. Next, the cross-attention module is applied, which is commonly used in neural networks to strengthen the association between different features, helping the model better capture important information and patterns. Then, the downsampling process mentioned in the figure is usually applied in convolutional neural networks, which reduces the resolution of images or spatial data step by step to reduce computational complexity while preserving important features. Next, the calculation of TSDF is involved, which is a way to represent object geometry commonly used in 3D reconstruction, which can effectively handle the estimation of object surface distance. After that, the latent code and 3D-UNet are used for further feature learning and extraction of geometric information, respectively. 3D-UNet is a neural network architecture specifically designed for three-dimensional data processing, commonly used in tasks such as medical image segmentation. Finally, after processing through these modules, the system can generate accurate object geometry models and accurately model and segment objects in 3D space. Its diffusion sampling goal is to accurately reconstruct the object geometry and be consistent with the semantic category, and its formula is as follows:

[0086] where, represents the shape loss function, represents the predicted value and the true value of the th sample, respectively, represents the object category predicted by the model, represents the true object category label, and CE represents the cross-entropy loss. The network structure uses a conditional variational autoencoder combined with diffusion residual blocks for modeling.

[0087] where, as shown in Figure 5 , the implementation of the layout branch module is as follows: First, the process begins with a convolution operation using a 1x5 kernel, which extracts features from the input data through batch normalization and ReLU activation. This operation helps the model extract effective spatial features from the raw data. Next, max pooling 1x2 and down-sampling 1x2 are used to further reduce the spatial resolution, which helps reduce computational load while preserving important high-level features in the image. The skip connection plays an important role in this process, allowing low-level features to be directly passed to the high-level network to preserve more detailed information and improve network performance. The process also includes a one-dimensional convolution 1x1 operation, which is typically used to further refine features and generate output probability distributions representing the model's prediction for each location. The generation of noisy bounding boxes is a key step, where the model generates the bounding boxes of objects and adds noise information to improve the robustness and accuracy of the model. Finally, the 1D-UNet architecture is used to process information such as the location, rotation, and size of objects. The UNet architecture is typically used for image segmentation tasks, but here it is extended for one-dimensional data processing to generate accurate object location and size predictions. The loss function includes the localization error and spatial consistency regularization formula as follows:

[0088] where represents the loss function of the layout, represents the predicted location and the true location of the th sample, represents the predicted rotation and the true rotation of the th sample, represents the predicted scaling and the true scaling of the th sample, represents the prior loss, lambda is the regularization term weight.

[0089] It is understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, modifications can be made to these features and embodiments to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A multi-modal semantic graph based dual-branch diffusion three-dimensional scene generation method, characterized in that, The method comprises the following steps: Step 1: first receive data from multi-modal data sources, and extract and process features of input multi-modal data through different processing modules; Step 2: the graph neural network enhances the features of the multi-modal semantic graph through a multi-layer attention mechanism; Step 3: complete the missing visual modal in the graph; meanwhile, complete the missing relationship in the graph structure, and enhance the structural representation capability of the graph through triple relationship learning in the relationship processing between different modalities; Step 4: based on the semantic information, spatial relationship and physical constraint in the graph, score the generated three-dimensional scene; Step 5: generate the spatial layout and geometric shape in the three-dimensional scene through a double-branch diffusion model.

2. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 1 first receives sketch instructions, text instructions, automatic completion instructions and scene common sense knowledge data, the sketch instructions are subjected to geometric information extraction and spatial relationship modeling through a graph neural network; the text instructions are converted into semantic features through a text encoder in a CLIP model to provide context information about the scene; the automatic completion instructions and the scene common sense knowledge are subjected to supplementary information extraction through a knowledge base and a semantic retrieval module, and finally, all modal information is unified into a multi-modal semantic graph through feature fusion and standardization processing to provide consistent feature representation for subsequent three-dimensional scene generation.

3. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 1 comprises the following specific steps: Step 1.1: Sketch-based scene composition editing instructions Processed by a graph neural network-based parser; natural language instructions Processed by a CLIP text encoder based on the Transformer architecture; DeepSeek autocomplete instructions Processed by a dedicated scene description encoder; scene common sense knowledge base Query and extraction through an Elasticsearch-based semantic retrieval system; Step 1.2: node feature fusion and standardization: constructing a unified feature representation for each object node; Step 1.3: edge relationship construction and completion: constructing an initial relationship network between objects based on input data; Step 1.4: multi-modal semantic graph generation: the finally output multi-modal semantic graph is converted into a Protocol Buffers format through a special serialization module.

4. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 2 updates each node according to the spatial relationship, semantic information and physical attribute in the graph, so that the node in the graph can obtain rich semantic information from the neighborhood; the physical constraint is introduced through a physical engine to optimize the node and edge structure in the graph.

5. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 3 comprises the following specific steps: Step 3.1: text-driven visual feature reconstruction: using the text embedding vector in the multi-modal semantic graph node, constructing high-quality reconstructed features of missing or incomplete visual modalities through a VQ-VAE model; Step 3.2: Multi-modal structure extension and dynamic enhancement map construction: final visual enhancement map , after Triplet-GCN completion, extended to hybrid enhancement map , for subsequent diffusion model driven layout generation.

6. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 4 calculates the pairing relationship score between objects to evaluate the rationality of the scene layout, and scores the scene according to the physical constraint; group relationship modeling and object co-occurrence analysis further help the system to evaluate the coordination and consistency of each part in the scene; the scoring system comprehensively considers the physical, semantic and spatial factors to provide optimization feedback for the subsequent generation process, and ensures that the finally output three-dimensional scene meets the expected requirements.

7. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The double-branch diffusion model of step 5 includes a layout branch and a shape branch; the layout branch generates the positions, orientations and scales of various objects in the scene through a Transformer diffusion network; the shape branch generates the geometric shapes of the objects through a conditional variational autoencoder and keeps consistent with the semantic categories; during the diffusion process, a cross-branch consistency mechanism is used to ensure the coordination of information between the layout branch and the shape branch; finally, the system outputs a complete three-dimensional scene graph including the spatial layout, geometric shape and physical properties of all objects for subsequent rendering or simulation systems.

Images