A multi-modal semantic graph-based dual-branch diffusion three-dimensional scene generation method

Abstract

The application belongs to the technical field of three-dimensional scene modeling, and discloses a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph, which comprises the following steps: firstly, multi-modal data such as sketches, texts, automatic completion instructions and scene common sense knowledge are received, features are extracted and fused into a unified multi-modal semantic graph; the semantic graph features are enhanced by using a graph neural network and an attention mechanism, and physical constraints are optimized through a physical engine; missing visual modalities and graph structure relationships are completed; the scene is scored in quality based on semantics, spatial relationships and physical constraints; finally, spatial layout and geometric shape are respectively generated through a double-branch diffusion model, ensuring the coordination of the layout and the shape. The advantages of the application include multi-modal information fusion, physical rationality guarantee, strong structure completion capability, high-quality score optimization and efficient generation process, and the application is suitable for three-dimensional scene modeling requirements in the fields of virtual reality, augmented reality and robots.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional scene modeling, and particularly relates to a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph. BACKGROUND

[0002] With the wide application of large models (such as GPT, BERT, CLIP, etc.) in natural language processing, computer vision and multi-modal learning, especially in virtual reality, augmented reality and robotics, three-dimensional scene generation technology has become increasingly important. Especially in image generation and scene modeling, convolutional neural networks and graph neural networks play an important role in automated three-dimensional scene generation. Existing three-dimensional scene generation methods mostly rely on single modal input (such as images, sketches or texts) and use traditional computer vision techniques for object selection and spatial layout design. Although these methods have made breakthroughs in specific applications, they often have problems such as information fragmentation, insufficient physical constraints and insufficient consideration of object interaction. For example, existing multi-modal learning methods fail to fully integrate information from different modalities (such as vision and language), resulting in poor semantic relevance of the generated scene, and the generated scene may not conform to actual physical constraints, resulting in object overlap, collision or instability.

[0003] How to effectively integrate multi-modal input and generate a three-dimensional scene that is physically reasonable and semantically consistent has become a core problem in current technology development. SUMMARY

[0004] The application aims to provide a double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph to solve the above technical problems.

[0005] To solve the above technical problems, the specific technical solution of the double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph is as follows:

[0006] A double-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph includes the following steps:

[0007] Step 1: First, receive data from multi-modal sources, and extract and process the input multi-modal data through different processing modules;

[0008] Step 2: The graph neural network enhances the features of the multi-modal semantic graph through multi-layer attention mechanisms;

[0009] Step 3: Complete the missing visual modal in the graph; at the same time, complete the missing relationship in the graph structure, and through triple relationship learning, enhance the structure representation ability of the graph in the relationship processing between different modalities;

[0010] Step 4: Quality scoring of the generated three-dimensional scene based on semantic information, spatial relationships, and physical constraints in the graph.

[0011] Step 5: Generating spatial layout and geometry in the three-dimensional scene through a dual-branch diffusion model.

[0012] Further, step 1 first receives data of sketch instructions, text instructions, automatic completion instructions, and scene common sense knowledge. The sketch instructions are processed through a graph neural network for geometric information extraction and spatial relationship modeling. The text instructions are converted into semantic features through a text encoder in the CLIP model, providing context information about the scene. The automatic completion instructions and scene common sense knowledge are extracted through a knowledge base and semantic retrieval module to supplement information. Finally, through feature fusion and standardization processing, all modal information is unified into a multi-modal semantic graph, providing consistent feature representation for subsequent three-dimensional scene generation.

[0013] Further, step 1 includes the following specific steps:

[0014] Step 1.1: Scene combination editing instructions based on sketches Processed through a graph neural network-based parser; natural language instructions Processed through a CLIP text encoder based on the Transformer architecture; DeepSeek automatic completion instructions Processed through a dedicated scene description encoder; scene common sense knowledge base Query and extraction through an Elasticsearch-based semantic retrieval system;

[0015] Step 1.2: Node feature fusion and standardization: constructing a unified feature representation for each object node;

[0016] Step 1.3: Edge relationship construction and completion: constructing an initial relationship network between objects based on input data;

[0017] Step 1.4: Multi-modal semantic graph generation: the final output multi-modal semantic graph is converted into Protocol Buffers format through a special serialization module.

[0018] Further, step 2 updates each node based on spatial relationships, semantic information, and physical properties in the graph, enabling nodes in the graph to obtain rich semantic information from the neighborhood. Physical constraints are introduced through a physics engine to optimize node and edge structures in the graph.

[0019] Further, step 3 includes the following specific steps:

[0020] Step 3.1: Text-driven visual feature reconstruction: Utilizing the text embedding vectors in the multi-modal semantic graph nodes, high-quality reconstructed features for missing or incomplete visual modalities are constructed through the VQ-VAE model.

[0021] Step 3.2: Multi-modal structure extension and dynamic augmented graph construction: The final visual augmented graph , which is extended after Triplet-GCN completion to a hybrid augmented graph , for subsequent diffusion model-driven layout generation.

[0022] Further, the step 4 calculates the pairing relationship scores between objects, evaluates the rationality of the scene layout, and scores the scene according to physical constraints; group relationship modeling and object co-occurrence analysis further help the system evaluate the coordination and consistency of each part in the scene; the scoring system considers physical, semantic, and spatial factors to provide optimization feedback for the subsequent generation process, ensuring that the final output three-dimensional scene meets the expected requirements.

[0023] Further, the step 5 double-branch diffusion model includes a layout branch and a shape branch, the layout branch generates the position, orientation, and scale of each object in the scene through the Transformer diffusion network; the shape branch generates the geometric shape of the object through the conditional variational autoencoder, and keeps consistent with the semantic category; during the diffusion process, the cross-branch consistency mechanism ensures the coordination of information between the layout branch and the shape branch; finally, the system outputs a complete three-dimensional scene graph, including the spatial layout, geometric shape, and physical properties of all objects, for subsequent rendering or simulation systems.

[0024] The multi-modal semantic graph-based double-branch diffusion three-dimensional scene generation method of the present application has the following advantages:

[0025] Multi-modal information fusion: The present application effectively fuses multiple modal information such as text, image, sketch, and physical knowledge, solving the information fragmentation problem in the prior art when processing a single modality, making the generated three-dimensional scene more consistent in semantics.

[0026] Physical rationality guarantee: By introducing the combination of graph neural networks and physical engines, the physical constraint verification and optimization of the scene are enhanced. The generated three-dimensional scene can effectively avoid physical irrational phenomena such as collision and overlap between objects, ensuring its operability and real-world applicability.

[0027] Enhanced structure completion and perception: Through the VQ-VAE and Triplet-GCN models, missing visual and structural information in the scene can be repaired and completed, improving the integrity and accuracy of the generation process, ensuring that the final output scene meets both visual and spatial structure requirements.

[0028] High-quality scoring and optimization: The present application adopts a multi-dimensional scoring system to comprehensively evaluate the quality, semantic consistency and physical reasonableness of the generated scene, ensuring that the generated three-dimensional scene meets the multiple requirements of users and application scenarios.

[0029] Efficient generation process: Through the double-branch diffusion generation mechanism, the spatial layout and geometric shape are optimized respectively, further improving the efficiency and accuracy of generating three-dimensional scenes. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a method flowchart of an embodiment of the present application.

[0031] Figure 2 is a structural schematic diagram of knowledge base import in an embodiment of the present application.

[0032] Figure 3 is a structural schematic diagram of the physical constraint module in an embodiment of the present application.

[0033] Figure 4 is a structural schematic diagram of the scoring system in an embodiment of the present application.

[0034] Figure 5 is a shape branch structural schematic diagram of the double-branch diffusion model in an embodiment of the present application.

[0035] Figure 6 is a layout branch structural schematic diagram of the double-branch diffusion model in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to better understand the purpose, structure and function of the present application, the present application of a double-branch diffusion three-dimensional scene generation method based on multi-modal semantic graph will be further described in detail below in conjunction with the drawings.

[0037] As shown in Figure 1 , a double-branch diffusion three-dimensional scene generation method based on multi-modal semantic graph of the present application includes the following steps:

[0038] Step 1: The system first receives data from multiple modalities, including sketch instructions, text instructions, autocomplete instructions, and scene common sense knowledge. These multi-modal data inputs are processed through different modules for feature extraction and processing. Specifically, sketch instructions are processed through a graph neural network for geometric information extraction and spatial relationship modeling; text instructions are converted into semantic features through a text encoder in the CLIP model, providing contextual information about the scene; autocomplete instructions and scene common sense knowledge are extracted through a knowledge base and semantic retrieval module. Finally, through feature fusion and standardization processing, the system unifies all modal information into a multi-modal semantic graph, providing consistent feature representation for subsequent three-dimensional scene generation.

[0039] Step 1.1: The system first normalizes and extracts features from the four heterogeneous input data: sketch instructions, text instructions, autocomplete instructions, and scene common sense knowledge. The sketch-based scene composition editing instructions are processed through a graph neural network-based parser that uses a hierarchical attention mechanism to first identify basic geometric elements of the sketch, then locates each instance through an object detection network, and finally outputs a structured layout set Natural language instructions Processed through a CLIP text encoder based on the Transformer architecture. The CLIP text encoder first tokenizes and embeds the input text, and after 12 layers of self-attention transformation, extracts the final semantic feature representation The feature vector has strong encoding ability for scene description sentences such as furniture attributes and spatial relationships, obtained through pre-training. DeepSeek autocomplete instructions Processed through a dedicated scene description encoder. The scene description encoder uses a bidirectional LSTM structure and is pre-trained on a large indoor scene corpus to output features Consistent with the CLIP feature dimension to achieve feature compatibility. Scene common sense knowledge base Query and extract through an Elasticsearch-based semantic retrieval system. Given the preliminary analysis results of the current scene, the system retrieves a set of related spatial layout rules Then convert them into numerical features through a rule encoder The rule encoder uses a hierarchical processing method: first uses BERT to extract the semantic features of each rule, then aggregates related rules through a graph attention network, and finally outputs a 256-dimensional edge feature vector. The conversion formula for the input is as follows:

[0040]

[0041] In this formula, is a predefined set of object categories C is the category label of the object, represents the center position coordinates of the object in the normalized scene coordinate system, represents the rotation angle of the object around the vertical axis, represents the total number of object instances in the current scene, is a fixed feature dimension, contains professional specifications such as "the main traffic path needs to maintain at least 80 cm clear width", "the sleeping area and the contaminated area need to maintain a distance of more than 2 meters" and so on.

[0042] Step 1.2: Node feature fusion and standardization.

[0043] The system constructs a unified feature representation for each object node. First, feature alignment and dimension unification are performed: text features and are projected linearly to 128 dimensions; geometric features form a 64-dimensional vector after supplementing category embeddings; knowledge base features remain 256-dimensional; visual features and physical attributes are 64-dimensional and 32-dimensional respectively. The final standardized node features retain the semantic information of the original features while having better numerical characteristics.

[0044] Step 1.3: Edge relationship construction and completion.

[0045] The system constructs an initial relationship network between objects based on input data, which encodes various associations between elements in the scene. Spatial relationship edges use a 3-layer MLP with 6-dimensional relative position features as input, output 128-dimensional edge features, and use residual connections and batch normalization. Semantic rule edges directly use knowledge base encoding results, and filter the most relevant 3 rules through an attention mechanism for feature fusion. The completion edge uses zero initialization, allowing the subsequent graph network to automatically learn potential relationships, and the final initial edge set is stored in a sparse matrix to support efficient batch processing. In particular, the system detects and filters redundant edges (such as mutually symmetric relationships) to maintain the simplicity of the graph structure.

[0046] Step 1.4: Multi-modal semantic graph generation.

[0047] The final output multi-modal semantic graph The multi-modal characteristics of the input data are fully preserved by converting to the Protocol Buffers format through a specialized serialization module: node features combine semantic and geometric information, and edge features encode spatial and logical constraint relationships. This representation not only preserves all the key information of the original input, but also converts it into a structured format suitable for computer processing, providing a high-quality input basis for subsequent visual enhancement, relationship reasoning, and scene generation processing stages. The following data structures are used to represent:

[0048] Node set : Each node contains 256-dimensional features and object metadata; Edge set : Stored as a tensor of N × N × 128, supporting multiple relationship types; Graph attributes: Record feature indexes, timestamps, version numbers, and other meta-information of the original input.

[0049] Step 2: The graph neural network enhances the multi-modal semantic graph through a multi-layer attention mechanism. The system updates each node based on the spatial relationship, semantic information, and physical attributes in the graph, allowing the nodes in the graph to obtain rich semantic information from the neighborhood. To ensure the physical rationality of the scene, the system introduces physical constraints such as collision detection, path accessibility analysis, and object stability evaluation through a physics engine, optimizing the node and edge structure in the graph. In this way, the system can real-time correct parts that do not conform to physical constraints and generate a three-dimensional scene that conforms to physical rules.

[0050] Step 2.1: Graph structure feature enhancement.

[0051] After receiving the user's multi-modal input and constructing the initial multi-modal semantic graph, the system introduces a graph neural network structure based on the multi-head attention mechanism to enhance the node features layer by layer. The graph neural network consists of L = 4 layers of graph attention networks, each containing K= 8 independent attention heads. For each attention head, the network first performs a linear transformation on the node features, projecting them into a shared feature space. Then, the strength of the association between node pairs is evaluated by calculating attention coefficients, which are non-linearized using the LeakyReLU activation function. To ensure the rationality of weight allocation, the system performs softmax normalization on the original attention coefficients, so that the sum of the attention weights of each node to its neighbors is 1. In the feature aggregation stage, each attention head performs a weighted summation of the features of its neighboring nodes based on the calculated attention weights. This mechanism allows the network to adaptively focus on neighbor information that is more important to the current node. The outputs of all attention heads are fused in the last layer, typically using a concatenation operation to preserve the different feature patterns learned by each attention head. To alleviate the vanishing gradient problem in deep network training, residual connections are introduced between layers, adding the features of the previous layer to the transformed features of the current layer. Furthermore, the output of each layer undergoes layer normalization to stabilize the training process and accelerate convergence. After four layers of this graph attention network processing, the initial node features are progressively enhanced into deep representations containing rich contextual information. These enhanced features not only encode the node's own attributes but also incorporate global graph structure information, enabling them to more accurately reflect the semantic relationships and spatial constraints between objects in a 3D scene.

[0052] Step 2.2: Physical constraint verification and correction.

[0053] To ensure the generated graph structure conforms to the physical plausibility of the 3D scene, the system introduces an integrated physics engine to perform three types of physical constraint analysis on candidate relationships in the graph: collision detection, layout constraints, and path reachability evaluation. The formulas are as follows:

[0054]

[0055] in, This represents the collision detection score evaluation function. They represent the first 3D bounding box of an object This represents the three-dimensional intersection-to-union ratio between two objects; The function represents the score evaluation function for layout constraints. It is a room floor plan. Represents the total number of 3D bounding boxes of objects. This represents the total number of wall enclosure boxes. Indicates the first A wall enclosure box; The representative path reachability score evaluation function selects paths on the path. indivual Location, corresponding yes 3D bounding box.

[0056] Step 2.3: Dynamic graph structure optimization.

[0057] After the physical verification module completes the initial scoring, the system dynamically corrects and enhances the edge weights and node features of the graph. First, the weights of all edges in the graph are adjusted based on the current physical compliance score. The system is then updated. Subsequently, a lightweight multilayer perceptron is used to correct the semantic features of nodes in physically violated areas. To improve the sampling resolution of physically sensitive areas, the system also introduces an adaptive sampling strategy to increase sampling points in areas with frequent physical conflicts.

[0058] Step 2.4: Multi-scale feature fusion.

[0059] To capture spatial semantic information of different granularities in graph structures, the system constructs a three-level hybrid feature pyramid.

[0060] First, object-level local features are extracted using a standard graph neural network. Second, region-level functional clustering features are achieved through cluster pooling. Third, global scene features are modeled using a self-attention mechanism encoder. Finally, all three are uniformly encoded and standardized through a linear fusion layer.

[0061] Step 2.5: Iterative optimization convergence.

[0062] The system sets convergence conditions during graph augmentation to ensure stable termination of the optimization phase. These conditions mainly include the following three points: first, the physical violation rate is less than 5%; second, the change rate of graph features in two consecutive iterations is [not specified]. Third, the maximum number of iteration rounds. =20 times. The overall optimization goal is:

[0063]

[0064] in, For feature reconstruction loss, For collision damage, For accessibility loss, For static stability loss, This is the sum of the losses from the three physical constraints. This refers to the graph semantic consistency loss.

[0065] Step 3: The system uses the VQ-VAE model to complete the missing visual modalities in the graph, ensuring that the visual features of the graph are fully recovered. Simultaneously, Triplet-GCN (Graph Convolutional Network) is used to complete the missing relationships in the graph structure. Particularly in handling relationships between different modalities, the system enhances the structural representation capability of the graph through triplet relationship learning. Through this module, the system can more completely understand and express the multimodal information in the graph, providing richer feature inputs for subsequent 3D scene generation.

[0066] Step 3.1: Text-driven visual feature reconstruction:

[0067] The system utilizes text embedding vectors from multimodal semantic graph nodes to construct high-quality reconstructed features for missing or incomplete visual modalities using the VQ-VAE model. The process includes three stages: latent encoding, quantization, and decoding.

[0068] The encoder will use text features The mapping is performed as a latent representation, and then, the codebook is selected to match the representation. Recent Each vector is quantized, and the decoder maps the quantized vectors back to the image feature space. To optimize the generation effect, the model maximizes the lower bound of evidence.

[0069] Step 3.2: Multimodal structure expansion and dynamic enhancement graph construction:

[0070] The final visual enhancement image After completion with Triplet-GCN, it is expanded into a hybrid enhancement map. This is used for subsequent layout generation driven by the diffusion model. Its structure contains a set of node features: , Represent a A 3D real vector space, This represents the total number of objects in the graph. Feature dimension; Edge feature set: ,in The edge feature dimension is used. A physical scoring map of the graph was also constructed simultaneously to provide physical guidance signals for the next step of the diffusion model.

[0071] Step 4: The system scores the quality of the generated 3D scene based on semantic information, spatial relationships, and physical constraints in the graph. Specifically, the system calculates the pairing relationship scores between objects, evaluates the rationality of the scene layout, and scores the scene according to physical constraints. Group relationship modeling and object co-occurrence analysis further help the system evaluate the coordination and consistency of various parts of the scene. By comprehensively considering physical, semantic, and spatial factors, the scoring system provides optimization feedback for subsequent generation processes, ensuring that the final output 3D scene meets the expected requirements.

[0072] Step 4.1: Pairwise relationship modeling:

[0073] The system first constructs a pairwise relationship vector in the unit of any object pair in the scene graph, which integrates object semantic features, spatial relative positions, and relationship type labels to measure the direct logical relationship between two objects.

[0074] Step 4.2: Group relationship modeling:

[0075] To further capture the local clustering structure and functional partition semantics of objects, the system adopts a dual clustering strategy based on semantic labels and spatial density to group objects in the scene graph. Each group of objects is encoded as a Gaussian distribution, and the system defines an interaction matrix between groups, where is the total number of groups in the scene, and the element represents the semantic-spatial coupling strength between group and group , which is formulated as follows:

[0076]

[0077] where represents the mean vector of cluster , represents the mean of all points belonging to cluster , , represents the covariance matrix of cluster , represents the covariance of all points belonging to cluster , , represents the similarity between cluster and cluster , represents the Frobenius norm of the covariance matrix difference between cluster and cluster , which measures the difference between the two covariance matrices. is the pre-defined semantic template direction vector.

[0078] Step 4.3: Object co-occurrence modeling:

[0079] The system constructs an object co-occurrence statistical map to determine whether the combination pattern of the current input graph conforms to semantic common sense. For any object pair , the system uses the co-occurrence matrix and the standard co-occurrence matrix Kullback-Leibler divergence matching is performed, and its co-occurrence probability formula is defined as follows:

[0080]

[0081] wherein represents the number of scenes in which and co-occur in the knowledge base, is the total number of reference scenes, represents Kullback-Leibler divergence ( KL divergence) used to measure the difference between two distributions and .

[0082] Step 4.4: Structural Similarity Assessment:

[0083] The system compares and evaluates the global consistency between the input graph structure and the reference graph structure through graph embedding comparison, and adopts graph neural network to encode the graph structure and then calculate the embedding distance, while considering the co-occurrence logical deviation to form a joint scoring function formula as follows:

[0084]

[0085] wherein represents the structural difference measure, which measures the difference between scene features and reference features , represents divergence, which quantifies the logical deviation of the input graph and the reference graph in node or edge distribution, represents the relationship-related score, and are weight factors, and the empirical value is 0.5.

[0086] Step 4.5: Multi-dimensional Score Calculation and Fusion:

[0087] According to the above model, the system constructs three main score items for comprehensive evaluation of scene rationality:

[0088] Object relationship score reflects the semantic consistency of paired relationships; layout topology score is used to evaluate whether the overall spatial distribution is balanced and coherent; and basic relationship score comprehensively evaluates whether there are uncommon combinations and connection omissions by combining object co-occurrence difference and edge connectivity rate. The final scene comprehensive score fuses the three indicators to form the final score formula as follows:

[0089]

[0090] where, denotes the number of samples, denote the relationship vectors between objects in the scene and reference, and . denotes the similarity score of the layout, is a tuning factor that controls the influence of the bias and density on the score, denotes the average positional bias between all nodes and the reference graph; denotes the local density standard deviation of the nodes are weight coefficients, with default values of 0.4, 0.3, and 0.3, respectively.

[0091] The Top-K scene graphs sorted by the comprehensive score are used as input for the dual-branch diffusion generation model in Step 5.

[0092] Step 5: The system generates the spatial layout and geometric shapes in the three-dimensional scene through a dual-branch diffusion model. The dual-branch diffusion model includes a layout branch and a shape branch. The layout branch generates the positions, orientations, and scales of each object in the scene through a Transformer diffusion network, ensuring a reasonable spatial layout of the scene. The shape branch generates the geometric shapes of the objects through a conditional variational autoencoder, ensuring consistency with the semantic categories. During the diffusion process, the system ensures information coordination between the two branches through a cross-branch consistency mechanism, avoiding mismatch between the layout and shape. Finally, the system outputs a complete three-dimensional scene graph, including the spatial layout, geometric shapes, and physical properties of all objects, for subsequent rendering or simulation systems.

[0093] Step 5.1: Initial diffusion modeling:

[0094] The system first initializes a geometric feature representation tensor for each node of the multi-modal semantic graph. In each branch, the training target is to learn an inverse denoising model .

[0095] Step 5.2: Layout branch diffusion generation:

[0096] The layout branch is responsible for generating spatial positions , orientations , and scales parameters for each scene object. The model structure uses a Transformer diffusion network that integrates time step embeddings and local graph attention information. The loss function includes positioning errors and spatial consistency regularizations, as follows:

[0097]

[0098] where denotes the loss function of the layout, pred and gt are the predicted and ground truth positions of the i-th sample, pred and gt are the predicted and ground truth positions of the i-th sample, pred and gt are the predicted and ground truth rotations of the i-th sample, pred and gt are the predicted and ground truth rotations of the i-th sample, pred and gt are the predicted and ground truth scales of the i-th sample, pred and gt are the predicted and ground truth scales of the i-th sample, represents the prior loss, lambda is the weight of the regularization term.

[0099] Step 5.3: Shape branch diffusion generation:

[0100] The shape branch focuses on modeling the local geometric properties and class semantics of the object. Each node corresponds to a shape latent variable , which represents the generation parameters of its 3D model.

[0101] The diffusion sampling target is to accurately reconstruct the object geometry and be consistent with the semantic category, and its formula is as follows:

[0102]

[0103] where, represents the loss function of the shape, pred and gt are the predicted and ground truth values of the i-th sample, pred and gt are the predicted and ground truth values of the i-th sample, represents the object category predicted by the model, represents the true object category label, and CE represents the cross-entropy loss. The network structure uses a conditional variational autoencoder combined with diffusion residual blocks for modeling.

[0104] Step 5.4: Cross-branch consistency modeling:

[0105] To maintain the collaborative relationship between the layout and the shape, the system exchanges information through the cross-branch attention module in each round of diffusion. The layout features are used to modulate the shape branch generation, while a consistency loss is defined to constrain the alignment of shape and layout information, keeping the predicted shape orientation consistent with the object pose.

[0106] Step 5.5: Scene fusion and output reconstruction:

[0107] Finally, the system combines the output features of the two branches to generate a complete structured three-dimensional scene graph, and its formula is as follows:

[0108]

[0109] where: is the set of nodes (object instances), is the set of interaction edges (guided by physical constraints or common sense graphs), a position set, an orientation set, a scale set, a shape latent variable set, an object semantic class.

[0110] The synthesized scene result will be physically corrected (such as collision detection, scale correction, etc.) by a post-processing module and output as a three-dimensional scene file that can be used by a rendering or simulation system.

[0111] Embodiment:

[0112] A multi-modal semantic graph-based double-branch diffusion three-dimensional scene generation method, comprising the following steps:

[0113] Step 1: Perform feature extraction and fusion on input data of multiple modalities to generate a unified multi-modal semantic graph.

[0114] The input includes sketch-based scene combination editing instructions, such as this is a sketch of an indoor bedroom. The central part of the picture is a large bed, and on both sides of the bed there is a bedside table and a table lamp. A long chair or low cabinet is placed against the wall on the right, and a set of sofa and tea table is placed on the left, and a TV cabinet or desk is placed near the foreground; DeepSeek-based efficient scene combination editing instruction generation, such as scheme: the living room adopts a symmetrical design, a double sofa is placed in the center, and a single chair is placed on both sides, and a long tea table is placed in the middle to form a reception area. The bedroom and living room are separated by a semi-transparent screen or bookshelf, the bed is placed near the window to make use of natural light, and a small wardrobe can be placed at the end of the bed to ensure storage function; user requirements, such as I want 6 schemes, the space area is 50 square meters, the sofa is located in the living room part, the sofa is a right-angle sofa around the whole living room, the color is gray, the style of the sofa is a certain home type, the bed is placed in the room opposite the living room, the color of the bed is blue, and the style of the bed is a fabric bed; scene common sense knowledge base, such as the main traffic path (such as the dynamic line from the door to the balcony) needs to maintain at least 80 cm clear width, the sofa, cabinet, etc. Furniture should not cut off the channel horizontally to ensure barrier-free access for people. The sleeping area (bed) and the pollution area (kitchen stove) need to be physically isolated, with a straight-line distance of not less than 2 meters, to avoid interference from oil smoke and comply with residential health standards.

[0115] As shown in Figure 2 , the implementation of the scene common sense knowledge base import is as follows:

[0116] First, a multimodal knowledge base is imported from an external source, containing both structured and unstructured data. This data covers common-sense knowledge of scenarios, physical constraints, and descriptions of spatial relationships. Next, a scene graph and a knowledge graph are constructed based on this data. The scene graph abstractly represents various objects in the physical space and their relative positions, while the knowledge graph uses nodes and edges to represent objects and their semantic relationships, providing a reasoning basis for spatial modeling. Subsequently, the system integrates the scene graph and the knowledge graph to form a multi-layered, cross-domain knowledge-scene graph network, and performs verification and reasoning by embedding common-sense knowledge and rule constraints (such as building design codes and hygiene standards). For example, the system automatically checks whether the width of the path from the door to the balcony meets the minimum net width requirement of 80 centimeters. Next, each scene element is further refined by combining it with a scene object library. The scene object library provides more detailed object descriptions, including their functional attributes, size specifications, and other information, enhancing the accuracy of the system in scene analysis. Then, the scene graph and knowledge graph data, after reasoning and verification, are transmitted to downstream application systems, such as smart home and automation design tools, ensuring the accurate flow of knowledge and information and providing efficient support for intelligent decision-making systems. Finally, the system feeds back the optimized scene design scheme to the user, ensuring that all spatial constraints and common-sense knowledge are met, thereby achieving intelligent spatial design and optimization.

[0117] The semantic capabilities of scene understanding are enhanced by fusing entities and relationships from the knowledge graph with a command-generated multimodal semantic graph. The knowledge graph contains a wealth of entities, relationships, and attributes based on common sense about the world (e.g., the relationship between "table" and "chair"). This knowledge is crucial for understanding objects in a 3D scene and their interrelationships. In this step, the model maps each entity and relationship in the common sense knowledge graph to the same embedding space as the features of the multimodal semantic graph. In this way, the common sense knowledge graph and the point cloud feature space are aligned in the same embedding space, enabling the model to combine these two types of information for reasoning. Given the initial parsing results of the current scene, the system retrieves a set of relevant spatial layout rules. Then through the rule encoder Convert it into numerical features :

[0118]

[0119] in, It includes professional standards such as "main passageways must maintain a net width of at least 80 centimeters" and "sleeping areas and contaminated areas must maintain a distance of more than 2 meters".

[0120] Step 2: Use graph neural networks to enhance the features of the multimodal semantic graph, and at the same time optimize the physical constraints of the scene through a physics engine to ensure physical rationality.

[0121] whereas the implementation of the physical constraints module is as follows: Figure 3

[0122] First, the system identifies and defines the main elements in the scene, such as rooms, furniture, doors, windows, etc., and determines their basic positions and roles in the space. Then, collision constraints are applied, and the system checks whether there is any overlap or collision between objects in the space. By calculating the minimum distance between objects, unnecessary overlaps are avoided, ensuring that each object has enough space. Next, the system applies room layout constraints, which ensure that the placement of objects in the scene meets physical specifications, such as the distance between furniture and walls or the width of passageways. Subsequently, the system further introduces accessibility constraints to ensure smooth connections between different areas, such as ensuring that the path from the room entrance to the balcony is not blocked by any objects. The shortest distance of the path can be calculated using the Euclidean distance formula to ensure the smoothness of the path. At this point, the U-Net model is introduced to further process and optimize the physical constraints, adjusting the positions of objects to better meet various constraint requirements through deep learning model optimization of space layout. Finally, through repeated optimization, the system generates a space layout design that meets all physical constraints, ensuring that the final design not only meets functional requirements but also satisfies the physical rules and logical constraints of space layout.

[0123] wherein,

[0124]

[0125] wherein, represents the collision detection score evaluation function, represent the 3D bounding box of the th object, represents the three-dimensional intersection ratio between two objects; represents the layout constraint score evaluation function, is the room plan, represents the total number of object 3D bounding boxes, represents the total number of wall bounding boxes, represents the th wall bounding box; represents the path accessibility score evaluation function, and positions are selected on the path, corresponding to is the 3D bounding box of .

[0126] ​​​​Step 3: Complete the missing information in the graph and enhance the expression ability of the graph structure through the multi-modal perception module to make the scene more complete.

[0127] Step 4: Evaluate the semantic consistency, spatial layout rationality and physical feasibility of the generated scene through the scoring system to provide feedback for optimization.

[0128] As shown in Figure 4 , the scoring system for multi-modal semantic graph scoring is implemented as follows:

[0129] First, a pair relationship model is used for preliminary analysis, which is used to evaluate the pairing relationship between elements in the scene (such as objects, furniture, etc.), and analyze whether they meet the preset layout rules. Then, the system applies a group relationship model to evaluate the mutual relationship between multiple objects to determine whether they are reasonably clustered together. For example, the sofa and coffee table in the living room should be placed together to form an effective functional area. Subsequently, the object co-occurrence model is applied to analyze the frequency of different objects appearing in space and their co-occurrence patterns. This helps the system determine which elements often appear together in the same space, further optimizing the layout. Next, the relationship model similarity module calculates the similarity between objects to ensure that similar objects are arranged in a suitable manner, avoiding incompatible or discordant elements from appearing together. Then, the system scores the scene layout by combining the results of multiple modules for comprehensive analysis. The basic relationship analysis and object relationship scoring module will score the layout of each object in the scene in detail, evaluating whether its position and function in space meet the design requirements.

[0130] Step 5: Generate the spatial layout and object shape of the three-dimensional scene through the layout branch and shape branch respectively, and finally output the complete three-dimensional scene graph.

[0131] As shown in Figure 5 , the implementation of the shape branch module is as follows:

[0132] First, the normalization process involves converting input data to a uniform scale or range for the accuracy and stability of subsequent processing. Next, the cross-attention module is applied, which is commonly used in neural networks to strengthen the association between different features, helping the model better capture important information and patterns. Then, the downsampling process mentioned in the figure is usually applied in convolutional neural networks, which reduces the resolution of images or spatial data step by step to reduce computational complexity while preserving important features. Next, the calculation of TSDF is involved, which is a way to represent object geometry commonly used in 3D reconstruction, which can effectively handle the estimation of object surface distance. After that, the latent code and 3D-UNet are used for further feature learning and extraction of geometric information, respectively. 3D-UNet is a neural network architecture specifically designed for three-dimensional data processing, commonly used in tasks such as medical image segmentation. Finally, after processing through these modules, the system can generate accurate object geometry models and accurately model and segment objects in 3D space. Its diffusion sampling goal is to accurately reconstruct the object geometry and be consistent with the semantic category, and its formula is as follows:

[0133]

[0134] where, represents the shape loss function, represents the predicted value and the true value of the th sample, respectively, represents the object category predicted by the model, represents the true object category label, and CE represents the cross-entropy loss. The network structure uses a conditional variational autoencoder combined with diffusion residual blocks for modeling.

[0135] where, as shown in Figure 5 , the implementation of the layout branch module is as follows:

[0136] First, the process begins with a convolution operation using a 1x5 kernel, which extracts features from the input data through batch normalization and ReLU activation. This operation helps the model extract effective spatial features from the raw data. Next, max pooling 1x2 and down-sampling 1x2 are used to further reduce the spatial resolution, which helps reduce computational load while preserving important high-level features in the image. The skip connection plays an important role in this process, allowing low-level features to be directly passed to the high-level network to preserve more detailed information and improve network performance. The process also includes a one-dimensional convolution 1x1 operation, which is typically used to further refine features and generate output probability distributions representing the model's prediction for each location. The generation of noisy bounding boxes is a key step, where the model generates the bounding boxes of objects and adds noise information to improve the robustness and accuracy of the model. Finally, the 1D-UNet architecture is used to process information such as the location, rotation, and size of objects. The UNet architecture is typically used for image segmentation tasks, but here it is extended for one-dimensional data processing to generate accurate object location and size predictions. The loss function includes the localization error and spatial consistency regularization formula as follows:

[0137]

[0138] where represents the loss function of the layout, represents the predicted location and the true location of the th sample, represents the predicted rotation and the true rotation of the th sample, represents the predicted scaling and the true scaling of the th sample, represents the prior loss, lambda is the regularization term weight.

[0139] It is understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, modifications can be made to these features and embodiments to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A multi-modal semantic graph based dual-branch diffusion three-dimensional scene generation method, characterized in that, The method comprises the following steps: Step 1: first receive data from multi-modal data sources, and extract and process features of input multi-modal data through different processing modules; Step 2: the graph neural network enhances the features of the multi-modal semantic graph through a multi-layer attention mechanism; Step 3: complete the missing visual modal in the graph; meanwhile, complete the missing relationship in the graph structure, and enhance the structural representation capability of the graph through triple relationship learning in the relationship processing between different modalities; Step 4: based on the semantic information, spatial relationship and physical constraint in the graph, score the generated three-dimensional scene; Step 5: generate the spatial layout and geometric shape in the three-dimensional scene through a double-branch diffusion model.

2. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 1 first receives sketch instructions, text instructions, automatic completion instructions and scene common sense knowledge data, the sketch instructions are subjected to geometric information extraction and spatial relationship modeling through a graph neural network; the text instructions are converted into semantic features through a text encoder in a CLIP model to provide context information about the scene; the automatic completion instructions and the scene common sense knowledge are subjected to supplementary information extraction through a knowledge base and a semantic retrieval module, and finally, all modal information is unified into a multi-modal semantic graph through feature fusion and standardization processing to provide consistent feature representation for subsequent three-dimensional scene generation.

3. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 1 comprises the following specific steps: Step 1.1: Sketch-based scene composition editing instructions Processed by a graph neural network-based parser; natural language instructions Processed by a CLIP text encoder based on the Transformer architecture; DeepSeek autocomplete instructions Processed by a dedicated scene description encoder; scene common sense knowledge base Query and extraction through an Elasticsearch-based semantic retrieval system; Step 1.2: node feature fusion and standardization: constructing a unified feature representation for each object node; Step 1.3: edge relationship construction and completion: constructing an initial relationship network between objects based on input data; Step 1.4: multi-modal semantic graph generation: the finally output multi-modal semantic graph is converted into a Protocol Buffers format through a special serialization module.

4. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 2 updates each node according to the spatial relationship, semantic information and physical attribute in the graph, so that the node in the graph can obtain rich semantic information from the neighborhood; the physical constraint is introduced through a physical engine to optimize the node and edge structure in the graph.

5. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 3 comprises the following specific steps: Step 3.1: text-driven visual feature reconstruction: using the text embedding vector in the multi-modal semantic graph node, constructing high-quality reconstructed features of missing or incomplete visual modalities through a VQ-VAE model; Step 3.2: Multi-modal structure extension and dynamic enhancement map construction: final visual enhancement map , after Triplet-GCN completion, extended to hybrid enhancement map , for subsequent diffusion model-driven layout generation.

6. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The step 4 calculates the pairing relationship score between objects to evaluate the rationality of the scene layout, and scores the scene according to the physical constraint; group relationship modeling and object co-occurrence analysis further help the system to evaluate the coordination and consistency of each part in the scene; the scoring system comprehensively considers the physical, semantic and spatial factors to provide optimization feedback for the subsequent generation process, and ensures that the finally output three-dimensional scene meets the expected requirements.

7. The multi-modal semantic graph based dual-branch diffusion 3D scene generation method according to claim 1, characterized in that, The double-branch diffusion model of step 5 includes a layout branch and a shape branch; the layout branch generates the positions, orientations and scales of various objects in the scene through a Transformer diffusion network; the shape branch generates the geometric shapes of the objects through a conditional variational autoencoder and keeps consistent with the semantic categories; during the diffusion process, a cross-branch consistency mechanism is used to ensure the coordination of information between the layout branch and the shape branch; finally, the system outputs a complete three-dimensional scene graph including the spatial layout, geometric shape and physical properties of all objects for subsequent rendering or simulation systems.

Images