Natural language driven cross-modal three-dimensional scene generation method and system

Abstract

The invention relates to a natural language driven cross-modal three-dimensional scene generation method and system, and the method comprises the steps: receiving scene description information inputted by a user in a natural language, and carrying out the semantic analysis, so as to extract an object element, an attribute element and a spatial relation element which are used for scene construction; generating a two-dimensional layout prior image of the scene according to a semantic analysis result; constructing a semantic structure intermediate expression based on the two-dimensional layout prior image; generating corresponding three-dimensional representation and initial space layout according to the two-dimensional layout prior image, and determining a local scene; integrating the semantic structure intermediate expression corresponding to the local scene into the semantic structure intermediate expression of the global scene, selecting an object as a semantic anchor point in the global scene, and guiding the local scene to be fused into the global scene; and receiving incremental editing information of the user, and executing the scene updating process until scene generation is completed. Compared with the prior art, the intelligent and automatic level of three-dimensional scene modeling is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of 3D scene generation technology, and in particular to a natural language-driven cross-modal 3D scene generation method and system. Background Technology

[0002] 3D scene generation technology has wide applications in virtual reality, augmented reality, game development, digital twins, and architectural visualization. With technological advancements, an increasing number of 3D scene generation methods are adopting generative artificial intelligence models, especially automated generation methods based on text, images, and semantic data. However, most existing 3D scene generation methods rely on stitching together existing 3D scene libraries or generating content based on simple templates. While these methods can save modeling time, the lack of effective spatial structure control and semantic consistency maintenance often results in problems with object hierarchy, spatial layout, and physical consistency in the generated 3D scenes.

[0003] Furthermore, existing methods have weak support for multi-round incremental editing, making it difficult to flexibly adjust and update scenes dynamically. More importantly, most of these methods are based on single-modal generation, such as relying solely on images or text, lacking multimodal interaction and generation capabilities, which greatly limits the flexibility and controllability of scene generation. For example, the invention disclosed in CN120353371A generates 3D digital content from natural language requests; through a 3D modeling system based on natural language requests, it uses natural language processing technology to analyze the phrases input by the user, generates entity-command representations, and maps them to the semantic scene graphics of existing 3D scenes to quickly generate or modify 3D scenes. However, this solution does not consider the semantic structural correspondence between the phrases input by the user and the existing 3D scenes, and cannot guarantee semantic consistency and rationality.

[0004] Therefore, how to achieve automated generation of 3D scenes with reasonable structure, semantic consistency, and support for multi-round incremental editing by using cross-modal generation technology and combining natural language, images, and semantic structures has become an important issue in current technology. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology, such as reliance on manual modeling in the 3D scene generation process, low expression efficiency, lack of semantic controllability, and difficulty in multi-round interactive editing, and to provide a natural language-driven cross-modal 3D scene generation method and system.

[0006] The objective of this invention can be achieved through the following technical solutions: A natural language-driven cross-modal 3D scene generation method includes: S1: Receive scene description information input by the user in natural language; S2: Perform semantic parsing on the scene description information to extract object elements, attribute elements, and spatial relationship elements for scene construction; S3: Generate a two-dimensional layout prior image of the scene based on the semantic parsing results of the scene description information; S4: Construct an intermediate semantic structure representation based on the two-dimensional layout prior image to express the hierarchical relationship and spatial dependency between objects; generate a corresponding three-dimensional representation and initial spatial layout based on the two-dimensional layout prior image; optimize the initial spatial layout to determine the local scene; S5: Integrate the intermediate semantic structure representation of the local scene into the pre-built intermediate semantic structure representation of the global scene, and select objects as semantic anchors in the global scene to guide the local scene to merge into the global scene; S6: Receive incremental editing information from the user, obtain incremental scene description information, and execute steps S2-S5 until the user completes the generation of the 3D scene.

[0007] Furthermore, the scene description information includes scene targets, core objects, positional relationships, and related constraints.

[0008] Furthermore, the process of generating the two-dimensional layout prior image includes: The appearance of the item is generated based on the object elements and attribute elements, the corresponding spatial layout is generated based on the spatial relationship elements, and the two-dimensional layout prior image is constructed based on the isometric view constraint.

[0009] Furthermore, the semantic structure is expressed as a graph structure or a hierarchical structure, which is used to dynamically represent the subordinate relationships and spatial semantic connections of objects.

[0010] Furthermore, the construction process of the intermediate semantic structure representation includes: The two-dimensional layout prior image is subjected to object detection and segmentation using an object detection algorithm to obtain the bounding box position, category label, and segmentation mask based on the object boundary for each object in the two-dimensional layout prior image. The segmentation mask is combined with category labels to generate a scene segmentation map with semantic information, so as to show the spatial location of each object and the category information corresponding to different segmentation regions; The objects in the two-dimensional layout prior image are matched in pairs to generate multiple object matching pairs. A bounding box is drawn for each object in each object matching pair. The bounding box is input into a preset visual model. Spatial relationship inference is performed based on preset relationship type prompts. The relationship prediction result of each object matching pair is obtained, and the spatial relationship set between all objects is obtained. Combined with the appearance information and attribute elements of the objects, it is transformed into a structured semantic structure intermediate expression.

[0011] Furthermore, the construction process of the intermediate semantic structure representation also includes: The pose prediction model is used to predict the position of each object in the segmentation mask of the two-dimensional layout prior image, and the pose attribute set of each object is obtained. The pose attribute set includes spatial position, orientation and size, which is used in the construction process of the semantic structure intermediate expression.

[0012] Furthermore, before generating the corresponding three-dimensional representation based on the two-dimensional layout prior image, the process also includes: For each object in the two-dimensional layout prior image, obtain the segmentation map of the object in the two-dimensional layout prior image, generate the complete outline and full shape mask of the object, and complete the two-dimensional layout prior image.

[0013] Furthermore, when integrating the intermediate semantic structure representation corresponding to the local scene into the pre-built intermediate semantic structure representation of the global scene, a subordinate relationship is established between the local scene and the semantic anchor point through the hierarchical structure and semantic constraints of the intermediate semantic structure representation of the global scene.

[0014] Furthermore, in step S6, during the local update of the global scene, a global scene semantic constraint is introduced to generate a two-dimensional layout prior image. This global scene semantic constraint is generated based on the intermediate expression of the semantic structure of the global scene, including spatial type, key object category, functional area label and style keywords.

[0015] The present invention also provides a cross-modal 3D scene generation system that implements the natural language-driven cross-modal 3D scene generation method described above, comprising: The language parsing module is used to receive scene description information input by the user in natural language; and to perform semantic parsing on the scene description information to extract object elements, attribute elements and spatial relationship elements for scene construction. The 2D prior generation module is used to generate a 2D layout prior image of the scene based on the semantic parsing results of the scene description information. The semantic structure construction module is used to construct an intermediate semantic structure representation based on the two-dimensional layout prior image to express the hierarchical relationship and spatial dependency between objects; A 3D content generation module is used to generate a corresponding 3D representation and an initial spatial layout based on the 2D layout prior image; The spatial layout optimization module is used to optimize the initial spatial layout and determine the local scene; The scene fusion module is used to integrate the intermediate semantic structure expression corresponding to the local scene into the intermediate semantic structure expression of the pre-built global scene, and select objects in the global scene as semantic anchors to guide the fusion of the local scene into the global scene; The interactive extension module is used to receive incremental editing information from the user, obtain incremental scene description information, and repeatedly execute the language parsing module, the two-dimensional prior generation module, the semantic structure construction module, the three-dimensional content generation module, the spatial layout optimization module, and the scene fusion module until the user completes the generation of the three-dimensional scene.

[0016] Compared with the prior art, the present invention has the following advantages: (1) The natural language-driven cross-modal 3D scene generation method provided by this invention adopts a cross-modal generation and progressive optimization mechanism. Based on the scene description information input by the user in natural language, semantic parsing and intermediate semantic structure expression are generated to realize the 3D generation of local scenes and integrate them into the 3D scene. It also supports receiving incremental editing information from the user to realize incremental generation of 3D scenes, thus comprehensively improving the intelligence level and flexibility of 3D scene generation. Overall, based on the natural language description input by the user, the automatic generation from 2D image prior to 3D scene is realized through cross-modal generation, and dynamic and interactive incremental scene adjustment is realized through progressive optimization mechanism.

[0017] (2) By introducing a semantic structure intermediate representation, the present invention accurately integrates the local semantic structure intermediate expression generated in the current step into the previously constructed global semantic structure intermediate expression, and establishes a clear subordinate relationship between local objects and semantic anchors through the hierarchical structure and semantic constraints of the global semantic structure intermediate expression, thereby forming a consistent global three-dimensional scene. This not only ensures the semantic controllability and structural consistency of the generated content, but also enables object-based local updates and multi-round interactive generation, significantly improving the efficiency and intelligence level of three-dimensional modeling.

[0018] Unlike traditional technical solutions that rely on single-time overall generation or simple template splicing, this invention has significant advantages in terms of spatial structure expression, semantic coordination, and dynamic adaptability. It can flexibly handle the generation and optimization of complex scenes, providing users with a more natural, accurate, and practical 3D modeling experience. Attached Figure Description

[0019] Figure 1This is a flowchart illustrating a natural language-driven cross-modal 3D scene generation method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the processing flow of a natural language-driven cross-modal 3D scene generation system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a language parsing module structure provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] Example 1 like Figure 1 As shown, this embodiment provides a natural language-driven cross-modal 3D scene generation method, including: S1: Receive scene description information input by the user in natural language. The scene description information includes scene objectives, core objects, positional relationships, and related constraints. Language information can be input through text boxes, voice recognition interfaces, or other human-computer interaction methods.

[0024] The natural language content input by users includes, but is not limited to, scene objectives (such as "living room" and "office area"), core objects (such as "sofa" and "monitor"), positional relationships (such as "left side" and "near the window"), and other constraints (such as "modern style" and "large size").

[0025] S2: Perform semantic parsing on the scene description information to extract object elements, attribute elements, and spatial relationship elements for scene construction; Semantic parsing includes extracting scene targets, object categories, scene styles, attribute descriptions, and spatial relationships based on language models.

[0026] The parsing task involves extracting key object, attribute, and spatial relationship information from user input. Specifically, the user's text description (e.g., "I want to generate a modern-style living room") will be interpreted by the system as structured information containing several semantic elements. The system will identify object categories (e.g., "living room," "sofa"), object attributes (e.g., "modern style"), and spatial relationships (e.g., "the sofa is to the left of the TV cabinet") in the scene.

[0027] Semantic parsing relies on a pre-trained language model and employs a two-stage generation strategy, such as... Figure 2 As shown, first, the system analyzes the input scene description to generate a candidate list of objects, which will become the basic building blocks of the subsequently generated image and 3D scene. Then, based on the candidate list, the system constructs cue words for image generation, which will provide the contextual information needed by the image generation model.

[0028] S3: Based on the semantic parsing results of the scene description information, generate a two-dimensional layout prior image of the scene; the generation process of the two-dimensional layout prior image includes: The appearance of the item is generated based on object elements and attribute elements, the corresponding spatial layout is generated based on spatial relationship elements, and a two-dimensional layout prior image is constructed based on isometric view constraints.

[0029] Specifically, based on the object candidate list and semantic description obtained from semantic parsing, a scene image matching the user's description is generated. During image generation, not only the appearance of individual objects needs to be considered, but also a reasonable spatial layout must be generated based on the spatial relationships between objects. In the initial scene generation, the system uses isometric perspective constraints to construct a two-dimensional image to ensure that the generated image clearly displays the spatial layout of the entire scene and the relative positions of objects. For subsequent rounds of scene expansion, the generated prompts focus more on detail supplementation and semantic enrichment, employing a perspective view for image generation.

[0030] S4: Construct an intermediate semantic structure representation based on the 2D layout prior image to express the hierarchical relationship and spatial dependency between objects; generate the corresponding 3D representation and initial spatial layout based on the 2D layout prior image; optimize the initial spatial layout and determine the local scene; The semantic structure is expressed as a graph structure or hierarchical structure, which is used to dynamically represent the subordinate relationships and spatial semantic connections of objects.

[0031] The process of constructing intermediate semantic structures includes: Object detection and segmentation are performed on the two-dimensional layout prior image using an object detection algorithm to obtain the bounding box position, category label, and segmentation mask based on the object boundary for each object in the two-dimensional layout prior image. By combining segmentation masks with category labels, a scene segmentation map with semantic information is generated to show the spatial location of each object and the category information corresponding to different segmentation regions; The objects in the two-dimensional layout prior image are matched pairwise to generate multiple object matching pairs. A bounding box is drawn for each object in each object matching pair. The bounding box is input into a preset visual model. Spatial relationship inference is performed based on preset relationship type prompts. The relationship prediction result of each object matching pair is obtained, and the set of spatial relationships between all objects is obtained. Combined with the appearance information and attribute elements of the objects, it is transformed into a structured semantic structure intermediate expression.

[0032] Specifically, this step is responsible for accurately identifying the spatial positions and semantic relationships between objects in the scene and transforming this information into an operable 3D representation. First, it performs in-depth scene analysis based on the input 2D scene image, and generates intermediate semantic structure representations to express the hierarchical relationships and spatial dependencies between objects. The semantic structure modeling module receives the 2D scene image generated by the previous module and outputs the 3D structural representation of each object in the scene, as well as the spatial relationships between objects.

[0033] Before this step, the input 2D scene image is preprocessed. Specifically, the system uses an object detection algorithm to detect and segment objects in the input image, and combines this with the object candidate list generated in the previous step as a category prior. This process ensures that every object in the image is identified. After this preprocessing step, the system obtains the bounding box position, corresponding category label, and segmentation mask based on the object boundary for each object in the image.

[0034] Subsequently, the scene parsing process utilizes a visual model to perform in-depth analysis of the preprocessed image. The input to this scene parsing process includes a 2D scene image, category labels, and preprocessed object-level segmentation data. The system combines the object segmentation mask with the category labels to generate a scene segmentation map with semantic information. This map not only shows the spatial location of each object but also indicates the category information corresponding to different segmentation regions. Through this generated segmentation map, the system can clearly understand the objects in the scene and their spatial distribution.

[0035] During scene understanding, to identify the spatial relationships between objects, the system performs pairwise matching of objects in the scene, generating multiple object matching pairs. To avoid interference between different matching pairs, the system performs relationship prediction for each pair separately. On the semantic segmentation map, a bounding box is drawn for each pair of objects and provided as input to the visual model. The system then infers the spatial relationship based on preset relationship type prompts. The visual model outputs the spatial relationship prediction result between the matching pairs. Finally, the relationship prediction results of all matching pairs are aggregated to generate a set of spatial relationships between objects.

[0036] In addition to the spatial relationships between objects, scene understanding also requires the system to perceive the spatial position, orientation, size, and other geometric attributes of each object. To this end, this invention utilizes a pose prediction model to process the segmentation mask of each object individually, thereby obtaining a set of pose attributes for each object. These pose attributes describe the object's spatial orientation, size, and relative position.

[0037] Through the above steps, this step transforms the visual information, spatial relationship information, and geometric attribute information of objects into a structured semantic intermediate expression, forming the hierarchical relationship and spatial dependency between objects in the scene.

[0038] This step is responsible for converting the segmentation information extracted from the 2D image of each object generated in the previous step into a 3D structural representation. For each object detected in the 2D image, its segmentation map in the scene image is first used as input. Due to the potential occlusion problem of objects in the image, the obtained object segmentation image is usually incomplete. This means that relying solely on this image for 3D reconstruction may lead to geometric defects in the generated 3D model, resulting in inconsistent geometric structure throughout the scene. To address this issue, this invention employs an improved method that uses an image completion model to explicitly complete the 2D segmentation image of each object during the object modeling stage. Specifically, based on the scene image generated in the current step, for each target object, its segmentation map in the image is first obtained. Then, the image completion model is used to generate the complete outline and full-shape mask of the object, thereby solving the problem of incomplete shape caused by occlusion. The completed object segmentation map is then used as input to the image-to-3D generation model, ultimately generating a 3D representation of the corresponding object, representing its spatial coordinates and physical properties.

[0039] The spatial layout optimization process is responsible for optimizing the initial placement of objects, allowing their orientation to be adaptively adjusted. This ensures that the generated local scene layout conforms to the consistency of the spatial structure and closely resembles the actual physical environment. This module infers the spatial position and orientation of each object based on the intermediate semantic structure representation provided by the aforementioned semantic parsing and structural modeling modules. During the initial scene generation phase, the spatial layout optimization module corrects the local scene layout using an orthogonalization algorithm, ensuring that the structural objects in the scene are orthogonal in their primary orientation.

[0040] S5: Integrate the intermediate semantic structure representation of the local scene into the pre-built intermediate semantic structure representation of the global scene, and select objects as semantic anchors in the global scene to guide the local scene to merge into the global scene; When integrating the intermediate semantic structure representation of a local scene into the pre-built intermediate semantic structure representation of a global scene, a subordinate relationship is established between the local scene and the semantic anchor point through the hierarchical structure and semantic constraints of the intermediate semantic structure representation of the global scene.

[0041] This step achieves efficient fusion and adjustment of the 3D scene by constructing and merging intermediate semantic structures of local and global scenes. To obtain a complete 3D scene, this step precisely integrates the intermediate local semantic structure generated in the current step into the previously constructed intermediate global semantic structure, thus forming a consistent global 3D scene. In each round of local scene generation, the system uses an object in the global scene as a semantic anchor point to guide the generation of the local scene. For example, in the initial generation stage, the system assumes the ground as a semantic anchor point and places objects based on it. This mechanism ensures the consistency of the assumption that "any scene generation is based on the extension of semantic anchor points." Through the hierarchical structure and semantic constraints of the intermediate global semantic structure, a clear subordinate relationship is established between local objects and semantic anchor points.

[0042] S6: Receive incremental editing information from the user, obtain incremental scene description information, and execute steps S2-S5 until the user completes the generation of the 3D scene.

[0043] During the local update of the global scene, a global scene semantic constraint is also introduced to generate a two-dimensional layout prior image. This global scene semantic constraint is generated based on the intermediate expression of the semantic structure of the global scene, including spatial type, key object category, functional area label and style keywords.

[0044] During the incremental user interaction in step S6, users can select an object in an existing scene as an anchor point, such as "put a mug on the coffee table". The system can automatically identify the semantic node corresponding to "coffee table" and its spatial location, repeat the steps of the cross-modal 3D scene automatic generation system, and generate and update the spatial layout of the local area.

[0045] Preferably, in the multi-round scene generation process, to ensure the consistency between locally generated content and the global scene, this invention introduces a global scene semantic constraint. Whenever the user provides a new scene description, the system not only uses the semantic information of the current round to generate image prompts, but also introduces the global semantic state of the current scene as auxiliary input. This global semantic constraint is automatically generated from the intermediate expression of the semantic structure of the current scene, including space type, key object categories, functional area labels, and style keywords.

[0046] Example 2 This embodiment provides a cross-modal 3D scene generation system that implements a natural language-driven cross-modal 3D scene generation method as described in Embodiment 1, comprising: The language parsing module is used to receive scene description information input by the user in natural language; and to perform semantic parsing on the scene description information to extract object elements, attribute elements and spatial relationship elements for scene construction. The 2D prior generation module is used to generate a 2D layout prior image of the scene based on the semantic parsing results of the scene description information. The semantic structure construction module is used to construct intermediate semantic structure representations based on two-dimensional layout prior images to express the hierarchical relationships and spatial dependencies between objects; The 3D content generation module is used to generate a corresponding 3D representation and initial spatial layout based on the 2D layout prior image. The spatial layout optimization module is used to optimize the initial spatial layout and determine the local scene. The scene fusion module is used to integrate the intermediate semantic structure expression corresponding to the local scene into the intermediate semantic structure expression of the pre-built global scene, and select objects in the global scene as semantic anchors to guide the fusion of the local scene into the global scene; The interactive extension module is used to receive incremental editing information from the user, obtain incremental scene description information, and repeatedly execute the language parsing module, the two-dimensional prior generation module, the semantic structure construction module, the three-dimensional content generation module, the spatial layout optimization module, and the scene fusion module until the user completes the generation of the three-dimensional scene.

[0047] Optionally, the language parsing module includes semantic analysis and structural extraction based on a large language model using Transformer.

[0048] The 3D content generation module integrates the processes of image region segmentation, image completion, and 3D reconstruction.

[0049] like Figure 2 As shown, in specific implementation, the present invention can be carried out according to the following steps: The system receives natural language scene descriptions input by the user. This language information can be input via text boxes, speech recognition interfaces, or other human-computer interaction methods. The natural language content input by the user includes, but is not limited to, scene objectives (e.g., "living room," "office area"), core objects (e.g., "sofa," "monitor"), positional relationships (e.g., "left side," "near the window"), and other constraints (e.g., "modern style," "large size").

[0050] After receiving the user's natural language description, the system first processes it through a language parsing module. The parsing task is to extract key object, attribute, and spatial relationship information from the user's input. Specifically, the user's text description (e.g., "I want to generate a modern-style living room") will be understood by the system as structured information containing several semantic elements. The system will identify object categories (e.g., "living room," "sofa"), object attributes (e.g., "modern style"), and spatial relationships (e.g., "the sofa is to the left of the TV cabinet") in the scene.

[0051] The language parsing module relies on a pre-trained language model and employs a two-stage generation strategy, such as... Figure 3 As shown, first, the system analyzes the input scene description to generate a candidate list of objects, which will become the basic building blocks of the subsequently generated image and 3D scene. Then, based on the candidate list, the system constructs cue words for image generation, which will provide the contextual information needed by the image generation model.

[0052] The 2D prior generation module generates a preliminary 2D layout image based on the extracted semantic information. The generative image model then generates a scene image that matches the user's description, based on the object candidate list and semantic description obtained from the language parsing module. During image generation, the generative model not only considers the appearance of individual objects but also generates a reasonable spatial layout based on the spatial relationships between objects. In the initial scene generation, the system uses isometric perspective constraints to construct the 2D image, ensuring that the generated image clearly displays the spatial layout of the entire scene and the relative positions of objects. For subsequent rounds of scene expansion, the generated prompts focus more on detail supplementation and semantic enrichment, employing a perspective view for image generation.

[0053] The semantic structure modeling module is responsible for accurately identifying the spatial positions and semantic relationships between objects in a scene and transforming this information into an operable 3D representation. This module performs in-depth scene analysis based on the input 2D scene image and generates intermediate semantic structure representations to express the hierarchical relationships and spatial dependencies between objects. The semantic structure modeling module receives the 2D scene image generated by the previous module and outputs the 3D structural representation of each object in the scene, as well as the spatial relationships between objects.

[0054] Before the semantic structure modeling module processes the data, the input 2D scene image is preprocessed. Specifically, the system uses an object detection algorithm to detect and segment objects in the input image, and combines this with the object candidate list generated in the previous module as a category prior. This process ensures that every object in the image is identified. After this preprocessing step, the system obtains the bounding box position, corresponding category label, and segmentation mask based on the object boundary for each object in the image.

[0055] Next, the scene parsing submodule uses a visual model to perform in-depth analysis of the preprocessed image. This submodule receives input including a 2D scene image, category labels, and preprocessed object-level segmentation data. The system combines the object segmentation mask with the category labels to generate a scene segmentation map with semantic information. This map not only shows the spatial location of each object but also indicates the category information corresponding to different segmentation regions. Through this generated segmentation map, the system can clearly understand the objects in the scene and their spatial distribution.

[0056] During scene understanding, to identify the spatial relationships between objects, the system performs pairwise matching of objects in the scene, generating multiple object matching pairs. To avoid interference between different matching pairs, the system performs relationship prediction for each pair separately. On the semantic segmentation map, a bounding box is drawn for each pair of objects and provided as input to the visual model. The system then infers the spatial relationship based on preset relationship type prompts. The visual model outputs the spatial relationship prediction result between the matching pairs. Finally, the relationship prediction results of all matching pairs are aggregated to generate a set of spatial relationships between objects.

[0057] In addition to the spatial relationships between objects, scene understanding also requires the system to perceive the spatial position, orientation, size, and other geometric attributes of each object. To this end, this invention utilizes a pose prediction model to process the segmentation mask of each object individually, thereby obtaining a set of pose attributes for each object. These pose attributes describe the object's spatial orientation, size, and relative position.

[0058] Through the above steps, the semantic structure modeling module transforms the visual information, spatial relationship information, and geometric attribute information of objects into a structured semantic structure intermediate expression, forming the hierarchical relationship and spatial dependency between objects in the scene.

[0059] The 3D content generation module is responsible for converting the segmentation information extracted from the 2D image of each object generated by the previous module into a 3D structural representation. For each object detected in the 2D image, its segmentation map in the scene image is first used as input. Due to the potential occlusion problem of objects in the image, the obtained object segmentation image is usually incomplete. This means that relying solely on this image for 3D reconstruction may lead to geometric defects in the generated 3D model, resulting in inconsistent geometric structures throughout the scene. To address this issue, this invention employs an improved method: using an image completion model to explicitly complete the 2D segmentation image of each object during the object modeling stage. Specifically, based on the scene image generated in the current step, for each target object, its segmentation map in the image is first obtained. Then, the image completion model is used to generate the object's complete outline and full-shape mask, thereby solving the problem of incomplete shape caused by occlusion. The completed object segmentation map is then used as input to the image-to-3D generation model, ultimately generating a 3D representation of the corresponding object, representing its spatial coordinates and physical properties.

[0060] The spatial layout optimization module is responsible for optimizing the initial placement of objects, allowing their orientation to be adaptively adjusted. This ensures that the generated local scene layout conforms to the consistency of the spatial structure and closely resembles the actual physical environment. This module infers the spatial position and orientation of each object based on the intermediate semantic structure representation provided by the aforementioned semantic parsing and structural modeling modules. During the initial scene generation phase, the spatial layout optimization module corrects the local scene layout using an orthogonalization algorithm, ensuring that the structural objects in the scene are orthogonal in their primary orientation.

[0061] The scene fusion module achieves efficient fusion and adjustment of 3D scenes by constructing and merging intermediate semantic structures of local and global scenes. To obtain a complete 3D scene, this module precisely integrates the intermediate local semantic structure generated in the current step into the previously constructed intermediate global semantic structure, thus forming a consistent global 3D scene. In each round of local scene generation, the system uses an object in the global scene as a semantic anchor point to guide the generation of the local scene. For example, in the initial generation stage, the system assumes the ground as a semantic anchor point and uses it as a reference for object placement. This mechanism ensures the consistency of the assumption that "any scene generation is based on the extension of semantic anchor points." Through the hierarchical structure and semantic constraints of the intermediate global semantic structure, a clear subordinate relationship is established between local objects and semantic anchor points.

[0062] The interactive extension module responds to subsequent user input, performing local editing and incremental generation operations. The interactive control module allows users to select an object in an existing scene as an anchor point, such as "put a mug on the coffee table." The system can automatically identify the semantic node corresponding to "coffee table" and its spatial position, repeating the steps of the cross-modal 3D scene automated generation system to generate and update the spatial layout of that local area.

[0063] To ensure consistency between locally generated content and the global scene during multi-round scene generation, this invention introduces global scene semantic constraints. Whenever a user provides a new scene description, the system not only generates image prompts using the semantic information of the current round, but also incorporates the global semantic state of the current scene as auxiliary input. This global semantic constraint is automatically generated from the intermediate expression of the current scene's semantic structure, including spatial type, key object categories, functional area labels, and style keywords.

[0064] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A natural language driven cross-modal three-dimensional scene generation method, characterized in that, The method comprises the following steps: S1: receiving a scene description information input by a user in natural language; S2: performing semantic analysis on the scene description information to extract object elements, attribute elements and spatial relationship elements for scene construction; S3: generating a two-dimensional layout prior image of the scene according to the semantic analysis result of the scene description information; S4: constructing a semantic structure intermediate expression based on the two-dimensional layout prior image to express the hierarchical relationship and spatial dependence between objects; generating a corresponding three-dimensional representation and an initial spatial layout according to the two-dimensional layout prior image; optimizing the initial spatial layout to determine a local scene; S5: integrating the semantic structure intermediate expression corresponding to the local scene into the semantic structure intermediate expression of a pre-constructed global scene, and selecting an object as a semantic anchor point in the global scene to guide the fusion of the local scene into the global scene; S6: receiving incremental editing information of the user to obtain incremental scene description information, and performing steps S2-S5 until the user completes the generation of the three-dimensional scene.

2. The natural language driven cross-modal three-dimensional scene generation method according to claim 1, wherein, The scene description information includes scene targets, core objects, location relationships and related constraints.

3. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, The generation process of the two-dimensional layout prior image comprises: generating the appearance of the object according to the object elements and attribute elements, generating the corresponding spatial layout according to the spatial relationship elements, and constructing the two-dimensional layout prior image based on the equidistant perspective constraint.

4. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, The semantic structure intermediate expression is a graph structure or a hierarchical structure, which is used to dynamically represent the subordinate relationship and spatial semantic connection of the objects.

5. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, The construction process of the semantic structure intermediate expression comprises: performing object detection and segmentation on the two-dimensional layout prior image through a target detection algorithm to obtain the bounding box position, class label and segmentation mask based on the object boundary of each object in the two-dimensional layout prior image; combining the segmentation mask and the class label to generate a scene segmentation map with semantic information to show the spatial position of each object and the class information corresponding to different segmentation regions; matching each object in the two-dimensional layout prior image in pairs to generate a plurality of object matching pairs, drawing a bounding box for each object of each object matching pair, inputting the object matching pairs into a preset visual model, performing spatial relationship speculation according to a preset relationship type prompt word, obtaining a relationship prediction result of each object matching pair, obtaining a spatial relationship set between all objects, and converting the spatial relationship set into a structured semantic structure intermediate expression in combination with the appearance information and attribute elements of the objects.

6. The natural language driven cross-modal three-dimensional scene generation method according to claim 5, wherein, The construction process of the semantic structure intermediate expression further comprises: using a pose prediction model to predict the position of the segmentation mask of each object in the two-dimensional layout prior image to obtain a pose attribute set of each object, which includes a spatial position, an orientation and a size, and is used in the construction process of the semantic structure intermediate expression.

7. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, Before generating a corresponding three-dimensional representation according to the two-dimensional layout prior image, the following step is further included: for each object in the two-dimensional layout prior image, obtaining a segmentation map of the object in the two-dimensional layout prior image, generating a complete contour and a full-form mask corresponding to the object, and completing the completion of the two-dimensional layout prior image.

8. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, When integrating the semantic structure intermediate representation corresponding to the local scene into the semantic structure intermediate representation of the pre-constructed global scene, the hierarchical structure and semantic constraints of the semantic structure intermediate representation of the global scene are used to establish the subsidiary relationship between the local scene and the semantic anchor.

9. The natural language driven cross-modal three-dimensional scene generation method of claim 1, wherein, In the local updating process of the global scene in step S6, the global scene semantic constraints are also introduced for generating the two-dimensional layout prior image. The global scene semantic constraints are generated based on the semantic structure intermediate representation of the global scene, and include the space type, the key object category, the functional area label, and the style keyword.

10. A cross-modal three-dimensional scene generation system implementing the natural language driven cross-modal three-dimensional scene generation method of any one of claims 1-9, characterized in that, Comprise: A language analysis module configured to receive scene description information input by a user in natural language; Perform semantic analysis on the scene description information to extract object elements, attribute elements, and spatial relationship elements for scene construction; A two-dimensional prior generation module configured to generate a two-dimensional layout prior image of a scene according to the semantic analysis result of the scene description information; A semantic structure construction module configured to construct a semantic structure intermediate representation based on the two-dimensional layout prior image to express the hierarchical relationship and spatial dependency between objects; A three-dimensional content generation module configured to generate a corresponding three-dimensional representation and initial spatial layout according to the two-dimensional layout prior image; A spatial layout optimization module configured to optimize the initial spatial layout to determine a local scene; A scene fusion module configured to integrate the semantic structure intermediate representation corresponding to the local scene into the semantic structure intermediate representation of the pre-constructed global scene, and select an object as a semantic anchor in the global scene to guide the fusion of the local scene into the global scene; An interaction expansion module configured to receive incremental editing information of a user to obtain incremental scene description information, and repeatedly execute the language analysis module, the two-dimensional prior generation module, the semantic structure construction module, the three-dimensional content generation module, the spatial layout optimization module, and the scene fusion module until the user completes the generation of the three-dimensional scene.

Images