Scene generation and interaction method, apparatus, electronic device, medium, and program product

Abstract

Embodiments of the present disclosure disclose a scene generation and interaction method and device, electronic equipment, medium and program product, which generate a three-dimensional scene (a first three-dimensional scene image) from a single image (a prompt image), extract a multi-modal two-dimensional feature map (multi-modal semantic features) from the prompt image by calling a pre-trained multi-modal feature extraction model, fuse the multi-modal two-dimensional feature map and the three-dimensional scene (the first three-dimensional scene image) into a multi-modal feature field through a distillation and fusion algorithm, and finally render a target scene including the multi-modal two-dimensional feature field, so that a user can interact with the target scene through a user operation instruction. In this way, in the single-image interactive target scene generation, multi-modal semantic features are embedded in real time and high-quality rendering is maintained, and based on the multi-modal semantic features, user operation instruction analysis, task decomposition, and the like can be completed to realize complex scene query and combinable editing of the target scene.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for scene generation and interaction. Background Technology

[0002] Interactive 3D (three-dimensional) scene generation technology driven by a single image or a limited number of views refers to the technique of generating interactive 3D scenes using a single image or a limited number of views as input. This technology combines the latest advancements in deep learning and computer vision, making it possible to convert 2D (two-dimensional) images into 3D scenes, and the generated 3D scenes possess high interactivity and realism. Specifically, it uses deep learning and computer vision techniques to take a single image or a limited number of views as input, and uses AI algorithms to process the input single image or limited number of views to generate a 3D model. This process involves the construction of basic elements such as points, lines, and surfaces, as well as texture mapping and lighting rendering of object surfaces. The generated 3D scene has a high degree of interactivity, allowing users to freely explore and manipulate in a virtual environment. Real-time rendering technology ensures a smooth and realistic user experience during interaction.

[0003] In recent years, interactive 3D scene generation technologies driven by single images or a small number of views (such as WonderJourney and WonderWorld) have achieved significant breakthroughs in appearance and geometric quality. However, they cannot simultaneously embed diverse features such as CLIP (Contrastive Language-Image Pre-training), SAM (Segment Anything Model), DINO (DIstillation with NO labels), and VLM (Vision-Language Model) during the interactive generation of a single image. As a result, the generated virtual environments (3D scenes) generally lack deep semantic annotations, which greatly limits the performance of complex tasks or fine-grained editing in the virtual environment. Summary of the Invention

[0004] To address the technical problems in related technologies, embodiments of this disclosure provide a scene generation and interaction method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] According to a first aspect of the present disclosure, a scene generation and interaction method is provided, the method comprising:

[0006] In response to any received prompt image, a first three-dimensional scene image is generated based on the prompt image, the prompt image including scene prompt content for generating the target scene;

[0007] A multimodal two-dimensional feature map is extracted from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature map includes multimodal semantic features.

[0008] The multimodal two-dimensional feature map and the first three-dimensional scene image are fused using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field;

[0009] The first multimodal feature field is rendered to obtain a second three-dimensional scene image, which is the target scene.

[0010] In response to a received user operation instruction, the system performs interactive operations on the target scene based on the user operation instruction, which includes text prompts and reference images.

[0011] As an optional embodiment, in response to any received cue image, a first three-dimensional scene image is generated based on the cue image, wherein the cue image includes scene cue content for generating the target scene, including:

[0012] In response to any received prompt image, the prompt image is parsed to obtain image depth estimation and image normal estimation;

[0013] Edge detection is performed based on the image depth estimation and the image normal estimation to determine the geometric structure and initial appearance of the scene prompt content in the prompt image;

[0014] A layered Gaussian surface representation is constructed based on the geometric structure and initial appearance of the scene prompt content in the prompt image to obtain the first three-dimensional scene image.

[0015] As an optional embodiment, the step of performing feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field includes:

[0016] The multimodal two-dimensional feature map is sampled and transformed into a feature vector of a preset dimension;

[0017] The first three-dimensional scene image is transformed to obtain the two-dimensional projected coordinates of the Gaussian surface element representation of the first three-dimensional scene image;

[0018] The two-dimensional projection coordinates are associated with the feature vector of the corresponding pixel of the first three-dimensional scene image in the preset dimension, and the first multimodal feature field is obtained by joint optimization through backpropagation. The first multimodal feature field represents the semantic association between the first three-dimensional scene image and the multimodal two-dimensional feature map. The semantic association is to associate the feature maps of the same scene content under multiple views.

[0019] As an optional embodiment, the step of sampling the multimodal two-dimensional feature map and converting it into a feature vector of a preset dimension includes:

[0020] The multimodal two-dimensional feature map is subjected to feature alignment processing to obtain a unified feature representation;

[0021] The unified feature representation is distilled and fused using a preset distiller to adjust its dimension to a preset dimension, thereby obtaining a feature vector of the preset dimension.

[0022] As an optional embodiment, in response to a received user operation instruction, interactive operations are performed on the target scene based on the user operation instruction, wherein the user operation instruction includes text prompts and reference images, including:

[0023] In response to receiving a user operation instruction, the user's operation intention is determined from the text prompt content and the reference image;

[0024] The user's operation intent is decomposed into a sequence of subtasks, which includes multiple subtasks executed sequentially.

[0025] The field plugin is invoked via a preset field-driven function call protocol to execute the sub-task sequence sequentially in the target scene and update the target scene.

[0026] The system uses text-based scene descriptions and visualizations to provide users with feedback on the updated results for the target scene.

[0027] As an optional embodiment, the step of determining the user's operation intent from the text prompt content and the reference image in response to receiving a user operation instruction includes:

[0028] The instruction feature vector is extracted from the text prompt content using a preset text encoding tool;

[0029] Image feature vectors are extracted from the reference image using a preset image encoding tool;

[0030] The instruction feature vector and the image feature vector are fused across modal features to obtain a joint semantic feature vector, which represents the user's operation intention.

[0031] As an optional embodiment, the decomposition of the user's operation intent into a sequence of subtasks, the sequence of subtasks including multiple subtasks executed sequentially, including:

[0032] The user's operation intent is classified to obtain at least one intent type;

[0033] Based on any intent type, subtasks matching that intent type and their order are generated using preset task chain generation rules to obtain the subtask sequence.

[0034] As an optional embodiment, the step of calling the field plugin through a preset field-driven function call protocol to sequentially execute the sub-task sequence on the target scene and update the target scene includes:

[0035] For any given subtask, determine the field plugin type that executes the subtask;

[0036] The preset field driving function calls the corresponding field plugin according to the determined field plugin type;

[0037] The subtask sequence is executed on the first multimodal feature field of the target scene using the corresponding field plugin to obtain the second multimodal feature field;

[0038] The target scene is updated based on the second multimodal feature field rendering.

[0039] According to a second aspect of the present disclosure, a scene generation and interaction apparatus is provided, the apparatus comprising:

[0040] A 3D scene generation module is used to generate a first 3D scene image in response to any received prompt image, wherein the prompt image includes scene prompt content for generating the target scene;

[0041] The feature map extraction module is used to extract multimodal two-dimensional feature maps from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature maps include multimodal semantic features.

[0042] The feature fusion module is used to perform feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field;

[0043] The image rendering module is used to render the first multimodal feature field to obtain a second three-dimensional scene image, which is the target scene.

[0044] An interactive operation module is used to respond to received user operation instructions and perform interactive operations on the target scene based on the user operation instructions. The user operation instructions include text prompts and reference images.

[0045] As an optional embodiment, the 3D scene generation module includes:

[0046] An image parsing unit is configured to perform image parsing on any received prompt image to obtain image depth estimation and image normal estimation;

[0047] The structure and appearance determination unit is used to perform edge detection based on the image depth estimation and the image normal estimation, and to determine the geometric structure and initial appearance of the scene prompt content in the prompt image;

[0048] A Gaussian rendering unit is used to construct a layered Gaussian polygon representation of the geometric structure and initial appearance of the scene prompt content in the prompt image, thereby obtaining the first three-dimensional scene image.

[0049] As an optional embodiment, the feature fusion module includes:

[0050] The feature vector unit is used to sample the features of the multimodal two-dimensional feature map and convert it into a feature vector of a preset dimension.

[0051] The coordinate transformation unit is used to transform the coordinates of the first three-dimensional scene image to obtain the two-dimensional projected coordinates of the Gaussian surface element representation of the first three-dimensional scene image;

[0052] The feature association unit is used to associate the two-dimensional projection coordinates with the feature vector of the corresponding pixel of the first three-dimensional scene image of the preset dimension, and to obtain a first multimodal feature field through backpropagation joint optimization. The first multimodal feature field represents the semantic association between the first three-dimensional scene image and the multimodal two-dimensional feature map. The semantic association is to associate the feature maps of the same scene content under multiple views.

[0053] As an optional embodiment, the feature vector unit is further used for:

[0054] The multimodal two-dimensional feature map is subjected to feature alignment processing to obtain a unified feature representation;

[0055] The unified feature representation is distilled and fused using a preset distiller to adjust its dimension to a preset dimension, thereby obtaining a feature vector of the preset dimension.

[0056] As an optional embodiment, the interactive operation module includes:

[0057] An operation intent determination unit is used to determine the user's operation intent from the text prompt content and the reference image in response to receiving a user operation instruction;

[0058] The task decomposition unit is used to decompose the user's operation intention into a sequence of subtasks, wherein the sequence of subtasks includes multiple subtasks to be executed in sequence.

[0059] The task execution unit is used to invoke the field plugin to execute the sub-task sequence in the target scene sequentially through a preset field-driven function call protocol, and to update the target scene;

[0060] The structural feedback unit is used to provide users with feedback on the update results of the target scene using text scene descriptions and visualizations.

[0061] As an optional embodiment, the operation intent determination unit is further configured to:

[0062] The instruction feature vector is extracted from the text prompt content using a preset text encoding tool;

[0063] Image feature vectors are extracted from the reference image using a preset image encoding tool;

[0064] The instruction feature vector and the image feature vector are fused across modal features to obtain a joint semantic feature vector, which represents the user's operation intention.

[0065] As an optional embodiment, the task decomposition unit is further configured to:

[0066] The user's operation intent is classified to obtain at least one intent type;

[0067] Based on any intent type, subtasks matching that intent type and their order are generated using preset task chain generation rules to obtain the subtask sequence.

[0068] As an optional embodiment, the task execution unit is further configured to:

[0069] For any given subtask, determine the field plugin type that executes the subtask;

[0070] The preset field driving function calls the corresponding field plugin according to the determined field plugin type;

[0071] The subtask sequence is executed on the first multimodal feature field of the target scene using the corresponding field plugin to obtain the second multimodal feature field;

[0072] The target scene is updated based on the second multimodal feature field rendering.

[0073] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0074] Memory, used to store computer program products;

[0075] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0076] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0077] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0078] In this embodiment, a three-dimensional scene (first three-dimensional scene image) is generated using a single image (prompt image). A pre-trained multimodal feature extraction model is used to extract a multimodal two-dimensional feature map (multimodal semantic features) from the prompt image. A distillation and fusion algorithm is then used to fuse the multimodal two-dimensional feature map with the three-dimensional scene (first three-dimensional scene image) to form a multimodal feature field. Finally, a target scene including the multimodal two-dimensional feature field is rendered. Users can interact with the target scene using user operation commands. This achieves real-time embedding of multimodal semantic features while maintaining high-quality rendering in the generation of an interactive target scene from a single image. Based on the multimodal semantic features, user operation command parsing and task decomposition can be performed to enable complex scene queries and composable editing of the target scene.

[0079] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0080] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0081] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0082] Figure 1 This is a flowchart illustrating one embodiment of the scene generation and interaction method disclosed herein.

[0083] Figure 2 This is a second flowchart illustrating a scene generation and interaction method according to an embodiment of the method disclosed herein.

[0084] Figure 3 This is a structural block diagram of a scene generation and interaction device according to an embodiment of the present disclosure.

[0085] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0086] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0087] Figure 1 This is a flowchart illustrating one embodiment of a scene generation and interaction method according to the present disclosure. Figure 1 As shown, a scene generation and interaction method may include the following steps:

[0088] Step 101: In response to any received cue image, generate a first three-dimensional scene image based on the cue image, wherein the cue image includes scene cue content for generating the target scene.

[0089] The cue image can be a two-dimensional image that includes scene cue content for generating the target scene. For example, if the cue image is a two-dimensional image including a blue sky, green land, and a red tower, then the blue sky, green land, and red tower can be the scene cue content, and the generated target scene will include the blue sky, green land, and red tower. The first three-dimensional scene image is the initial scene image of the target scene.

[0090] In this embodiment of the disclosure, starting with a single two-dimensional image, an initial three-dimensional scene for interactive exploration, namely the first three-dimensional scene image, can be constructed using a pre-trained image-to-world generation model. Specifically, this can be implemented as follows: in response to any received prompt image, image parsing is performed on the prompt image to obtain image depth estimation and image normal estimation; edge detection is performed based on the image depth estimation and image normal estimation to determine the geometric structure and initial appearance of the scene prompt content in the prompt image; and a layered Gaussian surface representation is constructed based on the geometric structure and initial appearance of the scene prompt content in the prompt image to obtain the first three-dimensional scene image.

[0091] Step 102: Extract multimodal two-dimensional feature maps from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature maps include multimodal semantic features.

[0092] Pre-trained multimodal feature extraction models include, but are not limited to, Contrastive Language-Image Pre-training (CLIP), Segment Anything Model (SAM), DIstillation with NO labels (DINO), and Vision-Language Model (VLM). Among them:

[0093] Among them, the contrastive language-image pre-trained model CLIP can achieve cross-modal matching between images and text, such as associating the text description of "wooden coffee table" with objects in a 3D scene. The general segmentation model SAM can achieve zero-shot image instance segmentation, such as accurately separating the tabletop and leg areas of a coffee table. The unlabeled self-distilling visual model DINO can self-supervised learn to extract object structural features, such as matching the 3D structure of a coffee table from different viewpoints. And the visual-language model VLM can jointly understand and reason about images and text, such as resolving the spatial and material requirements in "place a metal table lamp on the left".

[0094] For example, by calling the pre-trained multimodal feature extraction model described above, we can generate, for example, a global semantic feature map (implemented via CLIP), an instance segmentation mask feature map (implemented via SAM), a local fine-grained feature map (implemented via DINO), and a visual-language aligned feature map (implemented via VLM). The multimodal two-dimensional feature map is a multimodal two-dimensional tensor that preserves positional relationships, that is, it preserves a specific region of the scene (e.g., a first three-dimensional scene image) corresponding to each pixel.

[0095] Step 103: Use a preset feature distillation and fusion algorithm to perform feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image to obtain the first multimodal feature field.

[0096] In this embodiment, the multimodal feature field can be a vector database in 3D space, where each three-dimensional Gaussian surface element (or primitive) stores feature vectors such as CLIP / SAM / DINO. That is, through this step, the multimodal two-dimensional feature map is fused with the first three-dimensional scene image to achieve the fusion of the RGB color field and the multimodal semantic field (CLIP / SAM / DINO / VLM embedding), thereby enabling each three-dimensional Gaussian surface element (or primitive, Surfel) in 3D space to carry a multimodal feature vector.

[0097] As one implementation method, step 103 can be implemented as follows: The multimodal two-dimensional feature map is sampled and transformed into a feature vector of a preset dimension; the first three-dimensional scene image is transformed to obtain the two-dimensional projected coordinates of the Gaussian surface representation of the first three-dimensional scene image; the two-dimensional projected coordinates are associated with the feature vector of the corresponding pixel in the first three-dimensional scene image of the preset dimension; and joint optimization is performed through backpropagation to obtain the first multimodal feature field. The first multimodal feature field represents the semantic association between the first three-dimensional scene image and the multimodal two-dimensional feature map. Semantic association associates feature maps of the same scene content from multiple perspectives. It is understandable that different base models (i.e., pre-trained multimodal feature extraction models, including CLIP, SAM, DINO, etc.) output feature map resolutions are different. Therefore, feature alignment can be performed through resampling to obtain feature vectors of a unified dimension (preset dimension). Specifically, feature alignment can be achieved by, for example, constructing a unified feature pyramid and using adaptive interpolation, ultimately achieving automatic multi-view feature alignment.

[0098] As another embodiment, the multimodal 2D feature map can be projected onto a unified 3D space using camera parameters, and then feature sampling can be performed to transform (or compress) the feature responses of different modalities into feature vectors of a preset dimension (e.g., 128-dimensional). Then, Gaussian facets of the first 3D scene image are associated with the preset-dimensional feature vectors at corresponding pixels, so that the Gaussian facets of the first 3D scene image store the preset-dimensional feature vectors corresponding to the multimodal 2D feature map, resulting in a first multimodal feature field. Using a feature field to manage feature vectors allows for real-time updates as the 3D scene image changes, improving the efficiency of interactive operations on the 3D scene image.

[0099] Furthermore, feature sampling of the multimodal 2D feature map and its transformation into a feature vector of a preset dimension can be achieved by performing feature alignment processing on the multimodal 2D feature map to obtain a unified feature representation. This unified feature representation is then distilled and fused using a preset distiller to adjust its dimension to the preset dimension, resulting in a feature vector of the preset dimension. For example, a two-stage compression method can be used to unify heterogeneous features into a 32-dimensional vector, transforming the multimodal 2D feature map into a feature vector of the preset dimension. Assuming CLIP (512 dimensions), SAM (1024 dimensions), DINO (384 dimensions), and VLM (256 dimensions), a shared fully connected layer (128 dimensions) is used to transform CLIP, SAM, DINO, and VLM into a unified intermediate representation of 128 dimensions. This is then distilled and fused using a lightweight distiller (e.g., a 3-layer MLP (Multilayer Perceptron)) to obtain a unified feature vector of 32 dimensions (the preset dimension).

[0100] Step 104: Render the first multimodal feature field to obtain a second three-dimensional scene image, which is the target scene.

[0101] In this step, image rendering is performed on the first multimodal feature field, including color view rendering and multi-feature field rendering (i.e., rendering of the multimodal two-dimensional feature map), ultimately obtaining the second three-dimensional scene image. That is, the second three-dimensional scene image is rendered by incorporating the multimodal two-dimensional feature map into the first three-dimensional scene image. In this way, the semantic feature vectors from the multimodal feature map are fused into the three-dimensional Gaussian elements in the second three-dimensional scene image, and stored as a feature field for easy retrieval during subsequent interactive operations, satisfying real-time access and updates.

[0102] Step 105: In response to the received user operation instruction, perform interactive operations on the target scene based on the user operation instruction. The user operation instruction includes text prompts and reference images.

[0103] In this embodiment of the disclosure, this step can specifically be implemented as follows: in response to receiving a user operation command, determining the user's operation intent from the text prompt content and reference image, decomposing the user's operation intent into a sequence of sub-tasks, wherein the sub-task sequence includes multiple sub-tasks to be executed sequentially, and then invoking a field plugin through a preset field-driven function call protocol to execute the sub-task sequence sequentially on the target scene and update the target scene. Furthermore, the update result of the target scene can be fed back to the user through text scene description and visualization.

[0104] In response to receiving user operation instructions, determining the user's operation intent from the text prompt content and the reference image can be achieved by extracting the instruction feature vector from the text prompt content using a preset text encoding tool, extracting the image feature vector from the reference image using a preset image encoding tool, and fusing the instruction feature vector and the image feature vector across modal features to obtain a joint semantic feature vector, where the joint semantic feature vector represents the user's operation intent.

[0105] The user's operation intent is decomposed into a sequence of subtasks, which includes multiple subtasks executed in sequence. Specifically, the user's operation intent is classified to obtain at least one intent type. Based on any intent type, a subtask matching the intent type and the order between the subtasks are generated using a preset task chain generation rule to obtain the subtask sequence.

[0106] For example, if a user inputs a text prompt and a reference image, and the text prompt is "Place a potted plant in the southwest corner of the living room, adjusting it to 1 / 3 of the wall height," and the reference image is a picture of a plant, then the resulting sub-task sequence would be as follows: Step 1: Use CLIP+DINO to locate "southwest corner of the living room"; Step 2: Use SAM to segment the wall → calculate the 1 / 3 height coordinates; Step 3: Insert the 3D Gaussian polygons of the plant image and attach semantic features.

[0107] The pre-defined field-driven function call protocol invokes field plugins to sequentially execute a sequence of subtasks on the target scene and update the target scene. Specifically, for any subtask, the field plugin type for executing that subtask can be determined. The pre-defined field-driven function calls the corresponding field plugin according to the determined field plugin type. The corresponding field plugin executes the subtask sequence on the first multimodal feature field of the target scene to obtain the second multimodal feature field. The target scene is then rendered and updated based on the second multimodal feature field. For example, the pre-defined field-driven function call protocol can initialize and generate a standardized function call template (e.g., <operation type>(field type, parameter)), and then perform field plugin matching. The corresponding field plugin can be activated according to the field type (e.g., DINO-Field matching fine-grained matching plugin, SAM2-Field matching segmentation plugin), realizing plugin instantiation. Furthermore, the field plugin can perform a series of actions such as feature field query, feature decoding, and scene operation to complete the execution of the subtask sequence. The feature field query process involves first encoding parameters into feature vectors (e.g., text → CLIP vector), then searching for matching 3D Gaussian facets in a unified feature field to obtain a target 3D Gaussian facet set. Based on this target 3D Gaussian facet set, it is possible to further render a Latent feature map (generating 2D feature visualization), extract 3D bounding boxes / masks, generate descriptive text using VLM, etc. Finally, the feature field data is modified (e.g., deleting 3D Gaussian facets to remove the target geometry, attaching new feature vectors to 3D Gaussian facets to add attributes, etc.), and then updating the target scene based on the modified feature field data (second multimodal feature field).

[0108] In summary, a 3D scene (first 3D scene image) is generated from a single image (prompt image). A pre-trained multimodal feature extraction model is used to extract multimodal 2D feature maps (multimodal semantic features) from the prompt image. These multimodal 2D feature maps are then fused with the 3D scene (first 3D scene image) using a distillation and fusion algorithm to form a multimodal feature field. Finally, a target scene including the multimodal 2D feature field is rendered, allowing users to interact with the target scene using user commands. This achieves real-time embedding of multimodal semantic features while maintaining high-quality rendering in the generation of an interactive target scene from a single image. Based on these multimodal semantic features, user command parsing and task decomposition can be performed, enabling complex scene queries and composable editing of the target scene.

[0109] To enable those skilled in the art to accurately understand the technical solutions of the embodiments of this disclosure, the following is combined with... Figure 2 Further details. Figure 2 This is a second flowchart illustrating a scene generation and interaction method according to an embodiment of the method disclosed herein, as shown below. Figure 2 The scene generation and interaction method shown in this embodiment may include stage 1 and stage 2, wherein stage 1 is the scene generation stage and stage 2 is the scene interaction stage.

[0110] Specifically, in Stage 1, starting with a single "Input Image," an initial 3D scene (i.e., the first 3D scene image) for interactive exploration is constructed through the "Image-to-World Generation" module. Then, through the "Render" module, not only is a regular color view of the scene generated, but also "Versatile Field Rendering" is performed. In this embodiment, prior knowledge provided by various foundation models (i.e., pre-trained multimodal feature extraction models) is utilized to simultaneously distill and embed the semantic features of different modalities into the 3D representation (i.e., feature fusion of multimodal 2D feature maps and the first 3D scene image). Specifically, feature map A is obtained from the CLIP model, feature map B from the SAM2 model, feature map C from the DINOv2 model, and feature map D from the InternVL model. These feature maps, together with the color map (the first 3D scene image), constitute a unified, diverse feature field (i.e., the first multimodal feature field). The target scene generated in this stage supports direct manipulation and interacts bidirectionally with the rendering process.

[0111] In Phase 2, initiated by the "User," user submits user operation instructions (which may include text and images, such as text prompts and reference images) to the "Controller." Upon receiving the instructions, the controller first performs task planning (②), decomposing the user's intent into a series of ordered subtasks (a sequence of subtasks) within its internal Reasoning & Planning module. For example, it first uses DINO-Field for feature matching, then uses VLM-Field to generate a description, and finally uses SAM2-Field and CLIP-Field for segmentation and localization to perform a deletion operation. After planning, the controller executes these subtasks through a tool calling (③) stage. In this embodiment, the "Field Plug-in" of the Field-driven Function Call Protocol is used to transform the subtasks into standardized calls to underlying functions, such as feature retrieval or language-guided segmentation. In this way, the feature field (first multimodal feature field) corresponding to the target scene generated in stage 1 can be directly applied to achieve embedded invocation. As an optional embodiment, the result of each tool invocation (standardized invocation) can be returned to the controller through ④ Result Feedback. After collecting and integrating the result feedback of all intermediate steps, the controller provides the user with the final "feedback". As shown in the example in the figure, the feedback can be either text describing the scene or a series of visual results such as "Retrieval", "Tracking" and "Delete the "tower"", that is, the update results of the target scene are fed back to the user in a text scene description and visualization manner. Through this embodiment, the execution process of scene generation and interaction of the present disclosure embodiment can be fully presented, realizing an interactive three-dimensional scene intelligent agent that can perform perception, reasoning and execution closed loop.

[0112] Correspondingly, this disclosure also provides device embodiments corresponding to the foregoing method embodiments. The device embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 3 This is a structural block diagram of a scene generation and interaction device according to one embodiment of the present disclosure, such as... Figure 3 As shown, a scene generation and interaction device may include:

[0113] The 3D scene generation module 301 is used to generate a first 3D scene image in response to any received prompt image, wherein the prompt image includes scene prompt content for generating the target scene;

[0114] The feature map extraction module 302 is used to extract a multimodal two-dimensional feature map from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature map includes multimodal semantic features.

[0115] The feature fusion module 303 is used to perform feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field;

[0116] Image rendering module 304 is used to render the first multimodal feature field to obtain a second three-dimensional scene image, wherein the second three-dimensional scene image is the target scene;

[0117] The interactive operation module 305 is used to respond to the received user operation command and perform interactive operations on the target scene based on the user operation command. The user operation command includes text prompts and reference images.

[0118] In summary, a 3D scene (first 3D scene image) is generated from a single image (prompt image). A pre-trained multimodal feature extraction model is used to extract multimodal 2D feature maps (multimodal semantic features) from the prompt image. These multimodal 2D feature maps are then fused with the 3D scene (first 3D scene image) using a distillation and fusion algorithm to form a multimodal feature field. Finally, a target scene including the multimodal 2D feature field is rendered, allowing users to interact with the target scene using user commands. This achieves real-time embedding of multimodal semantic features while maintaining high-quality rendering in the generation of an interactive target scene from a single image. Based on these multimodal semantic features, user command parsing and task decomposition can be performed, enabling complex scene queries and composable editing of the target scene.

[0119] As an optional embodiment, the 3D scene generation module includes:

[0120] An image parsing unit is configured to perform image parsing on any received prompt image to obtain image depth estimation and image normal estimation;

[0121] The structure and appearance determination unit is used to perform edge detection based on the image depth estimation and the image normal estimation, and to determine the geometric structure and initial appearance of the scene prompt content in the prompt image;

[0122] A Gaussian rendering unit is used to construct a layered Gaussian polygon representation of the geometric structure and initial appearance of the scene prompt content in the prompt image, thereby obtaining the first three-dimensional scene image.

[0123] As an optional embodiment, the feature fusion module includes:

[0124] The feature vector unit is used to sample the features of the multimodal two-dimensional feature map and convert it into a feature vector of a preset dimension.

[0125] The coordinate transformation unit is used to transform the coordinates of the first three-dimensional scene image to obtain the two-dimensional projected coordinates of the Gaussian surface element representation of the first three-dimensional scene image;

[0126] The feature association unit is used to associate the two-dimensional projection coordinates with the feature vector of the corresponding pixel of the first three-dimensional scene image of the preset dimension, and to obtain a first multimodal feature field through backpropagation joint optimization. The first multimodal feature field represents the semantic association between the first three-dimensional scene image and the multimodal two-dimensional feature map. The semantic association is to associate the feature maps of the same scene content under multiple views.

[0127] As an optional embodiment, the feature vector unit is further used for:

[0128] The multimodal two-dimensional feature map is subjected to feature alignment processing to obtain a unified feature representation;

[0129] The unified feature representation is distilled and fused using a preset distiller to adjust its dimension to a preset dimension, thereby obtaining a feature vector of the preset dimension.

[0130] As an optional embodiment, the interactive operation module includes:

[0131] An operation intent determination unit is used to determine the user's operation intent from the text prompt content and the reference image in response to receiving a user operation instruction;

[0132] The task decomposition unit is used to decompose the user's operation intention into a sequence of subtasks, wherein the sequence of subtasks includes multiple subtasks to be executed in sequence.

[0133] The task execution unit is used to invoke the field plugin to execute the sub-task sequence in the target scene sequentially through a preset field-driven function call protocol, and to update the target scene;

[0134] The structural feedback unit is used to provide users with feedback on the update results of the target scene using text scene descriptions and visualizations.

[0135] As an optional embodiment, the operation intent determination unit is further configured to:

[0136] The instruction feature vector is extracted from the text prompt content using a preset text encoding tool;

[0137] Image feature vectors are extracted from the reference image using a preset image encoding tool;

[0138] The instruction feature vector and the image feature vector are fused across modal features to obtain a joint semantic feature vector, which represents the user's operation intention.

[0139] As an optional embodiment, the task decomposition unit is further configured to:

[0140] The user's operation intent is classified to obtain at least one intent type;

[0141] Based on any intent type, subtasks matching that intent type and their order are generated using preset task chain generation rules to obtain the subtask sequence.

[0142] As an optional embodiment, the task execution unit is further configured to:

[0143] For any given subtask, determine the field plugin type that executes the subtask;

[0144] The preset field driving function calls the corresponding field plugin according to the determined field plugin type;

[0145] The subtask sequence is executed on the first multimodal feature field of the target scene using the corresponding field plugin to obtain the second multimodal feature field;

[0146] The target scene is updated based on the second multimodal feature field rendering.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] Below, for reference Figure 4 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0149] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0150] like Figure 4 As shown, the electronic device includes one or more processors and memory.

[0151] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0152] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the scene generation and interaction methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0153] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0154] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0155] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0156] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0157] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the scene generation and interaction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0158] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the scene generation and interaction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0160] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0161] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0163] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0164] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0165] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0166] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0167] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A scene generation and interaction method, characterized in that, The method includes: In response to any received prompt image, a first three-dimensional scene image is generated based on the prompt image, the prompt image including scene prompt content for generating the target scene; A multimodal two-dimensional feature map is extracted from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature map includes multimodal semantic features. The multimodal two-dimensional feature map and the first three-dimensional scene image are fused using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field; The first multimodal feature field is rendered to obtain a second three-dimensional scene image, which is the target scene. In response to a received user operation instruction, interactive operations are performed on the target scene based on the user operation instruction, wherein the user operation instruction includes text prompts and reference images, including: In response to receiving a user operation instruction, the user's operation intention is determined from the text prompt content and the reference image; The user's operation intent is decomposed into a sequence of subtasks, which includes multiple subtasks executed sequentially. The field plugin is invoked via a preset field-driven function call protocol to execute the sub-task sequence sequentially in the target scene and update the target scene. The update results for the target scene are fed back to the user through text scene descriptions and visualizations. The multimodal feature extraction model includes at least a contrastive language-image pre-trained model, a general segmentation model, an unlabeled self-distillation visual model, and a visual-language model. The extraction of a multimodal two-dimensional feature map from the prompt image by calling the pre-trained multimodal feature extraction model, wherein the multimodal two-dimensional feature map includes multimodal semantic features, including: The contrastive language-image pre-trained model is invoked to extract a global semantic feature map from the prompt image; A general segmentation model is invoked to extract instance segmentation mask feature maps from the prompt image; The label-free self-distillation visual model is invoked to extract local fine-grained feature maps from the prompt image; The visual-language model is invoked to extract visual-language alignment feature maps from the prompt image.

2. The method according to claim 1, characterized in that, In response to any received cue image, a first three-dimensional scene image is generated based on the cue image, wherein the cue image includes scene cue content for generating the target scene, including: In response to any received cue image, the cue image is parsed to obtain image depth estimation and image normal estimation; Edge detection is performed based on the image depth estimation and the image normal estimation to determine the geometric structure and initial appearance of the scene prompt content in the prompt image; A layered Gaussian surface representation is constructed based on the geometric structure and initial appearance of the scene prompt content in the prompt image to obtain the first three-dimensional scene image.

3. The method according to claim 1, characterized in that, The step of performing feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field includes: The multimodal two-dimensional feature map is sampled and transformed into a feature vector of a preset dimension; The first three-dimensional scene image is transformed to obtain the two-dimensional projected coordinates of the Gaussian surface element representation of the first three-dimensional scene image; The two-dimensional projection coordinates are associated with the feature vector of the corresponding pixel of the first three-dimensional scene image in the preset dimension, and the first multimodal feature field is obtained by joint optimization through backpropagation. The first multimodal feature field represents the semantic association between the first three-dimensional scene image and the multimodal two-dimensional feature map. The semantic association is to associate the feature maps of the same scene content under multiple views.

4. The method according to claim 3, characterized in that, The step of sampling the multimodal two-dimensional feature map and converting it into a feature vector of a preset dimension includes: The multimodal two-dimensional feature map is subjected to feature alignment processing to obtain a unified feature representation; The unified feature representation is distilled and fused using a preset distiller to adjust its dimension to a preset dimension, thereby obtaining a feature vector of the preset dimension.

5. The method according to claim 1, characterized in that, The step of responding to receiving a user operation instruction and determining the user's operation intent from the text prompt content and the reference image includes: The instruction feature vector is extracted from the text prompt content using a preset text encoding tool; Image feature vectors are extracted from the reference image using a preset image encoding tool; The instruction feature vector and the image feature vector are fused across modal features to obtain a joint semantic feature vector, which represents the user's operation intention.

6. The method according to claim 1, characterized in that, The process of decomposing the user's operational intent into a sequence of subtasks, wherein the sequence of subtasks includes multiple subtasks executed sequentially, including: The user's operation intent is classified to obtain at least one intent type; Based on any intent type, subtasks matching that intent type and their order are generated using preset task chain generation rules to obtain the subtask sequence.

7. The method according to claim 1, characterized in that, The step of calling the field plugin through a preset field-driven function call protocol to sequentially execute the sub-task sequence in the target scene and update the target scene includes: For any given subtask, determine the field plugin type that executes the subtask; The preset field driving function calls the corresponding field plugin according to the determined field plugin type; The subtask sequence is executed on the first multimodal feature field of the target scene using the corresponding field plugin to obtain the second multimodal feature field; The target scene is updated based on the second multimodal feature field rendering.

8. A scene generation and interaction device, characterized in that, The device includes: A 3D scene generation module is used to generate a first 3D scene image in response to any received prompt image, wherein the prompt image includes scene prompt content for generating the target scene; The feature map extraction module is used to extract multimodal two-dimensional feature maps from the prompt image by calling a pre-trained multimodal feature extraction model. The multimodal two-dimensional feature maps include multimodal semantic features. The feature fusion module is used to perform feature fusion processing on the multimodal two-dimensional feature map and the first three-dimensional scene image using a preset feature distillation and fusion algorithm to obtain a first multimodal feature field; The image rendering module is used to render the first multimodal feature field to obtain a second three-dimensional scene image, which is the target scene. The interactive operation module is used to respond to received user operation instructions and perform interactive operations on the target scene based on the user operation instructions. The user operation instructions include text prompts and reference images, specifically including: In response to receiving a user operation instruction, the user's operation intention is determined from the text prompt content and the reference image; The user's operation intent is decomposed into a sequence of subtasks, which includes multiple subtasks executed sequentially. The field plugin is invoked via a preset field-driven function call protocol to execute the sub-task sequence sequentially in the target scene and update the target scene. The update results for the target scene are fed back to the user through text scene descriptions and visualizations. The multimodal feature extraction model includes at least a contrastive language-image pre-trained model, a general segmentation model, an unlabeled self-distillation visual model, and a visual-language model. The feature map extraction module is used to extract multimodal two-dimensional feature maps from the prompt image by calling the pre-trained multimodal feature extraction model. The multimodal two-dimensional feature maps include multimodal semantic features, specifically: The contrastive language-image pre-trained model is invoked to extract a global semantic feature map from the prompt image; A general segmentation model is invoked to extract instance segmentation mask feature maps from the prompt image; The label-free self-distillation visual model is invoked to extract local fine-grained feature maps from the prompt image; The visual-language model is invoked to extract visual-language alignment feature maps from the prompt image.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images