Virtual environment data processing method, electronic device, and storage medium

Abstract

The virtual environment data processing method, the electronic device and the storage medium provided by the embodiments of the present application, the method comprises: responding to a virtual data control instruction of a user in a virtual environment scene; when the virtual data control instruction represents updating the virtual environment scene, obtaining placement object information of a to-be-placed object, structure feature information and visual information of the virtual environment scene, and inputting the placement object information, the structure feature information and the visual information into a visual language model for data processing to obtain optimized arrangement data of the to-be-placed object; generating the to-be-placed object in the virtual environment scene based on the optimized arrangement data to obtain an adjusted virtual scene; and based on a plurality of physical placement constraints, performing at least one pose iteration correction on the to-be-placed object in the adjusted virtual scene to generate a target virtual scene that can be adjusted by the user, thereby significantly improving the reliability, physical realism and user demand of the scene data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a virtual environment data processing method, an electronic device and a storage medium. BACKGROUND

[0002] Virtual reality (VR) technology creates a physically realistic and credible experience for users in a virtual environment by providing an immersive interactive way. In the prior art, the creation of a virtual environment scene mainly relies on manual operation by professional software or automatic layout by programmatic content generation (PCG) and artificial intelligence (AI) driven methods. Among them, some emerging AI generation methods can directly output a complete three-dimensional scene layout scheme according to text or image prompts using large models. This kind of method is usually a "one-click" black box process, that is, after the AI receives high-level instructions, it will predict and generate an ideal object arrangement result that it considers to be the best, thereby achieving a certain degree of automation of scene generation.

[0003] However, the layout result generated by the above-mentioned AI driven scene generation method is only an ideal prediction based on data and logic, which cannot completely guarantee the physical reality and semantic rationality of the final scene, resulting in problems such as mutual penetration, suspension or placement position not conforming to physical common sense and functional logic between objects in the scene, which seriously damages the physical reality of the virtual scene and the immersive experience of the user, and makes the reliability of the generated virtual environment data low. SUMMARY

[0004] The virtual environment data processing method, the electronic device and the storage medium provided by the embodiments of the present application can improve the reliability of the generated virtual environment data.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application proposes a virtual environment data processing method, which comprises:

[0006] responding to a virtual data control instruction of a user in a virtual environment scene;

[0007] when the virtual data control instruction represents updating the virtual environment scene, obtaining placement object information of a to-be-placed object and structure feature information and visual information of the virtual environment scene, and inputting the placement object information, the structure feature information and the visual information into a visual language model for data processing to obtain optimized arrangement data of the to-be-placed object in the virtual environment scene;

[0008] generating the to-be-placed object in the virtual environment scene based on the optimized arrangement data to obtain an adjusted virtual scene;

[0009] Based on the plurality of physical placement constraints, a pose adjustment amount of the to-be-placed object in the adjusted virtual scene is calculated, and the to-be-placed object in the adjusted virtual scene is pose corrected based on the pose adjustment amount, to obtain an updated virtual scene, the updated virtual scene is taken as a new adjusted virtual scene, and the to-be-placed object in the adjusted virtual scene is iteratively pose corrected at least once.

[0010] The updated virtual scene obtained after the last iteration of pose correction is taken as a target virtual scene, and the target virtual scene is displayed or responds to a user's modification operation.

[0011] In some embodiments, the to-be-placed object information is obtained, including:

[0012] The object identifier of the to-be-placed object is determined from the virtual data control instruction;

[0013] The scene style information of the virtual environment scene is obtained;

[0014] The style object retrieval information is generated based on the object identifier and the scene style information;

[0015] The to-be-placed object information is obtained by matching in a three-dimensional model library based on the style object retrieval information.

[0016] In some embodiments, the structure feature information, the structure feature information, and the visual information are input into a visual language model for data processing to obtain optimized arrangement data of the to-be-placed object in the virtual environment scene, including:

[0017] When the virtual data control instruction includes reference position information of the to-be-placed object, the reference position information, the to-be-placed object information, the structure feature information, and the visual information are input into the visual language model for data processing to obtain the optimized arrangement data near the reference position information;

[0018] When the virtual data control instruction does not include reference position information of the to-be-placed object, the to-be-placed object information, the structure feature information, and the visual information are input into the visual language model for data processing to obtain the optimized arrangement data that conforms to the position logic in the virtual environment scene.

[0019] In some embodiments, the pose adjustment amount of the to-be-placed object in the adjusted virtual scene is calculated based on the plurality of physical placement constraints, including:

[0020] Perform placement state matching on the to-be-placed object in the adjusted virtual scene based on each of the physical placement constraints, to obtain a state matching result;

[0021] When the state matching result indicates that the to-be-placed object has a position error, generate an adjustment thrust based on position error information;

[0022] When the state matching result indicates that the to-be-placed object has an orientation error, generate an adjustment rotation based on orientation error information;

[0023] Obtain the pose adjustment amount based on the adjustment thrust and / or the adjustment rotation.

[0024] In some embodiments, the method further comprises:

[0025] Perform analysis processing on the target virtual scene to obtain scene summary information of the target virtual scene;

[0026] In response to inquiry information of a user;

[0027] Perform retrieval matching in a layout rule knowledge base based on the inquiry information and the scene summary information, to obtain design rule information;

[0028] Input the scene summary information, the inquiry information, and the design rule information into a question and answer model, to obtain reply result data corresponding to the inquiry information.

[0029] In some embodiments, the inputting the scene summary information, the inquiry information, and the design rule information into a question and answer model, to obtain reply result data corresponding to the inquiry information, comprises:

[0030] Confirm target structure feature information and target visual information of the target virtual scene;

[0031] Input the scene summary information, the inquiry information, the design rule information, the target structure feature information, and the target visual information into a question and answer model, to obtain reply result data corresponding to the inquiry information.

[0032] In some embodiments, the method further comprises:

[0033] When the reply result data includes new object information of a new to-be-placed object, determine new arrangement data of the new to-be-placed object in the target virtual scene based on the new object information;

[0034] Generate the new to-be-placed object in the target virtual scene based on the new arrangement data.

[0035] In some embodiments, the method further comprises:

[0036] in response to a direct modification instruction of a target object in the target virtual scene by a user;

[0037] adjust the target object in the target virtual scene based on the direct modification instruction;

[0038] generate and save updated scene information based on the adjusted target virtual scene.

[0039] To achieve the above object, a second aspect of embodiments of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the virtual environment data processing method according to the first aspect when executing the computer program.

[0040] To achieve the above object, a third aspect of embodiments of the present application provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the virtual environment data processing method according to the first aspect.

[0041] The virtual environment data processing method, the electronic device and the storage medium provided by the embodiments of the present application, the method comprises the following steps: first, responding to a virtual data control instruction of a user in a virtual environment scene; then, when the virtual data control instruction represents updating the virtual environment scene, obtaining placement object information of a to-be-placed object, structure feature information and visual information of the virtual environment scene, and inputting the placement object information, the structure feature information and the visual information into a visual language model for data processing to obtain optimized arrangement data of the to-be-placed object in the virtual environment scene; next, generating the to-be-placed object in the virtual environment scene based on the optimized arrangement data to obtain an adjusted virtual scene; secondly, calculating a pose adjustment amount of the to-be-placed object in the adjusted virtual scene based on a plurality of physical placement constraints, and correcting the pose of the to-be-placed object in the adjusted virtual scene based on the pose adjustment amount to obtain an updated virtual scene, taking the updated virtual scene as a new adjusted virtual scene, and performing at least one iteration of pose correction on the to-be-placed object in the adjusted virtual scene; finally, taking the updated virtual scene obtained after the last iteration of pose correction as a target virtual scene. The embodiments of the present application combine the intelligent prediction of the visual language model with the subsequent multi-constraint iterative optimization, effectively solving the two technical problems of physical unreliability and uncontrollable interaction caused by the 'black box' mode in the existing AI scene generation technology. The scheme provided by the present application does not take the idealized layout output by the AI as the final result, but on this basis, the intelligent prediction of the visual language model, the multi-constraint physical iterative optimization, and the user's dominant control in the loop are combined to ensure that the finally generated virtual environment data not only strictly follows the physical laws in micro details, but also realizes efficient and transparent human-computer collaborative process in macro creative process. In the iterative process, through an iterative physical optimization engine, the pose of the object in the scene is continuously corrected based on various physical placement constraints such as horizontal collision, vertical support and room boundary until a stable state is reached. This process can automatically find and repair physical errors such as model penetration, suspension and insufficient support in the AI layout, and more importantly, realizes efficient and transparent human-computer collaboration in the macro creative process, changes the creative process from a rigid 'instruction-execution' mode to a flexible 'dialogue-collaboration' mode, thereby significantly improving the reliability, physical realism and immersive creative experience of the scene data.

[0042] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of a virtual environment data processing method provided by an embodiment of the present application.

[0044] Figure 2 is a flowchart of acquiring placement object information of a to-be-placed object provided by another embodiment of the present application.

[0045] Figure 3 is a flowchart of acquiring optimized arrangement data of a to-be-placed object in a virtual environment scene provided by another embodiment of the present application.

[0046] Figure 4 is a flowchart of calculating a pose adjustment amount of a to-be-placed object in adjusting a virtual scene provided by another embodiment of the present application.

[0047] Figure 5 is a flowchart of adjusting optimization of a multi-physical placement constraint optimization algorithm provided by another embodiment of the present application.

[0048] Figure 6 is a schematic diagram of preparing optimization data provided by another embodiment of the present application.

[0049] Figure 7 is a flowchart of responding to inquiry information of a user provided by another embodiment of the present application.

[0050] Figure 8 is a flowchart of step 704 in the method. Figure 7

[0051] Figure 9 is a flowchart of generating a newly added placement object provided by another embodiment of the present application.

[0052] Figure 10 is a flowchart of responding to a direct modification instruction provided by another embodiment of the present application.

[0053] Figure 11 is a schematic diagram of the overall architecture of a virtual environment data processing system provided by another embodiment of the present application.

[0054] Figure 12 is a flowchart of placement object position prediction and optimization in a virtual environment data method provided by another embodiment of the present application.

[0055] Figure 13 is a flowchart of generating a scene from zero in a virtual environment data method provided by another embodiment of the present application.

[0056] Figure 14 is a schematic diagram of the hardware structure of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0057] ​In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0058] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.

[0060] Virtual reality (VR) technology creates a physically realistic and credible experience for users in a virtual environment by providing an immersive interactive way. In the prior art, the creation of a virtual environment scene mainly relies on manual operation by professional software or automatic layout through procedural content generation (PCG) and artificial intelligence (AI) driven methods. Among them, some emerging AI generation methods can directly output a complete three-dimensional scene layout scheme according to text or image prompts using large models. This kind of method is usually a "one-click" black box process, that is, after the AI receives high-level instructions, it will predict and generate an ideal object arrangement result that it considers to be the best, thereby achieving a certain degree of automation of scene generation.

[0061] However, the layout result generated by the above-mentioned AI driven scene generation method is only an ideal prediction based on data and logic, which cannot completely guarantee the physical reality and semantic reasonableness of the final scene, resulting in problems such as mutual penetration, suspension or placement position not conforming to physical common sense and functional logic between objects in the scene, which makes the reliability of the generated virtual environment data low and seriously damages the physical realism of the virtual scene and the immersive experience of the user.

[0062] To improve the reliability of the generated virtual environment data, this application combines intelligent prediction of a visual language model with subsequent multi-constraint iterative optimization. This effectively solves the two major technical problems of physical unreliability and uncontrollable interaction caused by the 'black box' mode in existing AI scene generation technologies. The solution provided in this application does not take the idealized layout output by AI as the final result. Instead, it combines intelligent prediction of a visual language model, multi-constraint physical iterative optimization, and user-led control within the loop to ensure that the final generated virtual environment data not only strictly adheres to physical laws in microscopic details but also achieves [the desired effect] in the macroscopic creation process. It achieves an efficient and transparent human-machine collaboration process. During the iteration process, an iterative physics optimization engine continuously corrects the pose of objects in the scene based on various physical placement constraints such as horizontal collision, vertical support, and room boundaries until a stable state is reached. This process can automatically detect and repair physical errors in AI layout such as clipping, suspension, and insufficient support. Furthermore, it achieves efficient and transparent human-machine collaboration in the macro-creation process, transforming the creation process from a rigid 'instruction-execution' mode to a flexible 'dialogue-collaboration' mode, thereby significantly improving the reliability of scene data, physical realism, and the user's immersive creation experience.

[0063] The following describes the virtual environment data processing method, electronic device, and storage medium provided in the embodiments of this application. The virtual environment data processing method provided in the embodiments of this application is based on a client-server architecture. The client (VR application) is responsible for user interaction and scene rendering, while the server is responsible for core computing. This method can be applied to any server or computing processor with computing resources.

[0064] The virtual environment data processing method in the embodiments of this application will be described in detail below. (Refer to...) Figure 1 This is an optional flowchart of the virtual environment data processing method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 105. It is also understood that this embodiment... Figure 1 The order of steps 101 to 105 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0065] Step 101: Respond to the user's virtual data control commands in the virtual environment scene.

[0066] Step 101 will be described in detail below.

[0067] In some embodiments, the starting point of the entire data processing flow is the virtual data control instruction issued by the user in the virtual environment scene (i.e., in the VR environment). Here, the virtual data control instruction refers to an input signal generated by the user within the immersive virtual environment scene to express his creative intention through natural interaction methods such as voice instructions or spatial pointing, grabbing, etc. of VR handles (such as handle rays). This instruction is the basis for triggering all subsequent scene updates and layout generation operations.

[0068] The system first identifies and classifies the user's virtual data control instruction by calling a large language model that supports audio input, mainly into four intentions: Retrieve, the user wants to obtain a new object model; Place, the user confirms to place the retrieved object; Generate, the user wants to generate a complete scene from scratch or according to a theme; Reply, the user asks the system questions for conversational interaction.

[0069] When the "Retrieve" intention is identified, the system retrieves the most matching model in the three-dimensional model library according to the object name described by the user's voice (such as "a wooden table") and the style information of the current scene, through a picture-text matching model (such as CLIP). The specific process is described as follows.

[0070] Preparation of three-dimensional model library: The system uses 3D-Front as its three-dimensional model library, which contains more than 16000 objects, each of which has a corresponding model front view. Before the system starts running, the encoded vector of each model preview and the storage location of the model are stored in the Faiss vector database and a json file containing storage location information, which are used for subsequent picture-text matching retrieval.

[0071] Style prompt enhancement: Combine the object name in the object description (such as "a wooden table") with the style information of the current scene to form an object description with style information (such as "a wooden table with Chinese style"), which is used to retrieve object models with the same style as the existing scene.

[0072] Picture-text matching retrieval: Transmit the text information enhanced by the style prompt to the model retrieval server, which uses the CLIP model to vectorize and encode the text information enhanced by the style prompt, and uses the similarity calculation of the Faiss vector database to find the model preview with the highest similarity to the encoded vector of the current text information, and obtains the model related information.

[0073] Store to model cache list: after obtaining the model related information, the URL corresponding to the model compression package, the model name, the model overhead view, and the pointing position provided by the handle ray when the user issues the instruction are stored to a temporary model cache list. This step can be repeatedly executed to batch receive multiple objects.

[0074] Step 102: When the virtual data control instruction represents updating the virtual environment scene, obtain the placement object information of the to-be-placed object and the structural feature information and visual information of the virtual environment scene, and input the placement object information, the structural feature information and the visual information into a visual language model for data processing to obtain the optimized arrangement data of the to-be-placed object in the virtual environment scene.

[0075] The step 102 is described in detail below.

[0076] In some embodiments, when the system parses the virtual data control instruction of the user and determines that the intention is to update the virtual environment scene (for example, “place” or “generate” a new object), the system will enter a multi-modal information preparation and processing phase. First, the system obtains the placement object information (such as three-dimensional model data) of the to-be-placed object itself. Among them, the three-dimensional model data mainly includes the obj model of the object, the name of the object, and the overhead view of the object in the empty scene with scales, wherein the scales are used to determine the scale of the object in the scene.

[0077] At the same time, the system will collect two types of key information of the current virtual environment scene: one is the structural feature information, which usually refers to a structured data file (such as Scene.json), which is used to accurately describe the identification, parent-child relationship and physical parameters of all existing objects in the scene; the second is visual information, that is, a real-time rendering image of the scene, such as a real-time overhead view of a bird's eye view. Among them, the overhead view adds a plane coordinate scale to indicate the plane coordinates of each location in the overhead view.

[0078] Subsequently, the system inputs the three types of information into a visual language model trained for a specific task, which can comprehensively understand the structural and visual context of the scene, thereby performing data processing and outputting an optimized arrangement data. This data is a structured parameter set that defines the initial position, orientation and spatial relationship constraints of the to-be-placed object in the current scene that best meets the logical and aesthetic considerations.

[0079] It can be understood that if the identified virtual data control instruction is a "generate" intent, the system calls a large language model (i.e., a visual language model) to generate the overall plan of the scene. The model is given the role of a "VR scene architect" and combines retrieval-augmented generation (RAG) technology to retrieve relevant design knowledge from a preset layout rule knowledge base, and then outputs a detailed scene plan strictly following the JSON format according to the user's generation prompt word (such as "a modern style bedroom"), including a list of objects to be retrieved and text describing how to place these objects.

[0080] The following describes how to obtain placement object information of the to-be-placed object.

[0081] Referring to Figure 2 , the placement object information of the to-be-placed object is obtained, including the following steps 201 to 204.

[0082] Step 201: determining the object identifier of the to-be-placed object from the virtual data control instruction.

[0083] Step 202: obtaining scene style information of the virtual environment scene.

[0084] Step 203: generating style object retrieval information based on the object identifier and the scene style information.

[0085] Step 204: matching in a three-dimensional model library based on the style object retrieval information to obtain the placement object information.

[0086] The following describes steps 201 to 204 in detail.

[0087] In some embodiments, when the system parses the user's virtual data control instruction and determines that the intent is to update the virtual environment scene (for example, "place" or "generate" a new object), the system first parses and extracts key information from the user's virtual data control instruction. The core of this step is to determine the object identifier of the to-be-placed object, which specifically refers to the object name described by the user in natural language, such as "a wooden table" contained in the voice instruction "give me a wooden table". This object identifier is the most original and basic query basis for the subsequent model retrieval process.

[0088] Next, similar to the "retrieve" instruction described above, in order to ensure that the newly added object can be aesthetically consistent with the existing environment, the system needs to obtain the scene style information of the current virtual environment scene. This information is a variable that describes the overall aesthetic characteristics of the scene, such as "modern minimalist" or "Chinese style", which can be obtained by a visual language model analyzing the current scene and used as an important context parameter to guide the subsequent model retrieval.

[0089] Then, the system fuses the information obtained in the previous two steps to generate a more accurate and contextually aware search instruction. Specifically, the system combines the object identification with the scene style information to generate style object search information. For example, if the object identification is "a wooden table" and the scene style information is "Chinese style", the generated style object search information is "a wooden table with Chinese style", which will be the final query text input into the search model.

[0090] Then, the system uses the style object search information generated in the previous step to perform a matching operation in the pre-prepared three-dimensional model library. This matching process vectorizes the text information through a text-image matching model (such as the CLIP model), and then performs similarity calculation in a vector database (such as Faiss) to find the most matching model preview image to the text description, and thus obtains the specific data of the three-dimensional model. The final placement object information is a structured data containing the URL of the model compression package, the model name, etc., which provides all necessary data support for subsequent instantiation of the object in the scene.

[0091] Through the above steps 201 to 204, the above steps collectively constitute an intelligent and contextually aware model search method. The beneficial effects brought by this method are that it combines the user's direct intention (object identification) with the global environment of the scene (scene style information), significantly improving the accuracy and relevance of model search. Compared with simple keyword matching, this method can ensure that the retrieved object is highly coordinated and unified in style with the existing scene, thereby ensuring the overall aesthetic quality of the virtual environment at the source of creation and improving the intelligence of automatic layout and the harmony of the final effect.

[0092] Next, the client sends the scene information (JSON and overhead view) along with the object information in the object cache area (or the object list and description planned by the "generated" intention) to the visual language model in the backend. This model is assigned the role of "professional 3D room layout assistant" and adopts different working modes for corresponding processing according to whether the user provides "prior position" or not, as described below.

[0093] Referring to Figure 3 The structural feature information, the structural feature information, and the visual information are input into the visual language model for data processing to obtain optimized placement data of the object to be placed in the virtual environment scene, including the following steps 301 to 302.

[0094] Step 301: When the virtual data control instruction includes reference position information of the object to be placed, input the reference position information, the object to be placed information, the structural feature information and the visual information into the visual language model for data processing to obtain the optimized arrangement data near the reference position information.

[0095] Step 302: When the virtual data control instruction does not include reference position information of the object to be placed, input the object to be placed information, the structural feature information and the visual information into the visual language model for data processing to obtain the optimized arrangement data in the virtual environment scene that conforms to the position logic.

[0096] The steps 301 to 302 are described in detail below.

[0097] In some embodiments, when the user provides an explicit spatial guidance intention (the user provides reference position information), the reference position information here refers to a specific coordinate or a general area specified in the virtual environment scene by the user through the ray pointing of the VR handle, etc. as the prior position of the object to be placed. The model is instructed that its primary goal is to "place the object at the reference position or its vicinity", and on this basis, intelligent fine-tuning is performed, such as automatic adsorption to the support surface, collision avoidance, etc.

[0098] When the user's virtual data control instruction contains this reference position information, the system will provide this information together with the structural feature information and visual information of the current scene as input to the visual language model for data processing. In this mode, the visual language model is explicitly instructed that its primary task is to arrange the object to be placed at the position indicated by the reference position information or in the immediate vicinity of the reasonable range, and on this basis, intelligent fine-tuning is performed, such as automatic adsorption to the support surface or collision avoidance with other objects, to finally obtain the optimized arrangement data near the reference position information.

[0099] When the user does not provide specific spatial guidance, but completely entrusts the layout decision to the artificial intelligence creation, the model is instructed to "must decide the most logical and aesthetic position for the new object without any user prompt", thereby having greater creative freedom.

[0100] When the user's virtual data control instruction does not include reference position information of the to-be-placed object, the system only inputs the structural feature information and visual information of the scene into the visual language model for data processing. In this reference-free working mode, the visual language model is given greater freedom of creation. It is instructed to comprehensively analyze the existing layout, functional partition and aesthetic style of the scene without any user prompt, and autonomously determine the position where the new object is most consistent with the logic and aesthetic principles. Therefore, the optimized arrangement data finally output by the model is a result generated by its internal knowledge in the virtual environment scene, which conforms to the position logic.

[0101] Through the above steps 301 to 302, by providing two different working modes, an efficient human-computer collaborative creation paradigm is constructed, which embodies the core concept of "man-in-the-loop". When the user has a clear intention, the AI can accurately execute the user's spatial guidance, and the user's final control right is reserved. When the user wants to obtain creative suggestions or carry out rapid layout, the AI can play the role of an automatic designer, and fully exert the advantages of intelligent planning. This dual-mode mechanism enables the method to adapt to diversified creation needs, and achieves an ideal balance between user-led fine control and AI-driven automatic generation.

[0102] All the above AI models are guided by carefully designed System Prompts. These prompts set clear roles, tasks, input-output formats, and logical rules (such as "large furniture must be placed on the floor" and "parent object ID is only used for physical support relationships") for the models. Through this prompt engineering, general AI models are transformed into professional scene design tools, ensuring the stability and professionalism of their output. The model ultimately predicts the best parameters for each object to be placed in the scene and returns them in a structured JSON format; this JSON format defines the predicted output of the AI model, which includes the function instruction: batch_predictions, a list containing the prediction information for all objects to be placed; predicted_object, which contains the core physical parameters of an object; objectId, the unique string identifier of the object; parentId, the objectId of the parent object that supports the object (e.g., "floor" for the floor); position, the two-dimensional coordinates [x, z] of the object in the XZ plane; rotation, the rotation angle [x, y, z] of the object; scale, the scaling ratio [x, y, z] of the object; object_name, the natural language description of the object; point_towards, spatial relationship constraints that define the objectId of another object that the object needs to face; against_wall, spatial relationship constraints that define whether the object needs to be against a wall; adjacent, spatial relationship constraints that define the distance (distance) that the object needs to maintain with another object (target).

[0103] Step 103: Generating the to-be-placed object in the virtual environment scene based on the optimized arrangement data, to obtain an adjusted virtual scene.

[0104] The following describes step 103 in detail.

[0105] In some embodiments, the system will load and preliminarily place the to-be-placed object in the virtual environment scene based on the optimized arrangement data generated by the visual language model in the previous step. The scene obtained after this process is complete is called the adjusted virtual scene. It should be noted that this adjusted virtual scene is a temporary, intermediate state scene, which reflects the intelligent prediction of the artificial intelligence model for object layout, but has not yet undergone strict physical law verification, and therefore may have potential physical irrationality problems such as object penetration and suspension.

[0106] During the generation process, the system topologically sorts the objects based on the parent-child dependencies in the prediction results, ensuring that the supporters (parent objects) are loaded before the supported objects (child objects). Subsequently, all object models are loaded asynchronously in order to their initial positions predicted by the AI, and a data structure containing the physical and constraint information of each object to be optimized is created.

[0107] Step 104: Based on the plurality of physical placement constraints, a pose adjustment amount of the to-be-placed object in the adjusted virtual scene is calculated, and the to-be-placed object in the adjusted virtual scene is pose corrected based on the pose adjustment amount to obtain an updated virtual scene. The updated virtual scene is taken as a new adjusted virtual scene, and the to-be-placed object in the adjusted virtual scene is subjected to at least one iteration of pose correction.

[0108] The following describes step 104 in detail.

[0109] In some embodiments, after generating the initial adjusted virtual scene, the system starts a core, iteration-based physical optimization process on the adjusted virtual scene. This process is based on a plurality of physical placement constraints, which are a series of preset rules simulating the physical laws and spatial logic in the real world, such as seven categories of constraints, including horizontal collision avoidance, vertical support relationship, room boundary limitation, object proximity and orientation relationship, etc. In each iteration, the system traverses the to-be-placed objects and checks whether they violate any of the above constraints. If there is a violation, the system calculates a pose adjustment amount for correction, which is specifically manifested as a “push” or a rotation operation applied to the object. The system then pose corrects the object based on this pose adjustment amount, thereby obtaining an updated virtual scene with improved physical rationality. The key point is that this process is not completed at once. The system takes the updated virtual scene as the input of the next iteration (i.e., the new adjusted virtual scene) and repeats the constraint checking and pose correction, which is the iteration of pose correction. This cycle continues until there are no more constraint conflicts in the scene, or the maximum number of iterations is reached.

[0110] Before performing the iteration of pose correction, optimization data of the to-be-optimized objects in the scene need to be prepared, including three-dimensional bounding boxes, two-dimensional bounding boxes, etc. After preparation, a state iteration loop (e.g., up to 400 rounds) is entered.

[0111] Next, in each iteration, all to-be-optimized objects (i.e., to-be-placed objects) are traversed, and a resultant force is calculated for each to-be-placed object according to the seven different physical placement constraints. Each violation of a constraint is quantified as a pose adjustment amount (i.e., a push or a rotation operation) applied to the object, which is described as follows.

[0112] Reference Figure 4, based on a plurality of physical placement constraints, a pose adjustment amount of the to-be-placed object in the adjusted virtual scene is calculated, including the following steps 401 to 404.

[0113] Step 401: Based on each physical placement constraint, the placement state of the to-be-placed object in the adjusted virtual scene is matched one by one to obtain a state matching result.

[0114] Step 402: When the state matching result indicates that the to-be-placed object has a position error, an adjustment thrust is generated based on the position error information.

[0115] Step 403: When the state matching result indicates that the to-be-placed object has an orientation error, an adjustment rotation is generated based on the orientation error information.

[0116] Step 404: Based on the adjustment thrust and / or the adjustment rotation, the pose adjustment amount is obtained.

[0117] The steps 401 to 404 are described in detail below.

[0118] In some embodiments, the system will conduct a comprehensive state evaluation of the to-be-placed object in the adjusted virtual scene at the beginning of each round of iterative optimization. The physical placement constraints here refer to a series of pre-defined rules to ensure the physical authenticity and semantic reasonableness of the scene, such as horizontal collision, vertical support, room boundary, proximity relationship, etc. The system will measure the current state of the object with each physical placement constraint one by one, which is the placement state matching. The result of the placement state matching, i.e. the state matching result, will explicitly indicate whether the object currently violates a specific constraint rule and quantify the degree of violation (such as collision depth, distance beyond the boundary, etc.), as described below.

[0119] Horizontal collision optimization: Detect whether the two-dimensional bounding boxes (Rect) of objects on the same support plane overlap. The two-dimensional bounding box is a rectangle obtained by orthogonally projecting the complete three-dimensional bounding box (Bounds) of the object to the horizontal (XZ) plane. If they overlap, calculate a mutually repulsive thrust along the direction of the center of gravity of the two object bounding boxes according to the overlap depth to push the objects apart until the minimum separation distance is met.

[0120] Vertical collision optimization: Detect whether the bottom of the three-dimensional bounding box (Bounds) of an object penetrates the top of its support (parent object). If it penetrates, calculate an upward thrust according to the penetration depth to push the object back to a very small safety distance above the support surface.

[0121] Room boundary optimization: Detect whether the bounding box of an object exceeds the preset room boundary. If it exceeds, calculate a thrust to push it back within the boundary to prevent the object from penetrating the wall or being placed outside the scene.

[0122] Support optimization: Calculate the proportion of the projection area of the supported object and the support in the XZ plane. If the proportion is lower than a preset threshold (e.g., 90%), calculate a centripetal force pointing to the center of the support to prevent the object from "hanging" on the edge and ensure its physical stability.

[0123] Orientation optimization: This is a rotation operation. According to the point_towards constraint predicted by AI, calculate the rotation angle required to make the object face the target, and apply a small part of the rotation in each iteration to achieve smooth steering. The rotation is performed around the bounding box center of the object, avoiding displacement caused by different model anchor points.

[0124] Wall optimization: This is a combination of position and rotation operations. The rotation part is completed once at the creation of the object, making the object face away from the wall by ray detection of the wall normal. The position part calculates a force to push the object towards the wall in the iteration until the distance to the wall is within the preset wall distance tolerance.

[0125] Adjacent optimization: According to the adjacent constraint predicted by AI, calculate the difference between the current distance and the target distance between objects. If the distance is too far, generate an attractive horizontal push force; if the distance is too close, generate a repulsive horizontal push force until the distance between objects is within the target distance tolerance.

[0126] When the state matching result of the previous step indicates that the object to be placed has a position error, the system will start a correction mechanism. Here, the position error refers to any situation that violates the physical placement constraints related to spatial position, such as object overlap, object exceeding the room boundary, or object not being stably placed on its support. Based on the position error information (such as overlap depth, penetration distance, etc.) contained in the state matching result, the system will calculate an adjustment force vector through a preset algorithm. The direction and size of the adjustment force are carefully designed to push the object in a direction that can reduce or eliminate the position error.

[0127] Similarly, when the state matching result indicates that the object to be placed has an orientation error, the system will also handle it accordingly. Here, the orientation error specifically refers to the object's rotation pose not meeting specific semantic constraints, such as the AI-predicted constraint "a chair needs to face the table" not being met. Based on the orientation error information (such as the angle difference between the current orientation and the target orientation), the system will calculate an adjustment rotation to correct the direction. This adjustment rotation is a specific rotation operation parameter used to guide the object to smoothly turn to its preset target direction.

[0128] Then, the system will integrate all the correction amounts calculated for one object in a single iteration. Specifically, the system will accumulate all the adjustment thrusts generated due to different position errors to form a resultant force, and combine the adjustment rotations generated due to orientation errors, to finally obtain a comprehensive pose adjustment amount. This pose adjustment amount is the sum of all the pose transformations that the object needs to perform in this iteration, which contains both the translation in position and the rotation in orientation, and provides an accurate execution basis for subsequent unified pose correction of the object.

[0129] Through the above steps 401 to 404, a systematic and modular pose adjustment amount calculation method is formed, which decomposes the complex physical optimization problem into a clear process of "state matching" and "type-specific error handling" (position and orientation), so that the system can clearly identify and quantify all constraint violations. By generating adjustment thrusts and adjustment rotations for different types of errors, and finally unifying them into a comprehensive pose adjustment amount, this method ensures the accuracy, stability and controllability of the optimization process, and can handle multiple complex constraint conflicts simultaneously, thereby efficiently guiding the virtual scene to converge to a physically completely reasonable stable state.

[0130] In each iteration, all the horizontal thrusts calculated are accumulated and applied uniformly to the object to update its position in the XZ plane. Vertical thrusts and rotation operations are handled independently.

[0131] Finally, the boundary information of all objects is updated, and it is checked whether the scene has reached a stable state (i.e., no object generates thrust or rotation). If the scene is stable or the maximum number of iterations is reached, the loop terminates.

[0132] Step 105: Take the updated virtual scene obtained after the last iteration of pose correction as the target virtual scene, and display the target virtual scene or respond to user modification operations.

[0133] The following describes step 105 in detail.

[0134] In some embodiments, after the above pose iteration correction process terminates, the system performs a final physical reasonableness check and determines the updated virtual scene obtained after the last iteration, which is stable and complies with all physical constraints, as the target virtual scene. The target virtual scene is the final output of the entire data processing method, which is a high-quality and reliable virtual environment that not only reflects the intelligent layout of the artificial intelligence model, but also has been strictly verified by physical laws.

[0135] Then the target virtual scene is displayed to the user for viewing, or is adaptively modified in response to user modification operations.

[0136] In addition, after the optimization process, the system performs a final physical constraint verification on each object in the scene, checking whether it still has serious horizontal collision, insufficient support, etc. Any object that does not meet the basic physical rationality will be automatically removed from the scene.

[0137] Then update and record the scene state: after the final physical rationality inspection and removal of unreasonable objects, the scene reaches the final stable state. At this time, the final physical parameters of all valid objects in the system will be formatted and updated to the Scene.json file. This file has a dual key role including: as a structured input for the AI model, the file records the precise state of all objects in the scene, which will be structured data, and the scene top-down view Figure 1 is input into the visual language model. This enables the AI model to better integrate visual and spatial logic, providing accurate context judgment for subsequent scene summary generation, intelligent question answering, and new object layout prediction; as a dynamic object list for the client, the file clearly identifies all objects added by the user through this method. This enables the client application (such as the system built in Unity) to distinguish these "added objects" from the inherent game objects in the scene, so that when performing "delete" or "all delete" and other interactive operations, it can accurately operate only on the user-added objects.

[0138] The Scene.json file is a JSON array, and each object in the array represents a valid object in the scene, including: objectId, the unique string identifier of the object; parentId, the objectId of the parent object supporting the object; position, the final two-dimensional coordinate [x, z] of the object after optimization in the XZ plane; rotation, the final rotation angle [x, y, z] of the object after optimization; scale, the scaling ratio [x, y, z] of the object.

[0139] Reference Figure 5 is an adjustment and optimization flowchart of a multi-physical placement constraint optimization algorithm provided by an embodiment of the present application. As Figure 5The diagram shows a detailed flowchart of the multi-constraint optimization engine provided in this application, illustrating the core algorithm for transforming AI-predicted idealized layouts into physically realistic layouts. The process includes: preparing optimization data, starting with receiving the AI-predicted layout results and creating an optimization object containing the physical and constraint information for each object to be placed, while simultaneously calculating the bounding box information for each object for subsequent optimization calculations; initiating optimization and iteration loops, entering an iteration loop. At the beginning of each iteration, all thrusts are first reset; all thrusts are calculated in parallel, which is the core of the optimization. The system calculates thrusts generated by seven different constraints in parallel, including horizontal collision thrusts, vertical support thrusts, wall thrusts, proximity thrusts, and boundary thrusts. All these thrusts are accumulated to form a resultant force acting on the object; application and updating, applying the calculated resultant force to all thrusts and updating its position. Simultaneously, rotations caused by orientation optimization are handled independently. After the transformation is completed, object information (such as bounding boxes) is updated to prepare for the next iteration; convergence or reaching the maximum loop, checking whether the scene has converged (i.e., all thrusts approach zero). If stability is achieved, or the number of iterations reaches the preset limit, the optimization is complete. Physical constraint check and termination: After optimization, the system performs a final physical constraint check. If all objects meet the constraints, the process ends; if any objects do not meet the conditions, these unreasonable objects are deleted to ensure the stability and realism of the final output scene.

[0140] Reference Figure 6 This is a schematic diagram illustrating the preparation of optimized data provided in an embodiment of this application. Figure 6 As shown in the attached diagram, this figure visually illustrates how a 3D object is simplified into a bounding box for various physics calculations. First, for each 3D object, the system obtains its complete 3D bounding box by rendering a mesh, as shown in the middle image. This 3D bounding box will be used to calculate vertical collisions and constraints requiring height information for support relationships. Second, by orthogonally projecting the 3D bounding box onto the horizontal (XZ) plane, a 2D bounding box (Rect) is obtained, as shown in the right image. This 2D bounding box is specifically used for efficiently calculating planar constraints such as horizontal collisions and proximity distances. This data preparation process from 3D to 2D is a core method for balancing complex physics simulations with computational efficiency.

[0141] Reference Figure 7 The virtual environment data processing method also includes the following steps 701 to 704.

[0142] Step 701: Analyze and process the target virtual scene to obtain scene summary information of the target virtual scene.

[0143] Step 702: Respond to the user's query information.

[0144] Step 703: Retrieving matching in the layout rule knowledge base based on the query information and the scene summary information, to obtain the design rule information.

[0145] Step 704: Inputting the scene summary information, the query information, and the design rule information into the question and answer model, to obtain the reply result data corresponding to the query information.

[0146] The steps 701 to 704 are described in detail as follows.

[0147] In some embodiments, after the scene is updated, i.e., after a target virtual scene is generated, the system starts a scene understanding process. The system calls a multi-modal visual language model for analysis to comprehensively analyze the visual information (such as an overhead view) and structured data (such as Scene.json) of the current scene, to obtain a “style” variable and a “scene summary”. The “style” variable (selected from a pre-defined style list, such as “modern minimalist”) is used for subsequent model retrieval, to ensure style consistency; the “scene summary” is used as core context information, which is input into the intelligent question and answer system in the next step (such as “the room is a dining room scene, there is a black dining table in the center of the dining room, there are four silver metal chairs around the dining table, and there is a table lamp beside the dining table, which illuminates the central area including the dining table and chairs. There is a white sofa in the corner of the room, and there is a tea table beside the sofa”). This summary information is a descriptive natural language text, which accurately summarizes the layout, style, and spatial relationship of the core objects of the current scene, to provide a key context for subsequent intelligent interaction.

[0148] Then, the system continuously listens to and analyzes the user's virtual data control instructions, and when it identifies that the user's intention is “reply”, it means that the user is asking a question, and the system will respond to the user's query information. The query information here is a specific question asked by the user in the form of natural language, such as “this room looks a bit empty, what should I add?”. This step is the starting point of the entire context-aware question and answer and execution process.

[0149] To make the model's answers more professional and targeted, the system employs a Retrieval Augmented Generation (RAG) technique. The system combines the user's inquiry information and the generated scene summary information as query conditions for semantic retrieval matching in a pre-set layout rule knowledge base (vector database). This knowledge base is a vector database that stores a large amount of professional interior design principles and space layout examples. The most relevant knowledge fragments obtained after retrieval matching are the design rule information, which will be injected as important external knowledge into the next step of the language model. These retrieved contents will be provided as context to the AI model, making its answers more professional and accurate.

[0150] Then, the system will integrate all the context information and call the question and answer model to generate the final reply. Specifically, the system will provide the scene summary information, user inquiry information, and retrieved design rule information as input to the question and answer model. After receiving this comprehensive context, the model will generate a structured reply result data. This data is usually a JSON object that not only contains the text answer to the user's question in natural language, but also may contain scene modification instructions that can be directly executed based on the dialogue content.

[0151] Among them, referring to Figure 8 The scene summary information, inquiry information, and design rule information are input into the question and answer model to obtain the reply result data corresponding to the inquiry information, including steps 801 to 802.

[0152] Step 801: Confirm the target structure feature information and target visual information of the target virtual scene.

[0153] Step 802: Input the scene summary information, inquiry information, design rule information, target structure feature information, and target visual information into the question and answer model to obtain the reply result data corresponding to the inquiry information.

[0154] Steps 801 to 802 are described in detail below.

[0155] To provide the question and answer model with the most comprehensive and accurate scene context, the system will confirm the underlying data of the target virtual scene before generating the final reply. This step aims to obtain two types of core original scene data: first, the target structure feature information, which specifically refers to a complete structured data file (Scene.json) that records the precise physical parameters and parent-child relationships of all effective objects in the scene; second, the target visual information, which usually refers to real-time images that can intuitively reflect the overall layout of the current scene, such as a top-down view of the scene. These two pieces of information together constitute the "ground truth data" about the state of the target virtual scene.

[0156] After that, all the information prepared in the previous process, including high-level scene summary information, user's direct inquiry information, external design rule information retrieved from the layout rule knowledge base, and confirmed underlying target structure feature information and target visual information, are packaged together to form a comprehensive information package. Subsequently, this complete context information package containing multi-level, multi-modal data is input into the question and answer model, and the final reasoning and generation are performed by the model, so as to obtain the reply result data corresponding to the inquiry information.

[0157] Through the above steps 801 to 802, by constructing an information-rich, data-modal comprehensive input, an unprecedented depth context is provided for the question and answer model, ensuring that the model's decision is not only based on high-level text summary and external knowledge, but also anchored by the scene's underlying structured data and real-time visual information. This multi-level, multi-modal context aggregation strategy greatly improves the depth and accuracy of the model's understanding of the scene state, effectively reducing the risk of the model producing "illusion" replies that do not conform to the actual scene, thereby ensuring that the generated reply result data has high reliability, accuracy and scene relevance.

[0158] The system instructs the question and answer model to output a strict JSON format object through a precise prompting process, rather than a simple text. The JSON object contains three key fields: "answer" (a segment of natural language Chinese answer), "add_objects" (a Boolean value indicating whether the user's question implies the intention to add objects), and "objects" (a list of names of objects to be added).

[0159] After that, the client parses the returned JSON. The system plays the content of the "answer" field to the user through a text-to-speech (TTS) module.

[0160] Reference Figure 9 The virtual environment data processing method further includes the following steps 901 to 902.

[0161] Step 901: When the reply result data includes new object information of a new placed object, determine new placement data of the new placed object in the target virtual scene based on the new object information.

[0162] Step 902: Generate the new placed object in the target virtual scene based on the new placement data.

[0163] The steps 901 to 902 are described in detail as follows.

[0164] In some embodiments, this step deals with the specific situation where the answer model includes executable creation instructions in the reply. The reply result data is the structured information output by the answer model, and when this data contains new object information, which usually refers to an explicit Boolean flag (i.e., "add_objects" is true) and a list of object names to be added, the system triggers an automated scene addition process. Based on this new object information, the system calls its core intelligent layout prediction module in the context of the current target virtual scene. This module calculates the optimal new placement data for each new object to be added, including the position, orientation, and other precise physical parameters required for the new object in the scene.

[0165] After obtaining the new placement data calculated for all new placement objects, the system will perform instance update of the scene. Based on the new placement data, this step loads and generates three-dimensional models of the corresponding new placement objects in the target virtual scene. These newly generated objects are initially placed at the positions specified by the new placement data and then seamlessly integrated into the scene, usually after subsequent multi-constrained physical optimization processes to ensure that their relationships with existing objects in the scene also meet the physical reality requirements.

[0166] Through steps 901 to 902 above, the intelligent question answering and scene creation are connected, building a complete interactive closed loop from "dialogue understanding" to "scene execution", which makes the question answering system not only a passive information query tool, but also an intelligent assistant that can actively participate in and execute creation tasks. Users can seamlessly propose and implement scene modifications and content additions during natural language dialogue with the system, greatly improving the smoothness, intuitiveness, and efficiency of the creation process, thereby realizing a more advanced and deep human-computer collaborative interaction mode.

[0167] Through steps 701 to 704 above, an intelligent question answering system with deep scene understanding and professional knowledge enhancement is built, making the interaction not a simple instruction response, but an intelligent dialogue with context awareness. By combining real-time scene summaries and external professional design knowledge bases, the system can not only "understand" the current scene and accurately answer user questions, but also provide professional advice that meets design principles. Further, it can understand and execute user creation instructions from the dialogue, forming a complete intelligent interactive closed loop from analysis, understanding to execution, greatly improving the intelligent level and interaction depth of human-computer collaborative creation.

[0168] In addition, with reference to Figure 10 , the virtual environment data processing method further includes steps 1001 to 1003.

[0169] Step 1001: In response to the user's direct modification instruction on the target object in the target virtual scene.

[0170] Step 1002: Adjust the target object in the target virtual scene based on the direct modification instruction.

[0171] Step 1003: Generate and save updated scene information based on the adjusted target virtual scene.

[0172] The steps 1001 to 1003 are described in detail below.

[0173] In some embodiments, after calling the large language model and the visual language model to modify the scene, the user can use the handle grab key to modify the objects in the existing scene. When the user makes more detailed and subjective creative adjustments based on the target virtual scene generated by AI, the system will respond accordingly. The direct modification instruction here is a bottom-layer interaction command that is different from the high-level voice instruction. It usually refers to the user's physical operation through the grab key of the VR handle to directly select and act on a specific target object in the scene to initiate a manual modification intention, ensuring that the user still has the final editing right of any element in the scene after the automatic layout process.

[0174] After receiving the user's direct modification instruction, the system will make corresponding adjustments to the selected target object based on the instruction. These adjustment operations are completely user-directed and can include: changing the spatial position of the target object in the scene through physical movement of the handle; adjusting the orientation of the target object through rotation of the handle; or executing a delete instruction to completely remove the target object from the scene. This step realizes the function of manually fine-tuning or correcting the AI layout result.

[0175] In step 903 of some embodiments, when the user completes the manual adjustment of the target object, the system will start a background scene information synchronization process. This process will actively collect the latest state of the modified object in the scene, including its final position, orientation, and support relationship (i.e., using a ray to extend from the center of each object downward to collect the parent object information of the supporting object), and generate an updated scene information based on these adjusted information. This updated scene information will then be written and saved to the structured data file (Scene.json) of the scene, ensuring that any manual modification by the user is recorded permanently, facilitating further AI prediction and placement operations based on the user's modification of the scene.

[0176] Through the above steps 1001 to step 1003, a truly "man-in-the-loop" human-computer collaborative creation mode is constructed, by giving the user the ability to directly intervene manually after AI automatic generation, ensuring the user's ultimate creative dominance and control, the user can use AI to complete the tedious batch layout work, and then adjust it manually to fine-tune the art individually, more importantly, the automatic update and save mechanism of the scene information ensures that the user's "manual modification" and the AI's "automatic creation" can seamlessly link and alternate in the same scene, forming a hybrid interactive process that is efficient, flexible and sustainable iteration of creative achievements.

[0177] With reference to Figure 11 Fig. 1 is a schematic diagram of the overall architecture of a virtual environment data processing system provided by an embodiment of the present application. As shown in Fig. 1, the virtual environment data processing system includes a virtual environment data processing system 100, a virtual environment data processing system 200, a virtual environment data processing system 300, a virtual environment data processing system 400, a virtual environment data processing system 500, a virtual environment data processing system 600, a virtual environment data processing system 700, a virtual environment data processing system 800, a virtual environment data processing system 900, and a virtual environment data processing system 1000. Figure 11As shown in the figure, an intelligent agent system with intent recognition as the core scheduler is demonstrated. The system is a typical client-server architecture and combines multiple AI models to form an agent system. It includes: a user input layer, in which the user inputs information through voice audio instructions and handle ray pointing information in the VR environment; an intent routing layer, in which the intent recognition module serves as the overall scheduling of the system, calls a large language model supporting audio input to parse the user's voice, and routes the intent to one of the four main workflows, namely retrieve, place, generate or reply; the scene generation workflow includes: retrieve, the instruction triggers the model retrieval service in the model retrieval server, which uses the CLIP model to match images and texts, and finds the object described by the user from the model library. The retrieved OBJ model compressed file is stored in the model cache list; generate, the instruction first triggers the planning module, which performs scene planning by calling a large language model, and outputs the object list and layout description; place, whether it is a "place" triggered manually by the user or a subsequent step of the "generate" workflow, it will eventually call the placement module or the generation module, which receives the cached object information and the current scene snapshot (Scene) as input by calling the visual language model, performs multi-modal reasoning, and outputs accurate layout prediction; the question and answer interaction workflow, reply, the instruction triggers the reply module, which receives the user's question and the scene summary information generated by the scene analysis module by calling the visual language model, and answers in context awareness. If the answer contains an instruction to add an object, it will further call the add object module to continue adding new objects to the scene. The architecture of the add object module is the same as that of the placement module; feedback, the text output of the reply module is converted into audio by the TTS (text-to-speech) service and played to the user; scene analysis and closed loop, the scene analysis module calls the visual language model to analyze the new scene whenever the scene changes, and extracts the style variables and scene summary information. The style information is fed back to the model retrieval server to ensure the consistency of the subsequent retrieval style; the summary information provides the context for the question and answer system, forming a continuously optimized closed loop.

[0178] Reference Figure 12 It is a flowchart of the placement of object position prediction and optimization in a virtual environment data method provided by the embodiment of the present application. As shown in the figure, Figure 12The diagram illustrates the complete object position prediction and optimization algorithm flow of this application. This flow is the core of the invention's achievement of physical realism and semantic rationality in scene layout generation. The flow includes: Input: The flow receives two core pieces of information: object model cache information, including the name of the object to be placed, its top view, and the user-selectable prior position; and existing scene information, including the scene's JSON file and top view. AI Prediction: After receiving these multimodal inputs, the visual language model batch-predicts the position and spatial constraints of each object, forming a formatted object layout representation output. Instantiation and Optimization: After the object is initially placed in the new scene, it immediately enters the physical and semantic optimization stage. This is a local computation process based on an iterative push model, applying seven constraints (such as collision, support, proximity, etc.) to gradually adjust the object's position and orientation in a loop of up to 400 rounds. Final Verification: After optimization, the system performs a final physical rationality check, verifying through the algorithm whether each object meets the most basic physical laws (such as no collision and sufficient support). Any object that does not meet the conditions will be automatically removed to ensure the stability and realism of the final output scene.

[0179] Reference Figure 13 This is a flowchart illustrating the process of generating a scene from scratch in a virtual environment data method provided in this application embodiment. For example... Figure 13 The diagram illustrates the scene generation process from scratch in this application, demonstrating the application of Retrieval-Enhanced Generation (RAG) technology. The process includes: In the planning phase, the user inputs a high-level generation instruction (e.g., "a restaurant"). The Large Language Model (LLM) receives this instruction, first vectorizing the instruction text information using a vector embedding model, and then performing a semantic search in a vector database to retrieve relevant layout rules. This vector database contains a complete layout design rule document. Some of the retrieved rules are dynamically injected into the LLM's system prompts, enabling it to generate a more professional and reasonable scene plan, including a list of object models and a target scene description. In the execution phase, the system automatically retrieves all models based on the generated model list and inputs their information, along with the scene description and an image of the current empty scene, into the visual language model. The visual language model, acting as the "layout executor," predicts the positions and constraints of all objects based on the detailed text description. The subsequent processes are optimized and output. Figure 12 The optimization and physical inspection process shown is the same, ultimately generating a complete new scene that conforms to the user's high-level intent and is physically realistic.

[0180] The virtual environment data processing method provided in the application constructs a "man-in-the-loop" VR creation mode with the user as the core: the application changes the one-way relationship between man and tool in traditional VR content creation, and establishes an efficient man-machine collaboration mode. In the VR environment, the user provides high-level creation intentions and key spatial references through the most natural voice instructions and handle spatial pointing, and the AI is responsible for performing tedious batch retrieval, layout calculation and detail optimization. At the same time, in the process of automatic layout creation, the user can still perform interactive moving and deleting operations on objects and the like. This "man-in-the-loop" mode not only retains the dominance and final control of the creator, but also seamlessly combines the creativity of man and the execution of AI, so that the user can focus on design itself rather than tool operation.

[0181] The virtual environment data processing method provided in the embodiment of the application improves the efficiency and quality of VR scene layout: through AI-driven intelligent planning and automatic layout, the application compresses the manual modeling and adjustment work that may take several hours in the traditional method to the minute level. At the same time, with the help of the AI model enhanced by professional knowledge (through RAG technology), the layout generated by the system not only has high speed, but also has high rationality in terms of spatial relationship, functional division and aesthetic principle, effectively avoiding the randomness and irrationality of traditional programmatic content generation.

[0182] The virtual environment data processing method provided in the embodiment of the application ensures the physical realism and immersion of the virtual layout: the unique multi-constraint optimization engine based on iterative thrust of the application is the key to ensuring the realism of VR experience. The engine can automatically correct physical errors such as collision, penetration and suspension between objects on the basis of the AI-generated layout, ensuring that each object in the scene follows the physical laws of the real world. This guarantee of physical realism greatly enhances the immersion and credibility of the user in the VR environment.

[0183] The virtual environment data processing method provided in the embodiment of the application realizes dynamic intelligent interaction with context understanding: the application is not only a static layout tool, but also an intelligent assistant that can "observe" and "understand" the scene. Through the innovative question-and-answer and execution closed loop, the system can understand the user's dialogue by combining the visual information (bird's eye view) and structured data (Scene.json) of the scene, and dynamically and intelligently modify the scene according to the dialogue content. This provides a deeper interaction mode for complex and dynamic content creation in the VR environment.

[0184] The scheme provided in the application utilizes "man-in-the-loop", that is, the user can improve and correct errors on the basis of AI generation, realizing a truly man-machine collaborative interaction mode rather than a rigid and one-way man-machine interaction mode, while maintaining physical semantic reasonableness and real-time during generation.

[0185] The embodiment of the present application further provides an electronic device, comprising:

[0186] at least one memory;

[0187] at least one processor;

[0188] at least one program;

[0189] The program is stored in the memory, and the processor executes the at least one program to realize the virtual environment data processing method provided in the embodiment of the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0190] Please refer to Figure 14 , Figure 14 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0191] The processor 1401 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to realize the technical solutions provided in the embodiments of the present application.

[0192] The memory 1402 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device or a RAM (Random Access Memory), etc. The memory 1402 can store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 1402 and are called and executed by the processor 1401 to realize the virtual environment data processing method of the embodiments of the present application.

[0193] The input / output interface 1403 is used to realize information input and output.

[0194] The communication interface 1404 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0195] A bus 1405 is used to transmit information between the various components (e.g., the processor 1401, the memory 1402, the input / output interface 1403, and the communication interface 1404) of the device.

[0196] The processor 1401, the memory 1402, the input / output interface 1403, and the communication interface 1404 are communicatively connected to each other within the device through the bus 1405.

[0197] The embodiment of the present application also provides a storage medium, which is a computer readable storage medium, and stores a computer program. The computer program is executed by a processor to implement the virtual environment data processing method.

[0198] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0199] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0200] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0201] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0202] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0203] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0204] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0205] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0206] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the application.

[0207] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0208] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0209] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of processing virtual environment data, characterized by, The method comprises: in response to a virtual data control instruction of a user in a virtual environment scene; when the virtual data control instruction represents updating the virtual environment scene, obtaining placement object information of a to-be-placed object, structural feature information and visual information of the virtual environment scene, and inputting the placement object information, the structural feature information and the visual information into a visual language model for data processing to obtain optimization arrangement data of the to-be-placed object in the virtual environment scene; generating the to-be-placed object in the virtual environment scene based on the optimization arrangement data to obtain an adjusted virtual scene; based on a plurality of physical placement constraints, calculating a pose adjustment amount of the to-be-placed object in the adjusted virtual scene, and based on the pose adjustment amount, correcting the pose of the to-be-placed object in the adjusted virtual scene to obtain an updated virtual scene, taking the updated virtual scene as a new adjusted virtual scene, and performing at least one iteration of pose correction on the to-be-placed object in the adjusted virtual scene; taking the updated virtual scene obtained after the last iteration of pose correction as a target virtual scene, and displaying the target virtual scene or responding to a user's modification operation; the calculation of the pose adjustment amount of the to-be-placed object in the adjusted virtual scene based on a plurality of physical placement constraints comprises: based on each of the physical placement constraints, performing placement state matching on the to-be-placed object in the adjusted virtual scene to obtain a state matching result; when the state matching result represents that the to-be-placed object has a position error, generating an adjustment thrust based on position error information; when the state matching result represents that the to-be-placed object has an orientation error, generating an adjustment rotation based on orientation error information; based on the adjustment thrust and / or the adjustment rotation, obtaining the pose adjustment amount; wherein, in each iteration of pose correction, all thrusts are calculated in parallel, and all thrusts are accumulated to form a resultant force acting on the to-be-placed object.

2. The virtual environment data processing method of claim 1, wherein, The obtaining of the placement object information of the to-be-placed object comprises: determining an object identifier of the to-be-placed object from the virtual data control instruction; obtaining scene style information of the virtual environment scene; generating style object retrieval information based on the object identifier and the scene style information; based on the style object retrieval information, performing matching in a three-dimensional model library to obtain the placement object information.

3. The virtual environment data processing method of claim 1, wherein, The inputting of the placement object information, the structural feature information and the visual information into the visual language model for data processing to obtain the optimization arrangement data of the to-be-placed object in the virtual environment scene comprises: when the virtual data control instruction includes reference position information of the to-be-placed object, inputting the reference position information, the placement object information, the structural feature information and the visual information into the visual language model for data processing to obtain the optimization arrangement data near the reference position information; When the virtual data control instruction does not include reference position information of the to-be-placed object, the placement object information, the structure feature information and the visual information are input into the visual language model for data processing to obtain the optimized arrangement data in the virtual environment scene in accordance with position logic.

4. The virtual environment data processing method of claim 1, wherein, The method further comprises: analyzing and processing the target virtual scene to obtain scene summary information of the target virtual scene; in response to inquiry information of a user; based on the inquiry information and the scene summary information, performing retrieval matching in a layout rule knowledge base to obtain design rule information; inputting the scene summary information, the inquiry information and the design rule information into a question and answer model to obtain reply result data corresponding to the inquiry information.

5. The virtual environment data processing method of claim 4, wherein, The inputting the scene summary information, the inquiry information and the design rule information into the question and answer model to obtain the reply result data corresponding to the inquiry information comprises: confirming target structure feature information and target visual information of the target virtual scene; inputting the scene summary information, the inquiry information, the design rule information, the target structure feature information and the target visual information into the question and answer model to obtain the reply result data corresponding to the inquiry information.

6. The virtual environment data processing method of claim 4, wherein, The method further comprises: when the reply result data includes new object information of a new placement object, determining new arrangement data of the new placement object in the target virtual scene based on the new object information; generating the new placement object in the target virtual scene based on the new arrangement data.

7. The virtual environment data processing method of claim 1, wherein, The method further comprises: in response to a direct modification instruction of a user on a target object in the target virtual scene; adjusting the target object in the target virtual scene based on the direct modification instruction; generating and saving updated scene information based on the adjusted target virtual scene.

8. An electronic device, comprising: A device comprises a memory and a processor, the memory stores a computer program, and the processor implements the virtual environment data processing method of any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the virtual environment data processing method of any one of claims 1 to 7.

Images