Three-dimensional scene generation method and device, program product and medium

Abstract

The embodiment of the invention provides a three-dimensional scene generation method and device, a program product and a medium, which are used for analyzing multi-modal data input by a user, and determining a target three-dimensional model of an object from an asset library according to attribute information and a layout file of the object obtained through analysis, so as to generate a three-dimensional scene. The method comprises the steps that multi-modal data input by a user is acquired, the multi-modal data is used for describing a target scene, the target scene is a target three-dimensional scene expected by the user, and the multi-modal data comprises at least two types of data in texts, images or videos; analyzing the multi-modal data to obtain attribute information and a layout file of the object, the layout file being used for indicating position information of the object; according to the attribute information and the layout file, a target three-dimensional model of the object is determined from an asset library, a three-dimensional scene is obtained, and the asset library is used for storing the three-dimensional model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a three-dimensional scene generation method, device, program product and medium. Background Art

[0002] 3D Reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects. It is the basis for processing, operating and analyzing its properties in a computer environment. It is also a key technology for establishing virtual reality in computers to express the objective world.

[0003] In recent years, with the development of artificial intelligence technology, large language models (LLM) have been introduced into three-dimensional scene reconstruction technology. They mainly generate the final three-dimensional scene by extracting the generation requirements for the three-dimensional scene from the text content input by the user and performing reasoning through the large language model.

[0004] However, due to the limited description and constraints of the scene in the text, it is difficult to accurately describe information such as the position of objects in the three-dimensional scene, which will cause a large deviation between the three-dimensional scene generated by LLM based on the input text and user needs, thereby reducing user satisfaction. Summary of the invention

[0005] The present application provides a three-dimensional scene generation method, device, program product and medium for parsing multimodal data input by a user, determining a target three-dimensional model of an object from an asset library based on attribute information and layout files of the object obtained through the analysis, thereby generating a three-dimensional scene.

[0006] In a first aspect, the present application provides a three-dimensional scene generation method, comprising: first, obtaining multimodal data input by a user, the multimodal data is used to describe a target scene, the target scene is a target three-dimensional scene that the user expects to obtain, and in addition, the multimodal data includes at least two types of data in text, image or video. Subsequently, the multimodal data can be parsed to obtain the attribute information and layout file of the object, wherein the layout file is used to indicate the location information of the object; after extracting the attribute information and layout file of the object from the multimodal data input by the user, the target three-dimensional model of the object can be determined from the asset library to obtain the three-dimensional scene, wherein the asset library is used to store the three-dimensional model.

[0007] In an embodiment of the present application, the multimodal data includes two or more different types of data in text, image or video. By parsing the multimodal data input by the user, information related to the three-dimensional scene to be generated can be extracted from the multiple different types of data, thereby providing richer descriptive information for the generation of the three-dimensional scene. Compared with text data, image or video data can express more scene details and describe more accurately when describing the three-dimensional scene desired by the user, thereby improving the accuracy of the three-dimensional scene generation.

[0008] In a possible implementation, the method may further include: obtaining a random degree, where the random degree is used to control the degree of difference between the three-dimensional scene and the target scene.

[0009] In the embodiment of the present application, by introducing randomness, the degree of difference between the generated three-dimensional scene and the three-dimensional scene expected by the user can be adjusted, so that diversified three-dimensional scenes can be generated in batches according to the randomness to meet the different needs of users.

[0010] In a possible implementation, the aforementioned determining the target three-dimensional model of the object from the asset library based on the attribute information and the layout file to obtain the three-dimensional scene may include: determining multiple target three-dimensional models of the object from the asset library based on the randomness, attribute information and the layout file to obtain multiple three-dimensional scenes.

[0011] In a possible implementation, the aforementioned determining multiple target three-dimensional models of the object from the asset library based on the randomness, attribute information and layout file to obtain multiple three-dimensional scenes may include: determining multiple target three-dimensional models of the object from the asset library based on the randomness and the size of the object in the attribute information; placing the multiple target three-dimensional models based on the posture of the object in the layout file to obtain multiple three-dimensional scenes.

[0012] In a possible implementation, before determining multiple target three-dimensional models of the object from the asset library based on the randomness, attribute information and layout file to obtain multiple three-dimensional scenes, the method may also include: generating a bounding box diagram based on the size of the object and the layout file, the bounding box diagram including multiple bounding boxes, and the bounding box is used to represent the boundary of the object; analyzing the bounding box diagram through a visual language model to obtain an analysis result, and the analysis result is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene; when the analysis result indicates that there is a risk of collision between the first object and the second object, updating the size and / or posture of the object until there is no risk of collision between the first object and the second object, and stopping the updating.

[0013] In an embodiment of the present application, after obtaining the layout file of the object, the collision risk of the objects in the generated three-dimensional scene can also be judged. When there is a collision risk, the size and / or posture of the object can be adjusted to avoid overlapping of objects in the generated three-dimensional scene, so that the subsequently generated three-dimensional scene is more in line with the real scene.

[0014] In a possible implementation, after determining the target three-dimensional model of the object from the asset library based on the attribute information and the layout file to obtain the three-dimensional scene, the method may further include: when the orientation of the object in the three-dimensional scene does not meet the preset requirements, adjusting the orientation of the object until the orientation of the object meets the preset requirements, and stopping the adjustment; when the orientation of the object in the three-dimensional scene meets the preset requirements, outputting the three-dimensional scene.

[0015] In an embodiment of the present application, the rationality of the orientation of objects in the three-dimensional scene is judged, and objects with unreasonable orientations are adjusted until the orientation of objects in the three-dimensional scene meets preset requirements, so as to verify the three-dimensional scene and make the verified three-dimensional scene more consistent with the actual situation.

[0016] In a possible implementation, the aforementioned parsing of the multimodal data to obtain the attribute information and layout file of the object may include: parsing the multimodal data through a visual language model to obtain the attribute information and layout file.

[0017] In the second aspect, the present application proposes a three-dimensional scene generation device, including: an acquisition module, used to acquire multimodal data input by a user, the multimodal data is used to describe a target scene, the target scene is a target three-dimensional scene that the user expects to obtain, and the multimodal data includes at least two types of data in text, image or video; a parsing module, used to parse the multimodal data to obtain attribute information and a layout file of the object, the layout file is used to indicate the location information of the object; a generation module, used to determine the target three-dimensional model of the object from an asset library based on the attribute information and the layout file, and obtain a three-dimensional scene, and the asset library is used to store the three-dimensional model.

[0018] In a possible implementation, the aforementioned acquisition module is further used to acquire a random degree, and the random degree is used to control the degree of difference between the three-dimensional scene and the target scene.

[0019] In a possible implementation, the aforementioned generation module is specifically used to determine multiple target three-dimensional models of the object from the asset library according to the randomness, attribute information and layout file, and obtain multiple three-dimensional scenes.

[0020] In a possible implementation, the aforementioned generation module is specifically used to: determine multiple target three-dimensional models of the object from the asset library according to the degree of randomness and the size of the object in the attribute information; place the multiple target three-dimensional models based on the positions of the object in the layout file to obtain multiple three-dimensional scenes.

[0021] In a possible implementation, before the aforementioned determination of multiple target three-dimensional models of the object from the asset library based on the randomness, attribute information and layout file to obtain multiple three-dimensional scenes, the aforementioned generation module is also used to generate a boundary box diagram based on the size of the object and the layout file, the boundary box diagram including multiple boundary boxes, and the boundary box is used to represent the boundary of the object; the device may also include: an analysis module, used to analyze the boundary box diagram through a visual language model to obtain an analysis result, and the analysis result is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene; an update module, used to update the size and / or posture of the object when the analysis result indicates that there is a risk of collision between the first object and the second object, until there is no risk of collision between the first object and the second object, and stop updating.

[0022] In a possible implementation, after determining the target three-dimensional model of the object from the asset library based on the attribute information and the layout file to obtain the three-dimensional scene, the device may also include: an adjustment module, which is used to adjust the orientation of the object when the orientation of the object in the three-dimensional scene does not meet the preset requirements, until the orientation of the object meets the preset requirements and stops the adjustment; the aforementioned generation module is also used to output the three-dimensional scene when the orientation of the object in the three-dimensional scene meets the preset requirements.

[0023] In a possible implementation, the aforementioned parsing module is specifically used to: parse the multimodal data through a visual language model to obtain attribute information and a layout file.

[0024] In a third aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device executes a method such as the first aspect or any possible implementation of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes a method such as the first aspect or any possible implementation of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes a method such as the first aspect or any possible implementation of the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a method such as the first aspect or any possible implementation manner of the first aspect.

[0028] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a system framework provided for an embodiment of the present application; Figure 2 Generate a sample image of the scene for a three-dimensional scene; Figure 3 A schematic diagram of a process for generating a three-dimensional scene provided in this application; Figure 4 This is a diagram of the text parsing process; Figure 5 This is a schematic diagram of the image parsing process; Figure 6 This is a schematic diagram of the video analysis process; Figure 7 A schematic diagram of the process of verifying a table scene including a dinner plate, a dinner knife, and a dinner fork; Figure 8 A schematic diagram of a process for self-verification of a three-dimensional scene; Fig. 9 A schematic diagram of the adjustment process of the robot arm for desktop arrangement; Fig.10 A schematic diagram of the structure of a three-dimensional scene generation device provided in this application; Fig.11 A schematic diagram of the structure of a computing device in an embodiment of the present application; Fig.12 A schematic diagram of the structure of a computing device cluster in an embodiment of the present application; Fig.13 Another structural diagram of a computing device cluster in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] To facilitate understanding, some concepts involved in this application are first introduced.

[0032] 1. Multimodal Data Multimodal data refers to the expression or perception of things from multiple modes, usually including visual, auditory, text and other forms of information. In the fields of computer science, artificial intelligence and machine learning, multimodal technology enhances the model's understanding and reasoning capabilities by integrating data from different modes (such as images, text, audio, video, etc.). 2. Vision language model (VLM) Visual language models are broadly defined as multimodal models that can learn from images and text, taking images and text as input and generating text or images as output. Large visual language models perform well in zero-shot learning, have good generalization capabilities, and can handle many types of images, including documents, web pages, etc.

[0033] 3. 3D Scene A 3D scene is a virtualization technology that simulates the various material forms, spatial relationships and other information of the real world. The geometric structure of an object can be constructed using basic elements such as points, lines and surfaces through mathematical geometry principles to achieve a simulation of the real world.

[0034] 4. Bounding box The bounding box is also called the bounding box or the bounding box envelope. The bounding box can be understood as an imaginary outer frame around the object being detected. It can also be simply understood as a bounding box that uses a slightly larger geometric body with simpler characteristics to approximate a complex geometric object.

[0035] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the present application is introduced below with reference to the accompanying drawings.

[0036] See also Figure 1 , is a system architecture 100 provided by this application. Figure 1As shown, the system architecture 100 may include a computing device cluster 110, which includes at least one computing device. The computing device may be a server, such as a cloud server, a central server, an edge server, or a local server in a local data center. In some possible embodiments, the computing device may also be a desktop computer, a laptop computer, or other terminal devices. When the computing device cluster includes multiple computing devices, the types of different computing devices may be the same or different.

[0037] Exemplarily, the computing device cluster may include one or more servers located in the cloud, and after receiving the multimodal data, the three-dimensional scene generation method of the embodiment of the present application is implemented by calling the cloud server; alternatively, the computing device cluster may also be a client device, and the three-dimensional scene generation method of the embodiment of the present application is implemented through installed applications or plug-ins, etc.; alternatively, the computing device cluster may include a server and a client device located in the cloud, and the three-dimensional scene generation method of the embodiment of the present application is implemented through information interaction between the cloud and the client device.

[0038] It is worth noting that Figure 1 The system architecture shown is only an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, the system architecture 100 also includes an asset library, which is used to store three-dimensional models. After obtaining the attribute information and layout file of the object according to the multimodal data input by the user, the three-dimensional model corresponding to the object can be determined from the asset library, thereby generating a three-dimensional scene.

[0039] At present, the traditional way of generating 3D scenes usually requires professional modeling software and professional modelers, or using point cloud data collected by data acquisition equipment to generate 3D scenes. However, relying on professional modeling software and modelers is costly and time-consuming, and generating 3D scenes based on point cloud data will result in a large loss of accuracy in the process of converting point clouds into mesh models. With the development of large models, large language models have been introduced into 3D scene reconstruction. However, since the 3D scene is generated only based on the user input text, it is difficult for text data to accurately describe the user's desired 3D scene, resulting in a large deviation between the generated 3D scene and user needs.

[0040] Based on this, an embodiment of the present application proposes a three-dimensional scene generation method, which can parse the multimodal data input by the user, and determine the target three-dimensional model of the object from the asset library based on the attribute information and layout file of the object obtained by the analysis, thereby generating a three-dimensional model.

[0041] For example, Figure 2As shown in the example diagram of 3D scene generation, the user can input at least two types of data from image, text or video data, and then parse the multimodal data through the image parsing module and the video parsing module in the input parsing and control module to obtain the background, object, spatial position, and even style description or lighting description; then, the layout generation module can be used to generate a layout file corresponding to the object, which includes the position and orientation of the object, and then the target 3D model of the object is determined from the asset library according to the attribute information of the object and the layout file to generate a 3D scene. After obtaining the 3D scene, it can be verified through the 3D scene visual verification model, and a reasonable 3D scene can be output.

[0042] Among them, the layout generation module can be divided into a background generation module and a foreground object generation module. When generating a three-dimensional scene, the scene can be divided into background and foreground, and the background and foreground objects can be determined according to different scenes. For example, when the user requires the generation of a desktop scene, the table is the background, and tables of different sizes, shapes, materials, and colors can constitute different "backgrounds", and various objects placed on the table are foregrounds; when the user requires the generation of a room-level three-dimensional scene, the room and the inherent decoration fixed on the room can be regarded as the "background", and the different sizes, shapes, wall textures, floor tile styles, etc. of the room constitute different backgrounds. Objects in the room that "are not part of the room itself" can be identified as foreground objects. "Room itself" in general scenes refers to objects that cannot be moved or are almost not considered to be moved, such as doors, walls, floors, etc. At this time, the table that serves as the background in the desktop scene is a foreground object in the room-level scene.

[0043] The following is an introduction to the method flow provided by this application in conjunction with the aforementioned system architecture.

[0044] See also Figure 3 , a flow chart of a three-dimensional scene generation method provided in this application is as described below.

[0045] Step 301: Obtain multimodal data input by a user; Typically, a user inputs a text description and uses a large language model to extract information related to the target three-dimensional scene (which can be called the target scene) that the user expects to obtain from the text description. However, ordinary users or professionals who lack professional knowledge find it difficult to accurately describe information such as the position and shape of objects in the scene, which may lead to a large deviation between the three-dimensional scene generated by the large language model based on the text description and the target scene.

[0046] Since multimodal data usually includes two or more different types of data in text, images or videos, by inputting multiple different types of data, it is possible to avoid the defect that only inputting text data is difficult to accurately describe the scene, and provide information from multiple different sources for generating three-dimensional scenes, which can improve the accuracy of generating three-dimensional scenes. Therefore, in an embodiment of the present application, more accurate information related to the target scene can be obtained by acquiring multimodal data. Among them, multimodal data is used to describe the target scene. Exemplarily, multimodal data can be text data and image data, or text data and video data, and can also include text data, image data and video data, wherein text data can also be obtained by recognizing audio data conversion.

[0047] Specifically, multimodal data can be publicly searchable data sets on the Internet, including open source data sets, as well as various information such as text, pictures and videos on the Internet. Multimodal data can also be data uploaded by users. The format of text, pictures or videos is not limited, and all reasonable formats are acceptable.

[0048] Exemplarily, when text data is difficult to accurately describe the target scene, image data or video data can be used for supplementary description. For example, if the text data is "Please generate a desktop scene containing a dinner plate, a dinner knife, and a dinner fork", the orientation and position of the dinner knife and the dinner fork cannot be accurately known at this time. The three-dimensional scene generated only based on the text data may not conform to the actual situation. Based on this, image data can be input, and the image data may include the usual orientation and position of the dinner plate, dinner knife, and dinner fork. By adding image data, user needs can be more accurately represented, and the three-dimensional scene generated based on text data and image data will be more in line with user needs and actual conditions.

[0049] For another example, when the multimodal data is image data and video data, the image data can provide an image from a certain perspective, while the video data can provide an image from multiple perspectives, thereby obtaining images from a variety of different perspectives. Moreover, the image or video can express scenes that are difficult to describe clearly in text, and may include more information about the three-dimensional scene, so that the three-dimensional scene generated based on the image data and video data has a smaller deviation from the three-dimensional scene expected by the user and is more in line with the actual situation.

[0050] Step 302: parse the multimodal data to obtain object attribute information and layout files; After obtaining the multimodal data input by the user, the multimodal data can be parsed through the visual language model according to the prompt to extract the attribute information and layout file of the object, wherein the attribute information of the object is used to represent the properties and characteristics of the object, including the category, quantity, size or material of the object, etc., which are not limited here, and the layout file is used to indicate the location information of the object, for example, the layout file includes the position and posture of the object (including the coordinates and orientation of the object).

[0051] Among them, the layout file can also include the relative position relationship of objects. For example, the multimodal data indication includes object A and object B, and the relative position relationship can be expressed as "A is on the surface of B", "A is inside B" or "A is in front of B" and other relative position relationships.

[0052] Optionally, the lighting atmosphere of the target scene described by the user can also be extracted, where the lighting atmosphere can be divided into dim and bright, etc., and can also be divided into warm light and cold light according to color temperature, etc. When the multimodal data does not include a description of the lighting atmosphere, the multimodal data can be inferred through a visual language model to obtain a suitable lighting atmosphere. In addition, the scene style of the target scene described by the user can also be extracted, and the scene style is used to represent the overall atmosphere of the scene. For example, when used to generate a scene of room decoration style, the scene style can be "Nordic style decoration", "cute style desktop layout", etc. By the same token, when the scene style or the specific layout of the object is not represented in the multimodal data, the visual language model can be used to infer the multimodal data to obtain a suitable style description or object layout.

[0053] Specifically, when the multimodal data includes text data, the text data can be parsed using a visual language model. The text parsing process is as follows: Figure 4 As shown. Figure 4 It can be seen that when the text data includes the object category and specific quantity of the target scene, the object category and quantity can be extracted from the text data and stored; when the text data does not include the object category and specific quantity, the visual language model can generate a list of objects that conform to the description of the text data based on the text data. The size and coordinates (or relative position) of the object can also be inferred. For example, the user inputs "Please generate a desktop scene containing a plate, a knife, and a fork". Through the visual language model to parse the input text data, it can be inferred that the size of the plate is 25cm*25cm; the plate, knife, and fork are in a straight line, and the plate is in the middle, with the knife and fork on both sides. The absolute position of the object needs to be added with the corresponding offset value according to the actual situation.

[0054] When the multimodal data includes image data, the image parsing process is as follows: Figure 5 As shown. Figure 5It can be seen that the image data can be analyzed through the visual language model in the image analysis module to extract information such as object category and quantity, object layout or relative position relationship, scene style or lighting atmosphere from the image data.

[0055] When the multimodal data includes video data, the process of parsing the video is as follows: Figure 6 As shown. Figure 6 It can be seen that the video data can first be converted into image data through the video parsing module, and then the converted image data can be parsed through the visual language model. Specifically, multiple key frames of the video can be extracted, and then the features of the multiple key frames can be extracted through the feature extraction module to obtain a scene diagram (image data).

[0056] Optionally, when the multimodal data includes video data, the video data can also be directly parsed by a video parsing model, such as a Gemini model. Specifically, the model can be used to extract relevant information describing the scene (including the time information between two frames of images in the video) from the video data, without first converting the video data into image data and then reusing the image parsing module to parse the image data.

[0057] Step 303: According to the attribute information and the layout file, determine the target 3D model of the object from the asset library to obtain a 3D scene.

[0058] In the embodiment of the present application, the asset library is used to store three-dimensional models. The source of the three-dimensional model can be a 3D asset library uploaded by a user, an open source data set, or a three-dimensional model generated from an image or text using image-to-3d or text-to-3d technology, which is not limited here. In addition, the three-dimensional models in the asset library can include categories such as rigid bodies, hinged objects, material files, or lighting files.

[0059] Optionally, before determining the target 3D model of the object from the asset library according to the object's attribute information and layout file, a degree of randomness may be obtained, which is used to control the degree of difference between the generated 3D scene and the target scene, where the target scene is the target 3D scene expected by the user.

[0060] In the embodiment of the present application, by introducing randomness and combining the attribute information and layout file of the object, multiple three-dimensional scenes can be generated in batches, thereby improving the efficiency of generating three-dimensional scenes and providing users with a variety of three-dimensional scenes in batches at one time, thereby improving user satisfaction.

[0061] Optionally, after obtaining the randomness, multiple target three-dimensional models can be determined from the asset library according to the randomness, attribute information and layout file to obtain multiple three-dimensional scenes.

[0062] Specifically, multiple target 3D models of the object can be determined from the asset library according to the randomness and the size of the object in the attribute information. The value of the randomness is a value in the range of 0 to 1. The closer the value is to 1, the higher the difference between the generated 3D scene and the target scene. The closer the value is to 0, the lower the difference between the generated 3D scene and the target scene. That is, the generated 3D scene is required to be more similar (or close) to the target scene.

[0063] Based on this, first, the selection requirements of the target three-dimensional model of the object can be determined according to the value of randomness; then, the size of the object in the attribute information can be combined to select multiple target three-dimensional models that meet the selection requirements from the asset library. For example, when the user requires to generate a 3D scene containing a red leather sofa, when the value of randomness is 0.1, it can be considered that the value of randomness is low. When determining the target three-dimensional model from the asset library, it is necessary to strictly determine the target three-dimensional model according to keywords such as "red", "leather" and "sofa"; when the value of randomness is 0.9, it can be considered that the value of randomness is high, and the object keywords mentioned in the text data can be selected without strictly following them. At this time, selecting a white leather sofa can also be considered to meet the selection requirements, so that multiple target three-dimensional models such as white leather sofas, red leather sofas or red fabric sofas may be obtained. Among them, when the randomness is higher than the preset value, it can be considered that the value is high, and when it is lower than the preset value, it can be considered that the value is low.

[0064] After determining multiple target 3D models of the object from the asset library, the multiple target 3D models can be placed based on the pose (including coordinates and orientation) of the object according to the pose of the object in the layout file to obtain multiple 3D scenes. Among them, each 3D scene is obtained by placing the target 3D model according to the pose of the object. And the generated 3D scene will be saved in a common 3D model file format, including but not limited to FBX (Filmbox), OBJ (Wavefront Object), GLTF (GL Transmission Format) and DAE (Collada) formats. Among them, FBX is suitable for most game engines, such as Unity and Unreal Engine; OBJ is often used for static models; GLTF supports rich functions and good compression performance; DAE supports exchange between multiple 3D applications.

[0065] In the embodiment of the present application, the generated 3D scene is represented using a universal format and can be loaded into various simulators, which greatly increases its versatility.

[0066] Optionally, before generating multiple three-dimensional scenes, collision risk detection can also be performed on objects in the three-dimensional scene to be generated according to the size of the object and the layout file. First, a bounding box diagram can be generated according to the size of the object and the layout file, and the bounding box diagram includes multiple bounding boxes, and the bounding box is used to represent the boundary of the object. It should be understood that the bounding box can be a fictitious rectangular box surrounding the object. After obtaining the bounding box diagram, the bounding box diagram can be analyzed by a visual language model to obtain an analysis result, which is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene to be generated; when the analysis result indicates that there is a risk of collision between the first object and the second object, or indicates that the bounding boxes overlap each other, the size and / or posture of the object can be updated until there is no risk of collision between the first object and the second object, and the update is stopped. Among them, the first object and the second object are any two objects in the three-dimensional scene to be generated, and the specific details are not limited here.

[0067] Optionally, after generating a three-dimensional scene, the rationality of the orientation of objects in the generated three-dimensional scene can also be judged. When the orientation of objects in the three-dimensional scene does not meet preset requirements, the orientation of the objects can be adjusted until the orientation of the objects meets the preset requirements and the adjustment is stopped; when the orientation of objects in the three-dimensional scene meets the preset requirements, the three-dimensional scene can be directly output.

[0068] Specifically, the graphic design software (or game engine or robot simulator) will use the built-in camera function of the software to take photos of the generated 3D scene, and use the visual language model to judge the rationality of the photos of the 3D scene. When there is an unreasonable orientation of an object, the visual language model will generate an adjustment strategy (or plan) and adjust the orientation of the object according to the adjustment strategy until the orientation of the object meets the preset requirements and the adjustment stops. For example, Figure 7 As shown, Figure 7 To verify the table scene containing a plate, knife, and fork, Figure 7 It can be seen that the initial orientation of the fork, spoon and knife does not conform to the placement orientation in the actual environment. The orientation of the fork, spoon and knife is continuously adjusted through the visual language model, and finally a desktop scene that conforms to the actual environment is obtained.

[0069] Combining the aforementioned collision risk judgment and object orientation rationality judgment, the flowchart for self-verification of the three-dimensional scene is as follows: Figure 8As shown, specifically, the user can send the task details to the visual language model in the form of text, and possibly in combination with images or videos. In addition, in order to make the generated three-dimensional scene more in line with user needs, the user can also send expert knowledge to the visual language model by providing examples and other methods. The visual language model can extract the layout information of the object according to the user input, and the layout information includes the size, position and overall layout of the object; then the bounding box diagram can be drawn according to the size and posture of the object, so as to facilitate the subsequent collision judgment according to the bounding box diagram. When the judgment result indicates that there is a risk of object collision, the size, coordinates and layout of the object will be regenerated and the collision risk judgment will be performed again until the object does not have a collision risk. According to the layout file and attribute information of the object, the template target three-dimensional model corresponding to the object can be determined from the asset library to generate a three-dimensional scene. After the three-dimensional scene is generated, the rationality of the object orientation can be judged for the three-dimensional scene. First, the real scene picture of the generated three-dimensional scene can be obtained by the graphic design software (or game engine or robot simulator), and then the visual language model judges whether the orientation of the object in the real scene picture is reasonable. When there is unreasonableness, adjustment requirements are given and adjusted until the orientation of the object is reasonable, and the final three-dimensional scene is output.

[0070] In the embodiment of the present application, before generating a three-dimensional scene, the collision risk of the three-dimensional scene to be generated can be judged according to the size of the object and the layout file, and the objects with collision risk can be adjusted to avoid unreasonable situations such as overlapping of objects in the generated three-dimensional model. And after obtaining the three-dimensional scene, the generated three-dimensional scene can also be verified, and by judging the rationality of the orientation of the objects in the generated three-dimensional scene, the generated three-dimensional scene can be made more in line with the actual situation.

[0071] The embodiment of the present application can generate multiple three-dimensional scenes in batches according to the multimodal data input by the user, providing the user with a rich three-dimensional scene. In addition, the collision risk judgment and orientation rationality verification of the three-dimensional scene can be performed to make the generated three-dimensional scene more consistent with the real environment, reducing the difference between the generated simulation environment and the real environment. In addition, since the three-dimensional scene generation method proposed in the present application also includes the process of verifying the three-dimensional scene, the present application can also be applied to the scene of the robot tidying up the desktop.

[0072] For example, a common application scenario is to use a robotic arm to complete the desktop tidying task. Since it is difficult to determine "what is a properly tidy desktop" using any programming language or specific instructions, the visual language model can be used to obtain a photo of the current scene and determine whether it needs to be adjusted, and an adjustment plan can be given until the image of the desktop scene meets the requirements and the adjustment stops. Fig. 9 As shown, Fig. 9Schematic diagram of the adjustment process of the robot arm for desktop arrangement. Fig. 9 It can be seen that the user can first provide expert knowledge (the professional knowledge required for the task) and task details (which can be a detailed description of the task requirements in text, picture or video format) to the VLM. The robotic arm can use its own visual sensor to take real-scene photos and transmit the photos to the VLM. Then, the VLM can judge the current scene and give corresponding adjustment suggestions. The robotic arm changes the position of the object according to the adjustment suggestions and repeats the above steps, taking real-scene photos again and transmitting them to the VLM until the VLM determines that the current scene meets the task requirements, stops adjusting, and the task ends.

[0073] The above is an introduction to the method flow provided by the present application. Based on the above method flow, the following is an introduction to the device provided by the present application.

[0074] See also Fig.10 , a structural schematic diagram of a three-dimensional scene generation device provided in this application is described as follows.

[0075] An acquisition module 1001 is used to acquire multimodal data input by a user, where the multimodal data is used to describe a target scene, where the target scene is a target three-dimensional scene that the user expects to obtain, and the multimodal data includes at least two types of data among text, image or video; The parsing module 1002 is used to parse the multimodal data to obtain the attribute information and layout file of the object, and the layout file is used to indicate the location information of the object; The generation module 1003 is used to determine the target 3D model of the object from the asset library according to the attribute information and the layout file to obtain the 3D scene. The asset library is used to store the 3D model.

[0076] In a possible implementation, the aforementioned acquisition module 1001 is further used to acquire a random degree, and the random degree is used to control the degree of difference between the three-dimensional scene and the target scene.

[0077] In a possible implementation, the aforementioned generation module 1003 is specifically used to determine multiple target three-dimensional models of the object from the asset library according to the randomness, attribute information and layout file, and obtain multiple three-dimensional scenes.

[0078] In a possible implementation, the aforementioned generation module 1003 is specifically used to: determine multiple target three-dimensional models of the object from the asset library according to the randomness and the size of the object in the attribute information; place the multiple target three-dimensional models based on the posture of the object in the layout file to obtain multiple three-dimensional scenes.

[0079] In a possible implementation, before the aforementioned determination of multiple target three-dimensional models of the object from the asset library based on the randomness, attribute information and layout file to obtain multiple three-dimensional scenes, the aforementioned generation module 1003 is also used to generate a boundary box diagram based on the size of the object and the layout file, the boundary box diagram including multiple boundary boxes, and the boundary box is used to represent the boundary of the object; the device may also include: an analysis module 1004, used to analyze the boundary box diagram through a visual language model to obtain an analysis result, and the analysis result is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene; an update module 1005, used to update the size and / or posture of the object when the analysis result indicates that there is a risk of collision between the first object and the second object, until there is no risk of collision between the first object and the second object, and stop updating.

[0080] In a possible implementation, after determining the target three-dimensional model of the object from the asset library based on the attribute information and the layout file to obtain the three-dimensional scene, the device may also include: an adjustment module 1006, which is used to adjust the orientation of the object when the orientation of the object in the three-dimensional scene does not meet the preset requirements, until the orientation of the object meets the preset requirements and stops the adjustment; the aforementioned generation module 1003 is also used to output the three-dimensional scene when the orientation of the object in the three-dimensional scene meets the preset requirements.

[0081] In a possible implementation, the aforementioned parsing module 1002 is specifically configured to parse the multimodal data through a visual language model to obtain attribute information and a layout file.

[0082] Among them, the acquisition module, the parsing module, the generation module, the analysis module, the update module and the adjustment module can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the acquisition module is introduced below by taking the acquisition module as an example. Similarly, the implementation of the parsing module, the generation module, the analysis module, the update module and the adjustment module can refer to the implementation of the acquisition module.

[0083] As an example of a software functional unit, a module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

[0084] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0085] As an example of a hardware functional unit, the acquisition module may include at least one computing device, such as a server, etc. Alternatively, the acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0086] The multiple computing devices included in the acquisition module can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0087] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the three-dimensional scene generation method, the parsing module can be used to execute any step in the three-dimensional scene generation method, the generation module can be used to execute any step in the three-dimensional scene generation method, the analysis module can be used to execute any step in the three-dimensional scene generation method, the update module can be used to execute any step in the three-dimensional scene generation method, and the adjustment module can be used to execute any step in the three-dimensional scene generation method. The steps that the acquisition module, parsing module, generation module, analysis module, update module and adjustment module are responsible for implementing can be specified as needed. The full functions of the three-dimensional scene generation device are realized by respectively implementing different steps in the three-dimensional scene generation method through the acquisition module, parsing module, generation module, analysis module, update module and adjustment module.

[0088] The present application also provides a computing device 1100. Fig.11 As shown, the computing device 1100 includes: a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate through the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1100.

[0089] The bus 1102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (eg, the memory 1106, the processor 1104, and the communication interface 1108).

[0090] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0091] The memory 1106 may include a volatile memory, such as a random access memory (RAM). The processor 1104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0092] The memory 1106 stores executable program codes, and the processor 1104 executes the executable program codes to respectively implement the functions of the acquisition module, the parsing module, the generation module, the analysis module, the update module, and the adjustment module, thereby implementing the three-dimensional scene generation method. That is, the memory 1106 stores instructions for executing the three-dimensional scene generation method.

[0093] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0094] like Fig.12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 in one or more computing devices 1100 in the computing device cluster may store the same instructions for executing the three-dimensional scene generation method.

[0095] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store some instructions for executing the three-dimensional scene generation method. In other words, the combination of one or more computing devices 1100 may jointly execute the instructions for executing the three-dimensional scene generation method.

[0096] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the three-dimensional scene generation device. That is, the instructions stored in the memory 1106 in different computing devices 1100 can implement the functions of one or more modules of the acquisition module, the parsing module, the generation module, the analysis module, the update module and the adjustment module.

[0097] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.13 A possible implementation is shown. Fig.13 As shown, the first computing device and the second computing device are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1106 in the first computing device stores instructions for executing the functions of the acquisition module. At the same time, the memory 1106 in the second computing device stores instructions for executing the functions of the parsing module, the generating module, the analyzing module, the updating module, and the adjusting module.

[0098] It should be understood that Fig.13 The function of the first computing device shown in FIG. 1100 may also be completed by multiple computing devices 1100. Similarly, the function of the second computing device may also be completed by multiple computing devices 1100.

[0099] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.12 and Fig.13 The connection mode of the computing device cluster is different in that the memory 1106 in one or more computing devices 1100 in the computing device cluster may store the same instructions for executing the three-dimensional scene generation method.

[0100] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store some instructions for executing the three-dimensional scene generation method. In other words, the combination of one or more computing devices 1100 may jointly execute the instructions for executing the three-dimensional scene generation method.

[0101] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the three-dimensional scene generation device. That is, the instructions stored in the memory 1106 in different computing devices 1100 can implement the functions of one or more modules of the acquisition module, the parsing module, the generation module, the analysis module, the update module and the adjustment module.

[0102] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the three-dimensional scene generation method.

[0103] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the three-dimensional scene generation method.

[0104] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0105] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional scene generation method, characterized in that: include: Acquire multimodal data input by a user, wherein the multimodal data is used to describe a target scene, the target scene is a target three-dimensional scene desired by the user, and the multimodal data includes at least two types of data among text, image or video; Parsing the multimodal data to obtain attribute information and a layout file of the object, wherein the layout file is used to indicate location information of the object; According to the attribute information and the layout file, a target three-dimensional model of the object is determined from an asset library to obtain a three-dimensional scene, and the asset library is used to store three-dimensional models.

2. The method according to claim 1, characterized in that The method further comprises: A random degree is obtained, where the random degree is used to control the degree of difference between the three-dimensional scene and the target scene.

3. The method according to claim 2, characterized in that The step of determining the target three-dimensional model of the object from an asset library according to the attribute information and the layout file to obtain a three-dimensional scene includes: According to the randomness, the attribute information and the layout file, a plurality of target three-dimensional models of the object are determined from the asset library to obtain a plurality of the three-dimensional scenes.

4. The method according to claim 3, characterized in that The step of determining a plurality of target three-dimensional models of the object from the asset library according to the randomness, the attribute information and the layout file to obtain a plurality of the three-dimensional scenes comprises: Determine a plurality of target three-dimensional models of the object from the asset library according to the randomness and the size of the object in the attribute information; Placing the plurality of target three-dimensional models based on the positions and postures of the objects in the layout file to obtain the plurality of three-dimensional scenes.

5. The method according to claim 4, characterized in that Before determining the plurality of target three-dimensional models of the object from the asset library according to the randomness, the attribute information and the layout file to obtain the plurality of three-dimensional scenes, the method further comprises: Generate a bounding box diagram according to the size of the object and the layout file, wherein the bounding box diagram includes a plurality of bounding boxes, and the bounding boxes are used to represent the boundaries of the object; Analyzing the bounding box diagram using a visual language model to obtain an analysis result, wherein the analysis result is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene; When the analysis result indicates that there is a collision risk between the first object and the second object, the size and / or position of the object is updated until there is no collision risk between the first object and the second object, and then the updating is stopped.

6. The method according to claim 5, characterized in that After determining the target three-dimensional model of the object from the asset library according to the attribute information and the layout file and obtaining the three-dimensional scene, the method further includes: When the orientation of the object in the three-dimensional scene does not meet the preset requirement, adjusting the orientation of the object until the orientation of the object meets the preset requirement, and then stopping the adjustment; When the orientation of the object in the three-dimensional scene meets the preset requirement, the three-dimensional scene is output.

7. The method according to any one of claims 1 to 6, characterized in that The step of parsing the multimodal data to obtain the attribute information and layout file of the object includes: The multimodal data is parsed through a visual language model to obtain the attribute information and the layout file.

8. A three-dimensional scene generation device, characterized in that: include: An acquisition module, used to acquire multimodal data input by a user, wherein the multimodal data is used to describe a target scene, wherein the target scene is a target three-dimensional scene that the user expects to obtain, and wherein the multimodal data includes at least two types of data among text, image or video; A parsing module, used to parse the multimodal data to obtain attribute information and a layout file of the object, wherein the layout file is used to indicate the position information of the object; A generation module is used to determine the target three-dimensional model of the object from an asset library according to the attribute information and the layout file to obtain a three-dimensional scene, and the asset library is used to store three-dimensional models.

9. The device according to claim 8, characterized in that The acquisition module is further used to acquire a random degree, and the random degree is used to control the degree of difference between the three-dimensional scene and the target scene.

10. The device according to claim 9, characterized in that The generation module is specifically used for: According to the randomness, the attribute information and the layout file, a plurality of target three-dimensional models of the object are determined from the asset library to obtain a plurality of the three-dimensional scenes.

11. The device according to claim 10, characterized in that The generation module is specifically used for: Determine a plurality of target three-dimensional models of the object from the asset library according to the randomness and the size of the object in the attribute information; Placing the plurality of target three-dimensional models based on the positions and postures of the objects in the layout file to obtain the plurality of three-dimensional scenes.

12. The device according to claim 11, characterized in that Before determining the plurality of target three-dimensional models of the object from the asset library according to the randomness, the attribute information and the layout file to obtain the plurality of three-dimensional scenes, The generating module is further used to generate a bounding box diagram according to the size of the object and the layout file, wherein the bounding box diagram includes a plurality of bounding boxes, and the bounding boxes are used to represent the boundaries of the object; The device also includes: an analysis module, configured to analyze the boundary box diagram by using a visual language model to obtain an analysis result, wherein the analysis result is used to indicate whether there is a risk of collision between the first object and the second object in the three-dimensional scene; An updating module is used to update the size and / or position of the object when the analysis result indicates that there is a collision risk between the first object and the second object, and stop updating until there is no collision risk between the first object and the second object.

13. The device according to claim 12, characterized in that After determining the target three-dimensional model of the object from the asset library according to the attribute information and the layout file and obtaining the three-dimensional scene, the device further includes: An adjustment module, used for adjusting the orientation of the object in the three-dimensional scene when the orientation of the object does not meet the preset requirement, and stopping the adjustment until the orientation of the object meets the preset requirement; The generating module is further configured to output the three-dimensional scene when the orientation of the object in the three-dimensional scene meets the preset requirement.

14. The device according to any one of claims 8 to 13, characterized in that The analysis module is specifically used for: The multimodal data is parsed through a visual language model to obtain the attribute information and the layout file.

15. A computing device, characterized in that: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device executes the method according to any one of claims 1 to 7.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 7.

18. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method as claimed in any one of claims 1 to 7.

Images