Method for Converting a Three-Dimensional Scene from Rendering Reality to Physical Reality

Abstract

The present invention discloses a method for converting a three-dimensional scene from rendering reality to physical reality. Through steps such as lighting conversion, material conversion, material classification, model splitting and clustering, model semantic annotation, model collision body detection, and animation model conversion, the conversion efficiency from rendering a real scene to a physical real scene is greatly improved. The obtained physical real scene can be better displayed through VR / AR / MR. Due to the improvement of the conversion efficiency, a large number of rendered real scenes can become physical real scenes, providing more real data for the virtual simulation of robots.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional scene display, and particularly relates to a method for converting a three-dimensional scene from rendering reality to physical reality. Background Art

[0002] Traditional three-dimensional scenes are mainly displayed by rendering 2D effect drawings, and the display methods and application scopes are greatly limited. With the development of technology, the three-dimensional scene display method can be AR / VR / MR with interaction, and the application scope can be extended to the physical simulation environment of robots. Therefore, a set of conversion methods are needed to convert the traditional scenes aimed at rendering reality into physical reality scenes, so that the three-dimensional scenes can play a greater value.

[0003] For three-dimensional scenes aimed at rendering reality, the main contents involved are three-dimensional models, rendering materials, and lighting. For three-dimensional scenes aimed at physical reality, on the basis of rendering reality, physical materials, model splitting and clustering, semantic annotation, model collision bodies, and physical animations need to be added. Finally, these information are converted to an engine supporting physical rendering through an automated processing script for actual use.

[0004] When constructing three-dimensional scenes with physical reality, basically manual methods are adopted, with low efficiency and high costs. For example, in June 2021, Facebook AI Research launched the Habitat2.0 physical simulation dataset. In the Introduction part of the corresponding paper "Habitat2.0: Training Home Assistants to Rearrange their Habitat", it is introduced that designers carried out the conversion of physical reality scenes based on the Replica dataset. For the transformation of 111 scenes, involving 92 dynamic models, it took more than 900 hours in total.

[0005] Patent document CN109523629A discloses a method for generating an object semantics and pose dataset based on physical simulation, which relates to the field of deep learning applications for robot intelligent perception, and includes steps such as geometric modeling, initialization, simulation solution, rendering, reading, and generating a large-scale dataset. It mainly obtains scene data through physical simulation and image rendering, and at the same time obtains information such as semantics and pose from the internal parameter settings of the physical world and rendering settings. Although the scene data are obtained automatically, these scene data are not sufficient to support the physical reality application requirements. Summary of the Invention

[0006] In view of the above, the purpose of the present invention is to provide a method for converting a three-dimensional scene from rendering reality to physical reality, which greatly improves the efficiency of scene data conversion through automation.

[0007] To achieve the above-mentioned invention objective, a method for converting a three-dimensional scene from rendering reality to physical reality provided by an embodiment includes:

[0008] Step 1, extract lights, rendering materials, 3D models, and configuration information from the rendered real scene;

[0009] Step 2, light conversion: convert the extracted lights into the lights required for the physical real scene;

[0010] Step 3, material conversion: convert the extracted rendering materials into the rendering materials required for the physical real scene by using a data-driven material parameter domain conversion method;

[0011] Step 4, material classification: perform physical material classification on the rendering materials required for the physical real scene by using a material classifier to determine the physical materials corresponding to the rendering materials;

[0012] Step 5, model splitting and clustering: perform fine-grained splitting on the extracted 3D models based on grid connected domain recognition to obtain multiple sub-models, and perform model space clustering on each sub-model based on the Euclidean distance;

[0013] Step 6, model semantic annotation: perform semantic annotation on each sub-model by using a web-based model semantic annotation tool;

[0014] Step 7, model collision body detection: perform collision body detection on the sub-models by using an improved V-HACD algorithm to label the collision body models;

[0015] Step 8, animation model conversion: convert the animation information and sub-models in the configuration information into animation models;

[0016] Step 9, extract the processing results of Step 2 - Step 8 and encapsulate them into physical real scene data.

[0017] In one embodiment, during light conversion, according to the conversion formula T1 = a * S1, the conversion of the extracted light S1 to the light T1 required for the physical real scene is realized, where a is the conversion coefficient, and the value of a is different for different light parameters.

[0018] In one embodiment, the conversion of the extracted rendering materials into the rendering materials required for the physical real scene by using a data-driven material parameter domain conversion method includes:

[0019] First, denote the rendering material required for the physical real scene as Denote the rendering material extracted from the rendered real scene as θ, then the conversion target is to learn the mapping function f such that During learning, the constraint condition is multiple pairs of matched material parameters [[ID=

[0020]

[0021] Among them, and respectively represent the rendering results of a real renderer and a physically realistic renderer under the same lighting scene;

[0022] Convert the extracted rendering material θ into the rendering material required for the physically realistic scene by using the mapping function f learned in the above manner

[0023] In one embodiment, when classifying materials, the material classifier used is a classifier trained based on the data pairs formed by the rendering materials required for the physically realistic scene and the physical materials, where the physical materials include friction, density, and bounce coefficient.

[0024] In one embodiment, when splitting the model, the fine-grained splitting of the extracted 3D model based on grid connectivity domain recognition includes:

[0025] Abstract the mesh topology of the 3D model into an undirected graph describing the adjacency relationship of patches, calculate the connected components of the undirected graph, and split the mesh topology corresponding to each connected component into 1 sub-model to achieve fine-grained splitting of the original network of the 3D model;

[0026] When clustering the model, the model space clustering of each sub-model based on the Euclidean distance includes:

[0027] For the set of mesh vertices corresponding to each sub-model, calculate the nearest Euclidean distance between pairwise mesh vertices, construct an adjacency matrix based on the Euclidean distance, and perform hierarchical clustering based on the adjacency matrix to achieve spatial reorganization of the components of the sub-model according to the labels.

[0028] In one embodiment, when performing model semantic annotation, the web-based model semantic annotation tool used embeds annotation information. When performing semantic annotation through the web end, alternative semantic labels and combination functions are provided for the selected components, and the semantic annotation is completed through the confirmation function label and / or combination function.

[0029] In one embodiment, when performing model collision detection, an improved V-HACD algorithm is used to perform collision detection on the sub-models, including:

[0030] When using the V-HACD algorithm to perform collision detection on the sub-models, for the patch model, generate a cuboid collision body, and the cuboid collision body is 0 - 1 cm higher than the real object corresponding to the patch model;

[0031] When using the V-HACD algorithm to perform collision body detection on sub-models, by constraining the maximum number of voxels in the voxel space during the voxelization stage of vhacd, it is ensured that the length of the opened array does not exceed the allocated upper limit, and at the same time, it does not occupy too much memory, so as to ensure the normal operation of the program without crashing or blocking.

[0032] In one embodiment, during the conversion of the animation model, for the sub-model with animation information, the animation information in the configuration information is parsed to obtain the conversion information and the model hierarchy relationship information, and the conversion information and the model hierarchy relationship information are converted into physical constraint information, and the model information of the sub-model is bound to the physical constraint information to form an animation model.

[0033] Compared with the prior art, the beneficial effects of the present invention at least include:

[0034] Through steps such as lighting conversion, material conversion, material classification, model splitting and clustering, model semantic annotation, model collision body detection, and animation model conversion, the conversion efficiency from a rendered real scene to a physically real scene is greatly improved, and the obtained physically real scene can be better displayed through VR / AR / MR. Due to the improvement of the conversion efficiency, a large number of rendered real scenes can become physically real scenes, providing more real data for the virtual simulation of robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0036] Figure 1 It is a flowchart of the conversion method from a rendered real three-dimensional scene to a physically real one provided by the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0038] In order to improve the conversion efficiency of the conversion from a rendered real three-dimensional scene to a physically real one, the embodiment provides a conversion method from a rendered real three-dimensional scene to a physically real one, as Figure 1 shown, including the following steps:

[0039] M1, data extraction.

[0040] In the embodiments, data extraction refers to extracting data from a rendered real scene through an automated processing script to obtain lights (S1), rendering materials (S2), 3D models (S3), and configuration information (S4), so as to prepare for generating a physically realistic scene. Among them, the rendered real scene refers to a 3D scene for rendering 2D renderings, and its data format can be produced by common DCC software, such as 3DSMax, Maya, or a web-based 3D scene, such as the Kujiale Yuntu tool.

[0041] Among them, the lights (S1) refer to the lights in the rendered real scene, which are used to illuminate the scene. The types of lights generally include directional lights, point lights, area lights, spotlights, etc.

[0042] The rendering materials (S2) refer to the materials in the rendered real scene, which are generally offline rendering materials. The material parameters mainly include diffuse reflection, reflection, Fresnel, refraction, etc. Commonly used offline rendering materials include VRay materials. Mental Ray materials, etc.

[0043] The 3D models (S3) refer to the 3D models in the rendered real scene, such as models of doors, walls, dining tables, etc.

[0044] The configuration information (S4) includes scene light information, material information, model information, animation information, etc. Among them, the scene light information includes the light position, light intensity, light color, etc. The material information includes various types of materials, such as marble materials, metal materials, etc. The model information includes model rotation, scaling, position, and the material information used by the model. The animation information generally refers to Transform animations, which record the object hierarchy structure and Transform transformations.

[0045] M2, light conversion.

[0046] Since there are some differences between the lights in the rendered real scene and the physically realistic scene, light conversion is required. When performing light conversion, according to the conversion formula T1 = a * S1, the conversion of the extracted lights (S1) to the lights (T1) required for the physically realistic scene is realized. Among them, the lights (S1) include parameters such as light intensity and color, and the converted lights (T1) conform to physical reality. a is the conversion coefficient, and the value of a is different for different light parameters. For example, for color parameters, a = 1; for intensity parameters, the coefficient of a will be determined according to the light unit (such as lumen, candela, etc.) and the implementation method of the rendering engine.

[0047] M3, material conversion.

[0048] In the embodiment, the extracted rendering material (S2) is converted into the rendering material (T2) required for a physically realistic scene by using a data-driven material parameter domain conversion method. A physically realistic scene generally uses a real-time rendering engine as a carrier. Therefore, the rendering material (T2) is generally a PBR material, while the rendering material (S2) is an offline rendering material. The material conversion is to convert the materials in two formats. If the physically realistic scene uses an offline rendering engine as a carrier, the material conversion step does not need to be performed, and the rendering material (S2) can be directly used.

[0049] When performing material conversion, by establishing a parameter domain relationship, the rendering effects of rendering-real and physically-real materials are respectively rendered under the same lighting model, and the most accurate corresponding relationship is matched based on SSIM. Specifically, it includes:

[0050] First, denote the rendering material required for the physically realistic scene as Denote the rendering material extracted from the rendering-real scene as θ. Then the conversion target is to learn the mapping function f such that During learning, the constraint condition is multiple pairs of matched material parameters Satisfy formula (1):

[0051]

[0052] Wherein, and respectively represent the rendering results of the rendering-real renderer and the physically-real renderer under the same lighting scene;

[0053] Use the mapping function f learned through the above method to convert the extracted rendering material θ into the rendering material required for the physically realistic scene

[0054] M4: Material classification.

[0055] In the embodiment, a material classifier is used to perform physical material classification on the rendering material (T2) required for the physically realistic scene to determine the physical material (T3) corresponding to the rendering material.

[0056] The material classifier can use the tpot and random forest algorithms as classifiers, and use the data formed by the rendering materials and physical materials required for physically realistic scenes to train the classifier, so as to obtain a material classifier that maps rendering materials to physical materials. When using the material classifier for material classification, the input is the rendering material (T2), and the output is the physical material (T3) category. When the rendering material is an offline rendering material, it includes parameters such as diffuse reflection, reflection, refraction, and Fresnel; when the rendering material is a real-time rendering material, it includes parameters such as BaseColor, Metallic, and Roughness. The physical material parameters include friction, density, and bounce coefficient. For different categories of physical materials, their parameters are different. The physical material categories are divided into metals, plastics, woods, stones, etc.

[0057] The material classification model built based on AI can automatically label the physical materials of the scene, saving the time of manual labeling.

[0058] M5: Model splitting and clustering.

[0059] In rendering a real scene, when modeling, some sub-models will be created into a large model. For example, the mouse and keyboard are created into a large model, which does not affect the rendering image. However, in physical simulation, the mouse and keyboard need to be separated. Therefore, it is necessary to implement model splitting and clustering to separate the components of each component model.

[0060] In the embodiment, based on the identification of grid connected domains, the extracted three-dimensional model is finely split, including: abstracting the grid topology of the three-dimensional model (S3) into an undirected graph describing the adjacency relationship of patches, calculating the connected components of the undirected graph through a non-recursive DFS algorithm, and splitting the grid topology corresponding to each connected component into 1 sub-model (T4), so as to realize the fine-grained splitting of the original network of the three-dimensional model. The sub-model (T4) is presented in the smallest granularity, which is more conducive to convenient model semantic annotation and improving the annotation accuracy.

[0061] In the embodiment, model space clustering is performed on each sub-model based on the Euclidean distance, including:

[0062] For the set of grid vertices corresponding to each sub-model, calculate the nearest Euclidean distance between pairwise grid vertices, construct an adjacency matrix based on the Euclidean distance, and perform hierarchical clustering based on the adjacency matrix, so as to realize the spatial reorganization of the components of the sub-model according to the labels, so that the components belonging to the same label and close in space are clustered together.

[0063] M6: Model semantic annotation.

[0064] Since the 3D models for rendering real scenes do not have semantic information, it is necessary to perform semantic annotation on the models within the scene. In the embodiments, a web-based model semantic annotation tool is used to perform semantic annotation on each sub-model. The web-based model semantic annotation tool has annotation information embedded. When performing semantic annotation on the displayed model through the web side, alternative semantic labels and combination functions are provided for the box selection component, and the semantic annotation is completed through the confirmation function label and / or combination function.

[0065] For example, the handle and the grip of a knife are marked as a group, and the cutting board is marked as a group. The annotator only needs to box select the model, select the corresponding function labels, style labels, etc. in the display toolbar of the model semantic annotation tool, and click "group" to complete the grouping annotation and save the label data into the model semantics (T5). The UI design of the semantic annotation tool greatly improves the semantic annotation efficiency.

[0066] M7: Collision body detection.

[0067] When the simulation model collides, using the convex hull approximation model collision body can greatly improve the performance and efficiency of the simulation. In order to simplify a complex model into a convex hull, an improved V-HACD (Volumetric Hierarchical Approximate Convex Decomposition) algorithm is adopted. The V-HACD algorithm is a commonly used algorithm for quickly approximating the convex hull decomposition and simplification of non-convex hull models, which can simplify a single complex model into multiple simple approximate convex hulls with a small number of vertices to accelerate collision detection and processing.

[0068] When performing collision body detection on the sub-model using the improved V-HACD algorithm, it includes:

[0069] When using the V-HACD algorithm to perform collision body detection on the sub-model, for a mesh model, such as a carpet model, a cuboid collision body is generated, and its height is much higher than the real object, which does not conform to the real situation. By increasing the number of voxels in the Z direction, the generation of an accurate collision body is realized.

[0070] When using the V-HACD algorithm to perform collision body detection on the sub-model, by restricting the maximum voxel amount in the voxel space during the voxelization stage of vhacd, it is ensured that the length of the opened array does not exceed the allocated upper limit, and at the same time, it does not occupy too much memory, so as to ensure the normal operation of the program without crashing or blocking.

[0071] M8: Animation model conversion.

[0072] In the embodiment, the animation information and sub-models in the configuration information are converted into an animation model. Specifically, when converting the animation model, for the sub-models with animation information, the animation information in the configuration information is parsed to obtain conversion information and model hierarchy relationship information, and the conversion information and model hierarchy relationship information are converted into physical constraint information. The model information of the sub-models is bound to the physical constraint information to form a physical animation (T7).

[0073] Among them, the conversion information includes rotation angle, translation distance, etc. The model hierarchy relationship information records the object hierarchy structure. For example, the refrigerator body and the refrigerator door have hierarchy relationship information. The refrigerator door is embedded in the refrigerator body and has a rotation relationship and a corresponding rotation angle. Constraining this hierarchy relationship information to the model information converts the three-dimensional model into a model with a physical animation such as opening the refrigerator door. The robot can interact through this physical animation model, that is, it can open the refrigerator door to achieve physical simulation.

[0074] M9: Data assembly.

[0075] In the embodiment, the processing results of M2 - M8 (i.e., the physically realistic scene available data T1 - T7) are assembled in a physical simulation support engine through the configuration information (S4) to obtain physically realistic scene data. Among them, the physical simulation support engine includes the UE4 engine, the IssacSim engine, etc.

[0076] The embodiment compares the conversion efficiency of the conversion method provided by the present invention with the traditional method. The average time used to process one piece of data is statistically compared, and the results are shown in Table 1 as follows:

[0077] Table 1

[0078] The method of the present invention Traditional method Physical material classification (minutes) 0.1 5 Model semantic annotation (minutes) 1 2.5 Model collision detection (minutes) 10 12 Physical animation generation (minutes) 1 6

[0079] It can be analyzed from Table 1 that when using the conversion method provided by the present invention for scene conversion, a large amount of time can be saved, and the more data is processed to improve the conversion efficiency, the more obvious the improvement in conversion efficiency is.

[0080] The physically realistic scene obtained by using the conversion method from rendering reality to physical reality of the three-dimensional scene provided by the above embodiment can be displayed in a VR manner and can also be used for robot simulation training, that is, providing more realistic data for the virtual simulation of the robot.

[0081] When performing VR display, in the UE4 engine, a VR scene based on VR virtual reality game devices is developed. The observer observes the VR scene in an embedded manner and can interact with the objects in the physically realistic scene through the handle, such as picking up a book.

[0082] When the robot uses the physical real - world scenario for training simulation, in the IssacSim simulation engine, the robot can recognize the semantics of the scene model in the physical real - world scenario and interact with the models in the physical real - world scenario, such as opening the refrigerator door and moving the chair.

[0083] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the principle scope of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for converting a three-dimensional scene from rendering realism to physical realism, characterized in that, Including the following steps: Step 1, extract lights, rendering materials, 3D models, and configuration information from the rendered real scene; Step 2, light conversion: convert the extracted lights into the lights required for the physically real scene; Step 3, Material Conversion: Convert the extracted rendering material into the rendering material required for a physically realistic scene using a data-driven material parameter domain conversion method, including: First, denote the rendering material required for a physically realistic scene as Denote the rendering material extracted from the rendered real scene as θ, then the conversion target is to learn the mapping function f such that During learning, the constraint condition is multiple pairs of matched material parameters Satisfy formula (1): Among them, and respectively represent the rendering results of a physically accurate renderer and a realistic renderer under the same lighting scenario; Convert the extracted rendering material θ into the rendering material required for a physically realistic scene using the mapping function f learned in the above manner Step 4, material classification: use a material classifier to classify the rendering materials required for the physically real scene into physical materials to determine the physical materials corresponding to the rendering materials; Step 5, model splitting and clustering: based on grid connectivity domain recognition, perform fine-grained splitting on the extracted 3D model to obtain multiple sub-models, and perform model space clustering on each sub-model based on the Euclidean distance; Step 6, model semantic annotation: use a web-based model semantic annotation tool to perform semantic annotation on each sub-model; Step 7, model collision body detection: use an improved V-HACD algorithm to perform collision body detection on the sub-models to label the collision body models, including: When using the V-HACD algorithm to perform collision body detection on the sub-models, for the patch model, generate a cuboid collision body, and the cuboid collision body is 0 - 1 cm higher than the real object corresponding to the patch model; When using the V-HACD algorithm to perform collision body detection on the sub-models, by constraining the maximum voxel amount in the voxel space during the vhacd voxelization stage, ensure that the length of the opened array does not exceed the allocated upper limit and does not occupy too much memory, so as to ensure the normal operation of the program without crashing or blocking; Step 8, animation model conversion: convert the animation information and sub-models in the configuration information into an animation model, including: for the sub-models with animation information, parse the animation information in the configuration information to obtain conversion information and model hierarchy relationship information, and convert the conversion information and model hierarchy relationship information into physical constraint information, bind the model information of the sub-model with the physical constraint information to form an animation model; Step 9, extract the processing results of Step 2 - Step 8 and encapsulate them into physically real scene data.

2. The method for converting a three-dimensional scene from rendering realism to physical realism according to claim 1, characterized in that, During light conversion, according to the conversion formula T1 = a * S1, realize the conversion of the extracted light S1 to the light T1 required for the physically real scene, where a is the conversion coefficient, and the a value is different for different light parameters.

3. The method for converting a three-dimensional scene from rendering realism to physical realism according to claim 1, characterized in that, During material classification, the used material classifier is a classifier trained based on the data formed by the rendering materials required for the physically real scene and physical materials, where the physical materials include friction, density, and bounce coefficient.

4. The method for converting a three-dimensional scene from rendering reality to physical reality according to claim 1, characterized in that During model splitting, the fine-grained splitting of the extracted 3D model based on grid connectivity domain recognition includes: Abstract the grid topology of the 3D model into an undirected graph describing the adjacency relationship of patches, calculate the connected components of the undirected graph, and split the grid topology corresponding to each connected component into 1 sub-model to realize the fine-grained splitting of the original network of the 3D model; During model clustering, the model space clustering of each sub-model based on the Euclidean distance includes: For the set of grid vertices corresponding to each sub-model, calculate the nearest Euclidean distance between pairwise grid vertices, construct an adjacency matrix based on the Euclidean distance, and perform hierarchical clustering according to the adjacency matrix to realize the spatial reorganization of the components of the sub-model according to the labels.

5. The method for converting a three-dimensional scene from rendering reality to physical reality according to claim 1, characterized in that When performing model semantic annotation, the web-based model semantic annotation tool used embeds annotation information. When performing semantic annotation through the web end, alternative semantic labels and combination functions are provided for the boxed components, and semantic annotation is completed through the confirmation function label and / or combination function.

Images