Simulation rendering interaction modeling method and device, equipment and storage medium

Abstract

The invention relates to a simulation rendering interaction modeling method and device, equipment and a storage medium, and the method comprises the steps: obtaining the point cloud data of a target object, carrying out the three-dimensional simulation reconstruction of the target object according to the point cloud data, and obtaining a simulation three-dimensional model of the target object; according to the obtained visual data, calculating visual loss of the simulation three-dimensional model, and according to the obtained force sense data, calculating force sense loss of the simulation three-dimensional model; the visual data and the force sense data are captured simultaneously in the process of performing single interaction on the target object; calculating the comprehensive loss of the simulation three-dimensional model according to the visual loss and the haptic loss; and optimizing parameters of the simulation three-dimensional model based on the comprehensive loss to obtain a target simulation model. Combined end-to-end identification of geometric, appearance and physical attributes of an object is realized, reverse loss optimization is carried out on a simulation model, and the precision of model reconstruction is improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent modeling technology, and in particular to a simulation rendering interactive modeling method, apparatus, device and storage medium. Background Technology

[0002] With the accelerated transformation of mining areas towards automation and intelligence, robotic arms have been widely used in complex operational scenarios such as ore grabbing and equipment parts transfer. In these tasks, robotic arms need to frequently interact with unknown objects in the environment, including irregularly shaped piles of ore and equipment parts of various shapes. The core requirement is to achieve efficient operation through precise manipulation.

[0003] However, current robotic arm interaction simulation modeling technology has significant shortcomings, severely restricting operational reliability and efficiency. Existing modeling methods generally rely on manual measurement and pre-set models. Faced with the irregular shapes, uneven mass distribution, and large differences in surface friction coefficients of objects in mining areas, it is difficult to accurately obtain the true geometric shape and physical property parameters of the target objects. This results in the inability to accurately predict the force response and motion trajectory when the robotic arm interacts with the object, directly causing a high failure rate in robot operation planning. This not only reduces production efficiency but also easily leads to safety hazards such as equipment collisions. At the same time, traditional modeling processes are time-consuming and labor-intensive, with extremely poor generalization ability. Every time a new type of object is encountered, manual calibration and model construction must be carried out again, which is completely unsuitable for the complex and ever-changing mining environment and the high real-time requirements of operational tasks. In the field of robotics, although research has confirmed the importance of building predictive world models from sparse observations for task planning, existing differentiable modeling methods still have significant shortcomings: they cannot simultaneously optimize the geometry, appearance, and physical properties of the scene, and they lack an end-to-end processing pipeline for deep integration of visual and tactile data. Therefore, how to accurately estimate the three-dimensional shape, surface texture, mass, friction coefficient, and other key parameters of an object through a single interactive action such as pushing and turning of a robotic arm without relying on a prior model, and then generate a high-fidelity digital twin model that can be directly used for simulation and rendering, has become a technical bottleneck that urgently needs to be overcome in the current automated operation of robotic arms in mining areas. Summary of the Invention

[0004] This disclosure provides a simulation rendering interactive modeling method, apparatus, device, and storage medium. By constructing a complete pipeline that runs through perception and simulation, it directly realizes the joint end-to-end recognition of the geometry, appearance, and physical properties of an object. Furthermore, by capturing the visual and force data of the target object during a single interaction, it performs reverse loss optimization on the simulation model, thereby improving the accuracy of model reconstruction.

[0005] In a first aspect, this disclosure provides a simulation rendering interactive modeling method, comprising: acquiring point cloud data of a target object; performing three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object; calculating the visual loss of the simulated three-dimensional model based on acquired visual data, and calculating the force loss of the simulated three-dimensional model based on acquired force data; the visual data and the force data being captured simultaneously during a single interaction with the target object; calculating the comprehensive loss of the simulated three-dimensional model based on the visual loss and the force loss; and optimizing the parameters of the simulated three-dimensional model based on the comprehensive loss to obtain a target simulation model.

[0006] In some embodiments, acquiring point cloud data of a target object and performing three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object includes: generating a model mesh of the simulated three-dimensional model based on the point cloud data; acquiring the physical parameters of the target object itself, and inputting the physical parameters and the model mesh into a differentiable simulator to obtain a simulated three-dimensional model output by the differentiable simulator.

[0007] In some embodiments, generating the model mesh of the simulated 3D model based on the point cloud data includes: performing differentiable Poisson reconstruction on the point cloud data to obtain an indicator function mesh and a voxel mesh; extracting a triangular mesh from the indicator function mesh; and interpolating the voxel mesh to the vertices of the triangular mesh using trilinear interpolation to obtain the model mesh.

[0008] In some embodiments, the visual data includes actual color data and actual depth data extracted from an actual image captured during a single interaction of the target object; before calculating the visual loss of the simulated 3D model, the method further includes: simulating a simulated color-depth image of the target object using the simulated 3D model, and obtaining simulated color data and simulated depth data of the target object based on the color-depth image; calculating the visual loss of the simulated 3D model based on the acquired visual data includes: calculating the visual loss according to the actual color data, the actual depth data, the simulated color data, and the simulated depth data using the following expression:

[0009] In the formula, The actual color data is extracted from the actual image captured during a single interaction of the target object. The simulated color data is generated from the simulated color depth image through the simulated 3D model. The actual depth data is extracted from the actual image captured during a single interaction of the target object. To simulate the depth data in the simulated color depth image generated by the simulated 3D model.

[0010] In some embodiments, the force data includes the actual force captured on the target object during a single interaction; before calculating the force loss of the simulated 3D model, the method further includes: simulating the simulated force on the target object using the simulated 3D model; calculating the force loss of the simulated 3D model based on the acquired force data includes: calculating the force loss according to the following expression based on the actual force and the simulated force:

[0011] In the formula, The actual forces captured on the target object during this interaction; The simulated force is generated by the simulation three-dimensional model.

[0012] In some embodiments, optimizing the parameters of the simulated 3D model based on the comprehensive loss includes: calculating the comprehensive loss using the following expression:

[0013] In the formula, Visual weighting coefficient; For force perception weighting coefficients; The regularization weight coefficient; This is a priori regularization.

[0014] Secondly, this disclosure provides a simulation rendering interactive modeling device, comprising: a simulation reconstruction module, used to acquire point cloud data of a target object, and perform three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object; a loss calculation module, used to calculate the visual loss of the simulated three-dimensional model based on acquired visual data, and to calculate the force loss of the simulated three-dimensional model based on acquired force data; the visual data and the force data are simultaneously captured during a single interaction with the target object; a comprehensive calculation module, used to calculate the comprehensive loss of the simulated three-dimensional model based on the visual loss and the force loss; and a model optimization module, used to optimize the parameters of the simulated three-dimensional model based on the comprehensive loss to obtain a target simulation model.

[0015] Thirdly, this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the foregoing aspects.

[0016] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the methods described in the above aspects.

[0017] Fifthly, this disclosure provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the foregoing aspects.

[0018] This disclosure provides a simulation rendering interactive modeling method, apparatus, device, and storage medium that captures visual and force data of a target object during a single interaction. When performing 3D simulation modeling of the target object, it calculates visual and force losses during the simulation process and uses these losses to optimize the parameters of the simulated 3D model, ensuring that the model accurately reproduces its static appearance while possessing precise dynamic behavior. Simultaneously, by constructing a complete pipeline connecting perception and simulation, it directly achieves end-to-end joint recognition of the object's geometry, appearance, and physical properties. Furthermore, it enables 3D simulation reconstruction of the target object from sparse observations through a single interaction, improving data utilization efficiency and making it suitable for scenarios with limited data acquisition and dangerous, urgent situations. Attached Figure Description

[0019] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a simulation rendering interactive modeling method provided in an embodiment of this disclosure.

[0020] Figure 2 This is a schematic diagram of the structure of a simulation rendering interactive modeling device provided in an embodiment of the present disclosure.

[0021] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0025] With the accelerated transformation of mining areas towards automation and intelligence, robotic arms have been widely applied in complex operational scenarios such as ore handling and equipment component transfer. In these tasks, robotic arms frequently interact with unknown objects in the environment, including irregularly shaped piles of ore and equipment parts of various shapes. The core requirement is to achieve efficient operation through precise manipulation. However, traditional modeling processes are time-consuming, labor-intensive, and have extremely poor generalization capabilities. Every time a new type of object is encountered, manual calibration and model building must be carried out again, which is completely unsuitable for the complex and ever-changing mining environment and the high real-time requirements of operational tasks. In the field of robotics, although research has confirmed the importance of building predictive world models from sparse observations for task planning, existing differentiable modeling methods still have significant shortcomings: they cannot simultaneously optimize the geometric, appearance, and physical properties of the scene, and they lack an end-to-end processing pipeline for deep fusion of visual and tactile data. Therefore, how to accurately estimate the three-dimensional shape, surface texture, mass, friction coefficient, and other key parameters of an object through a single interactive action such as pushing or pulling of a robotic arm without relying on prior models, and then generate a high-fidelity digital twin model that can be directly used for simulation and rendering, has become a technical bottleneck that urgently needs to be overcome in the automated operation of robotic arms in mining areas. Therefore, a simulation rendering interactive modeling method, device, equipment, and storage medium are needed to solve the above-mentioned technical problems. The simulation rendering interactive modeling method, device, equipment, and storage medium provided in the embodiments of this disclosure will be described in detail below.

[0026] Example 1 Figure 1 This is a flowchart illustrating a simulation rendering interactive modeling method provided in an embodiment of this disclosure. Figure 1 As shown, a simulation rendering interactive modeling method includes: Step 101: Obtain the point cloud data of the target object, and perform three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object.

[0027] Specifically, point cloud data is a collection of massive points scattered in three-dimensional space. Each point has a spatial position and a normal vector, forming the geometric position of a real object's surface. In the embodiments of this disclosure, three-dimensional simulation reconstruction integrates two technical processes: three-dimensional reconstruction and simulation. By collecting data of the target object in the real world, a corresponding high-precision three-dimensional digital model is constructed in a computer, and this three-dimensional digital model is given physical properties (such as gravity, hardness, elasticity, etc.), enabling the three-dimensional digital model to simulate dynamic behavior in real or virtual environments.

[0028] In some embodiments, acquiring point cloud data of a target object and performing three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object includes: generating a model mesh of the simulated three-dimensional model based on the point cloud data; acquiring the physical parameters of the target object itself, and inputting the physical parameters and the model mesh into a differentiable simulator to obtain a simulated three-dimensional model output by the differentiable simulator.

[0029] Specifically, in this embodiment of the disclosure, a differentiable physics simulator and a differentiable renderer are integrated into the same optimization process to achieve joint end-to-end recognition of object geometry, appearance, and physical properties.

[0030] Furthermore, in some embodiments, generating the model mesh of the simulated 3D model based on the point cloud data includes: performing differentiable Poisson reconstruction on the point cloud data to obtain an indicator function mesh and a voxel mesh; extracting a triangular mesh from the indicator function mesh; and interpolating the voxel mesh to the vertices of the triangular mesh using trilinear interpolation to obtain the model mesh.

[0031] Specifically, in some embodiments, the point cloud data of the target object is... Generate a voxelized indicator function grid through differentiable Poisson reconstruction. Furthermore, in the voxelized indicator function grid... Based on this, the triangular mesh is further extracted using the differentiable MarchingCubes algorithm. triangular mesh It can automatically fine-tune with parameter optimization, thus achieving end-to-end differentiable shape modeling. Additionally, voxel meshes... Representing the three-dimensional voxelized color or texture field of the target object, the voxel mesh... Interpolation to the triangular mesh is performed using trilinear interpolation. The mesh vertices are used to combine geometry and high-resolution appearance, resulting in a model mesh of the target object to be simulated and rendered. This embodiment of the disclosure inputs a point cloud with normal vectors for differentiable Poisson reconstruction and Marching Cubes to generate a triangular mesh; simultaneously, a voxel mesh is used to store the color field of the object surface, and texture interpolation is performed at the mesh vertices, balancing geometric reconstruction accuracy and rendering effect, providing a flexible differentiable basis for subsequent simulation and rendering.

[0032] After obtaining the model mesh, differentiable physical simulation and rendering are performed on the model mesh. The physical parameters of the target object itself are obtained, and these physical parameters, along with the model mesh, are input into a differentiable simulator to obtain the simulated 3D model output by the differentiable simulator. Specifically, the physical parameters of the target object itself... Including but not limited to object mass coefficient of friction Etc., this disclosure does not impose any limitations. When performing simulation using a differentiable simulator, the current mesh is... voxel grid and physical parameters All of these are used as inputs to a differentiable simulator to obtain a simulated 3D model. This simulated 3D model is then used to simulate the RGB-D images of the target object from various viewpoints, thus obtaining simulated color data from the simulated visual data. and simulated depth data and simulated force data .

[0033] Step 102: Calculate the visual loss of the simulated 3D model based on the acquired visual data, and calculate the force loss of the simulated 3D model based on the acquired force data.

[0034] Specifically, the visual data and the force data are captured simultaneously during a single interaction with the target object. In this embodiment, a robotic arm of a mining equipment performs a single controlled pushing interaction with the target object, and simultaneously collects the visual and force data of the target object during the pushing process. Based on physical consistency constraints, high-fidelity joint modeling of the geometry, appearance, and physical properties of complex objects in the mining area is achieved. During the interaction between the robotic arm and the target object, visual and force data are collected using a multimodal sensor. Specifically, the multimodal sensor includes an RGB-D camera, used to collect visual data (the color-depth image of the target object) to obtain the actual color data and actual depth data of the target object from multiple viewpoints. The multimodal sensor also includes a torque sensor for acquiring force feedback. It should be noted that, in some embodiments, the multimodal sensor can also collect data such as the target object's pose changes, surface morphology, and force response during movement as an observation dataset. Those skilled in the art can adjust the collected data according to actual application requirements, and this disclosure does not impose any limitations.

[0035] In this embodiment of the disclosure, visual and tactile information can be acquired simultaneously with only one push-press interaction of the mining robotic arm, and a high-precision reconstruction of the object model can be completed based on this sparse observation, which can greatly improve the data acquisition efficiency and is especially suitable for dangerous mining environments or emergency scenarios.

[0036] In some embodiments, the visual data includes actual color data and actual depth data extracted from the actual image captured during a single interaction of the target object; before calculating the visual loss of the simulated 3D model, the method further includes: simulating a simulated color depth image of the target object using the simulated 3D model, and obtaining simulated color data and simulated depth data of the target object based on the color depth image.

[0037] In some embodiments, during the acquisition of visual data, an RGB-D camera acquires actual color data and actual depth data from multiple viewpoints. During simulation, the same viewpoints are also used to obtain simulated color data and simulated depth data for the corresponding viewpoints. When calculating the visual loss in a single pass at the same viewpoint, color loss and depth loss are calculated separately, and then the calculated color loss and depth loss are summed to obtain the single-pass calculation result of the visual loss at the same viewpoint. When calculating the visual loss, a single-pass loss calculation is performed for the visual loss at the same viewpoint, and the results of the single-pass loss calculations for all viewpoints are summed to obtain the final visual loss.

[0038] The step of calculating the visual loss of the simulated 3D model based on the acquired visual data includes: calculating the visual loss according to the actual color data, the actual depth data, the simulated color data, and the simulated depth data using the following expression:

[0039] In the formula, The actual color data is extracted from the actual image captured during a single interaction of the target object. The simulated color data is generated from the simulated color depth image through the simulated 3D model. The actual depth data is extracted from the actual image captured during a single interaction of the target object. To simulate the depth data in the simulated color depth image generated by the simulated 3D model, visual loss is used to constrain the rendered image of the simulated 3D model to match the real observation, ensuring consistency between the rendered image and the real observation.

[0040] In some embodiments, during the acquisition of force data, a torque sensor collects force data during a single interaction. During simulation, the same interaction process is also simulated to obtain simulated forces under the corresponding interaction actions. When calculating the force loss in a single instance, the force loss at the same moment is calculated. When calculating the visual loss, the force loss at all moments is calculated in a single instance, and the results of the single force loss calculations at all moments are summed to obtain the final force loss.

[0041] In some embodiments, the force data includes the actual force captured on the target object during a single interaction; before calculating the force loss of the simulated 3D model, the method further includes: simulating the simulated force on the target object using the simulated 3D model; calculating the force loss of the simulated 3D model based on the acquired force data includes: calculating the force loss according to the following expression based on the actual force and the simulated force:

[0042] In the formula, The actual forces captured on the target object during this interaction; The simulated forces are generated by the simulated 3D model. Force perception loss is used to constrain the simulated physical response of the 3D model.

[0043] After calculating the visual loss and force loss, the overall loss of the simulated 3D model can be further calculated.

[0044] Step 103: Calculate the overall loss of the simulated 3D model based on the visual loss and the force loss.

[0045] Specifically, the overall loss is determined by visual loss, force loss, and prior regularization and their corresponding weights.

[0046] In some embodiments, optimizing the parameters of the simulated 3D model based on the comprehensive loss includes: calculating the comprehensive loss using the following expression:

[0047] In the formula, Visual weighting coefficient; For force perception weighting coefficients; The regularization weight coefficient; This is a priori regularization.

[0048] Step 104: Optimize the parameters of the simulated 3D model based on the comprehensive loss to obtain the target simulation model. In this embodiment, the loss gradient is back-transmitted from the output (visual data / force data) to the parameters of the simulated 3D model for correction.

[0049] In some embodiments, the parameter optimization process includes two stages: The first stage involves optimizing the geometric appearance parameters of the simulated 3D model. Specifically, after fixing the physical parameters of the simulated 3D model, the point cloud data and voxel mesh are optimized using only the initial observation image frames from the visual data, so that the initial rendered simulation results are consistent with the observations.

[0050] After optimizing the geometric appearance parameters of the simulated 3D model, its physical properties are further optimized. Specifically, the physical parameters of the simulated 3D model are unlocked and updated and optimized based on force data to ensure that the simulated physical trajectory and contact force are consistent with actual observations, ultimately resulting in the target simulation model. The optimized target simulation model includes a high-precision geometric mesh, high-resolution appearance texture, and accurate physical parameters. It can be used for robotic arm simulation path planning and physical prediction, and can also generate realistic new perspective images in remote control and AI training.

[0051] This disclosure proposes a step-by-step optimization of the geometric appearance and physical properties of a simulated 3D model. First, the geometry and texture are fine-tuned on a still frame, and then the physical parameters are optimized during dynamic pushing. This ensures that the model accurately reproduces the static appearance while having precise dynamic behavior. The output object model meets both simulation and rendering requirements. The final target simulation model can be directly used for subsequent robotic arm path planning and operation effect prediction. Furthermore, more realistic images can be generated based on this target simulation model to assist in remote control or train the vision system, thus achieving a balance between simulation accuracy and visual realism.

[0052] This disclosure provides a simulation rendering interactive modeling method, apparatus, device, and storage medium that captures visual and force data of a target object during a single interaction. When performing 3D simulation modeling of the target object, it calculates visual and force losses during the simulation process and uses these losses to optimize the parameters of the simulated 3D model, ensuring that the model accurately reproduces its static appearance while possessing precise dynamic behavior. Simultaneously, by constructing a complete pipeline connecting perception and simulation, it directly achieves end-to-end joint recognition of the object's geometry, appearance, and physical properties. Furthermore, it enables 3D simulation reconstruction of the target object from sparse observations through a single interaction, improving data utilization efficiency and making it suitable for scenarios with limited data acquisition and dangerous, urgent situations.

[0053] Example 2 Based on the above embodiments, this embodiment provides a simulation rendering interactive modeling device, such as... Figure 2As shown, it includes: a simulation reconstruction module, used to acquire point cloud data of a target object, and perform three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object; a loss calculation module, used to calculate the visual loss of the simulated three-dimensional model based on the acquired visual data, and to calculate the force loss of the simulated three-dimensional model based on the acquired force data; the visual data and the force data are captured simultaneously during a single interaction with the target object; a comprehensive calculation module, used to calculate the comprehensive loss of the simulated three-dimensional model based on the visual loss and the force loss; and a model optimization module, used to optimize the parameters of the simulated three-dimensional model based on the comprehensive loss to obtain a target simulation model.

[0054] In some embodiments of this example, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0055] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0056] In some embodiments of this example, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0057] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component, for executing the methods in the above embodiments.

[0058] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media can include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CDs). ROM, DVD ROM, Blu-ray discs, etc.

[0059] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0060] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0061] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0062] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0063] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0064] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0065] While the embodiments disclosed herein are as described above, the foregoing content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope of this disclosure; however, the scope of patent protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A simulation rendering interactive modeling method, characterized in that, include: The point cloud data of the target object is acquired, and the target object is reconstructed by three-dimensional simulation based on the point cloud data to obtain a simulated three-dimensional model of the target object. Based on the acquired visual data, the visual loss of the simulated 3D model is calculated, and based on the acquired force data, the force loss of the simulated 3D model is calculated. The visual data and the force data are captured simultaneously during a single interaction with the target object. Calculate the overall loss of the simulated 3D model based on the visual loss and the force loss. The parameters of the simulated 3D model are optimized based on the comprehensive loss to obtain the target simulation model.

2. The method according to claim 1, characterized in that, The process of acquiring point cloud data of the target object and performing 3D simulation reconstruction of the target object based on the point cloud data to obtain a simulated 3D model of the target object includes: Based on the point cloud data, generate the model mesh of the simulated 3D model; The physical parameters of the target object are obtained, and the physical parameters and the model mesh are input into a differentiable simulator to obtain a simulated 3D model output by the differentiable simulator.

3. The method according to claim 2, characterized in that, The step of generating the model mesh of the simulation 3D model based on the point cloud data includes: The point cloud data is reconstructed using a differentiable Poisson function to obtain an indicator function grid and a voxel grid. The triangular mesh is extracted from the indicator function mesh, and the voxel mesh is interpolated to the vertices of the triangular mesh using trilinear interpolation to obtain the model mesh.

4. The method according to claim 1, characterized in that, The visual data includes actual color data and actual depth data extracted from the actual image captured during a single interaction of the target object. Before calculating the visual loss of the simulated 3D model, the method further includes: The simulated color depth image of the target object is simulated using the simulated 3D model, and the simulated color data and simulated depth data of the target object are obtained based on the color depth image. The step of calculating the visual loss of the simulated 3D model based on the acquired visual data includes: calculating the visual loss according to the actual color data, the actual depth data, the simulated color data, and the simulated depth data using the following expression: In the formula, The actual color data is extracted from the actual image captured during a single interaction of the target object. The simulated color data is generated from the simulated color depth image through the simulated 3D model. The actual depth data is extracted from the actual image captured during a single interaction of the target object. To simulate the depth data in the simulated color depth image generated by the simulated 3D model.

5. The method according to claim 4, characterized in that, The force data includes the actual forces captured on the target object during a single interaction; Before calculating the force perception loss of the simulated 3D model, the method further includes: The simulated forces acting on the target object are simulated using the simulated 3D model. The step of calculating the force perception loss of the simulated 3D model based on the acquired force perception data includes: calculating the force perception loss according to the following expression based on the actual force and the simulated force: In the formula, The actual forces captured on the target object during this interaction; The simulated force is generated by the simulation three-dimensional model.

6. The method according to claim 5, characterized in that, The optimization of the parameters of the simulated 3D model based on the comprehensive loss includes: The combined loss is calculated using the following expression: In the formula, Visual weighting coefficient; For force perception weighting coefficients; The regularization weight coefficient; This is a priori regularization.

7. A simulation rendering interactive modeling device, characterized in that, include: The simulation reconstruction module is used to acquire point cloud data of the target object, and to perform three-dimensional simulation reconstruction of the target object based on the point cloud data to obtain a simulated three-dimensional model of the target object. The loss calculation module is used to calculate the visual loss of the simulated 3D model based on the acquired visual data, and to calculate the force loss of the simulated 3D model based on the acquired force data; the visual data and the force data are captured simultaneously during a single interaction with the target object. The comprehensive calculation module is used to calculate the comprehensive loss of the simulated 3D model based on the visual loss and the force loss. The model optimization module optimizes the parameters of the simulation 3D model based on the comprehensive loss to obtain the target simulation model.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images