A Physical-Based Multimodal Contact Simulation System and Method for Visuotactile Sensors

Abstract

The present invention discloses a physical-based multi-mode contact simulation system and method for a visual-tactile sensor, including: an elastomer contact simulation unit: used to perform particle processing on the elastomer of the visual-tactile sensor, and obtain the three-dimensional coordinates of the particles on the object surface through elastomer contact simulation; a coordinate system conversion unit: used to calculate the coordinates of the marked points in the pixel coordinate system through the coordinate system conversion relationship, and obtain the marked point simulation image; a tactile image simulation unit: used to process the three-dimensional coordinates of the particles to generate a tactile simulation image; a tactile marked point joint simulation unit: used to merge the marked point simulation image and the tactile simulation image to obtain a tactile marked point joint simulation image. The present invention can efficiently complete the simulation of tactile images, marked point images, and tactile marked point joint images of the visual-tactile sensor under motion states such as pressing, sliding, and rotating.

Description

Technical Field

[0001] The present invention relates to the technical field of tactile sensor simulation, and particularly to a physical-based multi-mode contact simulation system and method for a visual tactile sensor. Background Art

[0002] With the continuous development of robot technology, robots need to perform tasks in more complex and variable environments, which puts higher requirements on the tactile perception of robots. However, current tactile sensors, such as piezoresistive and magnetic touch sensors, have disadvantages such as less information and instability. Visual tactile sensors are a new type of optical sensor that uses a camera to capture the deformation of an elastomer and finally presents it in the form of an image, with advantages such as rich information and stable signals. However, due to reasons such as technology, the data acquisition cost of visual tactile sensors is relatively high. How to use simulation technology to generate high-quality visual tactile sensor contact information and thus reduce the data acquisition cost is of great significance.

[0003] The existing simulation technologies for visual tactile sensors are mainly divided into two types: based on generative models and based on finite element analysis (FEM). The former requires a large amount of visual tactile data to be collected, which further increases the acquisition cost and is limited by the breadth of the dataset and is difficult to be robust to different visual tactile sensors. Although the method based on finite element analysis (FEM) can effectively overcome the problems existing in the generative model, FEM requires a large amount of computing resources, which will reduce the simulation efficiency. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a physical-based multi-mode contact simulation system and method for a visual tactile sensor. Through technologies such as the material point method, camera model, and ray tracing, the simulation of tactile images, marker point images, and combined tactile marker point images of the visual tactile sensor in motion states such as pressing, sliding, and rotating is completed, overcoming problems such as poor robustness of the generative model-based method and low efficiency of the FEM model-based method.

[0005] On the one hand, to achieve the above object, the present invention provides a physical-based multi-mode contact simulation system for a visual tactile sensor, including:

[0006] An elastomer contact simulation unit: used to perform particle processing on the elastomer of the visual tactile sensor and obtain the three-dimensional coordinates of the particles on the object surface through elastomer contact simulation;

[0007] A coordinate system conversion unit: used to calculate the coordinates of the marker points in the pixel coordinate system through the coordinate system conversion relationship and obtain the marker point simulation image;

[0008] A tactile image simulation unit: used to process the three-dimensional coordinates of the particles and generate a tactile simulation image;

[0009] Tactile Marking Point Joint Simulation Unit: used to merge the marking point simulation image and the tactile simulation image to obtain a tactile marking point joint simulation image.

[0010] Preferably, the elastomer contact simulation unit includes:

[0011] Elastomer Particleization Module: used to particleize the elastomer of the visual tactile sensor;

[0012] Camera Calibration Module: used to calibrate the internal and external parameters of the camera used by the visual tactile sensor, and the internal and external parameters are used for the conversion from the world coordinate system to the image coordinate system in the coordinate system conversion unit;

[0013] Marking Point Setting Module: used to mark the particles according to the actual number and marking point positions, and the information of the marking points is included in the marked particles;

[0014] Elastomer Contact Simulation Module: used to obtain the three-dimensional coordinates of the particles.

[0015] Preferably, obtaining the three-dimensional coordinates of the particles includes:

[0016] Initialize the particles and grid nodes, where the particles are used to represent the elastomer and the indenter, and the grid composed of several grid nodes is used to record the object information with a fixed coordinate system;

[0017] Introduce the deformation map φ p : R 3 →R 3 Record the initial position and the final position of each particle during the simulation process, and introduce the affine velocity matrix C p ∈R 3×3 , and perform initialization;

[0018] Where the deformation gradient is derived as:

[0019]

[0020] Where, F p Is set to the three-dimensional identity matrix I 3×3 , x p ∈R 3 , is the position of the particle;

[0021] Calculate the mass and the grid momentum of the i-th grid node respectively:

[0022] Calculate the mass M i :

[0023]

[0024] Wherein, m p represents the mass of the p-th particle, G i represents 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle within the j-th grid, ω jp is the weight parameter of the j-th grid node and the p-th particle for weighted interpolation;

[0025] The grid momentum MG of the i-th grid node i is obtained by calculating the momentum MM generated by particle motion i and the momentum ME generated by elasticity i :

[0026] MG i = MM i + ME i (3)

[0027] After obtaining M i and MG i the velocity of the object near the i-th grid node V i is calculated as:

[0028]

[0029] Preferably, obtaining the three-dimensional coordinates of the particle further includes:

[0030] Updating the state of the particle using the previous states of the grid node and the particle, assuming the state at the k-th step, i.e., the velocity affine velocity and the deformation gradient are known, then the velocity affine velocity and the deformation gradient at the k+1 step are expressed as:

[0031]

[0032]

[0033] wherein, G' p represents the 3×3×3 grid nodes around the p-th particle, ω ip is the weight parameter of the i-th grid node and the p-th particle for weighted interpolation, V i (k) is the velocity owned by the i-th grid node, ΔX 2 is the square of the grid node interval, X i represents the position of the j-th adjacent node of the i-th grid node, is the position of the p-th particle, I is the identity matrix, and t is the time interval between two adjacent steps;

[0034] By equations (6) and (7), update the state of the particle, obtain the particle movement speed, and define each particle The position at the k+1 step is

[0035] Finally, obtain the three-dimensional coordinates representing the particles on the object surface.

[0036] Preferably, updating the state of the particle includes:

[0037] Set the speed of the elastomer bottom layer particles to 0 and calculate its speed

[0038]

[0039] where v i represents the speed of the indenter, I' represents the particles representing the indenter, B represents the particles on the elastomer bottom layer, and p is the p-th particle.

[0040] Preferably, the coordinate system conversion unit includes:

[0041] World coordinate system to camera coordinate system conversion module: used to convert the position of the particle in the world coordinate system to the camera coordinate system;

[0042] Camera coordinate system to image coordinate system conversion module: used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship between the camera coordinate system and the image coordinate system and obtain the marker point simulation image through segmentation and interpolation.

[0043] Preferably, the tactile image simulation unit includes a ray rendering module, and the ray rendering module interpolates the surface discrete points into a depth image based on the three-dimensional coordinates of the particles, grids the depth image, and renders the depth image to generate the tactile simulation image.

[0044] Preferably, the tactile marker point joint simulation unit includes a tactile marker point merging module, and the tactile marker point merging module is used to accumulate the marker point simulation image and the tactile simulation image to obtain the tactile marker point joint simulation image.

[0045] On the other hand, to achieve the above object, the present invention also provides a physical-based visual tactile sensor multi-mode contact simulation method, which is applied to a physical-based visual tactile sensor multi-mode contact simulation system, including:

[0046] The elastomer of the visual tactile sensor is granulated through an elastomer contact simulation unit, and the three-dimensional coordinates of the particles on the object surface are obtained through elastomer contact simulation;

[0047] Based on the three-dimensional coordinates, the coordinates of the marked points in the pixel coordinate system are calculated through a coordinate system conversion unit to obtain a marked point simulation image;

[0048] The three-dimensional coordinates of the particles are processed through a tactile image simulation unit to generate a tactile simulation image;

[0049] The marked point simulation image and the tactile simulation image are merged through a tactile marked point joint simulation unit to obtain a tactile marked point joint simulation image.

[0050] Compared with the prior art, the present invention has the following advantages and technical effects:

[0051] The present invention patent proposes a physics-based visual tactile sensor multi-mode contact simulation system and method. Through the connection and use of each module, high-quality simulations of tactile images, marked point images, and tactile marked point joint images of the visual tactile sensor can be completed under motion states such as pressing, sliding, and rotating, which can greatly reduce the time and cost of visual tactile contact data acquisition. Description of the Drawings

[0052] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0053] Figure 1 It is a flowchart of a physics-based visual tactile sensor multi-mode contact simulation method according to an embodiment of the present invention. Detailed Embodiments

[0054] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail this application.

[0055] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0056] The present invention proposes a physics-based visual tactile sensor multi-mode contact simulation system, including:

[0057] Elastomer contact simulation unit: used to particleize the elastomer of the visual-tactile sensor, and obtain the three-dimensional coordinates of the particles on the object surface through elastomer contact simulation;

[0058] Coordinate system conversion unit: used to calculate the coordinates of the marker points in the pixel coordinate system through the coordinate system conversion relationship, and obtain the marker point simulation image;

[0059] Tactile image simulation unit: used to process the three-dimensional coordinates of the particles to generate a tactile image;

[0060] Tactile marker point joint simulation unit: used to merge the marker point simulation image and the tactile image to obtain a tactile marker point joint simulation image.

[0061] The present invention integrates technologies such as the material point method, camera model, and ray tracing, and can efficiently complete the simulation of tactile images, marker point images, and tactile marker point joint images of the visual-tactile sensor under motion states such as pressing, sliding, and rotating.

[0062] Further, the elastomer contact simulation unit includes:

[0063] Elastomer particleization module: used to particleize the elastomer of the visual-tactile sensor;

[0064] Specifically, in this embodiment, the object is represented by particles using the material point method. The movement of the particles can be the deformation of the elastomer, and each particle contains object information such as mass, velocity, and deformation.

[0065] Camera calibration module: used to calibrate the internal parameters and external parameters of the camera used by the visual-tactile sensor, and the internal parameters and external parameters are used for the conversion from the world coordinate system to the image coordinate system in the coordinate system conversion unit;

[0066] Specifically, the visual-tactile sensor uses a camera to capture the deformation of the elastomer, and then presents the touch in the form of an image. Since different visual-tactile sensors use different cameras, it is necessary to calibrate different cameras to improve the robustness of the method to different visual-tactile sensors. The purpose of camera calibration is to determine the external parameters and internal parameters of the camera, that is, the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, and the transfer matrix from the camera coordinate system to the image coordinate system. Due to the differences in cameras, these parameters are inconsistent. Therefore, before simulation, it is necessary to calibrate the parameters of the used camera with the help of a calibration board and a calibration algorithm, etc.

[0067] Marker point setting module: used to mark the particles according to the actual number and marker point positions, and the information of the marker points is included in the marked particles;

[0068] Specifically, since the number of marker points used by different visual-tactile sensors and their positions on the elastomer are different, in order to achieve more realistic simulation, it is necessary to label the particles according to the actual number and positions of the marker points. During the simulation process, the information of the marker points is included in these labeled particles.

[0069] Elastomer contact simulation module: used to obtain the three-dimensional coordinates of the particles;

[0070] Specifically, it includes four steps: initialization, particle-to-grid, grid-to-particle, and particle operation. These parts include the entire process of information exchange, deformation simulation, and object movement.

[0071] Furthermore, obtaining the three-dimensional coordinates of the particles includes:

[0072] Initialization: Initialize the particles and grid nodes. Among them, the particles are used to represent the elastomer and the indenter, and the grid composed of several grid nodes is used to record the information of the object with a fixed coordinate system;

[0073] Specifically, for the p-th particle, consider its position x p ∈R 3 and velocity v p ∈R 3 to simulate the movement of the object;

[0074] For deformation simulation, a deformation map φ p :R 3 →R 3 is introduced, which records the initial and final positions of each particle in the simulation step. The deformation gradient F p ∈R 3×3 is derived as:

[0075]

[0076] where F p is set as the three-dimensional identity matrix I 3×3 during initialization.

[0077] In addition, an affine velocity matrix C p ∈R 3×3 for recording the velocities of adjacent particles is introduced to reduce information loss during the information exchange process between particles and grid nodes. It is initialized to O 3×3 because the particles are static and will be updated later.

[0078] Particle to Grid: In each simulation step, particle information is first transferred to the grid by calculating the momentum and mass in each grid node. It can be regarded as simulating the parts of the object around the grid nodes through quadratic B-spline weighted interpolation. It should be noted that the mass and momentum of the attached particles are collected, and thus the mass M of the i-th grid node i is:

[0079]

[0080] where m p represents the mass of the p-th particle, G i represents the 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle in the j-th grid, and ω jp is the weight parameter of the j-th grid node and the p-th particle for weighted interpolation, calculated by quadratic B-spline.

[0081] By applying such weight parameters, the contribution of nearby particles to M i is greater than that of distant particles, and the same weight parameter is used for mass and momentum calculations.

[0082] The grid momentum MG of the i-th grid node i can be obtained by calculating the momentum MM i generated by particle motion and the momentum ME i generated by elasticity:

[0083] MG i = MM i + ME i (3)

[0084] In this embodiment, MM i is calculated by collecting the velocities and affine velocities of nearby particles:

[0085]

[0086] where X j represents the position of the j-th adjacent node of the i-th grid node.

[0087] ME i can be obtained in the following way:

[0088]

[0089] where Δt is the time interval between two adjacent steps, ΔX is the grid node interval, represents the initial particle volume, and S p is the elastic force of the p-th particle.

[0090] After obtaining M and MG using equations (2) and (3) respectively, i and MG, i the velocity of the object near the i-th grid node V i can be calculated as:

[0091]

[0092] Furthermore, obtaining the three-dimensional coordinates of the particles also includes:

[0093] Updating the state of the particles using the previous states of the grid nodes and the particles, which can be regarded as simulating some objects around the particles through quadratic B-spline weighted interpolation.

[0094] Assuming the state at the k-th step, i.e., the velocity affine velocity and deformation gradient are known, then in the k + 1 step, and can be expressed as:

[0095]

[0096] where G' p represents the 3×3×3 grid nodes around the p-th particle, ω ip is the weight parameter of the i-th grid node and the p-th particle for weighted interpolation, V i (k ) is the velocity owned by the i-th grid node, ΔX 2 is the square of the grid node interval, X i represents the position of the j-th adjacent node of the i-th grid node, is the position of the p-th particle, I is the identity matrix, and t is the time interval between two adjacent steps.

[0097] Through equations (8) and (9), the state of the particles can be updated.

[0098] However, there are two special cases: the velocity of the particles at the bottom layer of the elastomer is set to 0 because they are fixed to the sensor (the sensor is static in this embodiment); since the indenter is rigid, the particles representing the indenter share the same velocity, i.e., the velocity of the indenter. Therefore, the of these special particles is as follows:

[0099]

[0100] where v i represents the velocity of the indenter; I' represents the particles representing the indenter; B represents the particles on the bottom layer of the elastomer, and p is the p-th particle.

[0101] Finally, the particles move at the speed calculated above, and each particle The position at step k + 1 is:

[0102]

[0103] After completing the elastomer contact simulation module, the three-dimensional coordinate system of the surface particles representing the object can be obtained, and then it is divided into two routes: tactile simulation and marker point simulation.

[0104] Furthermore, the coordinate system conversion unit includes:

[0105] World coordinate system to camera coordinate system conversion module: used to convert the position of the particles in the world coordinate system to the camera coordinate system;

[0106] Camera coordinate system to image coordinate system conversion module: used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship between the camera coordinate system and the image coordinate system, and obtain the marker point simulation image through segmentation and interpolation.

[0107] Specifically, the world coordinate system to camera coordinate system conversion module: through the camera calibration process, the external parameters of the camera can be obtained, and then the position of the particles in the world coordinate system can be converted to the camera coordinate system:

[0108] P = RP w + t (12)

[0109] In the formula, R is the rotation matrix, t is the translation vector, P is the coordinate of a certain particle point in the camera coordinate system, and P w is the coordinate of a certain point in the world coordinate system, and P w is obtained by the simulator.

[0110] Camera coordinate system to image coordinate system conversion module: in the above process, the coordinate of a certain point in the camera coordinate system can be obtained. Assuming P w = [X, Y, Z], through the principle of similar triangles, it can be obtained:

[0111]

[0112] In the pixel coordinate system, the origin is the upper left corner of the image in this embodiment, and there is a scaling ratio difference between the pixel plane and the imaging plane, so it can be obtained:

[0113]

[0114] In the formula, u, v are the positions of the corresponding points in the pixel coordinate system, where f x = αf and f y= βf; α and β are the scaling ratios from the pixel plane to the imaging plane respectively, and f is the focal length of the camera.

[0115] From equations (13) and (14), the conversion relationship from the camera coordinate system to the image coordinate system can be obtained as:

[0116]

[0117] From equations (11) and (15), the total conversion relationship from the world coordinate system to the pixel coordinate system can be obtained as:

[0118]

[0119] In the above formula is the camera internal parameter matrix, and R and t are the rotation matrix and translation vector in the camera external parameters respectively.

[0120] After obtaining the coordinates of the marked points in the pixel coordinate system, the simulated images of the marked points can be obtained through segmentation and interpolation.

[0121] Furthermore, the tactile image simulation unit includes a ray rendering module. The ray rendering module interpolates the surface discrete points into a depth image based on the three-dimensional coordinates of the particles, meshes the depth image, and renders the depth image to generate a tactile simulation image.

[0122] Specifically, the ray rendering module adopts a physically based ray rendering method, such as ray tracing rendering, Phong model rendering, raster rendering, etc. In this embodiment, the ray tracing rendering method is selected. Ray tracing is a physically based rendering method that simulates the propagation of light in a medium and finally collects and summarizes it. Since the light path is reversible, light rays can be emitted from the camera at random angles, their trajectories can be tracked, and the light rays in their reverse directions can be calculated. By calculating the color attenuation and accumulation along these paths, the color of each pixel can be approximated. In fact, the light emitted from the source travels in a straight line until it encounters a surface that obstructs its path, where it can be absorbed, refracted, or reflected. In the case of a translucent medium, the light can undergo partial refraction and reflection. Due to surface roughness, each reflection generates several diffuse rays in different directions. After each reflection, the intensity of the light decreases. In this embodiment, the software Blender is used to model the scene and the physically based path tracer in Blender-Cycles for image rendering. After the rendering is completed, the color information of each pixel point on the tactile simulation image can be obtained, and thus the simulation of the tactile image is completed.

[0123] Furthermore, the tactile marked point joint simulation unit includes a tactile marked point merging module. The tactile marked point merging module is used to accumulate the marked point simulation image and the tactile image to obtain a tactile marked point joint simulation image.

[0124] Through the connection and use of each module, the present invention can complete the high-quality simulation of tactile images, marker point images, and combined tactile marker point images in motion states such as pressing, sliding, and rotating of the visual-tactile sensor, which can greatly reduce the time and cost of visual-tactile contact data acquisition. With the help of the elastomer particle module, camera calibration module, marker point setting module, elastomer contact simulation module, and coordinate system conversion unit, the marker point simulation image is completed. With the help of the elastomer particle module, camera calibration module, marker point setting module, elastomer contact simulation module, and ray tracing rendering module, the tactile simulation image is completed. Based on the tactile simulation image and the marker point simulation image, the combined tactile marker point image simulation is further completed.

[0125] This embodiment also provides a physical-based multi-mode contact simulation method for a visual-tactile sensor, which is applied to a physical-based multi-mode contact simulation system for a visual-tactile sensor, such as Figure 1 , including:

[0126] Process the elastomer of the visual-tactile sensor through the elastomer contact simulation unit, and present the deformation of the elastomer of the visual-tactile sensor in the form of an image. Through elastomer contact simulation, the three-dimensional coordinates of the particles on the object surface are obtained;

[0127] Based on the three-dimensional coordinates, calculate the coordinates of the marker points in the pixel coordinate system through the coordinate system conversion unit to obtain the marker point simulation image;

[0128] Process the three-dimensional coordinates of the particles through the tactile image simulation unit to generate a tactile simulation image;

[0129] Merge the marker point simulation image and the tactile simulation image through the combined tactile marker point simulation unit to obtain the combined tactile marker point simulation image.

[0130] In order to more clearly express the technical solution of the present invention, specific embodiments are provided below for scheme introduction:

[0131] First, in the elastomer particleization module, the probe in contact with the visual and tactile sensor and the elastomer of the visual and tactile sensor are respectively particleized as follows: the quantities are 40,000 and 80,000, and the grid is set to 128*128*128. In the camera calibration module, a calibration board is used to calibrate the camera used by the visual and tactile sensor to obtain the external and internal parameter matrices of the camera. Then, in the marker point setting module, the particles representing the elastomer of the visual and tactile sensor are set as marker points according to the actual array size of the real sensor marker point array. Next, in the elastomer contact simulation module, the particles representing the probe are moved to make them contact the particles representing the elastomer. The probe is a rigid body, and these points have information such as velocity. This information will complete information transmission through interaction with the particles representing the elastomer of the visual and tactile sensor, and this process is the operation process of the material point method. Then, in each step, the position of the particles is updated according to the information obtained by the particles themselves, and thus the three-dimensional coordinates of each particle can be obtained.

[0132] Marker point simulation image:

[0133] After obtaining the three-dimensional coordinates of each particle, the internal and external parameter information of the camera calibrated in advance can be used in the world coordinate system to camera coordinate system conversion module and the camera coordinate system to image coordinate system conversion module to map the particle three-dimensional coordinate information into the camera image coordinate system, thereby completing the marker point simulation image.

[0134] Tactile simulation image:

[0135] After obtaining the three-dimensional coordinates of each particle, the surface discrete points can be interpolated into a depth image of 640*480 using the method of linear interpolation. Then, the depth image is meshed and the optical tracking rendering module is used to render the depth image and finally generate the tactile simulation image.

[0136] Tactile marker point combined simulation image:

[0137] After obtaining the marker point simulation image and the tactile simulation image, the two images can be physically accumulated to obtain the tactile marker point combined simulation image.

[0138] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A physics-based visual-tactile sensor multi-mode contact simulation system, characterized in that: include: Elastic body contact simulation unit: used to particle the elastic body of the visual tactile sensor, and obtain the three-dimensional coordinates of the particles on the surface of the object through elastic body contact simulation; Coordinate system conversion unit: used to calculate the coordinates of the marking point in the pixel coordinate system through the coordinate system conversion relationship to obtain a simulated image of the marking point; Tactile image simulation unit: used to process the three-dimensional coordinates of particles and generate tactile simulation images; A tactile marker point joint simulation unit is used to merge the marker point simulation image with the tactile simulation image to obtain a tactile marker point joint simulation image; The elastic body contact simulation unit comprises: Elastic body particle module: used to particle the visual tactile sensor elastic body; wherein the visual tactile sensor uses a camera to capture the deformation of the elastic body and then presents the tactile sensation in the form of an image; Camera calibration module: used to calibrate the intrinsic and extrinsic parameters of the camera used by the visual tactile sensor, wherein the intrinsic and extrinsic parameters are used to convert the world coordinate system to the image coordinate system in the coordinate system conversion unit; Marking point setting module: used to mark particles according to the actual number and marking point position. The marking point information is included in the marked particles. Elastic contact simulation module: used to obtain the three-dimensional coordinates of particles; Obtaining the three-dimensional coordinates of the particle, comprising: Initializing particles and mesh nodes, wherein the particles are used to represent the elastic body and the indenter, and the mesh composed of a number of the mesh nodes is used to record object information with a fixed coordinate system; Introducing deformation graph φ p :R 3 →R 3 Record the initial and final positions of each particle during the simulation, and introduce an affine velocity matrix C that records the velocities of adjacent particles p ∈R 3×3 , initialize; The deformation gradient is derived as: Among them, F p It is set to the three-dimensional identity matrix I during initialization 3×3 , x p ∈R 3 , is the position of the particle; Calculate the mass and grid momentum of the i-th grid node respectively: Calculate the mass M of the i-th mesh node i : In the formula, m p represents the mass of the pth particle, G i represents the 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle in the jth grid, ω jp is the weight parameter of the jth grid node and the pth particle used for weighted interpolation; The mesh momentum MG of the i-th mesh node i By calculating the momentum MM generated by the particle motion i and the momentum ME generated by elasticity i get: MG i =MM i +ME i (3) Get M i and MG i Afterwards, near the i-th grid node V i The object velocity is calculated as: The tactile marker point joint simulation unit includes a tactile marker point merging module, and the tactile marker point merging module is used to accumulate the marker point simulation image and the tactile simulation image to obtain a tactile marker point joint simulation image.

2. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: Obtaining the three-dimensional coordinates of the particle also includes: The state of the particle will be updated using the grid node and the previous state of the particle. Assume that the state of the kth step is Affine Speed and deformation gradient is known, then the speed in step k+1 is Affine Speed and deformation gradient It is expressed as: Among them, G' p represents the 3×3×3 grid nodes around the pth particle, ω ip is the weight parameter of the i-th grid node and the p-th particle used for weighted interpolation, V i (k) is the velocity of the ith grid node, ΔX 2 is the square of the grid node spacing, X i represents the position of the jth neighboring node of the ith grid node, is the position of the pth particle, I is the identity matrix, and t is the time interval between two adjacent steps; Through equations (6) and (7), the particle state is updated, the particle moving speed is obtained, and each particle is defined The position in step k+1 is Finally, the three-dimensional coordinates of particles representing the surface of the object are obtained.

3. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 2, characterized in that: Update the state of the particle, including: Set the velocity of the particles at the bottom of the elastic body to 0 and calculate its velocity Among them, v i represents the speed of the indenter, I' represents the particle representing the indenter, B represents the particle on the bottom layer of the elastomer, and p is the pth particle.

4. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The coordinate system conversion unit comprises: World coordinate system to camera coordinate system conversion module: used to convert the position of particles in the world coordinate system to the camera coordinate system; The module for converting the camera coordinate system to the image coordinate system is used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship from the camera coordinate system to the image coordinate system and to obtain the simulated image of the marker points through segmentation and interpolation.

5. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The tactile image simulation unit includes a light rendering module, which interpolates surface discrete points into a depth image based on the three-dimensional coordinates of the particles, grids the depth image, and renders the depth image to generate the tactile simulation image.

6. A physics-based visual-tactile sensor multi-mode contact simulation method, applied to the physics-based visual-tactile sensor multi-mode contact simulation system according to any one of claims 1 to 5, characterized in that: include: The elastic body of the visual tactile sensor is particle-processed by the elastic body contact simulation unit, and the three-dimensional coordinates of the particles on the surface of the object are obtained by elastic body contact simulation. Based on the three-dimensional coordinates, the coordinates of the marking point in the pixel coordinate system are calculated by a coordinate system conversion unit to obtain a simulated image of the marking point; The three-dimensional coordinates of the particles are processed by a tactile image simulation unit to generate a tactile simulation image; The marker point simulation image and the tactile simulation image are combined through a tactile marker point joint simulation unit to obtain a tactile marker point joint simulation image.

Images