Physics-based visual tactile sensor multi-mode contact simulation system and method

Abstract

The invention discloses a physics-based visual tactile sensor multi-mode contact simulation system and method, and the system comprises an elastic body contact simulation unit which is used for carrying out the particle processing of an elastic body of a visual tactile sensor, and obtaining the three-dimensional coordinates of particles on the surface of an object through the contact simulation of the elastic body; the coordinate system conversion unit is used for calculating the coordinates of the mark points on the pixel coordinate system through the coordinate system conversion relation to obtain a mark point simulation image; the tactile image simulation unit is used for processing the three-dimensional coordinates of the particles to generate a tactile simulation image; and the tactile mark point joint simulation unit is used for combining the mark point simulation image and the tactile simulation image to obtain a tactile mark point joint simulation image. According to the method, simulation of the visual tactile sensor tactile image, the mark point image and the tactile mark point combined image in the motion states of pressing, sliding, rotating and the like can be efficiently completed.

Description

Technical Field

[0001] The present invention relates to the technical field of tactile sensor simulation, and in particular to a physics-based visual tactile sensor multi-mode contact simulation system and method. Background Art

[0002] With the continuous development of robotics technology, robots need to perform tasks in more complex and changeable environments, which puts higher requirements on the robot's tactile perception. However, current tactile sensors, such as piezoresistance and magnetic touch, have shortcomings such as low information content and instability. Visual tactile sensor is a new type of optical sensor that uses a camera to capture the deformation of the elastic body and finally presents it in the form of an image. It has the advantages of rich information and stable signals. However, due to process and other reasons, the data acquisition cost of visual tactile sensor is relatively high. How to use simulation technology to generate high-quality visual tactile sensor contact information and thus reduce the cost of data acquisition is of great significance.

[0003] Existing simulation technologies for visual-tactile sensors are mainly divided into two types: those based on generative models and those based on finite element analysis (FEM). The former requires the collection of a large amount of visual-tactile data, which further increases the collection cost, and is limited by the breadth of the data set, making it difficult to generate robustness for different visual-tactile sensors. Although the method based on finite element analysis (FEM) can effectively overcome the problems of the generative model, FEM requires a lot of computing resources, which reduces the efficiency of the simulation. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a physics-based visual-tactile sensor multi-modal contact simulation system and method, which completes the simulation of tactile images, marker point images and tactile marker point joint images of the visual-tactile sensor in motion states such as pressing, sliding and rotation through technologies such as material point method, camera model and ray tracing, overcoming the problems of poor robustness of generative models and low efficiency of FEM models.

[0005] On the one hand, to achieve the above-mentioned object, the present invention provides a physics-based visual tactile sensor multi-mode contact simulation system, comprising:

[0006] Elastic body contact simulation unit: used to particle the elastic body of the visual tactile sensor, and obtain the three-dimensional coordinates of the particles on the surface of the object through elastic body contact simulation;

[0007] Coordinate system conversion unit: used to calculate the coordinates of the marking point in the pixel coordinate system through the coordinate system conversion relationship to obtain a simulated image of the marking point;

[0008] Tactile image simulation unit: used to process the three-dimensional coordinates of particles and generate tactile simulation images;

[0009] Tactile marker point joint simulation unit: used for merging the marker point simulation image with the tactile simulation image to obtain a tactile marker point joint simulation image.

[0010] Preferably, the elastic body contact simulation unit comprises:

[0011] Elastic body particle module: used to particle the visual tactile sensor elastic body;

[0012] Camera calibration module: used to calibrate the intrinsic and extrinsic parameters of the camera used by the visual tactile sensor, wherein the intrinsic and extrinsic parameters are used to convert the world coordinate system to the image coordinate system in the coordinate system conversion unit;

[0013] Marking point setting module: used to mark particles according to the actual number and marking point position. The marking point information is included in the marked particles.

[0014] Elastic contact simulation module: used to obtain the three-dimensional coordinates of particles.

[0015] Preferably, obtaining the three-dimensional coordinates of the particle comprises:

[0016] Initializing particles and mesh nodes, wherein the particles are used to represent the elastic body and the indenter, and the mesh composed of a number of the mesh nodes is used to record object information with a fixed coordinate system;

[0017] Introducing deformation graph φ p : R 3 →R 3 Record the initial and final positions of each particle during the simulation, and introduce an affine velocity matrix C that records the velocities of adjacent particles p ∈R 3×3 , initialize;

[0018] The deformation gradient is derived as:

[0019]

[0020] Among them, F p It is set to the three-dimensional identity matrix I during initialization 3×3 , x p ∈R 3 , is the position of the particle;

[0021] Calculate the mass and grid momentum of the i-th grid node respectively:

[0022] Calculate the mass M of the i-th mesh node i :

[0023]

[0024] In the formula, m p represents the mass of the pth particle, G i represents the 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle in the jth grid, ω jp is the weight parameter of the jth grid node and the pth particle used for weighted interpolation;

[0025] The mesh momentum MG of the i-th mesh node i By calculating the momentum MM generated by the particle motion i and the momentum ME generated by elasticity i get:

[0026] MG i =MM i +ME i (3)

[0027] Get M i and MG i Afterwards, near the i-th grid node V i The object velocity is calculated as:

[0028]

[0029] Preferably, obtaining the three-dimensional coordinates of the particle further comprises:

[0030] The state of the particle will be updated using the grid node and the previous state of the particle. Assume that the state of the kth step is Affine Speed and deformation gradient is known, then the speed in step k+1 is Affine Speed and deformation gradient It is expressed as:

[0031]

[0032]

[0033] Among them, G' p represents the 3×3×3 grid nodes around the pth particle, ω ip is the weight parameter of the i-th grid node and the p-th particle used for weighted interpolation, V i (k) is the velocity of the ith grid node, ΔX 2 is the square of the grid node spacing, X i represents the position of the jth neighboring node of the ith grid node, is the position of the pth particle, I is the identity matrix, and t is the time interval between two adjacent steps;

[0034] Through equations (6) and (7), the particle state is updated, the particle moving speed is obtained, and each particle is defined The position in step k+1 is

[0035] Finally, the three-dimensional coordinates of particles representing the surface of the object are obtained.

[0036] Preferably, updating the state of the particle comprises:

[0037] Set the velocity of the particles at the bottom of the elastic body to 0 and calculate its velocity

[0038]

[0039] Among them, v i represents the speed of the indenter, I' represents the particle representing the indenter, B represents the particles on the bottom layer of the elastomer, and p is the pth particle.

[0040] Preferably, the coordinate system conversion unit comprises:

[0041] World coordinate system to camera coordinate system conversion module: used to convert the position of particles in the world coordinate system to the camera coordinate system;

[0042] The module for converting the camera coordinate system to the image coordinate system is used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship from the camera coordinate system to the image coordinate system and to obtain the simulated image of the marker points through segmentation and interpolation.

[0043] Preferably, the tactile image simulation unit comprises a light rendering module, which interpolates surface discrete points into a depth image based on the three-dimensional coordinates of the particles, grids the depth image, and renders the depth image to generate the tactile simulation image.

[0044] Preferably, the tactile marker point joint simulation unit comprises a tactile marker point merging module, and the tactile marker point merging module is used for accumulating the marker point simulation image and the tactile simulation image to obtain a tactile marker point joint simulation image.

[0045] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a physics-based visual-tactile sensor multi-mode contact simulation method, which is applied to a physics-based visual-tactile sensor multi-mode contact simulation system, comprising:

[0046] The elastic body of the visual tactile sensor is particle-processed by the elastic body contact simulation unit, and the three-dimensional coordinates of the particles on the surface of the object are obtained by elastic body contact simulation.

[0047] Based on the three-dimensional coordinates, the coordinates of the marking point in the pixel coordinate system are calculated by a coordinate system conversion unit to obtain a simulated image of the marking point;

[0048] The three-dimensional coordinates of the particles are processed by a tactile image simulation unit to generate a tactile simulation image;

[0049] The marker point simulation image and the tactile simulation image are combined through a tactile marker point joint simulation unit to obtain a tactile marker point joint simulation image.

[0050] Compared with the prior art, the present invention has the following advantages and technical effects:

[0051] The patent of this invention proposes a physics-based multi-mode contact simulation system and method for visual-tactile sensors. Through the connection and use of various modules, high-quality simulation of tactile images, marker point images and tactile marker point joint images of visual-tactile sensors in motion states such as pressing, sliding and rotation can be completed, which can greatly reduce the time and cost of visual-tactile contact data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 The present invention is a flowchart of a physics-based multi-mode contact simulation method for visual-tactile sensors according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] The present invention proposes a physics-based visual-tactile sensor multi-mode contact simulation system, comprising:

[0057] Elastic body contact simulation unit: used to particle the elastic body of the visual tactile sensor, and obtain the three-dimensional coordinates of the particles on the surface of the object through elastic body contact simulation;

[0058] Coordinate system conversion unit: used to calculate the coordinates of the marking point in the pixel coordinate system through the coordinate system conversion relationship to obtain a simulated image of the marking point;

[0059] Tactile image simulation unit: used to process the three-dimensional coordinates of particles and generate tactile images;

[0060] A tactile marker point joint simulation unit is used to merge the marker point simulation image with the tactile image to obtain a tactile marker point joint simulation image.

[0061] The present invention integrates the material point method, camera model, ray tracing and other technologies, and can efficiently complete the simulation of tactile images of visual tactile sensors, marker point images and tactile marker point joint images under motion states such as pressing, sliding and rotation.

[0062] Furthermore, the elastic body contact simulation unit comprises:

[0063] Elastic body particle module: used to particle the visual tactile sensor elastic body;

[0064] Specifically, this embodiment uses particles to represent objects through the material point method. The movement of particles can be the deformation of elastic bodies. Each particle contains object information, such as mass, velocity, and deformation.

[0065] Camera calibration module: used to calibrate the intrinsic and extrinsic parameters of the camera used by the visual tactile sensor, wherein the intrinsic and extrinsic parameters are used to convert the world coordinate system to the image coordinate system in the coordinate system conversion unit;

[0066] Specifically, the visual tactile sensor uses a camera to capture the deformation of the elastic body, and then presents the tactile sensation in the form of an image. Since different visual tactile sensors use different cameras, calibration is required for different cameras to improve the robustness of the method for different visual tactile sensors. The purpose of camera calibration is to determine the external and internal parameters of the camera, that is, the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, and the transfer matrix from the camera coordinate system to the image coordinate system. Due to the difference in cameras, these parameters are inconsistent. For this reason, before simulation, it is necessary to calibrate the parameters of the camera used with the help of a calibration plate and a calibration algorithm.

[0067] Marking point setting module: used to mark particles according to the actual number and marking point position. The marking point information is included in the marked particles.

[0068] Specifically, since different visual-tactile sensors use different numbers of marking points and different positions on the elastic body, in order to achieve a more realistic simulation, the particles need to be marked according to the actual number and position of the marking points. During the simulation process, the information of the marking points is included in these marked particles.

[0069] Elastic contact simulation module: used to obtain the three-dimensional coordinates of particles;

[0070] Specifically, it includes four steps: initialization, particle to grid, grid to particle, and particle operation. These parts include the entire process of information exchange, deformation simulation, and object movement.

[0071] Further, obtaining the three-dimensional coordinates of the particle comprises:

[0072] Initialization: Initializing particles and mesh nodes, wherein particles are used to represent elastic bodies and pressure heads, and a mesh composed of a number of mesh nodes is used to record object information with a fixed coordinate system;

[0073] Specifically, for the pth particle, consider its position x p ∈R 3 and speed v p ∈R 3 To simulate the movement of objects;

[0074] For deformation simulation, the deformation map φ is introduced p :R 3 →R 3 , which records the initial and final positions of each particle in the simulation step, the deformation gradient F p ∈R 3×3 Export as:

[0075]

[0076] Among them, F p It is set to the three-dimensional identity matrix I during initialization 3×3 .

[0077] In addition, an affine velocity matrix C is introduced to record the velocities of adjacent particles. p ∈R 3×3 , in order to reduce the information loss in the information exchange process between particles and grid nodes. It is initialized to O 3×3 , because the particles are static and will be updated later.

[0078] Particle to Grid: In each simulation step, the particle information is first transferred to the grid by calculating the momentum and mass in each grid node. It can be seen as simulating the part of the object around the grid node through quadratic B-spline weighted interpolation. It is worth noting that the mass and momentum of the surrounding particles are collected, so the mass M of the i-th grid node is i for:

[0079]

[0080] Among them, m p represents the mass of the pth particle, G i represents the 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle in the jth grid, ω jp is the weight parameter of the j-th grid node and the p-th particle used for weighted interpolation, calculated by quadratic B-spline.

[0081] By applying such a weight parameter, the close-range particle pair M i The contribution is greater than that of distant particles and has the same weight parameter for mass and momentum calculations.

[0082] The mesh momentum MG of the i-th mesh node i The momentum MM generated by the particle motion can be calculated by i and the momentum ME generated by elasticity i To obtain:

[0083] MG i =MM i +ME i (3)

[0084] In this embodiment, MM i is calculated by collecting the velocities and affine velocities of nearby particles:

[0085]

[0086] Among them, X j Represents the position of the jth neighboring node of the ith grid node.

[0087] ME i It can be obtained by:

[0088]

[0089] Where Δt is the time interval between two adjacent steps, ΔX is the grid node spacing, represents the initial particle volume, S p is the elastic force of the pth particle.

[0090] When using equations (2) and (3) to obtain M i and MG i Afterwards, near the i-th grid node V i The object velocity can be calculated as:

[0091]

[0092] Furthermore, obtaining the three-dimensional coordinates of the particles also includes:

[0093] The state of the particle will be updated using the grid nodes and the previous state of the particle, which can be seen as simulating part of the object around the particle through quadratic B-spline weighted interpolation.

[0094] Assume the state of step k, that is, the speed Affine Speed and deformation gradient is known, then in step k+1 and It can be expressed as:

[0095]

[0096] Among them, G' p represents the 3×3×3 grid nodes around the pth particle, ω ip is the weight parameter of the i-th grid node and the p-th particle used for weighted interpolation, V i (k ) is the velocity of the i-th grid node, ΔX 2 is the square of the grid node spacing, X i represents the position of the jth neighboring node of the ith grid node, is the position of the pth particle, I is the identity matrix, and t is the time interval between two adjacent steps.

[0097] Through equations (8) and (9), the state of the particle can be updated.

[0098] However, there are two special cases: the velocity of the elastomer underlying particles is set to 0 because they are fixed to the sensor (the sensor is static in this example); since the indenter is rigid, the particles representing the indenter share the same velocity, i.e., the velocity of the indenter, so the velocity of these special particles As shown below:

[0099]

[0100] Among them, v i represents the speed of the indenter; I' represents the particle representing the indenter; B represents the particle on the bottom layer of the elastomer, and p is the pth particle.

[0101] Finally, the particles move at the speed calculated above, and each particle The position in step k+1 is:

[0102]

[0103] After completing the elastic body contact simulation module, the three-dimensional coordinate system of the surface particles representing the object can be obtained, which is then divided into two routes: tactile simulation and marker point simulation.

[0104] Furthermore, the coordinate system conversion unit includes:

[0105] World coordinate system to camera coordinate system conversion module: used to convert the position of particles in the world coordinate system to the camera coordinate system;

[0106] The module for converting the camera coordinate system to the image coordinate system is used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship from the camera coordinate system to the image coordinate system and to obtain the simulated image of the marker points through segmentation and interpolation.

[0107] Specifically, the module for converting the world coordinate system to the camera coordinate system: through the camera calibration process, the external parameters of the camera can be obtained, and then the position of the particle in the world coordinate system can be converted to the camera coordinate system:

[0108] P=RP w +t (12)

[0109] Where R is the rotation matrix, t is the translation vector, P is the coordinate of a particle point in the camera coordinate system, and P w is the coordinate of a point in the world coordinate system, P w is obtained by the simulator.

[0110] Camera coordinate system to image coordinate system conversion module: In the above process, the coordinates of a certain point in the camera coordinate system can be obtained. Assume that P w =[X,Y,Z], through the principle of similar triangles, we can get:

[0111]

[0112] In the pixel coordinate system, the origin is the upper left corner of the image in this embodiment, and there is a scaling factor difference between the pixel plane and the imaging plane, so:

[0113]

[0114] Where u and v are the positions of the corresponding points in the pixel coordinate system, and f x =αf and f y=βf; α and β are the scaling ratios from the pixel plane to the imaging plane, and f is the focal length of the camera.

[0115] From equations (13) and (14), we can get the transformation relationship from the camera coordinate system to the image coordinate system:

[0116]

[0117] From equations (11) and (15), the total transformation relationship from the world coordinate system to the pixel coordinate system is:

[0118]

[0119] In the above formula is the camera intrinsic parameter matrix, R, t are the rotation matrix and translation vector in the camera extrinsic parameters respectively.

[0120] After obtaining the coordinates of the marker points in the pixel coordinate system, the marker point simulation image can be obtained through segmentation and interpolation.

[0121] Furthermore, the tactile image simulation unit includes a light rendering module, which interpolates surface discrete points into a depth image based on the three-dimensional coordinates of the particles, grids the depth image, and renders the depth image to generate a tactile simulation image.

[0122] Specifically, the light rendering module adopts a physically based light rendering method, such as ray tracing rendering, Phong model rendering, raster rendering, etc. In this embodiment, the ray tracing rendering method is selected. Ray tracing is a physically based rendering method that simulates the propagation of light in a medium and finally collects and summarizes it. Since the light path is reversible, light can be emitted from the camera at random angles, their trajectories can be tracked, and the light in their reverse direction can be calculated. By calculating the color attenuation and accumulation along these paths, the color of each pixel can be approximated. In fact, the light emitted from the source propagates in a straight line until it encounters a surface that blocks its path, where it can be absorbed, refracted or reflected. In the case of a translucent medium, light can undergo partial refraction and reflection. Due to the surface roughness, each reflection produces several diffuse rays in different directions. After each reflection, the intensity of the light decreases. In this embodiment, the application software Blender models the scene in Blender-Cycles and the physically based path tracer for image rendering. After the rendering is completed, the color information of each pixel on the tactile simulation image can be obtained, thereby completing the simulation of the tactile image.

[0123] Furthermore, the tactile marker point joint simulation unit comprises a tactile marker point merging module, and the tactile marker point merging module is used to accumulate the marker point simulation image and the tactile image to obtain the tactile marker point joint simulation image.

[0124] The present invention can complete high-quality simulation of tactile images, marker point images and tactile marker point joint images of visual tactile sensors in motion states such as pressing, sliding and rotating through the connection and use of various modules, which can greatly reduce the time and cost of visual tactile contact data acquisition. The marker point simulation image is completed with the help of the elastic body particle module, camera calibration module, marker point setting module, elastic body contact simulation module and coordinate system conversion unit. The tactile simulation image is completed with the help of the elastic body particle module, camera calibration module, marker point setting module, elastic body contact simulation module and ray tracing rendering module. The tactile marker point joint image simulation is then completed with the help of the tactile simulation image and the marker point simulation image.

[0125] This embodiment also provides a physics-based visual-tactile sensor multi-mode contact simulation method, which is applied to a physics-based visual-tactile sensor multi-mode contact simulation system, such as Figure 1 ,include:

[0126] The visual tactile sensor elastic body is processed by an elastic body contact simulation unit, and the deformation of the visual tactile sensor elastic body is presented in the form of an image, and the three-dimensional coordinates of particles on the surface of the object are obtained by elastic body contact simulation;

[0127] Based on the three-dimensional coordinates, the coordinates of the marking point in the pixel coordinate system are calculated by a coordinate system conversion unit to obtain a simulated image of the marking point;

[0128] The three-dimensional coordinates of the particles are processed by a tactile image simulation unit to generate a tactile simulation image;

[0129] The marker point simulation image and the tactile simulation image are combined through a tactile marker point joint simulation unit to obtain a tactile marker point joint simulation image.

[0130] In order to more clearly express the technical solution of the present invention, the following specific embodiments are provided to introduce the solution:

[0131] First, in the elastic body particle module, the probe and the elastic body of the visual tactile sensor that are in contact with the visual tactile sensor are particle-ized into: the number is divided into 40,000 and 80,000, and the grid is set to 128*128*128. In the camera calibration module, the camera used by the visual tactile sensor is calibrated using the marker board to obtain the external and internal parameter matrices of the camera. Then, in the marker point setting module, the particles representing the elastic body of the visual tactile sensor are set as marker points according to the real sensor marker point array size. Then, in the elastic body contact simulation module, the particles representing the probe are moved to make them contact with the particles representing the elastic body. The probe is a rigid body, and these points have information such as speed. This information will complete the information transmission as it interacts with the particles representing the elastic body of the visual tactile sensor. This process is the calculation process of the material point method. Then, in each step, the position of the particle is updated according to the information obtained by the particle itself, and the three-dimensional coordinates of each particle can be obtained.

[0132] Marker point simulation image:

[0133] After obtaining the three-dimensional coordinates of each particle, the camera's intrinsic and extrinsic parameters that have been calibrated in advance can be used in the world coordinate system to camera coordinate system conversion module and the camera coordinate system to image coordinate system conversion module to map the particle's three-dimensional coordinate information to the camera image coordinate system, thereby completing the marker point simulation image.

[0134] Haptic simulation image:

[0135] After obtaining the three-dimensional coordinates of each particle, the surface discrete points can be interpolated into a 640*480 depth image using the linear interpolation method. Then the depth image is gridded and rendered using the optical tracing rendering module to finally generate a tactile simulation image.

[0136] Tactile marker joint simulation image:

[0137] After obtaining the marker point simulation image and the tactile simulation image, the two images may be physically accumulated to obtain a tactile marker point joint simulation image.

[0138] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A physics-based visual-tactile sensor multi-mode contact simulation system, characterized in that: include: Elastic body contact simulation unit: used to particle the elastic body of the visual tactile sensor, and obtain the three-dimensional coordinates of the particles on the surface of the object through elastic body contact simulation; Coordinate system conversion unit: used to calculate the coordinates of the marking point in the pixel coordinate system through the coordinate system conversion relationship to obtain a simulated image of the marking point; Tactile image simulation unit: used to process the three-dimensional coordinates of particles and generate tactile simulation images; Tactile marker point joint simulation unit: used for merging the marker point simulation image with the tactile simulation image to obtain a tactile marker point joint simulation image.

2. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The elastic body contact simulation unit comprises: Elastic body particle module: used to particle the visual tactile sensor elastic body; Camera calibration module: used to calibrate the intrinsic and extrinsic parameters of the camera used by the visual tactile sensor, wherein the intrinsic and extrinsic parameters are used to convert the world coordinate system to the image coordinate system in the coordinate system conversion unit; Marking point setting module: used to mark particles according to the actual number and marking point position. The marking point information is included in the marked particles. Elastic contact simulation module: used to obtain the three-dimensional coordinates of particles.

3. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: Obtaining the three-dimensional coordinates of the particle, comprising: Initializing particles and mesh nodes, wherein the particles are used to represent the elastic body and the indenter, and the mesh composed of a number of the mesh nodes is used to record object information with a fixed coordinate system; Introducing deformation graph φ p : R 3 →R 3 Record the initial and final positions of each particle during the simulation, and introduce an affine velocity matrix C that records the velocities of adjacent particles p ∈R 3×3 , initialize; The deformation gradient is derived as: Among them, F p It is set to the three-dimensional identity matrix I during initialization 3×3 , x p ∈R 3 , is the position of the particle; Calculate the mass and grid momentum of the i-th grid node respectively: Calculate the mass M of the i-th mesh node i : In the formula, m p represents the mass of the pth particle, G i represents the 3×3×3 grid nodes including the i-th grid node and its adjacent grid nodes, P j represents the particle in the jth grid, ω jp is the weight parameter of the jth grid node and the pth particle used for weighted interpolation; The mesh momentum MG of the i-th mesh node i By calculating the momentum MM generated by the particle motion i and the momentum ME generated by elasticity i get: MG i =MM i +ME i (3) Get M i and MG i Afterwards, near the i-th grid node V i The object velocity is calculated as:

4. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 3, characterized in that: Obtaining the three-dimensional coordinates of the particle also includes: The state of the particle will be updated using the grid node and the previous state of the particle. Assume that the state at step k, i.e., the velocity Affine Speed and deformation gradient is known, then the speed in step k+1 is Affine Speed and deformation gradient It is expressed as: Among them, G' p represents the 3×3×3 grid nodes around the pth particle, ω ip is the weight parameter of the i-th grid node and the p-th particle used for weighted interpolation, V i (k) is the velocity of the ith grid node, ΔX 2 is the square of the grid node spacing, X i represents the position of the jth neighboring node of the ith grid node, is the position of the pth particle, I is the identity matrix, and t is the time interval between two adjacent steps; Through equations (6) and (7), the particle state is updated, the particle moving speed is obtained, and each particle is defined The position in step k+1 is Finally, the three-dimensional coordinates of particles representing the surface of the object are obtained.

5. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 4, characterized in that: Update the state of the particle, including: Set the velocity of the particles at the bottom of the elastic body to 0 and calculate its velocity Among them, v i represents the speed of the indenter, I' represents the particle representing the indenter, B represents the particles on the bottom layer of the elastomer, and p is the pth particle.

6. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The coordinate system conversion unit comprises: World coordinate system to camera coordinate system conversion module: used to convert the position of particles in the world coordinate system to the camera coordinate system; The module for converting the camera coordinate system to the image coordinate system is used to obtain the coordinates of the marker points in the pixel coordinate system through the conversion relationship from the camera coordinate system to the image coordinate system and to obtain the simulated image of the marker points through segmentation and interpolation.

7. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The tactile image simulation unit includes a light rendering module, which interpolates surface discrete points into a depth image based on the three-dimensional coordinates of the particles, grids the depth image, and renders the depth image to generate the tactile simulation image.

8. The physics-based visual-tactile sensor multi-mode contact simulation system according to claim 1, characterized in that: The tactile marker point joint simulation unit includes a tactile marker point merging module, and the tactile marker point merging module is used to accumulate the marker point simulation image and the tactile simulation image to obtain a tactile marker point joint simulation image.

9. A physics-based visual-tactile sensor multi-mode contact simulation method, applied to the physics-based visual-tactile sensor multi-mode contact simulation system according to any one of claims 1 to 8, characterized in that: include: The elastic body of the visual tactile sensor is particle-processed by the elastic body contact simulation unit, and the three-dimensional coordinates of the particles on the surface of the object are obtained by elastic body contact simulation. Based on the three-dimensional coordinates, the coordinates of the marking point in the pixel coordinate system are calculated by a coordinate system conversion unit to obtain a simulated image of the marking point; The three-dimensional coordinates of the particles are processed by a tactile image simulation unit to generate a tactile simulation image; The marker point simulation image and the tactile simulation image are combined through a tactile marker point joint simulation unit to obtain a tactile marker point joint simulation image.

Images