Visual-tactile fusion feedback method and system for interaction between virtual hand and deformable body

Abstract

The present invention belongs to the field of virtual reality technology and discloses a visual-tactile fusion feedback method and system for the interaction between a virtual hand and a deformable object. The method includes: using a virtual model of the deformable object to fit the geometric state of the deformable object to obtain a spring-ball tree structure; after weighted association of the mesh model vertices of a mesh model with multiple balls of the spring-ball tree structure, establishing a mesh-ball coupling association; using a hybrid collision detection model to detect the contact state between the virtual hand and the deformable object in real time; determining the position update amount of each collision ball on the spring-ball tree structure based on the contact state; obtaining the virtual hand posture of the virtual hand; determining the deformation effect of the deformable object based on the position update amount, deforming the mesh model based on the deformation effect and the mesh-ball coupling association, and determining the feedback force based on the real posture and the virtual hand posture to achieve visual-tactile fusion feedback. In this way, the interaction between the virtual hand and the deformable object can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a visual-tactile fusion feedback method and system for interaction between a virtual hand and a deformable body. Background Art

[0002] A virtual hand geometric model describes the virtual hand's three-dimensional shape, degrees of freedom of motion, and geometric constraints of the palm and multiple fingers. This model meets the requirements of motion mapping, collision detection, pose optimization, feedback force calculation, and graphics rendering. Existing research has achieved fast discrete collision detection and pose optimization for virtual hand interactions with virtual objects by implementing high-dimensional geometric modeling of the virtual hand. However, existing research only supports interaction between the virtual hand and rigid objects and cannot address interaction between the virtual hand and deformable objects. Summary of the Invention

[0003] The main purpose of the present invention is to provide a visual-tactile fusion feedback method and system for the interaction between a virtual hand and a deformable body, aiming to solve the technical problem in the prior art that it only supports the interaction between a virtual hand and a rigid object but cannot cope with the interaction between a virtual hand and a deformable body.

[0004] To achieve the above object, the present invention provides a visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object, the method comprising the following steps:

[0005] Using a hybrid virtual hand model to simulate a physical hand to obtain a virtual hand;

[0006] The geometric state of the deformable body is fitted using the deformable body virtual model to obtain the spring ball tree structure;

[0007] After weighted association is performed between mesh model vertices of the mesh model and a plurality of balls of the spring ball tree structure, a mesh-ball coupling association is established;

[0008] Acquiring the real posture of the physical hand, mapping the real posture to a virtual scene, and then driving the virtual hand to perform an operation;

[0009] Detecting the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model;

[0010] Determine a position update amount of each collision ball on the spring ball tree structure according to the contact state;

[0011] Acquire a virtual hand posture of the virtual hand through a virtual hand posture optimization model;

[0012] The deformation effect of the deformable body is determined according to the position update amount, the deformation of the grid model is realized according to the deformation effect and the grid-sphere coupling association, and the feedback force is determined according to the real posture and the virtual hand posture to realize visual-tactile fusion feedback of the interaction process.

[0013] Optionally, the using the deformable body virtual model to fit the geometric state of the deformable body to obtain the spring ball tree structure includes:

[0014] The ball tree model is used to fit the geometric state of the deformed body to obtain the initial ball tree structure;

[0015] Obtaining the parent ball of the high-level ball in the initial ball tree structure according to the ball tree characteristics;

[0016] Determine and establish a threshold value according to the sphere radius and connection coefficient of the parent sphere;

[0017] When the distance between the centers of two balls in the initial ball tree structure is less than the establishment threshold, a spring is established between the two balls to obtain a target ball tree structure;

[0018] defining a fixed ball in the target ball tree structure, and determining a connected ball of the fixed ball along a spring;

[0019] Determining unconnected balls in the target ball tree structure, wherein the unconnected balls are balls in the target ball tree structure that cannot be found by following the spring to fix the balls;

[0020] The connected ball closest to the unconnected ball is used as the target ball, and a spring connection is established between the unconnected ball and the target ball in the target ball tree structure to obtain a spring ball tree structure.

[0021] Optionally, after weighted association of the mesh model vertices with the plurality of balls of the spring ball tree structure, establishing a mesh-ball coupling association comprises:

[0022] determining a plurality of close-range spheres for vertices of the mesh model;

[0023] Determine the weight of the close-range ball according to the distance between the close-range ball and the vertex of the mesh model;

[0024] After weighted association is performed between the mesh model vertices and the plurality of balls in the ball tree structure according to the weights, a mesh-ball coupling association is established.

[0025] Optionally, the hybrid collision detection model includes a sphere-sphere discrete-continuous collision detection model and a frustum-sphere discrete-continuous collision detection model; wherein,

[0026] The real-time detection of the contact state between the virtual hand and the deformable body by using a hybrid collision detection model includes:

[0027] Determining the contact state between the fingers of the virtual hand and the deformable body using the frustum-sphere discrete-continuous collision detection model;

[0028] The ball-ball discrete-continuous collision detection model is used to determine the contact state between the palm and fingertips of the virtual hand and the deformable body.

[0029] Optionally, the determining the contact state between the fingers of the virtual hand and the deformable body by using the frustum-sphere discrete-continuous collision detection model includes:

[0030] Approximating a frustum used to represent the fingers of the virtual hand into a plurality of small balls, and connecting the corresponding small balls between two adjacent moments into a plurality of capsules;

[0031] detecting whether the deformable ball in the ball tree structure collides with the frustum, and detecting whether the deformable ball collides with the plurality of capsules;

[0032] When a deformation body ball in the spring ball tree structure is detected to collide with the frustum or a deformation body ball is detected to collide with the plurality of capsule bodies, the deformation body ball is recorded in a collision ball array, and a contact state between the finger of the virtual hand and the deformation body is determined.

[0033] Optionally, determining the position update amount of each ball on the spring ball tree structure according to the contact state includes:

[0034] Determining, based on the contact state, the external force and internal force exerted on each collision ball on the spring-ball tree structure, wherein the external force is determined based on the embedding depth of a physical finger or a physical palm within the deformable body, and the internal force is the sum of the spring forces exerted by all balls connected to the ball;

[0035] Determine the resultant force of each collision ball according to the internal force and the external force;

[0036] Integrate the resultant force over a specified time step to obtain the velocity and displacement of each ball;

[0037] The position update amount of each colliding ball is determined based on the velocity and displacement.

[0038] Optionally, obtaining the virtual hand posture of the virtual hand through the virtual hand posture optimization model includes:

[0039] Establishing a sphere-sphere non-penetration constraint and a frustum-cone-sphere non-penetration constraint, wherein the sphere-sphere non-penetration constraint constrains the palm or the fingertip of the virtual hand to the mesh surface of the deformable body after a collision between the palm of the virtual hand and the fingertip of the virtual hand and the deformable body is detected, and the frustum-sphere non-penetration constraint constrains the finger of the virtual hand to the mesh surface of the deformable body after a collision between the finger and the deformable body is detected;

[0040] The palm pose is obtained by optimizing the ball-ball non-penetration constraint, and the finger pose is obtained by optimizing the frustum-sphere non-penetration constraint and the ball-sphere non-penetration constraint;

[0041] A virtual hand posture of the virtual hand is determined according to the palm posture and the finger posture.

[0042] In addition, to achieve the above-mentioned purpose, the present invention further proposes a visual-tactile fusion feedback system for interaction between a virtual hand and a deformable object, the visual-tactile fusion feedback system for interaction between a virtual hand and a deformable object comprising:

[0043] A simulation module, used to simulate a physical hand using a hybrid virtual hand model to obtain a virtual hand;

[0044] A fitting module is used to fit the geometric state of the deformable body using the deformable body virtual model to obtain a spring ball tree structure;

[0045] an association module, configured to establish a mesh-ball coupling association after weighted association between mesh model vertices of the mesh model and the plurality of balls of the spring-ball tree structure;

[0046] An acquisition module, configured to acquire the real posture of the physical hand, map the real posture to a virtual scene, and then drive the virtual hand to perform an operation;

[0047] a detection module, configured to detect the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model;

[0048] a determination module, configured to determine a position update amount of each collision ball on the spring ball tree structure according to the contact state;

[0049] The acquisition module is further configured to acquire the virtual hand posture of the virtual hand through a virtual hand posture optimization model;

[0050] The determination module is further used to determine the deformation effect of the deformable body based on the position update amount, realize the deformation of the grid model based on the deformation effect and the grid-sphere coupling association, and determine the feedback force based on the real posture and the virtual hand posture to achieve visual and tactile fusion feedback in the interaction process.

[0051] In addition, to achieve the above-mentioned objectives, the present invention also proposes a visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body, wherein the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body comprises: a memory, a processor, and a visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body stored in the memory and runnable on the processor, wherein the visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body is configured to implement the steps of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body as described above.

[0052] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which is stored a visual-tactile fusion feedback program for the interaction between a virtual hand and a deformable body. When the visual-tactile fusion feedback program for the interaction between a virtual hand and a deformable body is executed by a processor, the steps of the visual-tactile fusion feedback method for the interaction between a virtual hand and a deformable body as described above are implemented.

[0053] The present invention proposes a visual-tactile fusion feedback method and system for the interaction between a virtual hand and a deformable body, which obtains a virtual hand by simulating a physical hand using a hybrid virtual hand model; obtains a spring-ball tree structure by fitting the geometric state of the deformable body using a deformable body virtual model; establishes a grid-ball coupling association after weighted association between the grid model vertices of a grid model and multiple balls of the spring-ball tree structure; obtains the real posture of the physical hand, maps the real posture to a virtual scene, and then drives the virtual hand to perform an operation; detects the contact state of the virtual hand and the deformable body in real time through a hybrid collision detection model; determines the position update amount of each collision ball on the spring-ball tree structure according to the contact state; obtains the virtual hand posture of the virtual hand through a virtual hand posture optimization model; determines the deformation effect of the deformable body according to the position update amount, realizes the deformation of the grid model according to the deformation effect and the grid-ball coupling association, and determines the feedback force according to the real posture and the virtual hand posture to realize visual-tactile fusion feedback of the interaction process. Through the above method, we can solve the problem of fast, realistic and stable tactile rendering in the interaction between virtual hands and deformable bodies. We can innovate in three aspects: efficient physical model construction of deformable bodies, realistic coupling of multi-layer models of deformable bodies, and hybrid collision detection of virtual hands and deformable bodies. Based on the adaptive strategy, a ball-tree-spring model is constructed to achieve stable deformation, the multi-layer model coupling of deformable bodies based on the regional association strategy achieves good deformation effects, and the hybrid collision detection based on discrete-continuous cone-sphere contact processing achieves contact missed detection while ensuring efficient processing, thereby realizing the interaction between virtual hands and deformable bodies. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the structure of a visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body in a hardware operating environment according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the flow of the first embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object according to the present invention;

[0056] Figure 3 Schematic diagram of constructing a deformable body adaptive spring ball tree in the first embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body according to the present invention;

[0057] Figure 4Schematic diagram of the regionalized sphere-grid coupling strategy in the first embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body according to the present invention;

[0058] Figure 5 A depth diagram is embedded in the first embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object of the present invention;

[0059] Figure 6 Schematic diagram of the flow of the second embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object according to the present invention;

[0060] Figure 7 Schematic diagram of discrete and continuous collision detection in the second embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object according to the present invention;

[0061] Figure 8 This is a structural block diagram of the first embodiment of the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body according to the present invention.

[0062] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] Reference Figure 1 , Figure 1 This is a structural diagram of a visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body in a hardware operating environment according to an embodiment of the present invention.

[0065] like Figure 1As shown, the visual-tactile fusion feedback device for interaction between the virtual hand and the deformable body may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0066] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0067] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication model, a user interface model, and a visual-tactile fusion feedback program for the interaction between the virtual hand and the deformable body.

[0068] exist Figure 1 In the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body shown in the figure, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body of the present invention can be set in the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body, and the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body calls the visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body stored in the memory 1005 through the processor 1001, and executes the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body provided in an embodiment of the present invention.

[0069] Based on the above hardware structure, an embodiment of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body is proposed.

[0070] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object according to the present invention.

[0071] In this embodiment, the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object includes the following steps:

[0072] Step S10: using the hybrid virtual hand model to simulate the physical hand to obtain a virtual hand.

[0073] It should be noted that the execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a mobile phone, tablet computer, or personal computer, or an electronic device capable of implementing the aforementioned functions, or a visual-tactile fusion feedback device for interaction between a virtual hand and a deformable object. This embodiment and the following embodiments will be described below using the visual-tactile fusion feedback device for interaction between a virtual hand and a deformable object as an example.

[0074] It should be noted that the physical hand refers to the user's real hand, and the virtual hand refers to the hand in the virtual scene; the hybrid virtual hand model includes a ball-tree model, a frustum hinge model, and a hemisphere model. The palm of the physical hand is simulated by the ball-tree model, the fingers of the physical hand are simulated by the frustum hinge model, and the fingertips are simulated by the hemisphere.

[0075] Step S20: using the deformable body virtual model to fit the geometric state of the deformable body to obtain a spring ball tree structure.

[0076] It should be noted that the deformable body refers to a virtual object; the spring ball tree structure is a ball tree structure with springs added.

[0077] In the specific implementation, the ball tree model can be used to fit the geometric shape of the deformable body to obtain the deformable body ball tree model. In order to achieve the deformation effect, springs can be added between the highest-level balls of the deformable body ball tree model. Specifically, by setting a fixed threshold, a spring can be established between two balls whose ball center distance is less than the fixed threshold, and finally a spring ball tree structure can be obtained.

[0078] It is understandable that for a deformed body with complex shapes, the sizes of the highest-level balls corresponding to different parts are different, e.g. Figure 3As shown in the figure (left), when the deformable body is a rabbit, the radius of the ball at the rabbit's ears is smaller, while the radius of the ball at the rabbit's back is larger. If only a fixed threshold is set to establish the connection between the two balls, it will lead to uneven spring distribution. If the threshold is too small, the number of spring constraints established will increase. If the threshold is too large, some balls will not be able to establish spring constraints. This will cause these balls to be unable to be constrained in the deformable space when the virtual hand interacts with the interactive body, which will make the interactive system prone to instability. To prevent the interactive system from being prone to instability, the deformable body virtual model is further used to fit the geometric state of the deformable body to obtain a spring ball tree structure, including:

[0079] The ball tree model is used to fit the geometric state of the deformed body to obtain the initial ball tree structure;

[0080] Obtaining the parent ball of the high-level ball in the initial ball tree structure according to the ball tree characteristics;

[0081] Determine and establish a threshold value according to the sphere radius and connection coefficient of the parent sphere;

[0082] When the distance between the centers of two balls in the initial ball tree structure is less than the establishment threshold, a spring is established between the two balls to obtain a target ball tree structure;

[0083] defining a fixed ball in the target ball tree structure, and determining a connected ball of the fixed ball along a spring;

[0084] Determining unconnected balls in the target ball tree structure, wherein the unconnected balls are balls in the target ball tree structure that cannot be found by following the spring to fix the balls;

[0085] The connected ball closest to the unconnected ball is used as the target ball, and a spring connection is established between the unconnected ball and the target ball in the target ball tree structure to obtain a spring ball tree structure.

[0086] In the specific implementation, the ball-ball connection coefficient l (0≤l≤1) is defined, and the parent ball of the high-level ball is obtained according to the ball tree characteristics. The product rl of the parent ball's radius r and the connection coefficient l is used as the establishment threshold. The establishment threshold can be used to determine whether a spring can be established between the two balls. Specifically, if the distance between the ball centers of the two balls is less than the establishment threshold, a spring is established; otherwise, no spring is established.

[0087] It is understandable that for balls in the target ball tree structure that do not have spring connections, we can first define a portion of the balls in the target ball tree structure as fixed balls, and follow the springs to find the connected balls of the fixed balls (i.e., the balls connected to the fixed balls through springs). After traversing all the fixed balls, we find all the connected balls, and call the balls other than the connected balls unconnected balls (i.e., the balls that cannot be found as fixed balls by following the springs). Then, we traverse all the unconnected balls and establish a spring connection between each unconnected ball and the nearest connected ball, thereby ensuring that all balls on the deformable body are connected to the fixed balls through springs. Figure 3 As shown in the figure (right), since some balls in the target ball tree structure may be far away from other balls, they do not meet the conditions for adaptive spring establishment. For example, the black solid line ball (thicker line) in the figure is a fixed ball, and its position will not change during the interaction process. The black dotted line ball is close to the fixed ball, meets the conditions for spring establishment, and establishes a spring with the fixed ball, which is a connected ball; while the gray solid line ball (thinner line) is far away from other balls, does not meet the conditions for spring establishment, and is not a fixed ball. When adding springs to such balls, they need to be connected to the nearest connected ball (such as the black dotted line ball) to ensure that the deformable body is stably deformed under the action of external force during the interaction process, and finally a spring ball tree structure is obtained.

[0088] In this embodiment, constructing a spring ball tree structure through an adaptive distance threshold can avoid uneven distribution of springs in the ball tree model, which leads to instability of the deformation simulation and affects the deformation effect.

[0089] Step S30: After weighted association between mesh model vertices of the mesh model and the plurality of balls of the spring ball tree structure, a mesh-ball coupling association is established.

[0090] It should be noted that the visual display in the visual-tactile fusion feedback adopts a grid model. In order to make the grid model deform accordingly with the deforming body (that is, deform accordingly with the spring ball tree structure), it is necessary to establish a coupling association relationship between the grid model vertex of the grid model and the spring ball tree structure. However, directly associating the grid model vertex with the ball closest to the grid model vertex in the spring ball tree structure to achieve the deformation effect will cause the grid model to produce burrs and other uneven phenomena during the deformation process due to the different sizes and uneven distribution of the balls on the spring ball tree structure. In order to avoid burrs and other uneven phenomena during the deformation process, each grid model vertex on the grid model can be further associated with the N balls closest to the spring ball tree structure to establish a grid-ball coupling relationship. Specifically, after the grid model vertex is weightedly associated with the multiple balls of the spring ball tree structure, a grid-ball coupling association is established, including:

[0091] determining a plurality of close-range spheres for vertices of the mesh model;

[0092] Determine the weight of the close-range ball according to the distance between the close-range ball and the vertex of the mesh model;

[0093] After weighted association is performed between the mesh model vertices and the plurality of balls in the ball tree structure according to the weights, a mesh-ball coupling association is established.

[0094] In the specific implementation, we first traverse each mesh model vertex on the mesh model, use the priority queue to record the ball numbers and distances of the N balls closest to the spring ball tree structure, and weight the distances of the N balls. The farther the distance, the smaller the weight and the smaller the impact on the vertex. Specifically, Figure 4 As shown in the left figure, the finger just touches the virtual object (i.e., the deformable body). The horizontal line represents the plane formed by the mesh model, the black dot represents the mesh vertex P, and below are the three balls closest to the vertex P on the deformable body (i.e., the spring ball tree structure). The weights are w1, w2, and w3 according to the distance from the ball center to the vertex, as shown in the figure below. Figure 4 As shown in the figure (right), the ball moves downward under the action of external force and spring force. The resulting position offsets are d1, d2, and d3 respectively. The position change d of the mesh vertex P is calculated based on the distance weight. P =w1d1+w2d2+w3d3, so that the new position of the mesh model vertex P can be obtained. The deformation of the mesh model can be achieved by completing the position update of all mesh model vertices in the mesh structure.

[0095] In this embodiment, the association and coupling relationship between the grid and multiple balls in a certain area is established through distance weighting, so as to avoid burrs that affect the deformation effect by associating each vertex of the grid only with the nearest ball.

[0096] Step S40: Acquire the real posture of the physical hand, and drive the virtual hand to perform an operation after mapping the real posture to a virtual scene.

[0097] It should be noted that mapping the real posture of the physical hand to the virtual scene can drive the virtual hand in the virtual scene to perform operations. Specifically, the driving of the virtual hand can be achieved through computer vision methods (such as Leap motion) or data gloves (such as Noitom gloves).

[0098] Step S50: detecting the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model, wherein the hybrid collision detection model includes a sphere-sphere discrete-continuous collision detection model and a frustum-sphere discrete-continuous collision detection model.

[0099] It should be noted that in order to avoid the problem of missed detection of collisions between the virtual hand and the virtual object (deformation) when the physical hand moves faster or the virtual object (deformation) is thinner, a ball-ball discrete-continuous collision detection model can be used to determine the contact state between the palm and the deformation, and a frustum-cone-ball discrete-continuous collision detection model can be used to determine the contact state between the finger and the deformation. The collision ball in the spring ball tree structure (i.e., the ball that collides with the virtual hand) can be determined based on the contact state.

[0100] Step S60: determining a position update amount of each ball on the spring ball tree structure according to the contact state.

[0101] In a specific implementation, each ball on the spring ball tree structure can be determined by determining the external force and internal force on each ball.

[0102] In one embodiment, determining the position update amount of each collision ball on the spring ball tree structure according to the contact state includes:

[0103] Determining, based on the contact state, the external force and internal force exerted on each collision ball on the spring-ball tree structure, wherein the external force is determined based on the embedding depth of a physical finger or a physical palm within the deformable body, and the internal force is the sum of the spring forces exerted by all balls connected to the ball;

[0104] Determine the resultant force of each collision ball according to the internal force and the external force;

[0105] Integrate the resultant force over a specified time step to obtain the velocity and displacement of each ball;

[0106] The position update amount for each ball is determined based on the velocity and displacement.

[0107] It should be noted that the physical hand refers to the finger of the physical hand, and the physical palm refers to the palm of the physical hand. The geometric state of the deformable body is fitted by the spring ball tree structure, and the hybrid collision detection model is used to determine whether the virtual finger collides with the object. If a collision is detected and the virtual finger is embedded in the deformable body, it means that the virtual finger exerts an external force on the deformable body. For the i-th on the deformable body (i.e., the spring ball tree structure), the virtual finger is subjected to a force. th Collision balls, let the embedding depth of the virtual finger in the deformable body be d i ,like Figure 5 (a) shows the ball embedded in the deformable body by the frustum knuckle, d i The calculation formula is as follows

[0108]

[0109] Where R is the radius of the sphere on the deformed body, T is the radius between the side of the truncated cone and the The intersection of α and the side of the cone is the angle between the cone and its central axis. Draw a perpendicular line from the center of the sphere O on the deformed body to the side of the cone, and the intersection with the central axis l is point F. The direction of the external force direction, The calculation formula is as follows:

[0110]

[0111] Where k ps For penalty stiffness.

[0112] like Figure 5 As shown in (b), when the hemisphere of the fingertip is embedded in the ball on the deformable body, d i The calculation formula is as follows

[0113]

[0114] Where r2 is the radius of the bottom of the cone. In this case, the line connecting the two sphere centers is the external force. direction, The calculation formula is as follows:

[0115]

[0116] It should be noted that the i-th th The internal force on a collision ball is the sum of the spring forces applied by all the balls connected to it, and is calculated as follows

[0117]

[0118] Where N i is the i-th th The number of balls connected by the collision balls, L ij is the center of the sphere before deformation O i With the center O j The distance between them, i.e. L ij =O i -O j , L' ij is the distance between the centers of the two spheres after deformation. k sc is the spring coefficient, k dc is the damping coefficient, v i For the i th The velocity of the colliding balls can be obtained by numerical integration.

[0119] In a specific implementation, the sum of the external force and the internal force can be calculated (for example, Newton's second law) thThe acceleration of each collision ball is calculated, and then the displacement is solved to obtain the position update of the collision ball. The deformation effect of the deformable body can be determined based on the position update amounts of multiple collision balls.

[0120] In this embodiment, a method for calculating the embedding depth of a cone-sphere is proposed to obtain the external and internal forces acting on each sphere on the deformable body, and obtain the resultant force. Then, the position update amount of the sphere position is obtained by integration, and the deformation effect of the deformable body under the action of external force is realized according to the position update amount.

[0121] Step S70: Acquire the virtual hand posture of the virtual hand through the virtual hand posture optimization model.

[0122] It should be noted that the virtual hand posture refers to the virtual hand posture that is not embedded in the deformable body. The virtual hand posture includes the position of the virtual hand and the joint angles of the virtual hand fingers. According to the discrete cone-sphere and ball-sphere collision detection conditions in the hybrid collision detection model, non-embedded constraints on the contact between the finger knuckles and the deformable body, the fingertips and the deformable body, and the palm and the deformable body can be constructed. The constraint optimization target is established through the minimum potential energy principle. The palm posture is optimized first and then the finger posture is optimized to obtain the virtual hand posture that is not embedded in the deformable body.

[0123] In one embodiment, obtaining the virtual hand pose of the virtual hand through the virtual hand pose optimization model includes:

[0124] Establishing a sphere-sphere non-penetration constraint and a frustum-cone-sphere non-penetration constraint, wherein the sphere-sphere non-penetration constraint constrains the palm or the fingertip of the virtual hand to the mesh surface of the deformable body after a collision between the palm of the virtual hand and the fingertip of the virtual hand and the deformable body is detected, and the frustum-sphere non-penetration constraint constrains the finger of the virtual hand to the mesh surface of the deformable body after a collision between the finger and the deformable body is detected;

[0125] The palm pose is obtained by optimizing the ball-ball non-penetration constraint, and the finger pose is obtained by optimizing the frustum-sphere non-penetration constraint and the ball-sphere non-penetration constraint;

[0126] A virtual hand posture of the virtual hand is determined according to the palm posture and the finger posture.

[0127] It should be noted that the sphere-sphere non-penetration constraint is to constrain the palm or fingertips of the virtual hand to the mesh surface of the deformable body after detecting that the palm or fingertips of the virtual hand collides with the deformable body; the cone-sphere non-penetration constraint and the sphere-sphere non-penetration constraint are to constrain the fingers of the virtual hand to the mesh surface of the deformable body after detecting that the fingers of the virtual hand collide with the deformable body.

[0128] Step S80: Determine the deformation effect of the deformable body according to the position update amount, realize the deformation of the grid model according to the deformation effect and the grid-sphere coupling association, and determine the feedback force according to the real posture and the virtual hand posture to realize visual and tactile fusion feedback of the interaction process.

[0129] It should be noted that the deformation effect of the deformable body can be determined based on the position update amount of multiple collision balls; since the mesh model will deform accordingly with the geometric state of the deformable body, after obtaining the deformation effect of the deformable body, the deformation of the mesh model can be realized based on the mesh-ball coupling association; the deformation effect is used to realize the visual rendering of the interaction between the virtual hand and the deformable body, and the feedback force is used to realize the tactile rendering of the interaction between the virtual hand and the deformable body.

[0130] This embodiment obtains a virtual hand by simulating a physical hand using a hybrid virtual hand model; obtains a spring-ball tree structure by fitting the geometric state of the deformable body using a deformable body virtual model; establishes a grid-ball coupling association after weighted association between the grid model vertices of the grid model and multiple balls of the spring-ball tree structure; obtains the real posture of the physical hand, maps the real posture to a virtual scene, and then drives the virtual hand to perform an operation; detects the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model; determines the position update amount of each collision ball on the spring-ball tree structure according to the contact state; obtains the virtual hand posture of the virtual hand through a virtual hand posture optimization model; determines the deformation effect of the deformable body according to the position update amount, realizes the deformation of the grid model according to the deformation effect and the grid-ball coupling association, and determines the feedback force according to the real posture and the virtual hand posture to achieve visual-tactile fusion feedback in the interaction process. Through the above method, we can solve the problem of fast, realistic and stable tactile rendering in the interaction between virtual hands and deformable bodies. We can innovate in three aspects: efficient physical model construction of deformable bodies, realistic coupling of multi-layer models of deformable bodies, and hybrid collision detection of virtual hands and deformable bodies. Based on the adaptive strategy, a ball-tree-spring model is constructed to achieve stable deformation, the multi-layer model coupling of deformable bodies based on the regional association strategy achieves good deformation effects, and the hybrid collision detection based on discrete-continuous cone-sphere contact processing achieves contact missed detection while ensuring efficient processing, thereby realizing the interaction between virtual hands and deformable bodies.

[0131] refer to Figure 6 , Figure 6 This is a flow chart of a second embodiment of a visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object according to the present invention.

[0132] Based on the first embodiment described above, the hybrid collision detection model of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body in this embodiment includes a ball-ball discrete-continuous collision detection model and a frustum-ball discrete-continuous collision detection model; wherein,

[0133] The real-time detection of the contact state between the virtual hand and the deformable body by using a hybrid collision detection model includes:

[0134] Step S501: using the frustum-sphere discrete-continuous collision detection model to determine the contact state between the fingers of the virtual hand and the deformable body.

[0135] In one embodiment, the determining the contact state between the fingers of the virtual hand and the deformable body using the frustum-sphere discrete-continuous collision detection model includes:

[0136] Approximating a frustum used to represent the fingers of the virtual hand into a plurality of small balls, and connecting the corresponding small balls between two adjacent moments into a plurality of capsules;

[0137] detecting whether the deformable ball in the ball tree structure collides with the frustum, and detecting whether the deformable ball collides with the plurality of capsules;

[0138] When a deformation body ball in the spring ball tree structure is detected to collide with the frustum or a deformation body ball is detected to collide with the plurality of capsule bodies, the deformation body ball is recorded in a collision ball array, and a contact state between the finger of the virtual hand and the deformation body is determined.

[0139] It should be noted that if Figure 7 As shown in the figure, the circle represents the ball on the deformable body (spring ball tree structure), the trapezoid represents the frustum simulating the virtual hand finger; point C is the foot of the perpendicular from the center of the sphere O to the center axis l of the frustum, D and E represent the centers of the two circular bases of the frustum, and their coordinates are (x D ,y D ,z D ) and (x E ,y E ,z E ), the contact state between the deformable body and the finger can be compared with the current (i.e. the distance from the center O of the object ball to the center axis l of the virtual finger) and the threshold when the sphere is tangent to the side of the cone ( Figure 7 (a)) to determine if Then the deformable body collides with the finger. and This can be calculated in a variety of ways (i.e., performing collision detection at a certain moment based on the positional relationship between the frustum and the ball. This state, which does not consider the continuous motion of the virtual hand and the deformable body, is called discrete collision detection). The following is one of the calculation methods:

[0140]

[0141] Assume that the radii of the upper and lower bottoms of the cone are r2 and r1 (r2 <r1)

[0142]

[0143] Where R is the radius of the sphere O, T is the distance between the side of the cone and The intersection of the two cones is α, and α is the angle between the side of the cone and its central axis.

[0144] It should be noted that the update frequency of commercial sensor gloves is only 100Hz-200Hz. When the fingers bend too quickly, offline collision detection may miss detection, causing the virtual hand to be embedded in the deformed body. Continuous Collision Detection (CCD) detects whether a collision will occur during the movement of an object, which can avoid the problem of missed detection caused by rapid finger movement. The frustum is approximated as consisting of M spheres (all spheres have different radii and need to be adapted to the current frustum cross section). For example Figure 7 As shown in (b), the frustum knuckle is approximated as 4 small balls. The lines connecting the small balls at the corresponding positions on the frustum knuckle at two moments will form 4 capsules. During collision detection, not only is it detected whether the deformed ball collides with the frustum knuckle at time t, but also whether it collides with the 4 capsules. If there is a collision, the ball number on the deformed body is recorded in the collision ball array. Figure 7 In (b), the green object ball will be missed using discrete collision detection, but can be detected by continuous collision detection.

[0145] It is understandable that continuous state detection can simplify the complex problem of the collision between the ball and the cone trajectory into the collision detection between the ball and the capsule, such as Figure 7 As shown in (c), O represents the center of the ball on the deformed body, t-1 and t are the balls at the corresponding positions of the two frames before and after the articulated cone, and the capsule body composed of the balls in the two frames before and after represents the motion trajectory. A and B are the centers of the two ends of the capsule body respectively. The radius of the ball O and the capsule body are R1 and R2 respectively. According to geometric calculations, it can be determined whether the ball and the capsule body intersect. First, determine whether the ball O collides with the small ball at time t-1 and t. If so, directly add the ball O to the collision queue. If there is no collision, determine whether the projection of the center of the ball O falls within the line segment formed by AB, that is, it satisfies and The distance from O to the center line of the capsule can be calculated. If d O ≤R1+R2, that is, distance d O When it is smaller than the sum of the radius of ball O and the capsule body, ball O is added to the collision queue. In other cases, there is no collision.

[0146] It should be noted that, from Figure 7 (b) It can be seen that continuous collision detection requires multiple calculations of the contact between the ball O and the capsule, which increases the computational effort compared to discrete collision detection. This can especially affect interaction efficiency when M is large. However, in reality, the virtual hand pose optimization algorithm detects that the finger has collided with the deformable object, and the finger is constrained to the surface of the object, eliminating the need for continuous collision detection. When the virtual hand and the deformable object are not in contact (i.e., in the free state), the continuous collision detection algorithm is used for the physical finger (also the graphic finger, with the same pose). When the virtual hand and the deformable object are in contact (i.e., in the constrained state), the discrete collision detection algorithm is used for the graphic finger. This hybrid collision detection strategy improves computational efficiency while avoiding missed detections.

[0147] Step S502: using the ball-ball discrete-continuous collision detection model to determine the contact state between the palm and fingertips of the virtual hand and the deformable body.

[0148] It should be noted that for the collision detection between the fingertip hemisphere (according to the whole sphere) and the deformed body, and between the palm and the deformed body, discrete collision detection is achieved by comparing the distance between the center of the ball on the virtual hand and the ball on the deformed body with the radius of the two balls. If the former is less than or equal to the latter, a collision occurs, otherwise there is no collision. The continuous collision detection between the fingertip hemisphere (according to the whole sphere) and the deformed body, and between the palm and the deformed body, can be achieved by performing collision detection on the capsule body composed of the balls of the previous and next two frames and the ball on the deformed body. The specific method is the same as Figure 7 (c) Same. On this basis, the above hybrid collision detection idea is also used to improve the computational efficiency.

[0149] This embodiment uses the sphere-sphere discrete-continuous collision detection model to determine the contact state between the palm and fingertips of the virtual hand and the deformable object, and uses the frustum-sphere discrete-continuous collision detection model to determine the contact state between the fingers of the virtual hand and the deformable object. This approach avoids missed collision detections between the virtual hand and the virtual object when the real hand moves quickly or the virtual object is thin.

[0150] In addition, an embodiment of the present invention also proposes a storage medium, on which is stored a visual-tactile fusion feedback program for the interaction between a virtual hand and a deformable body. When the visual-tactile fusion feedback program for the interaction between a virtual hand and a deformable body is executed by a processor, the steps of the visual-tactile fusion feedback method for the interaction between a virtual hand and a deformable body as described above are implemented.

[0151] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the visual-tactile fusion feedback system for interaction between a virtual hand and a deformable body according to the present invention.

[0152] like Figure 8 As shown, the visual-tactile fusion feedback system for interaction between a virtual hand and a deformable body proposed in an embodiment of the present invention includes:

[0153] The simulation module 10 is used to simulate the physical hand using the hybrid virtual hand model to obtain a virtual hand.

[0154] The fitting module 20 is used to fit the geometric state of the deformable body using the deformable body virtual model to obtain the spring ball tree structure.

[0155] The association module 30 is configured to perform weighted association between mesh model vertices of the mesh model and the multiple balls of the spring ball tree structure, and then establish a mesh-ball coupling association.

[0156] The acquisition module 40 is used to acquire the real posture of the physical hand, map the real posture to the virtual scene, and then drive the virtual hand to perform an operation.

[0157] The detection module 50 is configured to detect the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model.

[0158] The determination module 60 is configured to determine a position update amount of each collision ball on the spring ball tree structure according to the contact state.

[0159] The acquisition module 40 is further configured to acquire the virtual hand posture of the virtual hand through a virtual hand posture optimization model.

[0160] The determination module 60 is further configured to determine the deformation effect of the deformable body based on the position update amount, implement deformation of the mesh model based on the deformation effect and the mesh-sphere coupling association, and determine the feedback force based on the real posture and the virtual hand posture to achieve visual-tactile fusion feedback in the interaction process.

[0161] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any limitation on this.

[0162] This embodiment obtains a virtual hand by simulating a physical hand using a hybrid virtual hand model; obtains a spring-ball tree structure by fitting the geometric state of the deformable body using a deformable body virtual model; establishes a grid-ball coupling association after weighted association between the grid model vertices of the grid model and multiple balls of the spring-ball tree structure; obtains the real posture of the physical hand, maps the real posture to a virtual scene, and then drives the virtual hand to perform an operation; detects the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model; determines the position update amount of each collision ball on the spring-ball tree structure according to the contact state; obtains the virtual hand posture of the virtual hand through a virtual hand posture optimization model; determines the deformation effect of the deformable body according to the position update amount, realizes the deformation of the grid model according to the deformation effect and the grid-ball coupling association, and determines the feedback force according to the real posture and the virtual hand posture to achieve visual-tactile fusion feedback in the interaction process. Through the above method, we can solve the problem of fast, realistic and stable tactile rendering in the interaction between virtual hands and deformable bodies. We can innovate in three aspects: efficient physical model construction of deformable bodies, realistic coupling of multi-layer models of deformable bodies, and hybrid collision detection of virtual hands and deformable bodies. Based on the adaptive strategy, a ball-tree-spring model is constructed to achieve stable deformation, the multi-layer model coupling of deformable bodies based on the regional association strategy achieves good deformation effects, and the hybrid collision detection based on discrete-continuous cone-sphere contact processing achieves contact missed detection while ensuring efficient processing, thereby realizing the interaction between virtual hands and deformable bodies.

[0163] In one embodiment, the fitting module 20 is further configured to:

[0164] The ball tree model is used to fit the geometric state of the deformed body to obtain the initial ball tree structure;

[0165] Obtaining the parent ball of the high-level ball in the initial ball tree structure according to the ball tree characteristics;

[0166] Determine and establish a threshold value according to the sphere radius and connection coefficient of the parent sphere;

[0167] When the distance between the centers of two balls in the initial ball tree structure is less than the establishment threshold, a spring is established between the two balls to obtain a target ball tree structure;

[0168] defining a fixed ball in the target ball tree structure, and determining a connected ball of the fixed ball along a spring;

[0169] Determining unconnected balls in the target ball tree structure, wherein the unconnected balls are balls in the target ball tree structure that cannot be found by following the spring to fix the balls;

[0170] The connected ball closest to the unconnected ball is used as the target ball, and a spring connection is established between the unconnected ball and the target ball in the target ball tree structure to obtain a spring ball tree structure.

[0171] In one embodiment, the association module 30 is further configured to:

[0172] determining a plurality of close-range spheres for vertices of the mesh model;

[0173] Determine the weight of the close-range ball according to the distance between the close-range ball and the vertex of the mesh model;

[0174] After weighted association is performed between the mesh model vertices and the plurality of balls in the ball tree structure according to the weights, a mesh-ball coupling association is established.

[0175] In one embodiment, the hybrid collision detection model includes a sphere-sphere discrete-continuous collision detection model and a frustum-sphere discrete-continuous collision detection model; wherein,

[0176] The detection module 50 is further configured to:

[0177] Determining the contact state between the fingers of the virtual hand and the deformable body using the frustum-sphere discrete-continuous collision detection model;

[0178] The ball-ball discrete-continuous collision detection model is used to determine the contact state between the palm and fingertips of the virtual hand and the deformable body.

[0179] In one embodiment, the detection module 50 is further configured to:

[0180] Approximating a frustum used to represent the fingers of the virtual hand into a plurality of small balls, and connecting the corresponding small balls between two adjacent moments into a plurality of capsules;

[0181] detecting whether the deformable ball in the ball tree structure collides with the frustum, and detecting whether the deformable ball collides with the plurality of capsules;

[0182] When a deformation body ball in the spring ball tree structure is detected to collide with the frustum or a deformation body ball is detected to collide with the plurality of capsule bodies, the deformation body ball is recorded in a collision ball array, and a contact state between the finger of the virtual hand and the deformation body is determined.

[0183] In one embodiment, the determining module 60 is further configured to:

[0184] Determining, based on the contact state, the external force and internal force exerted on each collision ball on the spring-ball tree structure, wherein the external force is determined based on the embedding depth of a physical finger or a physical palm within the deformable body, and the internal force is the sum of the spring forces exerted by all balls connected to the ball;

[0185] Determine the resultant force of each collision ball according to the internal force and the external force;

[0186] Integrate the resultant force over a specified time step to obtain the velocity and displacement of each ball;

[0187] The position update amount of each colliding ball is determined based on the velocity and displacement.

[0188] In one embodiment, the acquisition module 40 is further configured to:

[0189] Establishing a sphere-sphere non-penetration constraint and a frustum-cone-sphere non-penetration constraint, wherein the sphere-sphere non-penetration constraint constrains the palm or the fingertip of the virtual hand to the mesh surface of the deformable body after a collision between the palm of the virtual hand and the fingertip of the virtual hand and the deformable body is detected, and the frustum-sphere non-penetration constraint constrains the finger of the virtual hand to the mesh surface of the deformable body after a collision between the finger and the deformable body is detected;

[0190] The palm pose is obtained by optimizing the ball-ball non-penetration constraint, and the finger pose is obtained by optimizing the frustum-sphere non-penetration constraint and the ball-sphere non-penetration constraint;

[0191] A virtual hand posture of the virtual hand is determined according to the palm posture and the finger posture.

[0192] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the embodiment scheme, and no limitation is made here.

[0193] In addition, for technical details not fully described in this embodiment, please refer to the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body provided in any embodiment of the present invention, and will not be repeated here.

[0194] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0195] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0196] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0197] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A visual-tactile fusion feedback method for interaction between a virtual hand and a deformable object, characterized in that: The visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body comprises: Using a hybrid virtual hand model to simulate a physical hand to obtain a virtual hand; The geometric state of the deformable body is fitted using the deformable body virtual model to obtain the spring ball tree structure; After weighted association is performed between mesh model vertices of the mesh model and a plurality of balls of the spring ball tree structure, a mesh-ball coupling association is established; Acquiring the real posture of the physical hand, mapping the real posture to a virtual scene, and then driving the virtual hand to perform an operation; Detecting the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model; Determine a position update amount of each collision ball on the spring ball tree structure according to the contact state; Acquire a virtual hand posture of the virtual hand through a virtual hand posture optimization model; The deformation effect of the deformable body is determined according to the position update amount, the deformation of the grid model is realized according to the deformation effect and the grid-sphere coupling association, and the feedback force is determined according to the real posture and the virtual hand posture to realize visual-tactile fusion feedback of the interaction process.

2. The method according to claim 1, wherein The method of fitting the geometric state of the deformable body using the deformable body virtual model to obtain the spring ball tree structure includes: The ball tree model is used to fit the geometric state of the deformed body to obtain the initial ball tree structure; Obtaining the parent ball of the high-level ball in the initial ball tree structure according to the ball tree characteristics; Determine and establish a threshold value according to the sphere radius and connection coefficient of the parent sphere; When the distance between the centers of two balls in the initial ball tree structure is less than the establishment threshold, a spring is established between the two balls to obtain a target ball tree structure; defining a fixed ball in the target ball tree structure, and determining a connected ball of the fixed ball along a spring; Determining unconnected balls in the target ball tree structure, wherein the unconnected balls are balls in the target ball tree structure that cannot be found by following the spring to fix the balls; The connected ball closest to the unconnected ball is used as the target ball, and a spring connection is established between the unconnected ball and the target ball in the target ball tree structure to obtain a spring ball tree structure.

3. The method according to claim 1, wherein After weighted association of the mesh model vertices of the mesh model with the plurality of balls of the spring ball tree structure, a mesh-ball coupling association is established, comprising: determining a plurality of close-range spheres for vertices of the mesh model; Determine the weight of the close-range ball according to the distance between the close-range ball and the vertex of the mesh model; After weighted association is performed between the mesh model vertices and the plurality of balls in the ball tree structure according to the weights, a mesh-ball coupling association is established.

4. The method according to claim 1, wherein The hybrid collision detection model includes a sphere-sphere discrete-continuous collision detection model and a frustum-sphere discrete-continuous collision detection model; wherein, The real-time detection of the contact state between the virtual hand and the deformable body by using a hybrid collision detection model includes: Determining the contact state between the fingers of the virtual hand and the deformable body using the frustum-sphere discrete-continuous collision detection model; The ball-ball discrete-continuous collision detection model is used to determine the contact state between the palm and fingertips of the virtual hand and the deformable body.

5. The method according to claim 4, wherein The determining of the contact state between the fingers of the virtual hand and the deformable body by using the frustum-sphere discrete-continuous collision detection model includes: Approximating a frustum used to represent the fingers of the virtual hand into a plurality of small balls, and connecting the corresponding small balls between two adjacent moments into a plurality of capsules; detecting whether the deformable ball in the ball tree structure collides with the frustum, and detecting whether the deformable ball collides with the plurality of capsules; When a deformation body ball in the spring ball tree structure is detected to collide with the frustum or a deformation body ball is detected to collide with the plurality of capsule bodies, the deformation body ball is recorded in a collision ball array, and a contact state between the finger of the virtual hand and the deformation body is determined.

6. The method according to claim 1, wherein Determining the position update amount of each collision ball on the spring ball tree structure according to the contact state includes: Determining, based on the contact state, the external force and internal force exerted on each collision ball on the spring-ball tree structure, wherein the external force is determined based on the embedding depth of a physical finger or a physical palm within the deformable body, and the internal force is the sum of the spring forces exerted by all balls connected to the ball; Determine the resultant force of each collision ball according to the internal force and the external force; Integrate the resultant force over a specified time step to obtain the velocity and displacement of each ball; The position update amount of each colliding ball is determined based on the velocity and displacement.

7. The method according to claim 1, wherein The step of obtaining the virtual hand posture of the virtual hand by using the virtual hand posture optimization model includes: Establishing a sphere-sphere non-penetration constraint and a frustum-cone-sphere non-penetration constraint, wherein the sphere-sphere non-penetration constraint constrains the palm or the fingertip of the virtual hand to the mesh surface of the deformable body after a collision between the palm of the virtual hand and the fingertip of the virtual hand and the deformable body is detected, and the frustum-sphere non-penetration constraint constrains the finger of the virtual hand to the mesh surface of the deformable body after a collision between the finger and the deformable body is detected; The palm pose is obtained by optimizing the ball-ball non-penetration constraint, and the finger pose is obtained by optimizing the frustum-sphere non-penetration constraint and the ball-sphere non-penetration constraint; A virtual hand posture of the virtual hand is determined according to the palm posture and the finger posture.

8. A visual-tactile fusion feedback system for interaction between a virtual hand and a deformable object, characterized in that: The visual-tactile fusion feedback system for interaction between the virtual hand and the deformable body includes: A simulation module, used to simulate a physical hand using a hybrid virtual hand model to obtain a virtual hand; A fitting module is used to fit the geometric state of the deformable body using the deformable body virtual model to obtain a spring ball tree structure; an association module, configured to establish a mesh-ball coupling association after weighted association between mesh model vertices of the mesh model and the plurality of balls of the spring-ball tree structure; An acquisition module, configured to acquire the real posture of the physical hand, map the real posture to a virtual scene, and then drive the virtual hand to perform an operation; a detection module, configured to detect the contact state between the virtual hand and the deformable body in real time through a hybrid collision detection model; a determination module, configured to determine a position update amount of each collision ball on the spring ball tree structure according to the contact state; The acquisition module is further configured to acquire the virtual hand posture of the virtual hand through a virtual hand posture optimization model; The determination module is further used to determine the deformation effect of the deformable body based on the position update amount, realize the deformation of the grid model based on the deformation effect and the grid-sphere coupling association, and determine the feedback force based on the real posture and the virtual hand posture to achieve visual and tactile fusion feedback in the interaction process.

9. A visual-tactile fusion feedback device for interaction between a virtual hand and a deformable body, characterized in that: The device includes: a memory, a processor, and a visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body, which is stored in the memory and can be run on the processor. The visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body is configured to implement the steps of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body. When the visual-tactile fusion feedback program for interaction between a virtual hand and a deformable body is executed by a processor, the steps of the visual-tactile fusion feedback method for interaction between a virtual hand and a deformable body as described in any one of claims 1 to 7 are implemented.

Images