Real-time reconstruction method and device for physical hand-object interaction process

Abstract

The application relates to the technical field of human-computer interaction, in particular to a real-time reconstruction method and device for a hand-object interaction process based on physics, wherein the method comprises the following steps: acquiring geometry and pose information of an object and posture information of a hand based on depth input data; calculating physical properties and an initial contact state of the object according to the geometry and pose information of the object, and performing contact state optimization based on the physical properties and the initial contact state to obtain the coordinates and pressure of each fingertip after optimization; calculating the confidence of each fingertip according to the posture information of the hand, and calculating the final pose of the hand and the final coordinates of each fingertip according to the coordinates and pressure of each fingertip after optimization and the confidence. Therefore, the problems in the prior art that the interaction process is reconstructed in real time only by using a monocular depth camera, that physical constraints are lacked, that the reconstruction result does not conform to physical conditions, that the accuracy of the reconstruction result is reduced and that the requirement of real-time reconstruction of the hand and the object cannot be met are solved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method and apparatus for real-time reconstruction of the interaction process between a hand and an object based on physics. Background Technology

[0002] In related technologies, a joint learning network is used to process depth images to segment hand and object data points and predict hand key points. At the same time, LSTM (Long Short-Term Memory) is used to predict hand pose and optimize it to obtain hand pose and non-rigid motion of the object. Based on the hand pose, object motion and object model, the final reconstruction result is obtained.

[0003] However, the existing technologies that use only a monocular depth camera to reconstruct the interaction process in real time lack physical constraints, resulting in reconstruction results that do not conform to physical conditions, reducing the accuracy of the reconstruction results, and failing to meet the needs of real-time reconstruction of hands and objects. Therefore, improvements are urgently needed. Summary of the Invention

[0004] This application provides a physical-based real-time reconstruction method and apparatus for the interaction process between a hand and an object, which solves the problems in related technologies where only a monocular depth camera is used to reconstruct the interaction process in real time. This lack of physical constraints results in reconstruction results that do not conform to physical conditions, reducing the accuracy of the reconstruction results and failing to meet the needs of real-time reconstruction of the hand and object.

[0005] The first aspect of this application provides a real-time reconstruction method for the interaction process between a hand and an object based on physics, comprising the following steps: acquiring geometric and pose information of the object and pose information of the hand based on depth input data; calculating the physical properties and initial contact state of the object based on the geometric and pose information of the object, and optimizing the contact state based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip; calculating the confidence level of each fingertip based on the pose information of the hand, and calculating the final pose of the hand and the final coordinates of each fingertip based on the optimized coordinates and pressure of each fingertip and the confidence level.

[0006] Optionally, in one embodiment of this application, the physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

[0007] Optionally, in one embodiment of this application, the contact state optimization based on the physical properties and the initial contact state includes: solving for the motion of the object by force and contact state based on a preset constrained energy optimization problem, wherein the optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state.

[0008] Optionally, in one embodiment of this application, the confidence level is calculated using the following formula:

[0009] C i =min(1, N) i / N s ).

[0010] Among them, C i N represents the confidence level for each fingertip. i This represents the number of depth points near the fingertip i (within 3mm), where N is predefined. s =75 indicates the number of depth points needed to accurately predict the position of the fingertip.

[0011] A second aspect of this application provides a real-time reconstruction device for the interaction process between a hand and an object based on physics, comprising: an acquisition module for acquiring geometric and pose information of an object and pose information of a hand based on depth input data; a first calculation module for calculating the physical properties and initial contact state of the object based on the geometric and pose information of the object, and optimizing the contact state based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip; and a second calculation module for calculating the confidence level of each fingertip based on the pose information of the hand, and calculating the final pose of the hand and the final coordinates of each fingertip based on the optimized coordinates and pressure of each fingertip and the confidence level.

[0012] Optionally, in one embodiment of this application, the physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

[0013] Optionally, in one embodiment of this application, the first calculation module is further used to solve for the motion of the object based on a preset constrained energy optimization problem, wherein the optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state.

[0014] Optionally, in one embodiment of this application, the confidence level is calculated using the following formula:

[0015] C i =min(1, N) i / N s ).

[0016] Among them, C i N represents the confidence level for each fingertip. i This represents the number of depth points near the fingertip i (within 3mm), where N is predefined. s =75 indicates the number of depth points needed to accurately predict the position of the fingertip.

[0017] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a real-time reconstruction method for a physics-based hand-object interaction process as described in the above embodiments.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for real-time reconstruction of a physics-based hand-object interaction process.

[0019] This application's embodiments can calculate the object's physical properties and initial contact state based on the object's geometry and pose information, optimize the contact state, and obtain the optimized coordinates and pressure of each fingertip. Furthermore, it can calculate the confidence level of each fingertip based on the hand's pose information, and then calculate the final hand pose and the final coordinates of each fingertip based on the optimized coordinates, pressure, and confidence levels. This allows for optimization of the reconstruction results of any hand-object interaction, making the hand-object interaction more consistent with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time hand-object reconstruction. Therefore, it solves the problems in related technologies where only a monocular depth camera is used for real-time reconstruction of the interaction process, lacking physical constraints, resulting in reconstruction results that do not conform to physical conditions, reducing the accuracy of the reconstruction results, and failing to meet the needs of real-time hand-object reconstruction.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1 This is a flowchart of a real-time reconstruction method for a physics-based hand-object interaction process provided according to an embodiment of this application;

[0023] Figure 2A flowchart illustrating a physical-based real-time reconstruction method for hand-object interaction processes according to a specific embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the reconstruction results and predicted forces according to a specific embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a real-time reconstruction device for a physics-based hand-object interaction process according to an embodiment of this application.

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following describes a physical-based real-time reconstruction method and apparatus for hand-object interaction processes according to embodiments of this application, with reference to the accompanying drawings. Addressing the problem mentioned in the background section that the related technologies only use a monocular depth camera for real-time reconstruction of the interaction process, lacking physical constraints, resulting in reconstruction results that do not conform to physical conditions, reducing the accuracy of the reconstruction results, and failing to meet the needs of real-time hand-object reconstruction, this application provides a physical-based real-time reconstruction method for hand-object interaction processes. In this method, the physical properties and initial contact state of the object can be calculated based on the object's geometric and pose information, contact state optimization can be performed, and the optimized coordinates and pressure of each fingertip can be obtained. Furthermore, the confidence level of each fingertip can be calculated based on the hand's posture information. Then, based on the optimized coordinates, pressure, and confidence level of each fingertip, the final hand posture and the final coordinates of each fingertip can be calculated. This allows for optimization of the reconstruction results of any hand-object interaction, making the hand-object interaction more consistent with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time hand-object reconstruction. This solves the problems in related technologies where only a monocular depth camera is used to reconstruct the interaction process in real time, which lacks physical constraints, resulting in reconstruction results that do not conform to physical conditions, reducing the accuracy of reconstruction results, and failing to meet the needs of real-time reconstruction of hands and objects.

[0029] Specifically, Figure 1 This is a flowchart illustrating a real-time reconstruction method for the interaction between a hand and an object based on physics, as provided in an embodiment of this application.

[0030] like Figure 1As shown, this physics-based real-time reconstruction method for hand-object interaction includes the following steps:

[0031] In step S101, geometric and pose information of the object and pose information of the hand are obtained based on the depth input data.

[0032] In some embodiments, such as Figure 1 As shown, the embodiments of this application first perform kinematic reconstruction, which can be based on the kinematic hand-object interaction reconstruction method SingleDepth, for each frame of depth data. SingleDepth is used for reconstruction to obtain the geometric information of the object. Position information W t Information on hand gestures This can improve the accuracy of hand-object interaction reconstruction.

[0033] In step S102, the physical properties and initial contact state of the object are calculated based on the object's geometry and pose information, and the contact state is optimized based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip.

[0034] In actual implementation, such as Figure 1 As shown, the embodiments of this application can optimize the contact state based on the geometric information of the object. And pose information W t The physical properties and initial contact state of the object are calculated to optimize the contact state, thereby obtaining the optimized coordinates and pressure of each fingertip, resulting in a more physical and accurate reconstruction.

[0035] In one embodiment of this application, the physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

[0036] Specifically, the physical properties of an object include its motion properties (linear velocity and angular velocity v). t ω t linear acceleration and angular acceleration ) and inherent properties (mass m) o centroid c o And the inertial tensor I), where motion properties can be obtained by considering the object pose {W} in adjacent frames. t-2 W t-1 W t The intrinsic properties are obtained by performing a difference operation. Since the geometric surface of the object has been reconstructed by SingleDepth, it can be assumed that the mass of the object is a constant value and is uniformly distributed on the surface of the object, and the calculation is performed under this assumption.

[0037] Optionally, in one embodiment of this application, contact state optimization is performed based on physical properties and initial contact state, including: solving for the motion of the object by force and contact state based on a preset constrained energy optimization problem, wherein the optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state.

[0038] As one possible implementation, embodiments of this application can calculate the initial contact state, where only the contact state between the five fingertips and the object is considered. This saves time and improves efficiency, and since the fingertips are the most important parts in the interaction process, the contact state is defined as follows:

[0039]

[0040] Where, p i Let n represent the projection point of the i-th fingertip onto the surface of the object. i p i The interior normal vector of the object, d i Indicates fingertip and p i The distance between them, in order to calculate p i This requires sampling a portion of the surface of each fingertip, calculating the distance from each point to the object's surface, and then projecting the closest point onto the object's surface; this projected point is denoted as p. i .

[0041] Furthermore, the contact state optimization is solved through a constrained energy optimization problem. The goal of the optimization method is to find a reasonable set of forces and contact states that explain the motion of the object, where the optimization variable is f. i and The total energy function is:

[0042]

[0043]

[0044] Among them, f i This represents the force at the point of contact. This represents the distance from each fingertip to the object surface in the optimized contact state.

[0045] E f (f i One requirement is that the resultant force of the object's own weight and the forces exerted on the object by all points of contact should be able to explain the object's linear acceleration. The specific calculation method is as follows:

[0046]

[0047] Among them, A i= [x1, x2, x3, x4] represents the direction vector of the lateral edge of the square pyramid, and the optimization variable is f. i A represents the components on the four edges. i f i Indicates contact force.

[0048] In this embodiment, a linear approximation friction cone of a square pyramid is used to model the contact force. The direction vectors of the lateral edges of the square pyramid are represented by a matrix A. i = [x1, x2, x3, x4] represents the value, while the optimization variable f i These are the components on the four edges, A i f i Indicates contact force.

[0049] E m (f i The requirement is that the angular acceleration of the object can be explained by the net torque applied at the point of contact. The specific calculation method is as follows:

[0050]

[0051] in, τ represents the centroid coordinates of the object in frame t. t This represents the torque required to drive an object to produce angular acceleration.

[0052] τ t The calculation method is as follows:

[0053]

[0054] In this context, [.] represents the cross product matrix of the pointers.

[0055] E reg (f i ) is the regularization term that restricts the magnitude of the force, defined as:

[0056]

[0057] For the contact term, the constraint is that fingertips that are not in contact with the object cannot exert a force. This term is calculated as follows:

[0058]

[0059] The requirement is that the solved contact state should not differ too much from the initial value; it is defined as follows:

[0060]

[0061] In step S103, the confidence level of each fingertip is calculated based on the hand's posture information, and the final hand posture and the final coordinates of each fingertip are calculated based on the optimized coordinates, pressure, and confidence level of each fingertip.

[0062] In actual implementation, such as Figure 1 As shown, embodiments of this application can perform contact point motion optimization based on hand posture information. and depth input Calculate the confidence level C for each fingertip. i Based on the forces predicted at each contact point in the above steps, the sliding of contact points with high pressure is constrained, thereby avoiding physical slippage of contact points in the reconstruction results. Then, based on the optimized coordinates, pressure, and confidence of each fingertip, the final posture of the hand and the final coordinates of each fingertip are calculated, making the interaction between the hand and the object more in line with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time reconstruction of the hand and the object.

[0063] In one embodiment of this application, the confidence level C i The calculation formula is:

[0064] C i =min(1, N) i / N s ).

[0065] Among them, C i N represents the confidence level for each fingertip. i This represents the number of depth points near the fingertip i (within 3mm), where N is predefined. s =75 indicates the number of depth points needed to accurately predict the position of the fingertip.

[0066] Furthermore, the higher confidence level indicates that the kinematic reconstruction system can obtain enough data to fit the fingertip and produce accurate results, and the optimization process will tend to trust the results of the system more.

[0067] After the optimization steps above, the optimized coordinates of each fingertip can be obtained as follows:

[0068]

[0069] The pressure exerted by each fingertip on the object is:

[0070]

[0071] The coordinates of fingertip i in the current frame without any sliding. It should be:

[0072]

[0073] Among them, I 3×3 W represents a third-order identity matrix. t and W t-1 These represent the pose of the object in the current frame and the previous frame, respectively. This indicates the fingertip position in the final result of the solution in the previous frame.

[0074] Finger tip coordinates after contact point motion optimization for:

[0075]

[0076] in, This represents the optimized coordinates of each fingertip. G represents the coordinates of the fingertip i in the current frame. o =m o ||g|| represents the magnitude of the gravity acting on the object, α represents a fixed proportional parameter, β represents the strength of the confidence level, and γ represents the smoothing effect on the sliding process. Based on experience, α = 0.3, β = 5mm, and γ = 0.5 are set.

[0077] Finally, this embodiment of the application will be based on the optimized coordinates of the five fingertips. Perform an inverse dynamics solution to obtain the final hand pose θ. t The final coordinates of the five fingertips

[0078] like Figure 3 As shown, the embodiments of this application can obtain corresponding reconstruction results and predicted forces by referring to color images, thereby obtaining more physical and accurate reconstruction results. At the same time, a reasonable estimate of the contact force at each fingertip contact point can be given, making the interaction between the hand and the object more in line with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time reconstruction of the hand and the object.

[0079] The real-time reconstruction method for the physical interaction process between a hand and an object proposed in this application can calculate the physical properties and initial contact state of the object based on its geometric and pose information, optimize the contact state, and obtain the optimized coordinates and pressure of each fingertip. It can also calculate the confidence level of each fingertip based on the hand's posture information, and then calculate the final posture of the hand and the final coordinates of each fingertip based on the optimized coordinates, pressure, and confidence level of each fingertip. This method can optimize the reconstruction results of any hand-object interaction, making the interaction between the hand and the object more in line with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time reconstruction of hand-object interactions.

[0080] Next, referring to the accompanying drawings, a real-time reconstruction device for the physical hand-object interaction process proposed according to an embodiment of this application is described.

[0081] Figure 4 This is a block diagram of a physical-based real-time reconstruction device for hand-object interaction processes according to an embodiment of this application.

[0082] like Figure 4 As shown, the real-time reconstruction device 10 for the physical hand-object interaction process includes: an acquisition module 100, a first calculation module 200, and a second calculation module 300.

[0083] Specifically, the acquisition module 100 is used to acquire the geometric and pose information of the object and the posture information of the hand based on the depth input data.

[0084] The first calculation module 200 is used to calculate the physical properties and initial contact state of the object based on its geometric and pose information, and to optimize the contact state based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip.

[0085] The second calculation module 300 is used to calculate the confidence level of each fingertip based on the hand's posture information, and to calculate the final hand posture and the final coordinates of each fingertip based on the optimized coordinates, pressure, and confidence level of each fingertip.

[0086] Optionally, in one embodiment of this application, the physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

[0087] Optionally, in one embodiment of this application, the first calculation module 200 is further used to solve for the motion of an object based on a preset constrained energy optimization problem, wherein the optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state.

[0088] Optionally, in one embodiment of this application, the confidence level is calculated using the following formula:

[0089] C i =min(1, N) i / N s ).

[0090] Among them, C i N represents the confidence level for each fingertip. i This represents the number of depth points near the fingertip i (within 3mm), where N is predefined. s =75 indicates the number of depth points needed to accurately predict the position of the fingertip.

[0091] It should be noted that the foregoing explanation of the real-time reconstruction method for the physical hand-object interaction process also applies to the real-time reconstruction device for the physical hand-object interaction process in this embodiment, and will not be repeated here.

[0092] The real-time reconstruction device for the physical interaction process between a hand and an object proposed in this application can calculate the physical properties and initial contact state of the object based on its geometric and pose information, optimize the contact state, and obtain the optimized coordinates and pressure of each fingertip. It can also calculate the confidence level of each fingertip based on the hand's posture information, and then calculate the final posture of the hand and the final coordinates of each fingertip based on the optimized coordinates, pressure, and confidence level of each fingertip. This allows for the optimization of the reconstruction results of any hand-object interaction, making the interaction between the hand and the object more in line with physical laws, improving the accuracy of the reconstruction results, and effectively meeting the needs of real-time reconstruction of hand-object interactions.

[0093] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0094] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0095] When the processor 502 executes the program, it implements the real-time reconstruction method of the physical hand-object interaction process provided in the above embodiments.

[0096] Furthermore, electronic devices also include:

[0097] Communication interface 503 is used for communication between memory 501 and processor 502.

[0098] The memory 501 is used to store computer programs that can run on the processor 502.

[0099] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0100] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0101] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0102] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0103] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for real-time reconstruction of the physical-based hand-object interaction process.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0108] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0109] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0111] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A real-time reconstruction method for the interaction process between a hand and an object based on physics, characterized in that, Includes the following steps: Geometric and pose information of objects and hand posture information are obtained based on depth input data; The physical properties and initial contact state of the object are calculated based on its geometric and pose information, and the contact state is optimized based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip. After calculating the optimized coordinates of the five fingertips, the process further includes: performing an inverse dynamics solution based on the optimized coordinates of the five fingertips to obtain the final hand pose and the final coordinates of the five fingertips; and The confidence level of each fingertip is calculated based on the hand posture information, and the final hand posture and the final coordinates of each fingertip are calculated based on the optimized coordinates and pressure of each fingertip and the confidence level. The formula for calculating the confidence level is: in, This represents the confidence level for each fingertip. Indicates fingertips The number of nearby depth points, of which, Distance less than 3mm, predefined This represents the number of depth points needed to accurately predict the position of the fingertip. The contact state optimization based on the physical properties and initial contact state includes: Based on a pre-defined constrained energy optimization problem, the motion of the object is explained by force and contact state. The optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state. A linear approximation of a square pyramid friction cone is used to model the contact force. The total energy function of a constrained energy optimization problem includes: A requirement is that the resultant force of the object's own weight and the forces exerted on the object by all points of contact can explain the object's linear acceleration. The requirement is that the angular acceleration of the object can be explained by the resultant torque applied at the point of contact. For the regularization term that restricts the magnitude of the force, As a contact term, the constraint is that fingertips that are not in contact with the object cannot exert a force. The requirement is that the contact state obtained by solving the problem should not differ too much from the initial value.

2. The method according to claim 1, characterized in that, The physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

3. A real-time reconstruction device for the interaction process between a hand and an object based on physics, characterized in that, include: The acquisition module is used to acquire the geometric and pose information of the object and the posture information of the hand based on the depth input data; The first calculation module is used to calculate the physical properties and initial contact state of the object based on the geometric and pose information of the object, and to optimize the contact state based on the physical properties and initial contact state to obtain the optimized coordinates and pressure of each fingertip. After calculating the optimized coordinates of the five fingertips, the process further includes: performing an inverse dynamics solution based on the optimized coordinates of the five fingertips to obtain the final hand pose and the final coordinates of the five fingertips; and The second calculation module is used to calculate the confidence level of each fingertip based on the hand's posture information, and to calculate the final posture of the hand and the final coordinates of each fingertip based on the optimized coordinates and pressure of each fingertip and the confidence level. The formula for calculating the confidence level is: in, This represents the confidence level for each fingertip. Indicates fingertips The number of nearby depth points, of which, Distance less than 3mm, predefined This represents the number of depth points needed to accurately predict the position of the fingertip. The first calculation module is further used to solve the motion of the object based on a preset constrained energy optimization problem, where the optimization variables are the force at the contact point and the distance from each fingertip to the object surface in the optimized contact state, and a quadrangular pyramid linear approximation friction cone is used to model the contact force. The total energy function of a constrained energy optimization problem includes: A requirement is that the resultant force of the object's own weight and the forces exerted on the object by all points of contact can explain the object's linear acceleration. The requirement is that the angular acceleration of the object can be explained by the resultant torque applied at the point of contact. For the regularization term that restricts the magnitude of the force, As a contact term, the constraint is that fingertips that are not in contact with the object cannot exert a force. The requirement is that the contact state obtained by solving the problem should not differ too much from the initial value.

4. The apparatus according to claim 3, characterized in that, The physical properties include at least one of linear velocity and angular velocity, linear acceleration and angular acceleration, mass, center of mass, and inertial tensor.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the real-time reconstruction method for a physics-based hand-object interaction process as described in any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the real-time reconstruction method for the physical hand-object interaction process as described in any one of claims 1-2.

Images