Method and device for reconstructing hand-object interaction process, electronic equipment and medium

Abstract

The application relates to the technical field of three-dimensional reconstruction, in particular to a method and device for reconstructing a hand-object interaction process, electronic equipment and a medium, wherein the method comprises the following steps: collecting visual data and pressure data of the hand-object interaction process, and performing hand-object interaction reconstruction on the visual data to obtain an initial hand-object posture; and optimizing posture parameters of the hand and a six-degree-of-freedom pose of the object included in the initial hand-object posture according to the pressure data, to obtain a final hand-object posture, and obtaining a reconstruction result of the hand-object interaction process according to the final hand-object posture. Thus, the problem that the hand and the object are mutually occluded during the interaction process and observation data is missing is solved, the clarity of the reconstruction process is improved, and a more accurate reconstruction result is obtained.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to a method, device, electronic device and medium for reconstructing the interaction process between a hand and an object. Background Technology

[0002] In daily life, people frequently interact with objects in their environment using their hands. Therefore, reconstructing the interaction process between hands and objects is an important problem to be solved in the field of 3D reconstruction. This technology can generate computer animations or realistic interactive motion data, and can also help understand the semantics of actions during the interaction process, showing broad application prospects in virtual reality and human-computer interaction.

[0003] In related technologies, hand-object interaction reconstruction techniques are all vision-based methods. These methods can be divided into two main categories. The first category is optimization-based methods, which construct an energy function and optimize it to obtain the hand and object poses that best match the camera observation data. The second category is deep learning-based methods, which train a deep network with a large amount of data so that the network can directly estimate the hand and object poses from videos or images.

[0004] However, optimization-based methods require depth data to be collected via depth cameras, and this data itself is noisy, which can introduce errors into the reconstruction process. Deep learning-based methods often lack generalization ability for objects. In addition, both of these vision-based methods share a common drawback: the interaction between the hand and the object can cause mutual occlusion, resulting in missing observation data and making the reconstruction process ambiguous, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and medium for reconstructing the interaction process between a hand and an object, in order to solve the problem of mutual occlusion between the hand and the object during the interaction process, resulting in missing observation data, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0006] The first aspect of this application provides a method for reconstructing the interaction process between a hand and an object, comprising the following steps:

[0007] Collect visual and pressure data of the hand-object interaction process;

[0008] The visual data is used to reconstruct the hand-object interaction to obtain an initial hand-object pose, wherein the initial hand-object pose includes the hand's pose parameters and the object's six-degree-of-freedom pose; and

[0009] The hand's posture parameters and the object's six-degree-of-freedom pose are optimized based on the pressure data to obtain the final hand-object pose, and the reconstruction result of the interaction process between the hand and the object is obtained based on the final hand-object pose.

[0010] According to one embodiment of this application, optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data includes:

[0011] Based on a preset first energy optimization function, the hand's posture parameters and the object's six-degree-of-freedom pose are optimized according to the pressure data, wherein the preset first energy optimization function is:

[0012] E tot (θ, W) = E tac (θ,W)+E smo (θ,W)+E reg (θ);

[0013] Where θ is the pose parameter of the hand, W is the six-degree-of-freedom pose of the object, and E tot For the preset first energy optimization function, E tac For contact terms, E smo For the smoothing term, E reg This is a regularization term.

[0014] According to one embodiment of this application, E tac The specific calculation formula is as follows:

[0015]

[0016] Among them, SDF W J is the signed distance function of the object. i (θ) represents the three-dimensional coordinates of the i-th key point, F N α represents the pressure at the critical point, and α is the pressure threshold.

[0017] According to one embodiment of this application, E smo The specific calculation formula is as follows:

[0018] E smo (θ, W) = ||θ - θ0|| 2 +||Trans(W)-Trans(W0)|| 2 +||Rot(W)-Rot(W0)|| 2 ;;

[0019] Where Trans represents the translation component of the object's six-DOF pose, and Rot represents the rotation component of the object's six-DOF pose.

[0020] According to one embodiment of this application, the hand-object interaction reconstruction of the visual data includes:

[0021] Based on a preset hand-object interaction reconstruction system or a preset energy function, hand-object interaction reconstruction is performed on the visual data, wherein the preset energy function is:

[0022] E(θ, W) = E rec (θ,W)+E tac (θ, W);

[0023] Among them, E rec The total energy term used to reconstruct the existing hand-object interaction system.

[0024] According to one embodiment of this application, the pressure data includes multiple pressure values, and before optimizing the hand's posture parameters and the object's six-DOF pose based on the pressure data, it further includes:

[0025] The pressure values ​​that do not meet the preset pressure conditions are removed from the plurality of pressure values.

[0026] The hand-object interaction reconstruction method proposed in this application reconstructs the interaction process by reconstructing visual data to obtain an initial hand-object posture. Then, by optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on pressure data, a final hand-object posture is obtained. The reconstruction result of the hand-object interaction process is then derived from this final posture. This solves the problem of missing observation data caused by mutual occlusion between the hand and object during interaction, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0027] A second aspect of this application provides a device for reconstructing the interaction process between a hand and an object, comprising:

[0028] The acquisition module is used to collect visual and pressure data during the interaction between the hand and the object.

[0029] The reconstruction module is used to reconstruct the hand-object interaction from the visual data to obtain an initial hand-object pose, wherein the initial hand-object pose includes the hand's pose parameters and the object's six-degree-of-freedom pose; and

[0030] An optimization module is used to optimize the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data to obtain the final hand-object pose, so as to obtain the reconstruction result of the interaction process between the hand and the object based on the final hand-object pose.

[0031] According to one embodiment of this application, the optimization module is specifically used for:

[0032] Based on a preset first energy optimization function, the hand's posture parameters and the object's six-degree-of-freedom pose are optimized according to the pressure data, wherein the preset first energy optimization function is:

[0033] E tot (θ, W) = E tac (θ,W)+E smo (θ,W)+E reg (θ);

[0034] Where θ is the pose parameter of the hand, W is the six-degree-of-freedom pose of the object, and E tot For the preset first energy optimization function, E tac For contact terms, E smo For the smoothing term, E reg This is a regularization term.

[0035] According to one embodiment of this application, E tac The specific calculation formula is as follows:

[0036]

[0037] Among them, SDF W J is the signed distance function of the object. i (θ) represents the three-dimensional coordinates of the i-th key point, F N α represents the pressure at the critical point, and α is the pressure threshold.

[0038] According to one embodiment of this application, E smo The specific calculation formula is as follows:

[0039] E smo (θ, W) = ||θ - θ0|| 2 +||Trans(W)-Trans(W0)|| 2 +||Rot(W)-Rot(W0)|| 2 ;

[0040] Where Trans represents the translation component of the object's six-DOF pose, and Rot represents the rotation component of the object's six-DOF pose.

[0041] According to one embodiment of this application, the reconstruction module is specifically used for:

[0042] Based on a preset hand-object interaction reconstruction system or a preset energy function, hand-object interaction reconstruction is performed on the visual data, wherein the preset energy function is:

[0043] E(θ, W) = E rec (θ,W)+E tac (θ, W);

[0044] Among them, E rec The total energy term used to reconstruct the existing hand-object interaction system.

[0045] According to one embodiment of this application, the pressure data includes multiple pressure values. Before optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data, the optimization module is further configured to:

[0046] The pressure values ​​that do not meet the preset pressure conditions are removed from the plurality of pressure values.

[0047] The hand-object interaction reconstruction device proposed in this application reconstructs the interaction process by visual data to obtain an initial hand-object posture. Then, by optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on pressure data, a final hand-object posture is obtained. The reconstruction result of the hand-object interaction process is then derived from this final posture. This solves the problem of missing observation data caused by mutual occlusion between the hand and object during interaction, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0048] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for reconstructing the hand-object interaction process as described in the above embodiments.

[0049] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a method for reconstructing a hand-object interaction process as described in the above embodiments.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0051] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0052] Figure 1 This is a flowchart of a method for reconstructing a hand-object interaction process according to an embodiment of this application;

[0053] Figure 2 This is a schematic diagram showing the placement of a pressure sensor according to one embodiment of this application;

[0054] Figure 3 This is a block diagram of a device for reconstructing the interaction process between a hand and an object according to an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0057] The following description, with reference to the accompanying drawings, outlines a method, apparatus, electronic device, and medium for reconstructing the interaction process between a hand and an object, according to embodiments of this application. Addressing the problem mentioned in the background art where mutual occlusion between the hand and object during interaction leads to missing observation data, this application provides a method for reconstructing the interaction process between a hand and an object. In this method, an initial hand-object posture is obtained by reconstructing the hand-object interaction from visual data. Then, the hand posture parameters and the object's six-degree-of-freedom pose are optimized based on pressure data to obtain a final hand-object posture. The reconstruction result of the hand-object interaction process is then obtained based on the final hand-object posture. This solves the problem of missing observation data caused by mutual occlusion between the hand and object during interaction, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0058] Specifically, Figure 1 This is a flowchart illustrating a method for reconstructing the interaction process between a hand and an object, as provided in an embodiment of this application.

[0059] like Figure 1 As shown, the method for reconstructing the interaction process between the hand and the object includes the following steps:

[0060] In step S101, visual data and pressure data of the hand-object interaction process are collected.

[0061] Specifically, this application embodiment uses two different types of devices to collect data on the interaction process between the hand and the object. The first device is a camera, which is used to collect visual data on the interaction process between the hand and the object; the second device is a pressure sensor that can be worn on the hand, which is used to collect pressure data on the interaction process between the hand and the object.

[0062] In step S102, the visual data is reconstructed to obtain the initial hand-object pose, which includes the pose parameters of the hand and the six-degree-of-freedom pose of the object.

[0063] Specifically, in the embodiments of this application, the visual data acquired by the camera can be used to reconstruct the geometry of the object through an existing hand-object interaction reconstruction system, and a rough estimate of the pose of the hand and the object can be made to obtain the initial hand-object pose, wherein the initial hand-object pose includes the pose parameter θ of the hand and the six-degree-of-freedom pose W of the object.

[0064] Furthermore, in some embodiments, reconstructing hand-object interaction from visual data includes: reconstructing hand-object interaction from visual data based on a preset hand-object interaction reconstruction system or a preset energy function, wherein the preset energy function is:

[0065] E(θ, W) = E rec (θ,W)+E tac (θ, W);

[0066] Among them, E rec The total energy term used to reconstruct the existing hand-object interaction system.

[0067] It should be noted that E rec The signed distance function in the energy term can be replaced with an unsigned distance function or any other function that can represent the distance from a point in space to the nearest point on the surface of an object; no specific restrictions are imposed here.

[0068] In step S103, the hand posture parameters and the six-degree-of-freedom pose of the object are optimized based on the pressure data to obtain the final hand-object pose, so as to obtain the reconstruction result of the interaction process between the hand and the object based on the final hand-object pose.

[0069] Specifically, such as Figure 2 As shown, this embodiment employs 21 wearable pressure sensors, placed on different key points of the hand to collect pressure in different areas. During interaction, when a part of the hand contacts an object and applies pressure, the corresponding sensor can collect the pressure value at that point. The pressure values ​​of all key points are recorded as follows:

[0070] After obtaining the initial hand-object posture and pressure values ​​of all parts, this embodiment of the application optimizes the hand-object posture using pressure data so that the hand and object are in contact with each other where pressure is applied.

[0071] Furthermore, in some embodiments, optimizing the hand's posture parameters and the object's six-DOF pose based on pressure data includes: optimizing the hand's posture parameters and the object's six-DOF pose based on a preset first energy optimization function, wherein the preset first energy optimization function is:

[0072] E tot (θ, W) = E tac (θ,W)+Esmo (θ,W)+E reg (θ);

[0073] Where θ is the pose parameter of the hand, W is the six-degree-of-freedom pose of the object, and E tot For the preset first energy optimization function, E tac For contact terms, E smo For the smoothing term, E reg This is a regularization term.

[0074] It should be noted that the first item E tac The key points of the hand that generate pressure must be in contact with the object; Item E smo The final calculated hand and object poses should be close to the vision-based estimation results; the third term E reg The solved hand pose is constrained, requiring that the hand pose be within the limit of joint rotation angles and that the entire hand pose satisfy certain prior information within the PCA space of the hand pose.

[0075] In some embodiments, E tac The specific calculation formula is as follows:

[0076]

[0077] Among them, SDF W J is the signed distance function of the object. i (θ) represents the three-dimensional coordinates of the i-th key point, F N α represents the pressure at the critical point, and α is the pressure threshold.

[0078] In some embodiments, E smo The specific calculation formula is as follows:

[0079] E smo (θ, W) = ||θ - θ0|| 2 +||Trans(W)-Trans(W0)|| 2 +||Rot(W)-Rot(W0)|| 2 ;

[0080] Where Trans represents the translation component of the object's six-DOF pose, and Rot represents the rotation component of the object's six-DOF pose.

[0081] Furthermore, in some embodiments, the pressure data includes multiple pressure values, and before optimizing the hand posture parameters and the six-degree-of-freedom pose of the object based on the pressure data, the method further includes: removing pressure values ​​from the multiple pressure values ​​that do not meet the preset pressure conditions.

[0082] Understandably, in order to avoid errors in the pressure sensor that could lead to incorrect judgments about whether there is contact, this embodiment of the application sets a threshold α for the pressure magnitude. Only when the pressure value is greater than or equal to the threshold is it considered that there is contact between the hand and the object.

[0083] The hand-object interaction reconstruction method proposed in this application reconstructs the interaction process by reconstructing visual data to obtain an initial hand-object posture. Then, by optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on pressure data, a final hand-object posture is obtained. The reconstruction result of the hand-object interaction process is then derived from this final posture. This solves the problem of missing observation data caused by mutual occlusion between the hand and object during interaction, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0084] Next, with reference to the accompanying drawings, a reconstruction device for the hand-object interaction process proposed according to an embodiment of this application is described.

[0085] Figure 3 This is a block diagram of a device for reconstructing the interaction process between a hand and an object according to an embodiment of this application.

[0086] like Figure 3 As shown, the reconstruction device 10 for the interaction process between the hand and the object includes: a data acquisition module 100, a reconstruction module 200, and an optimization module 300.

[0087] The acquisition module 100 is used to acquire visual and pressure data during the interaction between the hand and the object.

[0088] The reconstruction module 200 is used to reconstruct the hand-object interaction from the visual data to obtain the initial hand-object pose, which includes the hand's pose parameters and the object's six-degree-of-freedom pose; and

[0089] The optimization module 300 is used to optimize the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data to obtain the final hand-object pose, so as to obtain the reconstruction result of the interaction process between the hand and the object based on the final hand-object pose.

[0090] Furthermore, in some embodiments, the optimization module 300 is specifically used for:

[0091] Based on a preset first energy optimization function, the hand's posture parameters and the object's six-degree-of-freedom pose are optimized according to pressure data. The preset first energy optimization function is as follows:

[0092] E tot (θ, W) = E tac (θ,W)+E smo (θ,W)+E reg (θ);

[0093] Where θ is the pose parameter of the hand, W is the six-degree-of-freedom pose of the object, and E tot For the preset first energy optimization function, E tac For contact terms, E smo For the smoothing term, E reg This is a regularization term.

[0094] Furthermore, in some embodiments, the specific formula for calculating Etac is:

[0095]

[0096] Among them, SDF W J is the signed distance function of the object. i (θ) represents the three-dimensional coordinates of the i-th key point, F N α represents the pressure at the critical point, and α is the pressure threshold.

[0097] Furthermore, in some embodiments, E smo The specific calculation formula is as follows:

[0098] E smo (θ, W) = ||θ - θ0|| 2 +||Trans(W)-Trans(W0)|| 2 +||Rot(W)-Rot(W0)|| 2 ;

[0099] Where Trans represents the translation component of the object's six-DOF pose, and Rot represents the rotation component of the object's six-DOF pose.

[0100] Furthermore, in some embodiments, the reconstruction module 200 is specifically used for:

[0101] Based on a preset hand-object interaction reconstruction system or a preset energy function, hand-object interaction reconstruction is performed on visual data. The preset energy function is:

[0102] E(θ, W) = E rec (θ,W)+E tac (θ, W);

[0103] Among them, E rec The total energy term used to reconstruct the existing hand-object interaction system.

[0104] Furthermore, in some embodiments, the pressure data includes multiple pressure values. Before optimizing the hand's posture parameters and the object's six-DOF pose based on the pressure data, the optimization module 300 is also used to:

[0105] Eliminate pressure values ​​that do not meet the preset pressure conditions from among multiple pressure values.

[0106] It should be noted that the explanation of the aforementioned method embodiment for reconstructing the interaction process between the hand and the object also applies to the device for reconstructing the interaction process between the hand and the object in this embodiment, and will not be repeated here.

[0107] The hand-object interaction reconstruction device proposed in this application reconstructs the interaction process by visual data to obtain an initial hand-object posture. Then, by optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on pressure data, a final hand-object posture is obtained. The reconstruction result of the hand-object interaction process is then derived from this final posture. This solves the problem of missing observation data caused by mutual occlusion between the hand and object during interaction, thereby improving the clarity of the reconstruction process and obtaining more accurate reconstruction results.

[0108] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0109] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0110] When the processor 402 executes the program, it implements the method for reconstructing the hand-object interaction process provided in the above embodiments.

[0111] Furthermore, electronic devices also include:

[0112] Communication interface 403 is used for communication between memory 401 and processor 402.

[0113] The memory 401 is used to store computer programs that can run on the processor 402.

[0114] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0115] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0117] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0118] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for reconstructing the hand-object interaction process.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0122] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0123] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0124] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for reconstructing the interaction process between a hand and an object, characterized in that, Includes the following steps: Visual and pressure data of the interaction between the hand and the object are collected, wherein the pressure data is collected by a pressure sensor worn on the hand; The visual data is used to reconstruct the hand-object interaction to obtain an initial hand-object pose, wherein the initial hand-object pose includes the hand's pose parameters and the object's six-degree-of-freedom pose; and The hand's posture parameters and the object's six-degree-of-freedom pose are optimized based on the pressure data to obtain the final hand-object pose, and the reconstruction result of the interaction process between the hand and the object is obtained based on the final hand-object pose. The step of optimizing the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data includes: Based on a preset first energy optimization function, the hand's posture parameters and the object's six-degree-of-freedom pose are optimized according to the pressure data, wherein the preset first energy optimization function is: ； in, Let be the posture parameters of the hand. Let the object be in its six-degree-of-freedom pose. The preset first energy optimization function is... For contact items, For smoothing terms, It is a regular term; ； in, Let be the signed distance function of the object. Let i be the three-dimensional coordinates of the i-th key point. Pressure at key points, This is the pressure threshold.

2. The method according to claim 1, characterized in that, ； in, For the translation component of the object's six-degree-of-freedom pose, This refers to the rotational component of the object's six-degree-of-freedom pose.

3. The method according to claim 1, characterized in that, The reconstruction of hand-object interaction from the visual data includes: Based on a preset hand-object interaction reconstruction system or a preset energy function, hand-object interaction reconstruction is performed on the visual data, wherein the preset energy function is: ； in, The total energy term used to reconstruct the existing hand-object interaction system.

4. The method according to claim 1, characterized in that, The pressure data includes multiple pressure values, and before optimizing the hand's posture parameters and the object's six-DOF pose based on the pressure data, it also includes: The pressure values ​​that do not meet the preset pressure conditions are removed from the plurality of pressure values.

5. A device for reconstructing the interaction process between a hand and an object, characterized in that, include: The acquisition module is used to acquire visual data and pressure data during the interaction between the hand and the object, wherein the pressure data is acquired through a pressure sensor worn on the hand. The reconstruction module is used to reconstruct the hand-object interaction from the visual data to obtain an initial hand-object pose, wherein the initial hand-object pose includes the hand's pose parameters and the object's six-degree-of-freedom pose; and An optimization module is used to optimize the hand's posture parameters and the object's six-degree-of-freedom pose based on the pressure data to obtain the final hand-object posture, so as to obtain the reconstruction result of the interaction process between the hand and the object based on the final hand-object posture. The optimization module is specifically used for: Based on a preset first energy optimization function, the hand's posture parameters and the object's six-degree-of-freedom pose are optimized according to the pressure data, wherein the preset first energy optimization function is: ； in, Let be the posture parameters of the hand. Let the object be in its six-degree-of-freedom pose. The preset first energy optimization function is... For contact items, For smoothing terms, It is a regular term; ； in, Let be the signed distance function of the object. Let i be the three-dimensional coordinates of the i-th key point. Pressure at key points, This is the pressure threshold.

6. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for reconstructing the hand-object interaction process as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for reconstructing the hand-object interaction process as described in any one of claims 1-4.

Images