Hand-object interaction three-dimensional reconstruction method, computer equipment and program product

Abstract

The invention relates to a hand-object interaction three-dimensional reconstruction method, computer equipment and a program product. Comprising the steps of obtaining a target video sequence and equipment attitude parameters; establishing a mapping relation between the two-dimensional coordinate system and the target three-dimensional coordinate system; mapping a plurality of pixel points included in each of the plurality of video frames to a target three-dimensional coordinate system according to the mapping relationship to obtain a plurality of target three-dimensional sampling points; respectively mapping the plurality of target three-dimensional sampling points into a hand reference space and an object reference space to obtain a plurality of hand target three-dimensional sampling points and a plurality of object target three-dimensional sampling points; performing field attribute fusion rendering processing on the plurality of hand target three-dimensional sampling points and the plurality of object target three-dimensional sampling points to obtain a three-dimensional reconstruction video frame; and arranging the three-dimensional reconstructed video frames to generate a three-dimensional reconstructed video sequence. Therefore, the data of the visible area can be complemented according to the structural adaptation rule of the hand and the object, and the hand and the object can still be accurately restored under the shielding scene.

Description

Technical Field

[0001] This disclosure relates to virtual imaging, and more particularly to a method for three-dimensional reconstruction of objects through hand-object interaction, computer equipment, and software products. Background Technology

[0002] In related technologies, two-dimensional video frames can be converted into three-dimensional reconstructed video sequences based on the target video sequence and the posture parameters of the image acquisition device, thereby realizing the three-dimensional reconstruction of the hand and the object.

[0003] However, during hand-object interaction, when there is contact occlusion between the hand and the object, the geometric and texture data of the occluded area in the video frame will be missing. The relevant technologies can only analyze and process the visible pixels in the video frame, and cannot supplement or repair these missing data, which leads to the final generated hand and object being incomplete in the occluded area. Summary of the Invention

[0004] In view of this, the present disclosure provides a hand-object interactive 3D reconstruction method, computer equipment, and program product to solve the problems existing in the related technology.

[0005] A first aspect of this disclosure provides a method for three-dimensional reconstruction of hand-object interaction. The method includes: during the interaction between a hand and an object, acquiring a target video sequence and device posture parameters corresponding to an image acquisition device, wherein the target video sequence includes multiple video frames arranged in chronological order; establishing a mapping relationship between a two-dimensional coordinate system corresponding to the target video sequence and a target three-dimensional coordinate system based on the device posture parameters; mapping multiple pixels included in each of the multiple video frames to the target three-dimensional coordinate system according to the mapping relationship, to obtain multiple target three-dimensional sampling points corresponding to each of the multiple video frames; mapping the multiple target three-dimensional sampling points corresponding to each video frame to a hand reference space and an object reference space, respectively, to obtain multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points; performing field attribute fusion rendering processing on the multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points corresponding to each video frame to obtain a three-dimensional reconstructed video frame corresponding to each video frame; and arranging the three-dimensional reconstructed video frames corresponding to each video frame in chronological order to generate a three-dimensional reconstructed video sequence corresponding to the target video sequence.

[0006] A second aspect of this disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described hand-object interaction three-dimensional reconstruction method.

[0007] A third aspect of this disclosure is to provide a computer program product that, when executed by a processor, implements the steps of the above-described hand-object interaction three-dimensional reconstruction method.

[0008] The at least one technical solution adopted in this disclosure can achieve the following beneficial effects: During the interaction between the hand and the object, the target video sequence and the device posture parameters corresponding to the image acquisition device are obtained, wherein the target video sequence includes multiple video frames arranged in chronological order; based on the device posture parameters, a mapping relationship is established between the two-dimensional coordinate system corresponding to the target video sequence and the target three-dimensional coordinate system; based on the mapping relationship, multiple pixels included in each of the multiple video frames are mapped to the target three-dimensional coordinate system to obtain multiple target three-dimensional sampling points corresponding to each of the multiple video frames; the multiple target three-dimensional sampling points corresponding to each video frame are mapped to the hand reference space and the object reference space respectively to obtain multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points; field attribute fusion rendering processing is performed on the multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points corresponding to each video frame to obtain a three-dimensional reconstructed video frame corresponding to each video frame; the three-dimensional reconstructed video frames corresponding to each video frame are arranged in chronological order to generate a three-dimensional reconstructed video sequence corresponding to the target video sequence.

[0009] Based on this, the embodiments of this disclosure can establish a mapping relationship between a two-dimensional coordinate system and a target three-dimensional coordinate system, transforming the pixels of a video frame into target three-dimensional sampling points, thereby achieving an effective transition from two-dimensional to three-dimensional, allowing the initial extraction of spatial information of the hand and object. Mapping the target three-dimensional sampling points to the hand reference space and the object reference space respectively enables clear differentiation and independent processing of the information in both, avoiding data confusion and ensuring the accuracy of subsequent field attribute fusion rendering processing. Furthermore, the three-dimensional reconstructed video frames generated after field attribute fusion rendering processing are arranged into a three-dimensional reconstructed video sequence according to time sequence, fully preserving the dynamic process and temporal continuity of the interaction between the hand and object. It is evident that by mapping the target three-dimensional sampling points to the hand reference space and the object reference space for independent processing, the embodiments of this disclosure can accurately extract and retain data from the visible areas of both. Based on this, when integrating the processed hand target three-dimensional sampling points and object target three-dimensional sampling points, the data in the visible area can be logically completed based on the structural adaptation rules of the hand and object, thus accurately restoring the geometric shape and texture features of the hand and object even in occluded scenes. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is a flowchart illustrating a hand-object interaction 3D reconstruction method according to an embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram of a process for preprocessing video data included in an initial video sequence, provided as an embodiment of the present disclosure.

[0013] Figure 3 This is a schematic diagram of the structure of an initial three-dimensional reconstruction model provided in an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram of a dense deformation field in a hand provided in an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram of the structure of a dense deformation field of an object provided in an embodiment of the present disclosure;

[0016] Figure 6 A flowchart illustrating another hand-object interaction 3D reconstruction method provided in an embodiment of this disclosure;

[0017] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0018] Figure 8 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure;

[0019] Figure 9 A schematic diagram of a computer program product provided according to an embodiment of this disclosure. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0022] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0025] In the field of virtual reality human-computer interaction, hand-object interaction is one of the core pathways for users to achieve natural interaction with the virtual environment. For head-mounted virtual imaging devices, the interaction between the user's hands and objects directly determines the strength of the user's immersion and the quality of the operating experience. Therefore, the value of high-precision hand-object interaction 3D reconstruction technology is particularly prominent. This technology can reconstruct the 3D geometric shape and surface texture features of the hand and object in real time, thereby providing users with more realistic interactive feedback and solidifying the foundation for natural interaction from a technical perspective. It should be understood that objects can include both virtual and physical objects.

[0026] In recent years, with the development of computer vision and deep learning technologies, significant progress has been made in hand and object reconstruction techniques. Among related technologies, since the introduction of neural radiation field technology, multi-view-based 3D reconstruction has rapidly become a research focus. Dynamic neural radiation field, developed on this basis, breaks through the limitations of the original static neural radiation field, enabling the 3D reconstruction of rigid objects and non-rigid human bodies through dynamic multi-view or monocular video sequences. This provides a more flexible technical path for dynamic scene modeling and further opens up new possibilities for high-precision hand-object interaction 3D reconstruction. Specifically, through the hand-object interaction process, monocular video sequences can naturally acquire multi-view observation information of objects and hands. In summary, based on dynamic multi-view or monocular video sequences, combined with neural radiation field and other technologies, joint reconstruction of objects and hands can be achieved. This not only meets the lightweight hardware requirements of head-mounted virtual imaging devices but also provides an efficient solution for 3D reconstruction in natural interaction scenarios.

[0027] However, the following problems still exist in the relevant technologies:

[0028] Problem 1: Hand-object interaction 3D reconstruction typically requires simultaneous modeling of the geometric and textural features of both the hand and the object. However, in actual hand-object interaction, occlusion occurs at the contact points between the hand and the object, leading to a loss of derivative information. Because of this occlusion-induced information loss in the acquired video sequence, subsequent modeling of the hand and object may fail to extract the geometric and textural data of the occluded areas from the video sequence. Consequently, when modeling the hand and object separately, the reconstruction results are highly susceptible to incomplete geometric shapes and illogical structural logic.

[0029] Question 2: During the preprocessing stage, the hand and object model parameters obtained through relevant models may differ from the actual hand and object pose parameters due to limitations in video sequence clarity or model computation. Therefore, directly using these parameters with discrepancies from the actual hand and object pose parameters for subsequent modeling will introduce ambiguous information into the subsequent learning of the dynamic neural radiation field. This will cause the neural radiation field to fit features based on incorrect pose information, ultimately leading to artifacts and insufficient geometric accuracy in the rendered image.

[0030] Question 3: Most of the related technologies are based on frequency coding and multilayer perceptron to build dynamic neural radiation field models. The paths involved in this process will cause the dynamic neural radiation field model to converge too slowly during training, and a lot of real-time calculations are required during the rendering stage. This will increase the generation time of a single frame of 3D reconstructed video, making it difficult to meet the efficiency requirements of practical application scenarios.

[0031] To address the aforementioned issues, this disclosure provides a method, computer device, and program product for 3D reconstruction of hand-object interaction. This method involves acquiring a target video sequence and device posture parameters corresponding to an image acquisition device during hand-object interaction. The target video sequence comprises multiple video frames arranged chronologically. Based on the device posture parameters, a mapping relationship is established between a 2D coordinate system corresponding to the target video sequence and a target 3D coordinate system. According to the mapping relationship, multiple pixels in each video frame are mapped onto the target 3D coordinate system to obtain multiple target 3D sampling points corresponding to each video frame. These multiple target 3D sampling points are then mapped to a hand reference space and an object reference space to obtain multiple hand target 3D sampling points and multiple object target 3D sampling points. Field attribute fusion rendering is performed on these multiple hand target 3D sampling points and multiple object target 3D sampling points corresponding to each video frame to obtain a 3D reconstructed video frame corresponding to each video frame. Finally, the 3D reconstructed video frames are arranged chronologically to generate a 3D reconstructed video sequence corresponding to the target video sequence.

[0032] Based on this, the embodiments of this disclosure can establish a mapping relationship between a two-dimensional coordinate system and a target three-dimensional coordinate system, transforming the pixels of a video frame into target three-dimensional sampling points, thereby achieving an effective transition from two-dimensional to three-dimensional, allowing the initial extraction of spatial information of the hand and object. Mapping the target three-dimensional sampling points to the hand reference space and the object reference space respectively enables clear differentiation and independent processing of the information in both, avoiding data confusion and ensuring the accuracy of subsequent field attribute fusion rendering processing. Furthermore, the three-dimensional reconstructed video frames generated after field attribute fusion rendering processing are arranged into a three-dimensional reconstructed video sequence according to time sequence, fully preserving the dynamic process and temporal continuity of the interaction between the hand and object. It is evident that by mapping the target three-dimensional sampling points to the hand reference space and the object reference space for independent processing, the embodiments of this disclosure can accurately extract and retain data from the visible areas of both. Based on this, when integrating the processed hand target three-dimensional sampling points and object target three-dimensional sampling points, the data in the visible area can be logically completed based on the structural adaptation rules of the hand and object, thus accurately restoring the geometric shape and texture features of the hand and object even in occluded scenes.

[0033] Figure 1 This is a flowchart illustrating a hand-object interaction 3D reconstruction method according to an embodiment of this disclosure. Figure 1 As shown, the hand-object interaction 3D reconstruction method of this disclosure specifically includes:

[0034] S101: Acquire the initial video sequence during the interaction between the hand and the object.

[0035] In some embodiments, the subject may wear a head-mounted image acquisition device to capture an initial video sequence from a first-person perspective during the interaction between the subject's hand and the target object. The initial video sequence may include various hand-object movements during interaction, such as holding the target object with all five fingers, holding the target object with two fingers, rotating the target object, rotating the wrist, and translating the target object. The subject may be a human, an animal, or a robotic hand.

[0036] Specifically, during the initial video sequence acquisition process, the acquiring subject can wear a head-mounted image acquisition device or hold a handheld image acquisition device, keeping the device stationary throughout the acquisition process. This allows video clips acquired within a certain time period to form the initial video sequence. During data acquisition, it's crucial to comprehensively demonstrate all interactive actions between the hand and the target object, and to ensure that all surfaces of the target object are effectively captured through action design, thereby improving the completeness and data coverage of the initial video sequence. It should be understood that image acquisition devices can include monocular and multi-view image acquisition devices, and are any devices used to acquire images.

[0037] S102, preprocess the video data included in the initial video sequence to obtain the training video data. Here, the initial video sequence may include multiple video frames ordered in a certain time sequence. A video frame may be a static image, while the training video data may include video frame data corresponding to multiple video frames. The video frame data may include hand model parameters corresponding to the hand mask image and object model parameters corresponding to the object mask image. The hand model parameters may include hand pose parameters and hand shape parameters, and the object model parameters may include object translation parameters and object rotation parameters.

[0038] Figure 2 This is a schematic diagram illustrating a process for preprocessing video data included in an initial video sequence, as provided in one embodiment of this disclosure. Figure 2 As shown, step S102 may specifically include steps S1021 to S1024, the details of which are as follows:

[0039] S1021, perform foreground segmentation processing on the multiple video frames included in the initial video sequence to obtain multiple foreground mask images corresponding to the multiple video frames. One video frame corresponds to one foreground mask image.

[0040] In some embodiments, foreground segmentation is performed on each video frame, which involves segmenting the background and foreground corresponding to each video frame, thereby obtaining a foreground mask image for each video frame. Here, a robust video matting (RVM) algorithm can be used to perform foreground segmentation on multiple video frames included in the acquired initial video sequence. It should be understood that the RVM algorithm takes the initial video sequence as input and outputs a segmented image of the extracted foreground subject, providing multiple output options such as original transparency prediction, original foreground prediction, and a composite video of the foreground against a green screen background.

[0041] Specifically, the initial video sequence contains background elements in the shooting environment, such as the hands and objects, in addition to the hands and objects themselves. Therefore, the Relative Visualization Machine (RVM) algorithm can be used to process consecutive video frames frame by frame. This means that by analyzing the features of each pixel in the video frame, the foreground regions belonging to the foreground in each frame can be accurately identified, namely the regions corresponding to the hands and objects, and then a corresponding foreground mask map can be generated. The foreground mask map is a binary or grayscale image with the same size as the original video frame; in the foreground mask map, pixels corresponding to the hands and objects are marked as valid pixels, while pixels corresponding to the background regions are marked as invalid pixels.

[0042] Based on this, redundant information in the background region of the video frame can be filtered out, avoiding interference from irrelevant elements to the foreground region in subsequent processing, thereby improving the accuracy and efficiency of subsequent processing. At the same time, during the frame-by-frame processing of the video sequence, the RVM algorithm can also maintain the temporal consistency of the foreground region in consecutive frames, ensuring that the motion trajectory of the hand and object is presented coherently in the foreground mask map, providing a reliable foreground region foundation for subsequent pose tracking and 3D reconstruction based on temporal information.

[0043] S1022, the foreground mask image corresponding to each video frame is segmented to obtain the hand mask image and object mask image corresponding to each foreground mask image.

[0044] Specifically, the foreground mask obtained after step S1021 has eliminated background interference, but there is still a mixed area of ​​hands and objects in the foreground region. In order to more clearly segment the hands and objects, an image semantic segmentation tool can be used to perform semantic segmentation on the foreground mask corresponding to each video frame to obtain the hand mask and object mask corresponding to each foreground mask.

[0045] The image semantic segmentation tool can be MMSegmentation. As an open-source image semantic segmentation library, MMSegmentation provides pre-trained models trained on a large amount of image data, capable of recognizing specific object categories. Therefore, when using MMSegmentation to further perform semantic segmentation on the foreground mask map corresponding to each video frame, the pre-trained model in MMSegmentation can analyze the semantic features of pixels in the foreground mask map, such as the shape differences, texture features, and spatial distribution of the hand and object, to further divide the foreground region into hand and object regions, generating corresponding hand and object masks respectively. Both the hand and object masks can be images with the same size as the original video frame. The hand mask can only label the effective pixels in the hand region, and the object mask can only label the effective pixels in the object region.

[0046] This allows the foreground region to be clearly distinguished into two independent parts: the hand region and the object region. This avoids errors caused by target confusion when further processing the hand region and the object region. It provides a clear regional boundary basis for the accurate pose calculation of the two types of targets, the hand and the object, and is an important prerequisite for modeling the hand and the object separately in subsequent 3D reconstruction.

[0047] S1023, identify and analyze the hand mask images corresponding to different foreground mask images to obtain the hand model parameters corresponding to different foreground mask images. The hand model parameters include hand pose parameters and hand shape parameters. Among them, the hand pose parameters can be used to describe the motion state of the hand in the current video frame, such as the parameter set of joint rotation angles and joint positions; the hand shape parameters are used to describe the inherent physiological morphology of the hand, such as the parameter set of individual differences in palm size, finger thickness, and joint proportions, which are the core indicators for distinguishing different static features of the hand.

[0048] In some embodiments, object detection algorithms can be used to further locate the hand in the hand mask image. The object detection algorithm can be the YOLO-X algorithm, a highly efficient object detection algorithm that can quickly locate the region containing the hand in the hand mask image and output a hand detection box. This hand detection box can mark the approximate position and range of the hand in the hand mask image in coordinate form, defining the boundaries for subsequent fine-grained processing focused on the hand region and avoiding interference from irrelevant areas.

[0049] Furthermore, the hand detection bounding box output by YOLO-X can be used as a constraint, and the Openpose feature point detector can be invoked to perform targeted detection within the area corresponding to the hand detection bounding box. Openpose can accurately identify and output the coordinates of multiple feature points corresponding to the hand by analyzing features such as the texture and contour of the hand. The aforementioned feature points can number up to 21. Specifically, they can include: 4 feature points on the thumb (thumb tip, first joint, second joint, and metacarpophalangeal joint); 4 feature points on the index finger (thumb tip, first joint, second joint, and metacarpophalangeal joint); 4 feature points on the middle finger (thumb tip, first joint, second joint, and metacarpophalangeal joint); 4 feature points on the ring finger (thumb tip, first joint, second joint, and metacarpophalangeal joint); 4 feature points on the little finger (thumb tip, first joint, second joint, and metacarpophalangeal joint); and feature points at the base of the hand, i.e., the feature points near the wrist. These feature points not only reflect the current shape of the hand, such as whether the fingers are open or bent, but also provide a basic spatial coordinate reference for subsequent 3D modeling of the posture.

[0050] Based on this, the data corresponding to the 21 feature points of each video frame can be input into the EasyMocap open-source tool, and processed using the Model of Animated Hand with Articulated Nails and Opposable Thumb (MANO). EasyMocap can perform temporal correlation and optimization on the data corresponding to the 21 feature points, ensuring the continuity of hand poses in consecutive video frames. The MANO model, a parametric model specifically designed for 3D hand modeling, can derive the 3D pose information of the hand based on the spatial distribution characteristics of the 21 feature points, ultimately outputting 48 hand pose parameters and 10 hand shape parameters for each video frame. EasyMocap, developed by a university 3D vision team, is an open-source tool and a label-free human motion capture toolkit designed to achieve human motion capture and new perspective synthesis through video. The MANO model is a parametric model used for accurate modeling and driving the 3D form and movement of the human hand, widely used in computer vision, graphics, and human-computer interaction.

[0051] The aforementioned 48 hand pose parameters can include 3 global pose parameters and 45 finger joint pose parameters. The global pose parameters describe the overall rotation of the hand in three-dimensional space, corresponding to the rotation angles of the hand relative to the target three-dimensional coordinate system, such as the rotation angles of the hand around the X, Y, and Z axes of the target three-dimensional coordinate system, used to determine the overall spatial orientation of the hand, i.e., the overall rotation direction of the wrist when holding an object. The finger joint pose parameters describe the local rotation of each joint in the hand. The 45 finger joint pose parameters can include pose parameters corresponding to 15 key joints. These 15 key joints include 3 joints corresponding to each of the 5 fingers, and each key joint can be described by multiple parameters to represent its rotation state, such as joint flexion angle and joint lateral tilt angle. The hand shape parameters describe the inherent physiological morphology of the hand, including but not limited to inherent attributes such as palm width, finger length, knuckle thickness, palm thickness, and finger spacing. It should be understood that the target's three-dimensional coordinate system can be the world coordinate system. The world coordinate system is a fixed coordinate system shared by all objects and image acquisition devices in the scene, and is usually defined as a right-handed coordinate system.

[0052] Thus, through steps S1021 to S1023, the transformation of hand features from image features to digital parameters is completed step by step, from locating the hand region to extracting feature points, and then converting the data corresponding to the feature points into hand pose parameters and hand shape parameters. This provides core input data for the subsequent training stage and is the foundation for achieving accurate hand reconstruction.

[0053] S1024: Identify and analyze the object mask images corresponding to different foreground mask images to obtain the object model parameters, object translation parameters, and object rotation parameters corresponding to different object mask images. It should be understood that steps S1023 and S1024 can be performed simultaneously or sequentially, and no specific limitation is made here.

[0054] In some embodiments, since the initial video sequence consists of consecutive video frames, the object mask images corresponding to different video frames will change as the object moves. These changes can reflect the differences in the object's pose in space. Therefore, based on the object mask image corresponding to each video frame, the object model parameters of the object in the object mask image can be estimated by combining the Bundle Adjustment Signed Distance Function (BundleSDF). That is, the BundleSDF algorithm can be used to calculate the position and angle changes of the object in three-dimensional space by analyzing features such as contour changes and pixel position offsets of the object mask image in consecutive video frames.

[0055] Specifically, the BundleSDF algorithm can be used to obtain the object model parameters corresponding to an object in any video frame. These parameters include three translation parameters and three rotation parameters. The three translation parameters describe the object's positional movement along the X, Y, and Z axes, while the three rotation parameters describe the object's rotation around the X, Y, and Z axes. It should be understood that BundleSDF (Bundle Shape and Density Field) is a neural 6-DOF tracking and 3D reconstruction method for unknown objects. It can perform near real-time 6-DOF tracking of unknown rigid objects from a monocular video sequence while simultaneously performing neural 3D reconstruction.

[0056] Based on this, by temporal correlation and pose calculation of the object mask images corresponding to different video frames, the object model parameters corresponding to the objects in each video frame can be obtained. These object model parameters can not only reflect the dynamic pose changes of the objects during the interaction process, but also serve as the core input data for the subsequent training stage, providing spatial pose basis for the geometric reconstruction and rendering of the objects, and ensuring that the reconstruction results are consistent with the motion state of the real objects.

[0057] S103, use the video data to be trained to train the initial 3D reconstruction model to obtain the target 3D reconstruction model.

[0058] Figure 3 This is a schematic diagram of the structure of an initial three-dimensional reconstruction model provided in one embodiment of this disclosure. Figure 3 As shown, an initial 3D reconstruction model can be trained frame-by-frame using multiple video frames from the initial video sequence, enabling the initial 3D reconstruction model to learn the object model parameters and hand model parameters of each video frame. Specifically, any video frame in the initial video sequence can be selected as the starting video frame for training. For the starting video frame, the object model parameters, hand pose parameters, and device pose parameters corresponding to the starting video frame can be input into the training module of the initial 3D reconstruction model to optimize the model parameters corresponding to the initial 3D reconstruction model. The device pose parameters can include internal and external parameters of the image acquisition device. Internal parameters include parameters related to the physical characteristics of the image acquisition device, such as focal length, principal point coordinates, and distortion coefficients. External parameters include parameters related to the position and pose of the image acquisition device in the target 3D coordinate system, used to describe the pose of the image acquisition device relative to the target 3D coordinate system, such as rotation matrices and translation vectors. Here, the principal point coordinates refer to the coordinates corresponding to the optical center on the image acquisition device.

[0059] Specifically, when analyzing the initial video frame, a certain number of pixels can be randomly selected from the initial video frame to construct training samples. This randomness in pixel selection ensures that the initial 3D reconstruction model learns the features of different regions of the initial video frame, such as hand edges, object surfaces, and hand-object contact points, avoiding feature learning bias caused by fixed sampling. Here, the number of pixels can be 2048.

[0060] Next, for the multiple pixels selected from the initial video frame, calculations can be performed using the internal and external parameters of the image acquisition device to obtain the rays corresponding to these pixels in the target 3D coordinate system. For any given pixel, the ray is determined by the position of the image acquisition device and the direction of the line connecting the image acquisition device to that pixel in the target 3D coordinate system. This integrates multiple parameters into the same target 3D coordinate system, ensuring a unified description of the spatial positions and interactions of the hand, object, and image acquisition device, avoiding positional deviations caused by coordinate system differences.

[0061] In some embodiments, the process of calculating the rays corresponding to multiple pixels in the target three-dimensional coordinate system by combining the internal and external parameters of the image acquisition device can be as follows: For multiple selected pixels in a video frame, lens distortion can be eliminated first using internal parameters to obtain the corrected pixel coordinates of each pixel; then, based on the focal length and principal point coordinates, the corrected pixel coordinates are converted into three-dimensional direction vectors in the coordinate system of the image acquisition device. Next, the coordinate system of the image acquisition device can be transformed to the target three-dimensional coordinate system using the rotation matrix and translation vector of the external parameters. Simultaneously, the principal point coordinates of the image acquisition device are used as the origin and mapped to the ray origin in the target three-dimensional coordinate system using the translation vector; thus, the rays corresponding to multiple pixels in the target three-dimensional coordinate system can be obtained.

[0062] In some embodiments, after obtaining the ray corresponding to each pixel, coarse sampling can be performed first to determine the key regions corresponding to each ray, followed by fine sampling of the key regions. This yields multiple key 3D sampling points, which are then used for subsequent training of the initial 3D reconstruction model. Specifically, coarse sampling can first be performed, uniformly selecting a small number of 3D sampling points across the entire range of multiple rays to quickly identify key regions on the rays where hands or objects may exist, excluding obviously empty spaces. Next, fine sampling is performed on the key regions identified by coarse sampling, densely selecting more key 3D sampling points within these regions to improve the sampling accuracy of the hand and object surfaces and surrounding space. The key 3D sampling points obtained through this layered sampling method can serve as inputs for subsequent dense deformation fields of the hand, objects, and neural radiation fields, providing foundational data for coordinate transformation and attribute prediction in these network modules. This sampling method avoids redundant sampling of invalid spaces to reduce computational load, while ensuring sampling density in key regions through fine sampling, thereby improving training efficiency and the learning effect of the initial 3D reconstruction model on details. It should be understood that a pixel is a discrete point on a 2D image plane, a ray is a 3D straight line originating from the optical center of the image acquisition device and passing through a pixel, and a sampling point is a spatial point selected on the ray according to the above strategy. Here, the three-dimensional observation space refers to the three-dimensional space corresponding to the visual information directly captured by the image acquisition device.

[0063] like Figure 3 As shown, the learning network of the initial 3D reconstruction model in this embodiment may include two parts: a hand branch learning network and an object branch learning network. Based on this, after obtaining multiple key 3D sampling points, candidate 3D sampling points of the hand located in the hand region can be input into the hand branch learning network, and candidate 3D sampling points of the object located in the object region can be input into the object branch learning network. The hand branch learning network includes a dense deformation field of the hand and a neural radiation field of the hand, while the object branch learning network includes a dense deformation field of the object and a neural radiation field of the object.

[0064] Specifically, for the hand-branch learning network, hand model parameters and multiple key 3D sampling points can be input into a dense deformation field of the hand. This field can then map these key 3D sampling points to a hand reference space based on the hand skeleton motion weights, obtaining the corresponding key 3D sampling points of the hand when located in the reference space. These key 3D sampling points can be further processed by inputting into a hand neural radiation field to obtain the hand's signed distance function (SDF), hand density value, and hand color weights. The hand density value describes whether the key 3D sampling point belongs to the hand surface, and the hand color parameters can be the red (R), green (G), and blue (B) color information corresponding to the key 3D sampling point. The hand reference space can be understood as a standard hand template. This template contains the basic shape of the hand, including the pre-defined skeletal structure and joint positions. Key hand sampling points from all key 3D sampling points in the 3D observation space are mapped to this space through a dense deformation field of the hand to uniformly describe the shape of the hand in different postures. The object reference space can be regarded as the space containing a standard object template. Key object sampling points from key 3D sampling points in the 3D observation space are mapped to this space to achieve standardized processing of the object's shape in different postures.

[0065] For the object branch learning network, the object model parameters and multiple key 3D sampling points can be input into the object dense deformation field. The object dense deformation field can predict the subtle deformation residuals of the object and map the multiple key 3D sampling points into the object reference space, obtaining the multiple key 3D sampling points corresponding to the object when they are located in the object reference space. These multiple key 3D sampling points are then input into the object neural radiation field for further processing to obtain the object SDF value, object density value, and object color parameter corresponding to each key 3D sampling point. The object density value describes whether the key 3D sampling point belongs to the object surface, and the object color parameter describes the object texture features. Both the hand neural radiation field and the object neural radiation field can adopt a "multi-resolution hash coding + small multilayer perceptron" structure, which enables the initial 3D reconstruction model to converge quickly and improves accuracy.

[0066] In some embodiments, since occlusion may occur during hand-object interaction, the final features of each key 3D sampling point can also be determined by field overlay. Specifically, the density value of the 3D sampling point can be taken from the hand nerve radiation field and the object nerve radiation field with the larger density value. At the same time, volume rendering is performed based on the SDF value and color weight corresponding to the field with the larger density value, thereby generating a rendered image that matches the device pose parameters.

[0067] Specifically, when implementing volume rendering, the following steps can be followed: First, for any key sampling point, the corresponding density value can be extracted from the hand's neural radiation field and the object's neural radiation field, respectively. By comparing these two density values, the larger value is selected as the final density value for that sampling point. Next, depending on the source of the density value, i.e., whether it comes from the hand's neural radiation field or the object's neural radiation field, the final SDF value output by the corresponding neural radiation field is called. Specifically, if the final density comes from the hand's neural radiation field, the hand's SDF value is used as the final SDF value; if it comes from the object's neural radiation field, the object's SDF value is used as the final SDF value. This final SDF value can help optimize the spatial position accuracy of the key sampling point.

[0068] Furthermore, color weight calculation rules can be matched based on the density source: if the hand's neural radiation field dominates, color weights are assigned according to the texture mapping logic of the hand's neural radiation field; if the object's neural radiation field dominates, color weights are calculated according to the object's own material properties. Finally, the final density value, the sampling position corresponding to any key sampling point optimized based on the final SDF value, and the corresponding color weight are substituted into the volume rendering formula, combined with the device pose parameters of the current image acquisition device, to synthesize the final rendered 3D reconstructed video frame pixel by pixel. In this process, areas with higher density will naturally become the visual focus, such as highlighting the outline of the hand when it occludes an object, and highlighting the details of the object when it is revealed, thereby achieving realistic rendering that conforms to the spatial occlusion logic.

[0069] In some embodiments, after obtaining a 3D reconstructed video frame that matches the device pose parameters through volume rendering, the model parameters of the initial 3D reconstruction model can be updated by backpropagation based on the mean squared error (MSE) and mean absolute error (MAE) loss between the 3D reconstructed video frame corresponding to the starting video frame and the starting video frame, thus completing a single training for the starting video frame.

[0070] Specifically, firstly, a 3D reconstructed video frame matching the device's pose parameters can be obtained through volume rendering. This 3D reconstructed video frame is a prediction of the real scene from the initial video frame. Next, the MAE and MSE between this 3D reconstructed video frame and the initial video frame are calculated. MSE is obtained by averaging the squared differences in the corresponding pixel color values ​​of the two images, while MAE is obtained by averaging the absolute differences in the corresponding pixel color values. Then, after obtaining the MSE and MAE loss values, these two loss values ​​are used as key optimization parameters. Through backpropagation, the loss information is passed from the output layer to the input layer to update the model parameters of the initial 3D reconstruction model, such as the weights and biases of the neural network. Finally, after updating the model parameters, a single training iteration with the initial video frame is complete.

[0071] In summary, the target 3D reconstruction model can be obtained by training the multiple video frames included in the initial video sequence frame by frame using the above method, starting with the initial video frame.

[0072] Figure 4 This is a schematic diagram of a dense deformation field in a hand, provided as an embodiment of the present disclosure. Figure 4 As shown, the structure of the dense deformation field of the hand includes a convolution generator, which can be a 3D convolution generator. This generator generates a Motion Weight Volume (MWV) to determine the influence of each bone movement on each candidate 3D sampling point in the hand's reference space. Specifically, the convolution generator spatially expands the globally learnable latent variables built into the initial 3D reconstruction model, transforming them from tensors of size (1, 1, 1) to tensors of size (32, 32, 32), while simultaneously transforming the feature dimensions. Each feature dimension represents the influence of a predefined bone movement transformation on the candidate 3D sampling points in the hand's reference space. Here, the built-in globally learnable latent variables are variables pre-defined in the model and automatically adjusted and optimized during training.

[0073] like Figure 4 As shown, the structure of the dense deformation field of the hand also includes a global skeleton calculation module, a hand transformation module, and a weighted summation module. Specifically, taking any candidate 3D sampling point of the hand in the initial video frame as an example, the 48 hand pose parameters corresponding to the initial video frame and the predefined hand dynamics map can be input into the global skeleton motion calculation module to obtain 16 rotation and translation variables of the skeleton relative to the target 3D coordinate system. By integrating these rotation and translation variables, the corresponding global skeleton transformation matrix, i.e., the SE3 matrix, can be generated. Then, the candidate 3D sampling points p = (x, y, z) of the hand in the hand reference space are transformed to a homogeneous coordinate system and coupled with the SE3 matrix. Multiplying these results in the candidate 3D sampling points of the hand after transformation by the hand transformation module; then, taking the i-th bone as an example, the positions of the transformed candidate 3D sampling points of the hand are... Normalized to the interval between -1 and 1, and then trilinear interpolation is performed on the MWV corresponding to the i-th bone to obtain the weight of the i-th bone motion. Then through The formula calculates the weight of the i-th bone motion in the hand reference space. Using the formula in the weighted summation module Weighted summation yields the key 3D sampling points of the hand in the hand reference space. Here, the hand transformation module can be an SE3 transformation module. The object SE3 transformation module is a functional component used to implement SE3 transformation. SE3 transformation is a special Euclidean group transformation in 3D space, which includes rotation and translation operations and is represented by a 4×4 homogeneous transformation matrix.

[0074] Figure 5 This is a schematic diagram of a dense deformation field of an object provided in one embodiment of this disclosure. For example... Figure 5 As shown, the structure of the dense deformation field of the object consists of a feature extraction network, a pose embedding network, a feature fusion network, a rotation and translation prediction network, a matrix multiplication module, and an object transformation module.

[0075] like Figure 5 As shown, the feature extraction network and the pose embedding network take the candidate 3D sampling points of the object in the 3D observation space and the object model parameters as inputs, respectively. The feature extraction network encodes the candidate 3D sampling points of multiple objects and extracts the geometric features of the point cloud, such as distribution patterns and local structures, and outputs multiple spatial feature vectors corresponding to the candidate 3D sampling points of multiple objects. The pose embedding network encodes the object model parameters into object pose feature vectors. The feature extraction network and the pose embedding network are each composed of a high-frequency information encoder and a multilayer perception mechanism containing 4 fully connected layers.

[0076] Next, the feature fusion component performs feature fusion on multiple spatial feature vectors and pose feature vectors through feature concatenation to obtain the local feature fusion vector corresponding to each candidate 3D sampling point of the object. Then, based on the local feature fusion vector corresponding to each candidate 3D sampling point of the object, a rotation and translation prediction network is used to predict the local dense deformation residual of each candidate 3D sampling point of the object in SE3 space, which is described by three rotation and three translation quantities. Subsequently, the object model parameters and the local dense deformation residual corresponding to each candidate 3D sampling point of the object can be transformed into SE3 space, and the two matrices are cross-multiplied by the matrix multiplication module to obtain the local dense deformation matrix for each candidate 3D sampling point of the object. Finally, the multiple candidate 3D sampling points of the object and the local dense deformation matrix can be transformed to the object reference space through the object transformation module to obtain multiple key 3D sampling points of the object corresponding to the multiple candidate 3D sampling points of the object. As mentioned above, the rotation and translation prediction network includes a rotation prediction network and a translation prediction network, both of which are composed of a multilayer perception mechanism containing two fully connected layers. Here, the object transformation module can be the SE3 transformation module. The object SE3 transformation module is a functional component used to implement the SE3 transformation. The SE3 transformation is a special Euclidean group transformation in three-dimensional space, which includes rotation and translation operations and is represented by a 4×4 homogeneous transformation matrix.

[0077] S104, acquire the target video sequence and input it into the target 3D reconstruction model to obtain the corresponding 3D reconstructed video sequence. Specifically, the target video sequence is input into the target 3D reconstruction model, which will process the target video sequence according to the relevant steps S102 to S103. For details, please refer to the following text. Figure 6 Related content.

[0078] Therefore, this disclosure provides a method, computer device, and program product for 3D reconstruction of hand-object interaction. It allows for the acquisition of a target video sequence and device posture parameters corresponding to an image acquisition device during hand-object interaction. The target video sequence includes multiple video frames arranged in chronological order. Based on the device posture parameters, a mapping relationship is established between a 2D coordinate system corresponding to the target video sequence and a target 3D coordinate system. According to the mapping relationship, multiple pixels in each video frame are mapped to the target 3D coordinate system to obtain multiple target 3D sampling points corresponding to each video frame. These multiple target 3D sampling points are then mapped to a hand reference space and an object reference space to obtain multiple hand target 3D sampling points and multiple object target 3D sampling points. Field attribute fusion rendering is performed on the multiple hand target 3D sampling points and multiple object target 3D sampling points corresponding to each video frame to obtain a 3D reconstructed video frame corresponding to each video frame. Finally, the 3D reconstructed video frames are arranged in chronological order to generate a 3D reconstructed video sequence corresponding to the target video sequence.

[0079] Based on this, the embodiments of this disclosure can establish a mapping relationship between a two-dimensional coordinate system and a target three-dimensional coordinate system, transforming the pixels of a video frame into target three-dimensional sampling points, thereby achieving an effective transition from two-dimensional to three-dimensional, allowing the initial extraction of spatial information of the hand and object. Mapping the target three-dimensional sampling points to the hand reference space and the object reference space respectively enables clear differentiation and independent processing of the information in both, avoiding data confusion and ensuring the accuracy of subsequent field attribute fusion rendering processing. Furthermore, the three-dimensional reconstructed video frames generated after field attribute fusion rendering processing are arranged into a three-dimensional reconstructed video sequence according to time sequence, fully preserving the dynamic process and temporal continuity of the interaction between the hand and object. It is evident that by mapping the target three-dimensional sampling points to the hand reference space and the object reference space for independent processing, the embodiments of this disclosure can accurately extract and retain data from the visible areas of both. Based on this, when integrating the processed hand target three-dimensional sampling points and object target three-dimensional sampling points, the data in the visible area can be logically completed based on the structural adaptation rules of the hand and object, thus accurately restoring the geometric shape and texture features of the hand and object even in occluded scenes.

[0080] The hand-object interaction 3D reconstruction method provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0081] For example, the aforementioned terminals may include one or more of the following: mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, handheld computers (PDAs), and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technologies. They may also include, but are not limited to, remote control devices, wearable devices, streetlights, home appliances, and other smart terminals. This disclosure does not impose specific limitations on these aspects.

[0082] Figure 6 This is a flowchart illustrating another hand-object interaction 3D reconstruction method provided in an embodiment of this disclosure. Figure 6 As shown, it specifically includes:

[0083] S601, during the interaction between the hand and the object, acquire the target video sequence and the device posture parameters corresponding to the image acquisition device, wherein the target video sequence includes multiple video frames arranged in chronological order.

[0084] In some embodiments, the subject may wear a head-mounted image acquisition device to acquire a target video sequence from a first-person perspective during the interaction between the subject's hands and the target object.

[0085] Specifically, during the acquisition of the target video sequence, the acquiring subject can wear a head-mounted image acquisition device or hold a handheld image acquisition device, keeping the device stationary throughout the acquisition process; thus, video segments acquired within a certain time period can be used as the target video sequence. It should be understood that image acquisition devices can include monocular and multi-view image acquisition devices, etc., and are devices used to acquire images.

[0086] S602, based on the device attitude parameters, establish a mapping relationship between the two-dimensional coordinate system corresponding to the target video sequence and the target three-dimensional coordinate system. It should be understood that the target three-dimensional coordinate system in this embodiment can be the world coordinate system, which is a fixed coordinate system shared by all objects and image acquisition devices in the scene, typically defined as a right-handed coordinate system.

[0087] In some embodiments, device attitude parameters may include internal and external parameters of the image acquisition device. These parameters can map pixels in a video frame to their spatial positions in a three-dimensional target coordinate system. Therefore, by establishing a mapping between a two-dimensional coordinate system and the target three-dimensional coordinate system, pixels in the video frame can be transformed into three-dimensional sampling points. This allows multiple parameters to be integrated into the same target three-dimensional coordinate system, ensuring a consistent description of the spatial positions of pixels in the video frame and avoiding positional deviations caused by coordinate system differences.

[0088] S603, according to the mapping relationship, the multiple pixels included in each of the multiple video frames are mapped to the target three-dimensional coordinate system, so as to obtain the multiple target three-dimensional sampling points corresponding to each of the multiple video frames.

[0089] In some embodiments, assuming that continuous video frames of a user's hand-object interaction process are acquired using a monocular image acquisition device, a mapping relationship between the imaging plane of the monocular image acquisition device and the target 3D coordinate system where the hand and object are located can first be established in advance. This mapping relationship can be calculated from the internal and external parameters of the monocular image acquisition device. When processing a certain video frame, pixels of key parts such as the user's finger joints can be extracted from the frame. Based on the above mapping relationship, the coordinates of each pixel are substituted into a preset transformation formula to obtain the corresponding target 3D sampling points. These target 3D sampling points can reflect the positional changes of the hand and object in different video frames in real time. The preset transformation formula can directly adopt existing classic formulas in the field of computer vision.

[0090] Therefore, by establishing a mapping relationship, the pixels of a video frame can be transformed into target 3D sampling points in the target 3D coordinate system, thus achieving a precise conversion from 2D visual information to 3D spatial information. In terms of effectiveness, on the one hand, each pixel in a video frame is assigned a clear 3D spatial coordinate, giving the originally scattered 2D pixel data a unified spatial positioning reference, laying the foundation for the spatiotemporal correlation of subsequent multi-frame data. On the other hand, the target 3D sampling points formed after mapping retain the pixel features and inter-frame temporal relationships of the original video frames, ensuring the integrity of spatial information and avoiding the loss of key data during the 2D-to-3D conversion process, providing high-quality basic data for downstream tasks such as 3D reconstruction and motion tracking. Furthermore, the unified processing method to the target 3D coordinate system eliminates differences in viewpoint and scale between different video frames, significantly reducing the technical difficulty of cross-frame data fusion and improving the efficiency and accuracy of the overall processing flow.

[0091] S604 maps multiple target 3D sampling points corresponding to each video frame to the hand reference space and the object reference space respectively, so as to obtain multiple hand target 3D sampling points and multiple object target 3D sampling points.

[0092] In some embodiments, hand reference space refers to a standardized three-dimensional spatial reference framework established for hand-related three-dimensional positioning, posture analysis, or interactive computation. Its core function is to provide a unified coordinate reference for the skeletal structure, motion trajectory, or interactive actions of the hand, thereby simplifying the processing, analysis, and application of hand-related data. Object reference space is a dedicated three-dimensional reference coordinate system established for a specific object, used to standardize the description of the object's own structure, posture, and spatial relationships with other objects, such as the hand.

[0093] Based on this, by mapping the corresponding target 3D sampling points to the hand reference space and the object reference space respectively, the spatial information of both the hand and the object in interaction can be "normalized" and "focused". Furthermore, this mapping provides a stable benchmark for subsequent analysis of video frames. The hand target 3D sampling points and the object target 3D sampling points of different video frames can be directly compared based on their respective benchmark spaces, significantly improving the continuity and accuracy of interactive motion tracking, and providing more reliable underlying data support for applications such as gesture interaction and virtual operation.

[0094] S605 performs field attribute fusion rendering on multiple 3D sampling points of hand targets and multiple 3D sampling points of object targets corresponding to each video frame to obtain the 3D reconstructed video frame corresponding to each video frame.

[0095] In some embodiments, for a scenario where a cup is picked up by hand, 30 3D sampling points of the hand target, including fingertip sampling points and knuckle sampling points, can be extracted from a certain video frame. 20 3D sampling points of the object target, including sampling points at the edge of the cup and sampling points on the body of the cup, can be extracted. Based on this, during field attribute fusion rendering, the "distance between sampling points" can be set as a field attribute. Thus, when the distance between the fingertip sampling point and the cup rim sampling point is less than 2cm, these two sampling points and their surrounding area can be rendered in red; if the distance is greater than 5cm, they can be rendered in blue; and if the distance is between 2-5cm, they can be rendered in a gradient orange-yellow. Simultaneously, a semi-transparent mesh is used to connect similar sampling points, allowing the outlines of the hand and cup to be naturally presented through the mesh. In the final 3D reconstructed video frame, the gripping state of "the red area of ​​the fingertip fitting with the red area of ​​the cup rim" can be clearly seen. The overall shape of the hand and cup is also intuitively displayed through the mesh and color, preserving 3D spatial position information while highlighting key interactive states through field attributes.

[0096] This allows the two originally discrete sampling points to be transformed into a holistic image with continuous field properties, making the spatial form and relative position of the hand and object more intuitive and perceptible. Simultaneously, the fusion rendering preserves and highlights the characteristics and interactions of the two types of sampling points in their respective reference spaces. For example, the motion trajectory of the hand's 3D sampling points is presented through dynamic changes in field properties, while the field property response of the corresponding area of ​​the object's 3D sampling points reflects the object's state under the influence of the hand. This makes the 3D reconstructed video frame not only a reconstruction of form but also a dynamic record of the interaction process. Furthermore, this processing provides a high-quality visual data foundation for subsequent interaction analysis and action recognition. Based on the prominent interactive features after rendering, actions such as grasping and touching can be identified more efficiently, improving the accuracy and efficiency of related applications.

[0097] S606, arrange the 3D reconstructed video frames corresponding to each video frame in chronological order to generate the 3D reconstructed video sequence corresponding to the target video sequence.

[0098] In some embodiments, arranging the images chronologically can string together isolated single-frame 3D reconstructed images into a continuous dynamic sequence, thus fully reproducing the entire process of hand-object interaction and allowing users to clearly perceive the dynamic logic of the interaction. Simultaneously, it enables clear correlation between the changes in field attributes of each frame in the 3D reconstructed video sequence, facilitating the tracing of key state transitions during the interaction and allowing for comparative analysis of the movement patterns of the hand and object through continuous frame comparison. Furthermore, this continuous sequence provides complete dynamic data samples for subsequent interactive behavior analysis and action pattern recognition. Based on the coherent field attribute features and morphological changes in the sequence, specific interactive actions such as "grasping," "moving," and "putting down" can be more accurately identified, providing more realistic support for applications such as virtual interactive feedback and human-computer collaborative control.

[0099] As can be seen, the embodiments of this disclosure provide a method, computer device, and program product for three-dimensional reconstruction of hand-object interaction. This method acquires a target video sequence and device posture parameters corresponding to the image acquisition device during the interaction between the hand and the object. The target video sequence includes multiple video frames arranged in chronological order. Based on the device posture parameters, a mapping relationship is established between the two-dimensional coordinate system corresponding to the target video sequence and the target three-dimensional coordinate system. According to the mapping relationship, multiple pixels in each video frame are mapped to the target three-dimensional coordinate system to obtain multiple target three-dimensional sampling points corresponding to each video frame. These multiple target three-dimensional sampling points are then mapped to the hand reference space and the object reference space to obtain multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points. Field attribute fusion rendering is performed on the multiple hand target three-dimensional sampling points and multiple object target three-dimensional sampling points corresponding to each video frame to obtain a three-dimensional reconstructed video frame corresponding to each video frame. Finally, the three-dimensional reconstructed video frames corresponding to each video frame are arranged in chronological order to generate a three-dimensional reconstructed video sequence corresponding to the target video sequence.

[0100] Based on this, the embodiments of this disclosure can establish a mapping relationship between a two-dimensional coordinate system and a target three-dimensional coordinate system, transforming the pixels of a video frame into target three-dimensional sampling points, thereby achieving an effective transition from two-dimensional to three-dimensional, allowing the initial extraction of spatial information of the hand and object. Mapping the target three-dimensional sampling points to the hand reference space and the object reference space respectively enables clear differentiation and independent processing of the information in both, avoiding data confusion and ensuring the accuracy of subsequent field attribute fusion rendering processing. Furthermore, the three-dimensional reconstructed video frames generated after field attribute fusion rendering processing are arranged into a three-dimensional reconstructed video sequence according to time sequence, fully preserving the dynamic process and temporal continuity of the interaction between the hand and object. It is evident that by mapping the target three-dimensional sampling points to the hand reference space and the object reference space for independent processing, the embodiments of this disclosure can accurately extract and retain data from the visible areas of both. Based on this, when integrating the processed hand target three-dimensional sampling points and object target three-dimensional sampling points, the data in the visible area can be logically completed based on the structural adaptation rules of the hand and object, thus accurately restoring the geometric shape and texture features of the hand and object even in occluded scenes.

[0101] In some embodiments, multiple target 3D sampling points corresponding to each video frame are mapped to a hand reference space and an object reference space, respectively, to obtain multiple hand target 3D sampling points and multiple object target 3D sampling points. This includes: for any video frame, segmenting the hand and object in that video frame to obtain a hand mask map and an object mask map; using a hand keypoint detection algorithm to identify the hand region corresponding to the hand in the hand mask map, and mapping multiple candidate hand 3D sampling points located within the hand region from the multiple target 3D samplings to the hand reference space through a hand dense deformation field, to obtain multiple hand target 3D sampling points; determining the object region corresponding to the object based on the object mask map, and mapping multiple candidate object 3D sampling points located within the object region from the multiple target 3D samplings to the object reference space through a preset object dense deformation field, to obtain multiple object target 3D sampling points.

[0102] Specifically, firstly, RVM can be used to perform foreground segmentation on any captured video frame; then, an image semantic segmentation tool can be used to perform semantic segmentation on the foreground mask map corresponding to any video frame, obtaining the hand mask map and object mask map corresponding to the foreground mask map; then, an object detection algorithm can be used to further locate the hand in the hand mask map to identify the hand region corresponding to the hand in the hand mask map. For details, please refer to steps S1021 to S1023 above.

[0103] In some embodiments, the dense deformation field for the hand includes a convolution generator, a global skeleton calculation module, a hand transformation module, and a weighted summation module. It maps multiple candidate 3D sampling points of the hand located within the hand region from multiple target 3D samples onto a hand reference space through the dense deformation field to obtain multiple target 3D sampling points of the hand. This includes: inputting a preset globally learnable latent variable into the convolution generator to obtain a skeleton weight field, wherein the skeleton weight field describes the relationship between each 3D sampling point in the hand reference space and multiple bones; inputting the hand model parameters corresponding to any video frame into the global skeleton calculation module to obtain a global skeleton transformation matrix corresponding to any video frame; for any candidate 3D sampling point of the hand, inputting the candidate 3D sampling point of the hand and the global skeleton transformation matrix into the hand transformation module to obtain a transformed candidate 3D sampling point of the hand; and inputting the transformed candidate 3D sampling point of the hand and the skeleton weight field into the weighted summation module to obtain the target 3D sampling point of the hand corresponding to the candidate 3D sampling point of the hand. It should be understood that the skeleton weight field here is the motion weight volume.

[0104] like Figure 4As shown, the structure of the dense deformation field of the hand includes a convolution generator, which can be a 3D convolution generator. This generator generates a Motion Weight Volume (MWV) to determine the influence of each bone movement on each target 3D sampling point in the hand's reference space. Specifically, the convolution generator spatially expands the globally learnable latent variables built into the initial 3D reconstruction model, transforming them from tensors of size (1, 1, 1) to tensors of size (32, 32, 32), while simultaneously transforming the feature dimensions. Each feature dimension represents the influence of a predefined bone movement transformation on the 3D sampling points in the hand's reference space. Here, the built-in globally learnable latent variables are variables pre-defined in the model and automatically adjusted and optimized during training.

[0105] like Figure 4 As shown, the structure of the dense deformation field of the hand also includes a global skeleton calculation module, a hand transformation module, and a weighted summation module. Specifically, taking any candidate 3D sampling point of the hand in any video frame as an example, the 48 hand pose parameters corresponding to any video frame and the predefined hand dynamics map can be input into the global skeleton motion calculation module to obtain 16 rotation and translation variables of the skeleton relative to the target 3D coordinate system. By integrating these rotation and translation variables, the corresponding global skeleton transformation matrix, i.e., the SE3 matrix, can be generated. Then, the candidate 3D sampling points p = (x, y, z) of the hand in the hand reference space are transformed to a homogeneous coordinate system and coupled with the SE3 matrix. Multiplying these results in the candidate 3D sampling points of the hand after transformation by the hand transformation module; then, taking the i-th bone as an example, the positions of the transformed candidate 3D sampling points of the hand are... Normalized to the interval between -1 and 1, and then trilinear interpolation is performed on the MWV corresponding to the i-th bone to obtain the weight of the i-th bone motion. Then through The formula calculates the weight of the i-th bone motion in the hand reference space. Using the formula in the weighted summation module Weighted summation yields the 3D sampling points of the hand target in the hand reference space. Here, the hand transformation module can be the SE3 transformation module. The object SE3 transformation module is a functional component used to implement the SE3 transformation. The SE3 transformation is a special Euclidean group transformation in 3D space, which includes rotation and translation operations and is represented by a 4×4 homogeneous transformation matrix.

[0106] In summary, the convolution generator can spatially expand and dimensionally transform globally learnable latent variables to generate motion weights, accurately quantifying the influence of each bone's motion on any candidate 3D sampling point of the hand, providing a refined correlation basis for subsequent deformation calculations. The global bone calculation module combines hand pose parameters and dynamic graphs to generate a global bone transformation matrix, accurately capturing the hand's motion state in video frames and ensuring the reliability of the transformation basis. The hand transformation module uses SE3 transformation to achieve rotation and translation operations in 3D space, accurately transforming any candidate 3D sampling point of the hand to its corresponding transformation state. The weighted summation module performs interpolation and weighted calculations based on motion weights, ultimately achieving accurate mapping of any candidate 3D sampling point of the hand to the hand's reference space. Overall, the structure of this dense hand deformation field achieves dynamic adaptation to the influence of bone motion through learnable motion weights, and ensures the continuity and accuracy of hand deformation through bone transformation and weighting mechanisms, effectively improving the accuracy and stability of mapping any candidate 3D sampling point of the hand, providing an efficient deformation mapping solution for tasks related to hand motion, such as 3D reconstruction and pose estimation.

[0107] In some embodiments, the hand model parameters corresponding to any video frame are input into the global skeleton calculation module to obtain the global skeleton transformation matrix corresponding to the video frame. The method further includes: acquiring multiple feature point data corresponding to multiple hand feature points in any video frame, and performing temporal correlation processing on the multiple feature point data; determining the hand model parameters corresponding to any video frame based on the distribution characteristics of multiple hand feature points in any video frame, the multiple feature point data corresponding to multiple hand feature points, and the corresponding temporal correlation processing results. Here, the hand model parameters include hand posture parameters and hand shape parameters. Among them, hand posture parameters can be used to describe the motion state of the hand in the current video frame, such as a set of parameters for joint rotation angles and joint positions; hand shape parameters are used to describe the inherent physiological morphology of the hand, such as a set of parameters for individual differences in palm size, finger thickness, and joint proportions, and are the core indicators for distinguishing different static features of the hand.

[0108] Specifically, the hand detection bounding box output by YOLO-X can be used as a constraint to further narrow down the range of the hand region by calling the Openpose feature point detector. This allows for targeted detection within the hand region corresponding to the hand detection bounding box. Openpose can analyze the texture and contour features of the hand within this region, accurately identifying and outputting the coordinates of multiple hand feature points. The number of hand feature points mentioned above can be 21, specifically including 4 feature points on the thumb: the thumb tip, the first joint of the thumb, the second joint of the thumb, and the metacarpophalangeal joint of the thumb; 4 feature points on the index finger: the index finger tip, the first joint of the index finger, the second joint of the index finger, and the metacarpophalangeal joint of the index finger; 4 feature points on the middle finger: the middle finger tip, the first joint of the middle finger, the second joint of the middle finger, and the metacarpophalangeal joint of the middle finger; 4 feature points on the ring finger: the ring finger tip, the first joint of the ring finger, the second joint of the ring finger, and the metacarpophalangeal joint of the ring finger; 4 feature points on the little finger: the little finger tip, the first joint of the little finger, the second joint of the little finger, and the metacarpophalangeal joint of the little finger; and feature points at the base of the palm, i.e., the hand feature points corresponding to the wrist.

[0109] Furthermore, the data corresponding to 21 hand feature points from any video frame can be input into the EasyMocap open-source tool, and processed using a Model of Animated Hand with Articulated Nails and Opposable Thumb (MANO). EasyMocap performs temporal correlation processing and optimization on the data corresponding to the 21 hand feature points to ensure the continuity of hand poses in consecutive video frames. The MANO model, a parametric model specifically designed for 3D hand modeling, can derive the 3D pose information of the hand based on the spatial distribution characteristics of the 21 hand feature points, multiple feature point data corresponding to multiple hand feature points, and the corresponding temporal correlation processing results. Ultimately, it outputs 48 hand pose parameters and 10 hand shape parameters corresponding to any video frame. Here, EasyMocap is an open-source tool developed by a university's 3D vision team. It is a label-free human motion capture toolkit designed to achieve human motion capture and new perspective synthesis through video. The MANO model is a parametric model used for accurate modeling and driving the 3D shape and movement of the human hand, and is widely used in the fields of computer vision, graphics, and human-computer interaction.

[0110] The aforementioned 48 hand pose parameters can include 3 global pose parameters and 45 finger joint pose parameters. The global pose parameters describe the overall rotation of the hand in three-dimensional space, corresponding to the rotation angles of the hand relative to the target three-dimensional coordinate system, such as the rotation angles of the hand around the X, Y, and Z axes of the target three-dimensional coordinate system, used to determine the overall spatial orientation of the hand, i.e., the overall rotation direction of the wrist when holding an object. The finger joint pose parameters describe the local rotation of each joint in the hand. The 45 finger joint pose parameters can include pose parameters corresponding to 15 key joints. These 15 key joints include 3 joints corresponding to each of the 5 fingers, and each key joint can be described by multiple parameters to represent its rotation state, such as joint flexion angle and joint lateral tilt angle. The hand shape parameters describe the inherent physiological morphology of the hand, including but not limited to inherent attributes such as palm width, finger length, knuckle thickness, palm thickness, and finger spacing.

[0111] In summary, the hand model parameter acquisition method of this disclosure, in the feature point detection stage, can use the YOLO-X detection box as a constraint and Openpose to specifically identify 21 hand feature points. This not only improves efficiency by narrowing the scope with the detection box, but also ensures the integrity and accuracy of feature data by accurately locating key points such as fingertips and joints, laying a high-quality data foundation for subsequent parameter derivation. In the temporal processing and parameter modeling stage, EasyMocap can optimize the temporal correlation of feature point data, effectively ensuring the continuity of hand posture in consecutive video frames and avoiding posture jumps caused by single-frame data fluctuations. The MANO model, combined with the structural characteristics of the hand with articulated nails, derives parameters based on the spatial distribution of feature points and temporal results. It can not only accurately output 48 hand posture parameters and 10 hand shape parameters, but also take into account the overall hand movement and individual physiological differences through parameterized expression. Overall, this method, through a closed-loop process of "detection-temporal optimization-parametric modeling," not only achieves accurate conversion of hand feature points to hand model parameters but also ensures the stability and physiological rationality of parameters in continuous video frames. This provides a highly reliable input basis for subsequent generation of the global transformation matrix of the skeleton and accurate mapping of the dense deformation field of the hand, further improving the accuracy and naturalness of hand 3D motion modeling.

[0112] In some embodiments, the object dense deformation field comprises a feature extraction network, a pose embedding network, a feature fusion network, a rotation and translation prediction network, a matrix multiplication module, and an object transformation module. It maps multiple candidate 3D sampling points of objects located within the object region from multiple target 3D samplings onto the object reference space through a preset object dense deformation field to obtain multiple target 3D sampling points of the object. This includes: inputting the multiple candidate 3D sampling points of the object into the feature extraction network to obtain multiple spatial feature vectors corresponding to the multiple candidate 3D sampling points of the object; inputting the object model parameters corresponding to any video frame into the pose embedding network to obtain the object pose feature vector corresponding to the object model parameters; and inputting the multiple candidate 3D sampling points of the object into the pose embedding network. Multiple spatial feature vectors and object pose feature vectors corresponding to the 3D sampling points are input into the feature fusion network to obtain local feature fusion vectors corresponding to multiple candidate 3D sampling points of the object. The local feature fusion vectors corresponding to multiple candidate 3D sampling points of the object are input into the rotation and translation prediction network to obtain local dense deformation residuals corresponding to multiple candidate 3D sampling points of the object. The local dense deformation residuals corresponding to multiple candidate 3D sampling points of the object and the object model parameters are input into the matrix multiplication module to obtain the local dense deformation matrix corresponding to multiple candidate 3D sampling points of the object. The multiple candidate 3D sampling points of the object and the corresponding local dense deformation matrix are input into the object transformation module to obtain multiple target 3D sampling points of the object.

[0113] like Figure 5 As shown, the feature extraction network and the pose embedding network take the target 3D sampling points and object model parameters in the 3D observation space as inputs, respectively. The feature extraction network encodes multiple candidate 3D sampling points of the object and extracts the geometric features of the point cloud, such as distribution patterns and local structures, and outputs multiple spatial feature vectors corresponding to multiple candidate 3D sampling points of the object. The pose embedding network encodes the object model parameters into object pose feature vectors. The feature extraction network and the pose embedding network are each composed of a high-frequency information encoder and a multilayer perception mechanism containing 4 fully connected layers.

[0114] Next, the feature fusion component fuses multiple spatial feature vectors and pose feature vectors through feature concatenation to obtain a local feature fusion vector corresponding to each candidate 3D sampling point of the object. Then, based on the local feature fusion vector corresponding to each candidate 3D sampling point, a rotation and translation prediction network is used to predict the local dense deformation residual of each candidate 3D sampling point in SE3 space, described by three rotation and three translation quantities. Subsequently, the object model parameters and the local dense deformation residual corresponding to each candidate 3D sampling point can be transformed into SE3 space, and the cross product of the two matrices is obtained through the matrix multiplication module to obtain the local dense deformation matrix for each candidate 3D sampling point. Finally, the multiple candidate 3D sampling points and the local dense deformation matrix can be transformed to the object reference space through the object transformation module to obtain multiple target 3D sampling points of the object corresponding to the multiple candidate 3D sampling points. As mentioned above, the rotation and translation prediction network includes a rotation prediction network and a translation prediction network, both of which are composed of a multilayer perception mechanism containing two fully connected layers. Here, the object transformation module can be the SE3 transformation module. The object SE3 transformation module is a functional component used to implement the SE3 transformation. The SE3 transformation is a special Euclidean group transformation in three-dimensional space, which includes rotation and translation operations and is represented by a 4×4 homogeneous transformation matrix.

[0115] In summary, during the feature extraction stage, the feature extraction network, leveraging a high-frequency information encoder and a multilayer perceptron, accurately extracts the geometric features of the candidate 3D sampling points, such as distribution patterns and local structures. The resulting spatial feature vectors provide a solid geometric foundation for subsequent processing. The pose embedding network, also composed of a high-frequency information encoder and a multilayer perceptron, encodes the object model parameters into object pose feature vectors, effectively capturing the object's pose information. The feature fusion network, through feature concatenation, fuses spatial and pose feature vectors into a local feature fusion vector, achieving an organic combination of geometric and pose features, ensuring that each candidate 3D sampling point possesses both spatial and pose attributes. The rotation and translation prediction network predicts local dense deformation residuals in SE3 space based on the local feature fusion vectors, accurately describing the rotation and translation changes of each candidate 3D sampling point. The matrix multiplication module transforms the object model parameters and local dense deformation residuals into SE3 space and performs a cross multiplication to obtain a local dense deformation matrix, further optimizing the deformation description. The object transformation module utilizes SE3 transformation to combine the candidate 3D sampling points and the local dense deformation matrix, successfully mapping them to the object's reference space. Based on this, the structure of the candidate 3D sampling points of the object in the embodiments of this disclosure realizes a complete process from sampling point feature extraction, posture information capture to feature fusion, deformation prediction and final mapping. It not only ensures the accuracy of feature extraction and the effectiveness of fusion, but also ensures the reliability of mapping results through precise deformation processing. It provides strong support for object-related 3D modeling, deformation analysis and other tasks, and improves the accuracy and rationality of 3D sampling point mapping.

[0116] In some embodiments, field attribute fusion rendering is performed on multiple hand target 3D sampling points and multiple object target 3D sampling points corresponding to each video frame to obtain a 3D reconstructed video frame corresponding to each video frame. This includes: for any video frame, inputting multiple hand target 3D sampling points corresponding to that video frame into a preset hand neural radiation field to obtain hand radiation field parameters corresponding to the multiple hand target 3D sampling points; inputting multiple object target 3D sampling points into a preset object neural radiation field to obtain object radiation field parameters corresponding to the multiple object target 3D sampling points; if the hand radiation field parameter is greater than the object radiation field parameter, determining the hand radiation field parameter as the target radiation field parameter; or, if the object radiation field parameter is greater than the hand radiation field parameter, determining the object radiation field parameter as the target radiation field parameter; constructing a target 3D mesh surface based on the target radiation field parameter, and performing rendering operations on the target 3D mesh surface based on the target radiation field parameter to obtain a 3D reconstructed video frame corresponding to any video frame.

[0117] like Figure 3As shown, the learning network of the target 3D reconstruction model in this embodiment of the present disclosure may include two parts: a hand branch learning network and an object branch learning network. Based on this, after obtaining multiple target 3D sampling points, these points can be input into the hand branch learning network and the object branch learning network respectively. The hand branch learning network includes a dense deformation field of the hand and a preset hand nerve radiation field, while the object branch learning network includes a dense deformation field of the object and a preset object nerve radiation field. Here, the target 3D reconstruction model in this embodiment of the present disclosure is the target 3D reconstruction model described above, and the specific construction process can be found in steps S101 to S103.

[0118] Specifically, for the hand-branch learning network, hand model parameters and multiple target 3D sampling points can be input into a dense deformation field of the hand. This field can map these points to a hand reference space based on hand bone motion weights, thus obtaining the corresponding hand target 3D sampling points when located in the hand reference space. These points can then be further processed by a preset hand neural radiation field to obtain the hand's signed distance function (SDF), hand density value, and hand color weights. The hand density value describes whether the target 3D sampling point belongs to the hand surface, and the hand color parameters can be the red (R), green (G), and blue (B) color information corresponding to the target 3D sampling point.

[0119] For object branch learning networks, object model parameters and multiple object target 3D sampling points can be input into an object dense deformation field. This field predicts subtle deformation residuals and maps these sampling points to the object's reference space, obtaining the corresponding 3D sampling points when located in the reference space. These sampling points are then further processed by a preset object neural radiation field to obtain the object's SDF value, density value, and color parameters. The object density value describes whether the sampling point belongs to the object's surface, and the color parameter describes the object's texture features. Both the preset hand neural radiation field and the preset object neural radiation field can employ a "multi-resolution hash coding + small multilayer perceptron" structure, which enables rapid convergence of the initial 3D reconstruction model and improves accuracy.

[0120] In some embodiments, since occlusion may occur during hand-object interaction, the final features of each target 3D sampling point can also be determined by field overlay. Specifically, a target 3D sampling point can be mapped to both the hand reference space and the object reference space simultaneously; that is, the target 3D sampling point can serve as both a hand target 3D sampling point and an object target 3D sampling point. In this case, the density value of the target 3D sampling point can be taken as the larger density value between the preset hand neural radiation field and the preset object neural radiation field. Simultaneously, volume rendering is performed based on the SDF value and color weight corresponding to the field with the larger density value, thereby generating a rendered image that matches the device pose parameters.

[0121] Specifically, when performing volume rendering, the following steps can be followed: First, for any target 3D sampling point, the corresponding density value can be extracted from the preset hand neural radiation field and the preset object neural radiation field, respectively. By comparing these two density values, the larger value is selected as the final density value of the target 3D sampling point. Next, depending on the source of the density value, i.e., whether it comes from the preset hand neural radiation field or the preset object neural radiation field, the final SDF value output by the corresponding neural radiation field is called. Specifically, if the final density comes from the preset hand neural radiation field, the hand SDF value is used as the final SDF value; if it comes from the preset object neural radiation field, the object SDF value is used as the final SDF value. This final SDF value can help optimize the spatial position accuracy of the 3D sampling point.

[0122] Furthermore, color weight calculation rules can be matched based on the density source: if the preset hand neural radiation field dominates, color weights are assigned according to the texture mapping logic of the preset hand neural radiation field; if the preset object neural radiation field dominates, color weights are calculated according to the material properties of the object itself. Finally, the final density value, the sampling position corresponding to any target sampling point optimized based on the final SDF value, and the corresponding color weight are substituted into the volume rendering formula, combined with the device pose parameters of the current image acquisition device, to synthesize the final rendered 3D reconstructed video frame pixel by pixel. In this process, areas with higher density will naturally become the visual focus, such as highlighting the outline of the hand when it occludes an object, and highlighting the details of the object when it is revealed, thereby achieving realistic rendering that conforms to the spatial occlusion logic.

[0123] In summary, in the processing of hand and object branches, the hand branch learning network and the object branch learning network respectively map the target sampling points to the corresponding reference space using the corresponding dense deformation fields. Then, they output radiation field parameters through preset neural radiation fields. The "multi-resolution hash encoding + small multilayer perceptron" structure also improves the accuracy of parameter output and model convergence speed, providing high-quality basic data for subsequent rendering. Regarding the occlusion problem in hand-object interaction, determining the dominant field by comparing the density values ​​of the hand and the object is crucial. Selecting the area with the higher density value as the visual focus not only conforms to the spatial occlusion logic but also optimizes the sampling point position accuracy through the SDF value of the corresponding field. Simultaneously, calculating color weights according to the rules of the dominant field ensures the realism of texture and material presentation in occluded scenes and avoids visual confusion in overlapping areas. During the volume rendering stage, video frames are generated by combining the final density, optimized sampling position, and color weight. This allows for the precise synthesis of images that match the device's posture parameters, highlighting the hand's outline when it occludes an object and showcasing details when the object is revealed. The final output 3D reconstructed video frames combine spatial accuracy and visual realism, effectively improving the overall effect of 3D reconstruction in hand-object interaction scenarios and providing highly reliable visual output for related applications.

[0124] In some embodiments, mapping multiple pixels in each of the multiple video frames to a target three-dimensional coordinate system according to a mapping relationship to obtain multiple target three-dimensional sampling points corresponding to each of the multiple video frames includes: for any video frame, converting the pixel coordinates corresponding to the multiple pixels in any video frame into multiple direction vectors in the target three-dimensional coordinate system according to the mapping relationship; and sampling on the multiple direction vectors to obtain multiple target three-dimensional sampling points corresponding to any video frame.

[0125] Specifically, when analyzing the initial video frames, a certain number of pixels can be randomly selected from any video frame. This randomness in pixel selection ensures that the target 3D reconstruction model learns the features of different regions within any video frame, such as hand edges, object surfaces, and hand-object contact points, avoiding feature learning bias caused by fixed sampling. Here, the number of pixels can be 2048.

[0126] Next, for multiple pixels selected from any video frame, the direction vectors corresponding to these pixels in the target 3D coordinate system can be calculated by combining the internal and external parameters of the image acquisition device. For any given pixel, the direction vector is determined by the position of the image acquisition device and the direction of the line connecting the image acquisition device and the pixel in the target 3D coordinate system. This integrates multiple parameters into the same target 3D coordinate system, ensuring a unified description of the spatial positions and interactions of the hand, object, and image acquisition device, avoiding positional deviations caused by coordinate system differences. It should be understood that the direction vector here refers to the ray mentioned above.

[0127] In some embodiments, the process of calculating the direction vectors corresponding to multiple pixels in the target three-dimensional coordinate system by combining the internal and external parameters of the image acquisition device can be as follows: For multiple selected pixels in a video frame, lens distortion can be eliminated first using internal parameters to obtain the corrected pixel coordinates of each pixel; then, based on the focal length and principal point coordinates, the corrected pixel coordinates are converted into three-dimensional direction vectors in the coordinate system corresponding to the image acquisition device. Next, the coordinate system corresponding to the image acquisition device can be transformed to the target three-dimensional coordinate system using the rotation matrix and translation vector of the external parameters. Simultaneously, the principal point coordinates of the image acquisition device are used as the origin and mapped to the origin of the direction vectors in the target three-dimensional coordinate system using the translation vector; thus, the direction vectors corresponding to multiple pixels in the target three-dimensional coordinate system can be obtained.

[0128] In summary, during the pixel selection stage, 2048 pixels are randomly selected from any video frame. This randomness covers features from different areas such as hand edges, object surfaces, and hand-object contact points, avoiding feature learning biases that may result from fixed sampling. This ensures that the target 3D reconstruction model can comprehensively identify and analyze features from various regions of the video frame, providing diverse foundational data for the accurate acquisition of subsequent 3D sampling points. In the direction vector calculation and sampling stage, the direction vector of each pixel in the target 3D coordinate system is obtained by combining the internal and external parameters of the image acquisition device. Integrating pixel coordinates and device parameters into the same coordinate system effectively eliminates positional deviations caused by coordinate system differences, ensuring a unified description of the spatial position and interaction relationship between the hand, object, and image acquisition device. Sampling based on the direction vector to obtain target 3D sampling points allows the sampling points to accurately correspond to pixel information in the video frame, providing a precise spatial sampling basis for subsequent 3D reconstruction of the hand and object, further improving the reliability of the entire 3D reconstruction process.

[0129] In some embodiments, sampling is performed on multiple direction vectors to obtain multiple target 3D sampling points corresponding to any video frame, including: performing a first sampling on multiple direction vectors to obtain multiple candidate 3D sampling points; determining a key region corresponding to a hand or object based on the multiple candidate sampling points; and performing a second sampling on multiple direction vectors located within the key region to obtain multiple target 3D sampling points.

[0130] In some embodiments, after obtaining the direction vector corresponding to each pixel, coarse sampling can be performed first to determine the key region corresponding to each direction vector, i.e. the region corresponding to the hand and / or the region corresponding to the object, and then fine sampling can be performed on the key region to obtain multiple target three-dimensional sampling points.

[0131] Specifically, coarse sampling can be performed first, uniformly selecting a small number of 3D sampling points across multiple directional vectors to quickly identify key regions where hands or objects might exist along the directional vectors, excluding obviously empty spaces. Next, fine sampling is performed on these key regions, densely selecting more target 3D sampling points to improve the sampling accuracy of the hand and object surfaces and surrounding space. The target 3D sampling points obtained through this layered sampling method can serve as input for subsequent dense deformation fields of the hand, object, and neural radiation fields, providing foundational data for coordinate transformation and attribute prediction in these network modules. This sampling method avoids redundant sampling of invalid spaces to reduce computation, while ensuring sampling density in key regions through fine sampling, thereby improving the detail recognition performance of the target 3D reconstruction model.

[0132] Based on this, the embodiments of this disclosure can establish a mapping relationship between a two-dimensional coordinate system and a target three-dimensional coordinate system, transforming the pixels of a video frame into target three-dimensional sampling points, thereby achieving an effective transition from two-dimensional to three-dimensional, allowing the initial extraction of spatial information of the hand and object. Mapping the target three-dimensional sampling points to the hand reference space and the object reference space respectively enables clear differentiation and independent processing of the information in both, avoiding data confusion and ensuring the accuracy of subsequent field attribute fusion rendering processing. Furthermore, the three-dimensional reconstructed video frames generated after field attribute fusion rendering processing are arranged into a three-dimensional reconstructed video sequence according to time sequence, fully preserving the dynamic process and temporal continuity of the interaction between the hand and object. It is evident that by mapping the target three-dimensional sampling points to the hand reference space and the object reference space for independent processing, the embodiments of this disclosure can accurately extract and retain data from the visible areas of both. Based on this, when integrating the processed hand target three-dimensional sampling points and object target three-dimensional sampling points, the data in the visible area can be logically completed based on the structural adaptation rules of the hand and object, thus accurately restoring the geometric shape and texture features of the hand and object even in occluded scenes.

[0133] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0134] This disclosure also provides an electronic device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is used to execute the instruction to implement the steps of the method disclosed in this disclosure.

[0135] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 includes at least one processor 701 and a memory 702 coupled to the processor 701. The processor 701 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0136] The processor 701 described above can also be referred to as a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 701's hardware or by software instructions. The processor 701 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 702, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 701 reads information from the memory 702 and, in conjunction with its hardware, completes the steps of the above method.

[0137] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 8 The computer system 800 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those mentioned above. Figure 8 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure.

[0138] Computer system 800 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 8As shown, the computer system 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the computer system 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0140] Multiple components in the computer system 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device capable of inputting information into the computer system 800. The input unit 806 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 808 may include, but is not limited to, a hard disk and an optical disk. The communication unit 809 allows the computer system 800 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.

[0141] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0142] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0143] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] Figure 9 This is a schematic diagram of a computer program product provided according to an embodiment of the present disclosure. Figure 9 As shown, the computer program product 900 includes a computer program 901, which, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0146] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0149] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0150] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0151] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for three-dimensional reconstruction of hand-object interaction, characterized in that, Comprise: In the interaction process of the hand and the object, obtain the device posture parameters corresponding to the target video sequence and the image acquisition device, wherein the target video sequence comprises a plurality of video frames arranged in time sequence; According to the device posture parameters, the mapping relationship between the two-dimensional coordinate system corresponding to the target video sequence and the target three-dimensional coordinate system is established; According to the mapping relationship, map a plurality of pixel points included in each of the plurality of video frames to the target three-dimensional coordinate system to obtain a plurality of target three-dimensional sampling points corresponding to each of the plurality of video frames respectively; Map the plurality of target three-dimensional sampling points corresponding to each of the video frames to the hand reference space and the object reference space respectively to obtain a plurality of hand target three-dimensional sampling points and a plurality of object target three-dimensional sampling points; Perform field attribute fusion rendering processing on the plurality of hand target three-dimensional sampling points and the plurality of object target three-dimensional sampling points corresponding to each of the video frames to obtain a three-dimensional reconstruction video frame corresponding to each of the video frames; Arrange the three-dimensional reconstruction video frame corresponding to each of the video frames in the time sequence to generate a three-dimensional reconstruction video sequence corresponding to the target video sequence.

2. The method of claim 1, wherein, The mapping of the plurality of target three-dimensional sampling points corresponding to each of the video frames to the hand reference space and the object reference space respectively to obtain a plurality of hand target three-dimensional sampling points and a plurality of object target three-dimensional sampling points comprises: For any video frame, perform segmentation processing on the hand and the object in the any video frame to obtain a hand mask graph and an object mask graph; Identify the hand region corresponding to the hand in the hand mask graph using a hand key point detection algorithm, and map a plurality of hand candidate three-dimensional sampling points located in the hand region in the plurality of target three-dimensional sampling points to the hand reference space through a hand dense deformation field to obtain a plurality of hand target three-dimensional sampling points; According to the object mask graph, determine the object region corresponding to the object, and map a plurality of object candidate three-dimensional sampling points located in the object region in the plurality of target three-dimensional sampling points to the object reference space through a preset object dense deformation field to obtain a plurality of object target three-dimensional sampling points.

3. The method of claim 2, wherein, The hand dense deformation field comprises a convolution generator, a skeleton global calculation module, a hand transformation module and a weighted summation module, and the mapping of the plurality of hand candidate three-dimensional sampling points located in the hand region in the plurality of target three-dimensional sampling points to the hand reference space through the hand dense deformation field to obtain a plurality of hand target three-dimensional sampling points comprises: Input a preset global learnable hidden variable into the convolution generator to obtain a skeleton weight field, wherein the skeleton weight field is used to describe the association relationship between each three-dimensional sampling point in the hand reference space and a plurality of skeletons; Input the hand model parameters corresponding to the any video frame into the skeleton global calculation module to obtain a skeleton global transformation matrix corresponding to the video frame; For any of the hand candidate three-dimensional sampling points, input any of the hand candidate three-dimensional sampling points and the bone global transformation matrix into the hand transformation module to obtain a transformed any of the hand candidate three-dimensional sampling points; Input the transformed any of the hand candidate three-dimensional sampling points and the bone weight field into the weighted summation module to obtain a hand target three-dimensional sampling point corresponding to the hand candidate three-dimensional sampling point.

4. The method of claim 3, wherein, The method further comprises: Obtain a plurality of feature point data corresponding to a plurality of hand feature points in the any video frame, and perform time sequence association processing on a plurality of the feature point data; Determine the hand model parameter corresponding to the any video frame according to the distribution characteristics of a plurality of the hand feature points in the any video frame, a plurality of feature point data corresponding to a plurality of hand feature points, and a corresponding time sequence association processing result.

5. The method of claim 2, wherein, The object dense deformation field comprises a feature extraction network, a pose embedding network, a feature fusion network, a rotation and translation prediction network, a matrix multiplication module, and an object transformation module. The plurality of object candidate three-dimensional sampling points located in the object region in the plurality of target three-dimensional sampling points are mapped into an object reference space through a preset object dense deformation field to obtain a plurality of object target three-dimensional sampling points, comprising: Input the plurality of object candidate three-dimensional sampling points into the feature extraction network to obtain a plurality of spatial feature vectors corresponding to the plurality of object candidate three-dimensional sampling points, respectively; Input the object model parameter corresponding to the any video frame into the pose embedding network to obtain an object pose feature vector corresponding to the object model parameter; Input the plurality of spatial feature vectors corresponding to the plurality of object candidate three-dimensional sampling points and the object pose feature vector into the feature fusion network to obtain a local feature fusion vector corresponding to each of the plurality of object candidate three-dimensional sampling points, respectively; Input the local feature fusion vector corresponding to each of the plurality of object candidate three-dimensional sampling points into the rotation and translation prediction network to obtain a local dense deformation residual corresponding to each of the plurality of object candidate three-dimensional sampling points, respectively; Input the local dense deformation residual corresponding to each of the plurality of object candidate three-dimensional sampling points and the object model parameter into the matrix multiplication module to obtain a local dense deformation matrix corresponding to each of the plurality of object candidate three-dimensional sampling points; Input the plurality of object candidate three-dimensional sampling points and the corresponding local dense deformation matrix into the object transformation module to obtain a plurality of object target three-dimensional sampling points.

6. The method of claim 1, wherein, The method further comprises: For any video frame, input the plurality of hand target three-dimensional sampling points corresponding to the any video frame into a preset hand neural radiance field to obtain a hand radiance field parameter corresponding to each of the plurality of hand target three-dimensional sampling points, respectively; For any video frame, input the plurality of hand target three-dimensional sampling points corresponding to the any video frame into a preset hand neural radiance field to obtain a hand radiance field parameter corresponding to each of the plurality of hand target three-dimensional sampling points, respectively; inputting a plurality of object target three-dimensional sampling points into a preset object neural radiance field to obtain object radiance field parameters corresponding to the plurality of object target three-dimensional sampling points respectively; determining the hand radiance field parameters as the target radiance field parameters in a case where the hand radiance field parameters are greater than the object radiance field parameters, or determining the object radiance field parameters as the target radiance field parameters in a case where the object radiance field parameters are greater than the hand radiance field parameters; constructing a target three-dimensional grid surface according to the target radiance field parameters, and performing a rendering operation on the target three-dimensional grid surface according to the target radiance field parameters to obtain the three-dimensional reconstruction video frame corresponding to the any video frame.

7. The method of claim 1, wherein, The mapping of the plurality of pixel points included in each of the plurality of video frames to the target three-dimensional coordinate system according to the mapping relationship to obtain a plurality of target three-dimensional sampling points corresponding to each of the plurality of video frames comprises: For any video frame, converting pixel coordinates corresponding to a plurality of pixel points included in the any video frame into a plurality of direction vectors in the target three-dimensional coordinate system according to the mapping relationship; sampling on the plurality of direction vectors to obtain a plurality of target three-dimensional sampling points corresponding to the any video frame.

8. The method of claim 7, wherein, The sampling on the plurality of direction vectors to obtain a plurality of target three-dimensional sampling points corresponding to the any video frame comprises: performing first sampling on the plurality of direction vectors to obtain a plurality of candidate three-dimensional sampling points; determining a key region corresponding to a hand or an object according to the plurality of candidate sampling points; performing second sampling on the plurality of direction vectors in the key region to obtain a plurality of target three-dimensional sampling points.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method of any one of claims 1-8.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the method of any one of claims 1-8.

Images