Three-dimensional hand reconstruction method and device based on multi-axis manbha space-time joint modeling

Abstract

The scheme provides a three-dimensional hand reconstruction method and device for multi-axis manba space-time joint modeling, comprising: obtaining a sample to be reconstructed; inputting the sample to be reconstructed into a pre-trained three-dimensional hand reconstruction model comprising a feature extraction unit, a space-time feature remodeling unit and a reconstruction unit; and performing three-dimensional hand reconstruction. The scheme adds a learnable space-time position coding to the initial space feature, clearly defines the position relationship of the hand movement time sequence frame and the spatial topological position of the hand vertex, and adopts a manba architecture to build a time axis + space axis double scanning module to realize long-range space-time dependence modeling of the hand with linear computational complexity, and adapt to the real-time reconstruction requirements of high-frequency and fine motion of the hand.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a method and apparatus for three-dimensional hand reconstruction using multi-axis Mamba spatiotemporal co-modeling. Background Technology

[0002] 3D hand reconstruction is a core technology for achieving immersive experiences in virtual reality, augmented reality, and natural human-computer interaction. Current mainstream methods rely on parametric models such as MANO to complete the reconstruction, and are mainly divided into two categories: static reconstruction and temporal reconstruction. Static reconstruction only performs pose estimation on a single frame of image. Single-view methods have inherent depth blur and occlusion ambiguity. Although multi-view geometric methods improve the accuracy of static geometry, they lack temporal memory capabilities. When processing video sequences frame by frame, independent errors between frames will continue to accumulate, resulting in non-physical jitter in the reconstruction results and extremely poor temporal continuity.

[0003] To address the issue of temporal jitter, temporal reconstruction methods have gradually become the mainstream research approach. Early temporal reconstruction schemes employed recurrent neural networks such as RNNs and LSTMs, which are prone to gradient instability during training. Furthermore, recurrent structures are inefficient at modeling long-range dependencies and struggle to capture high-frequency, fine-grained hand movements. Recent approaches have shifted to Transformer models, whose self-attention mechanism can efficiently capture global context, but their computational complexity increases quadratically with sequence length, creating a significant computational bottleneck. Even compromises such as window attention sacrifice the model's global receptive field, failing to achieve efficient and complete long-range temporal dependency modeling. Current technologies have consistently failed to achieve a balance between high geometric accuracy, high temporal coherence, and high computational efficiency, making it difficult to meet the demands of long-sequence, real-time 3D hand reconstruction applications. Summary of the Invention

[0004] This application provides a three-dimensional hand reconstruction method based on multi-axis Mamba spatiotemporal joint modeling. This method adds learnable spatiotemporal position codes to the initial spatial features, clarifies the positional relationship of the hand movement time sequence frames and the spatial topological position of the hand vertices, and uses the Mamba architecture to build a dual scanning module of time axis + space axis, so as to realize long-range spatiotemporal dependency modeling of the hand with linear computational complexity, which is suitable for the real-time reconstruction requirements of high-frequency and fine hand movements.

[0005] In a first aspect, embodiments of this application provide a three-dimensional hand reconstruction method based on multi-axis Mamba spatiotemporal co-modeling, the method comprising:

[0006] The sample to be reconstructed is obtained. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain the corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It then projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain the camera view vector corresponding to each camera view. It then fuses the camera view vectors of each camera view to obtain the fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features; the space axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

[0007] Secondly, embodiments of this application provide a three-dimensional hand reconstruction device for multi-axis Mamba spatiotemporal co-modeling, comprising: The acquisition unit is used to acquire the sample to be reconstructed. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain a camera view vector corresponding to each camera view. It fuses the camera view vectors of each camera view to obtain a fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features; the space axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

[0008] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a three-dimensional hand reconstruction method based on multi-axis Mamba spatiotemporal co-modeling.

[0009] The main contributions and innovations of this invention are as follows: This application embodiment acquires multiple 3D vertices of the hand corresponding to each time frame based on global features, and quickly constructs an initial 3D mesh that fits the topology of the human hand based on multi-view geometric constraints, solving the problems of large deviation in the initial 3D structure construction of the hand and inaccurate template matching. This application embodiment generates view vectors by projecting and sampling 3D vertices and adaptively fuses them to achieve accurate association between the 3D vertices of the hand and 2D images from each viewpoint, supplementing the hand observation information from different viewpoints, strengthening the spatial geometric constraints of the hand, and mitigating the reconstruction error caused by hand projection distortion. This application embodiment uses the Mamba architecture to build a dual scanning module of time axis + space axis, realizing long-range spatiotemporal dependency modeling of the hand with linear computational complexity, solving the problems of unstable gradients in traditional RNNs and high computational complexity of Transformers, and adapting to the real-time reconstruction requirements of high-frequency and fine hand movements. This application embodiment generates 3D vertex coordinate deviation correction initial vertices by generating a Transformer reconstruction unit, and makes precise position correction for fine joints of the hand, eliminating coarse errors in the initial reconstruction, making the hand joint angles, fingertip positions, and mesh surfaces more in line with the real hand shape, and meeting the requirements of VR / AR and human-computer interaction for the fineness of hand reconstruction.

[0010] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is an overall flowchart of a three-dimensional hand reconstruction model according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a feature extraction unit according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a spatiotemporal feature reshaping unit according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a reconstruction unit according to an embodiment of this application; Figure 5 This is a structural block diagram of a three-dimensional hand reconstruction device for multi-axis Mamba spatiotemporal co-modeling according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0013] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0014] Example 1 This application provides a method for constructing a 3D hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling. By adding learnable spatiotemporal position codes to the initial spatial features, the method clarifies the positional relationships of hand motion time-series frames and the spatial topological positions of hand vertices. It also employs a Mamba architecture to build a dual-scan module for both the time and spatial axes, achieving long-range spatiotemporal dependency modeling of the hand with linear computational complexity. This method is suitable for real-time reconstruction requirements of high-frequency, fine hand movements. Specifically, refer to... Figure 1 The method includes: Acquire samples to be reconstructed, which consist of consecutive time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained 3D hand reconstruction model. The 3D hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain corresponding multi-scale image features. All multi-scale image features in the corresponding time frame are fused to obtain global features. Based on the global features, multiple 3D vertices of the hand corresponding to each time frame are obtained. The multiple 3D vertices of the hand are projected onto each corresponding 2D hand image to obtain a camera view vector corresponding to each camera view. The camera view vectors of each camera view are fused to obtain a fused view vector. The multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame are stitched together to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features; the space axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

[0015] In this scheme, video images of the same hand are simultaneously acquired from multiple camera perspectives to obtain the sample to be reconstructed. This allows for the simultaneous use of hand geometric information provided by different perspectives, effectively alleviating the inherent depth blur and occlusion ambiguity problems under single-view acquisition, and providing a more complete spatial geometric basis for subsequent reconstruction.

[0016] Specifically, the receiving morphology of the three-dimensional hand reconstruction framework is as follows: The training samples are used as input, and the training samples are transformed into This allows the feature extraction unit to extract features from each two-dimensional hand image in each time frame in parallel, where B is the batch size, used to control the number of samples trained each time, and T is the number of time frames. The number of camera angles, H is the number of feature channels in the 2D hand image, H is the image height of the 2D hand image, and W is the image width of the 2D hand image.

[0017] In the current embodiment, the structure of the feature extraction unit is as follows: Figure 2 As shown, the feature extraction unit uses multiple convolutional feature extractors with shared parameters to extract features from each two-dimensional hand image in each time frame in parallel, and then fuses all the extracted multi-scale image features to obtain global features.

[0018] In the step of obtaining multiple hand 3D vertex coordinates corresponding to each time frame based on global features, a set of 2D keypoint coordinates is predicted based on global features. Based on the set of 2D keypoint coordinates, an initial set of 3D joint coordinates is calculated by direct linear transformation combined with camera intrinsic and extrinsic parameters. The initial set of 3D joint coordinates is then subjected to inverse linear skinning to obtain multiple hand 3D vertices.

[0019] Furthermore, a set of 2D keypoint coordinates is predicted from global features using a UV decoder and soft-maximization integral. Specifically, the global features are fed into a UV coordinate decoder to obtain a heatmap of hand keypoints, and the hand keypoint heatmap is transformed into a set of 2D keypoint coordinates based on the weighted integral of soft-maximization. In this scheme, the set of 2D keypoint coordinates contains a set of 2D hand keypoint coordinates from each camera viewpoint.

[0020] Furthermore, all standard 3D joints are obtained on the standard hand network. Based on the set of 2D key point coordinates, the 2D key point coordinates corresponding to each standard 3D joint are obtained. Based on the corresponding 2D key point coordinates, a homogeneous linear equation is constructed for each standard 3D joint. The homogeneous equation system is solved and combined with the camera intrinsic and extrinsic parameters to obtain the three-dimensional coordinates of each standard 3D joint to form the initial 3D joint coordinate set.

[0021] Furthermore, multiple 3D vertices of the hand are generated based on the initial set of 3D joint coordinates. Specifically, the relative coordinates of the initial 3D joints, with the root joint as a reference, are input into the vertex generation mapping module. A learnable linear mapping is used to establish the correspondence between the joints and the vertices of the hand mesh, thereby outputting multiple 3D vertices of the hand.

[0022] In the current embodiment, multiple 3D vertices of the hand are projected onto each two-dimensional hand image in the corresponding time frame based on the camera's intrinsic and extrinsic parameters to obtain the projection coordinates of each camera viewpoint. Bilinear interpolation mesh sampling is then performed on the corresponding two-dimensional hand image using the projection coordinates to obtain the camera viewpoint vector of the corresponding camera viewpoint.

[0023] Specifically, based on bilinear interpolation grid sampling, the position of the hand vertex in 3D space is matched with the 2D hand image acquired from each camera viewpoint, so that the observation information from different viewpoints can be effectively fused into the 3D vertex features, further supplementing the spatial geometric constraints and improving the accuracy of subsequent reconstruction.

[0024] Furthermore, an adaptive fusion network is used to fuse the camera view vectors of each camera view. For example, the adaptive fusion network used in this scheme is Merge Net.

[0025] In the current embodiment, 3D coordinate data of multiple hand 3D vertices are acquired as explicit geometric position information. The explicit geometric position information of the same time frame, the fused view vector, and global features are concatenated in the channel dimension to obtain the initial spatial features, expressed by the formula:

[0026] in, For initial spatial features, For splicing operations, As a global feature, To fuse view vectors, This is explicit geometric position information.

[0027] In the current embodiment, the structure of the spatiotemporal feature reshaping unit is as follows: Figure 3 As shown, after adding temporal and spatial coding features to each initial spatial feature, it is input into the spatiotemporal feature reshaping unit. The temporal coding feature represents the position of the current temporal frame in the entire temporal sequence; the spatial coding feature represents the spatial position of each hand 3D vertex in the initial spatial feature.

[0028] Specifically, in the step of adding temporal and spatial encoded features to each initial spatial feature, a temporal axis and a spatial axis are constructed for each initial spatial feature, learnable temporal and spatial location codes are initialized, the temporal location codes are broadcast to the temporal axis of each initial spatial feature, and the spatial location codes are broadcast to the spatial axis of each initial spatial feature.

[0029] In the current embodiment, the time axis scanning module constructs time sequence features based on the temporal relationship of the time sequence frames, using the features of the same hand 3D vertex within each initial spatial feature. The time axis scanning module performs forward scanning and reverse scanning on the time sequence features respectively, and fuses the results of forward scanning and reverse scanning to obtain the time scanning result. The time scanning results of all hand 3D vertices are fused to obtain the time enhancement feature.

[0030] Specifically, the temporal relationship of the time-series frames is obtained based on the temporal coding features within each initial spatial feature.

[0031] Specifically, the Mamba architecture captures the dependency relationship of the same 3D hand vertices between different temporal frames through the circular memory mechanism, solving the temporal occlusion problem. It uses historical frame information to complete the features of the current frame, which does not require global attention calculation of the entire frame features as in traditional temporal modeling methods, thus greatly reducing the computational complexity of temporal modeling.

[0032] In the current embodiment, the spatial axis scanning module is based on the spatial relationship of each hand 3D vertex within the initial spatial feature. It constructs a spatial sequence feature using the features of all hand 3D vertices within the same initial spatial feature. The spatial axis scanning module performs forward scanning and reverse scanning on the spatial sequence feature respectively, and fuses the results of forward scanning and reverse scanning to obtain a spatial scanning result. The spatial scanning result of fusing all initial spatial features is used to obtain a spatial enhancement feature.

[0033] Specifically, the spatial relationship of each hand 3D vertex within the initial spatial feature is obtained based on the spatial encoding features within each initial spatial feature.

[0034] Specifically, the structural constraint relationships between different hand 3D vertices within the same temporal frame are captured through the linear complexity feature interaction of the Mamba architecture, which completes the local vertex feature information and strengthens the modeling of the hand topology and long-range dependencies between points.

[0035] In the current embodiment, the time axis scanning module and the spatial axis scanning module are modeled independently, and both adopt the Mamba architecture but do not share parameters.

[0036] Specifically, the time axis scanning module and the spatial axis scanning module are instantiated as independent Mamba modules, which are used for time relationship modeling and spatial relationship modeling, respectively. The scanning method of the Mamba architecture can be implemented using existing technologies.

[0037] Specifically, the temporal enhancement features and spatial enhancement features are added element by element to obtain the refined spatiotemporal features.

[0038] In the current embodiment, the structure of the reconstruction unit is as follows: Figure 4 As shown, the refined spatiotemporal features are expanded into multiple refined sequence features according to the temporal order of the time-series frames. Each refined sequence feature corresponds to one time-series frame. Based on index slices, each refined sequence feature is separated into vertex features and joint features. Specifically, according to the arrangement order of the refined sequence features, the refined sequence features are segmented and sliced ​​based on a preset number of vertices and joints to obtain index slices. The feature slices located at the beginning of the index slice are used as vertex features, and the feature slices located at the end of the index slice are used as joint features. The vertex features are the feature representations of each vertex on the surface of the hand mesh, and the joint features are the feature representations of each joint in the hand skeleton.

[0039] In other words, this scheme constructs index slices based on a preset number of vertices and joints, and divides the refined sequence features into two parts. The feature slices located at the front of the index slices are vertex features, which are responsible for representing each point on the surface of the hand mesh and determining the shape, surface, and fingertip shape of the hand. The feature slices located at the back of the index slices are joint features, which are responsible for representing each joint of the hand skeleton and determining the finger bending angle, bone posture, and joint position.

[0040] Furthermore, the reconstruction unit includes an attention layer, a base weight layer, a motion basis weight layer, a bias layer, and an output layer. In the attention layer, a multi-layer Transformer encoder is used to perform multi-head self-attention interaction on vertex features and keypoint features to obtain attention features. These attention features are then input in parallel into the base weight layer, motion basis weight layer, and bias layer. The base weight layer uses a fully connected layer to map the attention features, generating a first weight matrix. This first weight matrix represents the combined weights of each vertex with multiple motion basis vectors. The motion basis weight layer uses a linear-convolutional layer structure to process the attention features, generating a second weight matrix. This second weight matrix represents multiple learnable 3D motion basis vectors. The bias layer generates a coordinate bias matrix based on the attention features. This coordinate bias matrix is ​​used for residual compensation of vertex coordinates. Matrix calculations are performed based on the first weight matrix, the second weight matrix, and the coordinate bias matrix to obtain the vertex coordinate changes. The output layer outputs the vertex coordinate changes as the corresponding hand 3D vertex coordinate deviations.

[0041] Specifically, the attention layer of this scheme consists of four Transformer encoders. In each Transformer encoder, a multi-head self-attention mechanism is used to enable the interaction between vertex features and keypoint features, thereby enhancing local geometric details.

[0042] Specifically, the base weight layer outputs a weight vector of dimension M through a fully connected layer, and then obtains the first weight matrix through Softmax normalization.

[0043] Specifically, the motion basis weight layer generates M motion basis vectors through a linear-convolutional layer structure, which constitute the second weight matrix.

[0044] Specifically, the bias layer directly regresses the bias matrix based on attention features.

[0045] Specifically, the formula for obtaining the change in vertex coordinates by performing matrix calculations on the first weight matrix, the second weight matrix, and the coordinate bias matrix is ​​expressed as follows:

[0046] in, The change in vertex coordinates. This is the first weight matrix. This is the second weight matrix. This is the bias matrix.

[0047] Specifically, in the output layer, the vertex coordinate changes are mapped to the same resolution as the corresponding multiple hand 3D vertices, thereby obtaining the hand 3D vertex coordinate deviation. The hand 3D vertex coordinate deviation is then used to correct the multiple hand 3D vertices in the corresponding time frame. The formula is expressed as follows:

[0048] in, These are the corrected 3D vertex coordinates for the hand. These are the multiple 3D vertices of the hand before correction. This represents the 3D vertex coordinate deviation of the hand.

[0049] In some embodiments, the three-dimensional hand reconstruction model used in this solution can be trained using any training method, and this solution does not limit the training method.

[0050] Example 2 Based on the same concept, referencing Figure 5 This application also proposes a three-dimensional hand reconstruction device based on multi-axis Mamba spatiotemporal co-modeling, comprising: The acquisition unit is used to acquire the sample to be reconstructed. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain a camera view vector corresponding to each camera view. It fuses the camera view vectors of each camera view to obtain a fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features; the space axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

[0051] Example 3 This embodiment also provides an electronic device, see reference. Figure 6 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0052] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0053] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0054] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0055] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the three-dimensional hand reconstruction methods of multi-axis Mamba spatiotemporal co-modeling in the above embodiments.

[0056] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0057] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0058] The input / output device 408 is used to input or output information. In this embodiment, the input information may be a sample to be reconstructed, and the output information may be the hand reconstruction result, etc.

[0059] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program: The sample to be reconstructed is obtained. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain the corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It then projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain the camera view vector corresponding to each camera view. It then fuses the camera view vectors of each camera view to obtain the fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features; the space axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

[0060] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0061] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0062] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 6 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0063] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A three-dimensional hand reconstruction method based on multi-axis Mamba spatiotemporal co-modeling, characterized in that, Includes the following steps: The sample to be reconstructed is obtained. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain the corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It then projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain the camera view vector corresponding to each camera view. It then fuses the camera view vectors of each camera view to obtain the fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features. The spatial axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

2. The method for constructing a 3D hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, In the step of obtaining multiple hand 3D vertex coordinates corresponding to each time frame based on global features, a set of 2D keypoint coordinates is predicted based on global features. Based on the set of 2D keypoint coordinates, an initial set of 3D joint coordinates is calculated by direct linear transformation combined with camera intrinsic and extrinsic parameters. The initial set of 3D joint coordinates is then subjected to inverse linear skinning to obtain multiple hand 3D vertices.

3. The method for constructing a 3D hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, Based on the camera's intrinsic and extrinsic parameters, multiple 3D vertices of the hand are projected onto each 2D hand image in the corresponding time frame to obtain the projection coordinates of each camera viewpoint. Bilinear interpolation mesh sampling is then performed on the corresponding 2D hand image using the projection coordinates to obtain the camera viewpoint vector of the corresponding camera viewpoint.

4. The method for constructing a three-dimensional hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, After adding temporal and spatial coding features to each initial spatial feature, the features are input into the spatiotemporal feature reshaping unit. The temporal coding feature represents the position of the current temporal frame in the entire temporal sequence, and the spatial coding feature represents the spatial position of each hand 3D vertex in the initial spatial feature.

5. The method for constructing a three-dimensional hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, The time axis scanning module constructs time sequence features based on the temporal relationship of the time sequence frames, using the features of the same hand 3D vertex within each initial spatial feature. The time axis scanning module performs forward scanning and reverse scanning on the time sequence features respectively, and fuses the results of forward scanning and reverse scanning to obtain the time scanning result. The time scanning results of all hand 3D vertices are fused to obtain the time enhancement feature.

6. The method for constructing a three-dimensional hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, The spatial axis scanning module is based on the spatial relationship of each hand 3D vertex within the initial spatial feature. It constructs a spatial sequence feature using the features of all hand 3D vertices within the same initial spatial feature. The spatial axis scanning module performs forward and reverse scanning on the spatial sequence feature respectively, and merges the results of the forward and reverse scanning to obtain the spatial scanning result. The spatial enhancement feature is obtained by fusing the spatial scanning results of all initial spatial features.

7. The method for constructing a three-dimensional hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 1, characterized in that, The refined spatiotemporal features are expanded into multiple refined sequence features according to the temporal order of the time-series frames. Each refined sequence feature corresponds to a time-series frame. Based on index slices, each refined sequence feature is separated into vertex features and joint features. Specifically, according to the arrangement order of the refined sequence features, the refined sequence features are segmented and sliced ​​based on a preset number of vertices and joints to obtain index slices. The feature slices located at the beginning of the index slice are used as vertex features, and the feature slices located at the end of the index slice are used as joint features. The vertex features are the feature representations of each vertex on the surface of the hand mesh, and the joint features are the feature representations of each joint in the hand skeleton.

8. The method for constructing a three-dimensional hand reconstruction model using multi-axis Mamba spatiotemporal co-modeling according to claim 7, characterized in that, The reconstruction unit includes an attention layer, a base weight layer, a motion basis weight layer, a bias layer, and an output layer. In the attention layer, a multi-layer Transformer encoder performs multi-head self-attention interaction on vertex features and keypoint features to obtain attention features. These attention features are then input in parallel into the base weight layer, motion basis weight layer, and bias layer. The base weight layer uses a fully connected layer to map the attention features, generating a first weight matrix. This first weight matrix represents the combined weights of each vertex with multiple motion basis vectors. The motion basis weight layer uses a linear-convolutional layer structure to process the attention features, generating a second weight matrix. This second weight matrix represents multiple learnable 3D motion basis vectors. The bias layer generates a coordinate bias matrix based on the attention features. This coordinate bias matrix is ​​used for residual compensation of vertex coordinates. Matrix calculations are performed based on the first weight matrix, the second weight matrix, and the coordinate bias matrix to obtain the vertex coordinate change. The output layer outputs the vertex coordinate change as the corresponding hand 3D vertex coordinate deviation.

9. A three-dimensional hand reconstruction device using multi-axis Mamba spatiotemporal co-modeling, characterized in that, include: The acquisition unit is used to acquire the sample to be reconstructed. The sample to be reconstructed consists of continuous time frames. Each time frame contains two-dimensional hand images synchronously acquired from multiple different camera perspectives under the same hand pose. The sample to be reconstructed is input into a pre-trained three-dimensional hand reconstruction model. The three-dimensional hand reconstruction model includes a feature extraction unit, a spatiotemporal feature reshaping unit, and a reconstruction unit. The feature extraction unit extracts features from each 2D hand image in each time frame in parallel to obtain corresponding multi-scale image features. It then fuses all multi-scale image features in the corresponding time frame to obtain global features. Based on the global features, it obtains multiple 3D vertices of the hand corresponding to each time frame. It projects the multiple 3D vertices of the hand onto each corresponding 2D hand image to obtain a camera view vector corresponding to each camera view. It fuses the camera view vectors of each camera view to obtain a fused view vector. Finally, it concatenates the multiple 3D vertices of the hand, the fused view vector, and the global features of each time frame to obtain the initial spatial features corresponding to each time frame. The spatiotemporal feature reshaping unit includes a time axis scanning module and a space axis scanning module of the Mamba architecture. The time axis scanning module captures the temporal relationship of each hand 3D vertex in all initial spatial features to obtain temporal enhancement features. The spatial axis scanning module captures the spatial relationship between all hand 3D vertices in each initial spatial feature to obtain spatial enhancement features. The temporal enhancement features and spatial enhancement features are added element by element to obtain refined spatiotemporal features. The reconstruction unit uses the Transformer architecture to generate the hand 3D vertex coordinate deviation corresponding to each time frame based on refined spatiotemporal features. The hand 3D vertex coordinate deviation is used to correct multiple hand 3D vertices in the corresponding time frame to complete the hand reconstruction.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a three-dimensional hand reconstruction method for multi-axis Mamba spatiotemporal co-modeling as described in any one of claims 1-8.

Images