Human body grid reconstruction method and system based on complementary learning

Abstract

The invention discloses a human body grid reconstruction method and system based on complementary learning, and the method comprises the steps: firstly extracting representation features with good generalization ability, strong discrimination and rich semantics through a large-scale visual model (LVM), and then designing a cross-modal complementary learning strategy to promote the bidirectional information interaction between two modals, then, a feature selection mechanism based on cross attention is introduced, features of different modes are adaptively fused, focusing is carried out on an area with more information amount, dense optical flow consistency loss is further provided, the loss function enforces the optical flow generated by the event flow to be consistent with the movement of the vertex of the grid, and the consistency of the optical flow generated by the event flow is improved. Therefore, the alignment effect of the dynamic movement of the human body in the time dimension is enhanced. A large number of experiments on two mainstream reference data sets MMHPSD and EPMD show that the performance of the EvGHMR is obviously superior to that of an existing latest method, the maximum improvement can reach 37.9%, and the effectiveness and advancement of the method are fully verified.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a human mesh reconstruction method and system based on complementary learning. BACKGROUND

[0002] Human Mesh Reconstruction (HMR) aims to estimate fine 3D human poses and shapes, and is becoming increasingly important, widely applied in virtual reality (VR), augmented reality (AR), robotics and human-computer interaction, etc. Traditional human 3D reconstruction methods mainly rely on images captured by RGB or RGB-D cameras, but such methods have inherent limitations, such as being susceptible to occlusion, sensitive to lighting conditions, and usually having high power consumption and memory consumption. In recent years, the advent of event cameras has brought new possibilities for HMR. Such cameras use a biomimetic design to detect brightness changes at the pixel level in an asynchronous manner, with high temporal resolution, low latency, high dynamic range and low power consumption, making them highly robust in complex scenes such as low light or high-speed motion. In fact, event cameras have shown great potential in a number of vision tasks, such as gesture recognition, gait recognition, 3D reconstruction, image segmentation and autonomous driving, etc. However, the application of event cameras in HMR is still in the exploratory stage.

[0003] Although event data to some extent overcomes the limitations of traditional image methods, it is still challenging to rely solely on event data to reconstruct fine 3D human meshes. This is because events themselves lack significant texture information required for HMR, and usually cannot retain enough texture and edge features. For example, when the human body remains relatively still, the foreground activity is sparse, and the event generation is insufficient, resulting in missing information. Combining asynchronous event streams with traditional image frames for HMR also faces many challenges. First, existing image feature extraction networks (such as ResNet50) are usually optimized for specific data distributions, and have poor generalization ability when facing distribution shifts or weakly supervised scenarios. Second, event streams and image frames differ significantly in data format, temporal distribution and visual information type, and how to design a completely adaptive multi-modal fusion strategy remains an unsolved problem. In addition, although existing methods attempt to constrain motion estimation by aligning optical flow with mesh vertex motion, due to the sparsity of mesh representation itself, the number of vertices is limited, making it difficult to establish a dense correspondence relationship with optical flow, and thus leading to insufficient supervision of motion modeling and poor alignment results. SUMMARY

[0004] To solve the above problems, the present application discloses a human mesh reconstruction method and system based on complementary learning.

[0005] To achieve the above purpose, the technical scheme of the present application is as follows:

[0006] A human body mesh reconstruction method based on complementary learning, comprising the following steps:

[0007] Step one, constructing an EvGHMR model; the EvGHMR model comprises an image frame feature extractor, a complementary learning module, a fusion module, a post backbone network and an SMPL module; the input of the EvGHMR model comprises an event stream I t and a gray image sequence; the event stream I t is a sequence formed by a plurality of event image frames, and the gray image sequence is a sequence formed by a plurality of gray image frames;

[0008] The image frame feature extractor is used to extract features from the gray image sequence and output an image feature representation f c ; the complementary learning module is used to input the image feature representation f c and the event stream I t , learn complementary features from the event stream I t and the image feature representation f c , and output a spliced image feature representation f c and alignment similarity information obtained from the event stream I t and splicing of the event stream and alignment similarity information obtained from the image feature representation

[0009] The fusion module inputs and , performs a splicing operation on and , then enhances the expression ability of the complementary features by using E c , and realizes effective feature fusion based on a cross-attention selection mechanism to output final fusion features F fusion ; E c is a convolutional layer composed of a series of 3*3 convolutional layers and 5*5 convolutional LeakyReLU activation functions;

[0010] The post backbone network inputs F fusion , performs deep feature extraction, and outputs deep features F final ;

[0011]

[0012] wherein PostBackbone represents the post backbone network, and FM represents the fusion module;

[0013] ​The backbone network ResNet-50 (B1, B2, B3, B4) is followed by a single-layer GRU (with 2048 hidden dimensions) as the backbone structure. Building upon this, we introduce early complementary learning and multimodal feature fusion in the B1 stage; therefore, the backbone network comprises B2, B3, B4, and a single-layer GRU.

[0014] The input to the SMPL module is deep feature F. final Then, the SMPL parameters are mapped to output a 3D human body mesh M. t ;

[0015] Step 2: Construct a dataset by collecting human event streams and grayscale image sequences and using SMPL parameters as labels to construct a training dataset.

[0016] Step 3: Train the EvGHMR model using the training dataset to obtain the trained EvGHMR model;

[0017] Step 4: Input the event flow and grayscale image sequence of the human body to be reconstructed into the trained EvGHMR model to obtain the corresponding 3D human body mesh M. t .

[0018] Further improvements include the loss function of the EvGHMR model. as follows:

[0019]

[0020] Ω={(x,y)\middle|||O t (x,y)||2>τ}

[0021]

[0022] Where b(x,y) represents the centroid coordinates of the pixel (x,y) with x-coordinate and y-coordinate within the corresponding mesh triangle. The corresponding mesh triangle is obtained as follows: for the 3D human body mesh M... t All grid triangles, using The algorithm calculates the intersection of the ray originating from the camera's optical center with the grid triangle, and takes the intersection point closest to the camera's optical center as the corresponding position of the pixel in three-dimensional space. The grid triangle containing the intersection point closest to the camera's optical center is the grid triangle corresponding to the pixel (x,y). Indicates the coordinates of the j-th centroid; Indicates the coordinates of the first centroid. Indicates the coordinates of the second centroid. Indicates the coordinates of the third barycenter; Ω represents the effective region; ||2 represents the L2 norm, O t(x, y) represents the optical flow estimated from the event stream, (x, y) represents the dense optical flow derived from the mesh, (x, y) represents the t-th mesh triangle vertex in the input sequence, and π() represents the projection equation.

[0023] Further improvement, the image frame feature extractor is a large visual model LVM; the post backbone network includes B2, B3, B4 of ResNet-50 and is followed by a single-layer GRU.

[0024] Further improvement, the data processing method of the complementary learning module is as follows:

[0025] Firstly, the gray image feature F g and the event image feature F e are extracted.

[0026]

[0027] E ap is a linear convolution layer with a kernel size of 1x1, an input channel number of 1536, and an output channel number of 16, which plays a role of dimension reduction compression; E mo represents a linear convolution layer with a kernel size of 1x1, represents a resnet50 standard fast corresponding to f c whose parameters are not shared; represents a resnet50 standard fast corresponding to I t whose parameters are not shared;

[0028] Then, in each spatial position (h, w), a square local region with a side length of 2r+1 is extracted from the gray image feature F g with (h, w) as the center, which is used for similarity calculation with the corresponding event feature F e :

[0029]

[0030] where K=(2r+1) 2 represents the total number of pixels contained in the square local region extracted with (h, w) as the center, represents the k-th neighborhood feature vector in the region, k∈[1, K]; represents a real number, and C represents a dimension; P (h,w) represents a square local region with a side length of 2r+1 with (h, w) as the center.

[0031] The cost vector is calculated for representing the similarity between the event feature and its corresponding gray image local patch P (h,w) .

[0032]

[0033] F e for event feature

[0034] define the backward cost vector

[0035]

[0036] F e from local region in F

[0037] stack the cost vector over the whole image, get

[0038]

[0039] C g→e denote - stacked cost vector, C e→g denote stacked cost vector, denote a real set with dimension

[0040] KxHxW, H denotes height, W denotes width

[0041] concatenate the cost volume with the corresponding modality feature in channel dimension:

[0042]

[0043] where Γ() denotes the concatenation operation.

[0044] Further improvement, the data processing method of the fusion module is as follows:

[0045]

[0046] E c composed of a series of convolutional layers and LeakyReLU activation functions; Softmax denotes a normalization function, Conv 1×1,3×3,1×1 denotes a convolutional layer; ⊙ denotes element-wise multiplication, F j denotes the feature of the jth input modality; d denotes the number of input modalities, score j denotes the score of the jth input feature.

[0047] Further improvement, the data processing method of the SMPL module is as follows: F final mapped to SMPL parameters by a multi-layer perceptron (MLP), denoted as [theta]t ,β t ,d t ]=FC(F final ), FC() represents a multi-layer perception machine; after obtaining the estimated pose θ t , shape β t , and global translation d t , a three-dimensional human mesh M t is obtained by a fixed SMPL regressor:

[0048] M t =(V t ,J 3D,t )=SMPL(θ t ,β t )+d t (2)

[0049] wherein, represents a triangular mesh vertex, represents a three-dimensional joint position.

[0050] An event stream and frame complementary learning-based human mesh reconstruction system is used to run the above-mentioned complementary learning-based human mesh reconstruction method The advantages of the present application are:

[0051] The present application first extracts a representation feature with good generalization ability, strong discrimination and rich semantics through a large visual model (LVM) to improve the robustness and adaptability of the model. In order to fully exploit the complementary characteristics between event streams and image frames, we design a cross-modal complementary learning strategy to promote bidirectional information interaction between the two modalities. The key module is the cross-modal cost volume (CCV), which explicitly encodes the correlation between modalities and effectively fuses spatial and temporal cues. On this basis, we introduce a cross-attention-based feature selection mechanism that can adaptively fuse features from different modalities and focus on areas with more information. In addition, we propose a dense optical flow consistency loss that forces the optical flow generated by the event stream to be consistent with the motion of the mesh vertices, thereby enhancing the alignment effect of human dynamic motion in the time dimension. Extensive experiments on two mainstream benchmark datasets (MMHPSD and EPMD) show that EvGHMR significantly outperforms the existing state-of-the-art methods, with a maximum improvement of 37.9%, fully verifying the effectiveness and advancement of our method. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1is the overall framework of the EvGHMR of the present application. It consists of four modules: a complementary learning module, a feature fusion module, a subsequent backbone network module, and an SMPL regression module. The complementary learning module first enhances the discriminativeness and generalization ability of frame features through LVM, and then uses the CCV model to realize cross-modal interaction and explicitly model the relationship between frames and events; the feature fusion module uses a selection mechanism based on cross-attention to realize effective fusion of frame-event features; the subsequent backbone network module uses a deep convolutional neural network to extract vectorized feature representation, and then captures the spatiotemporal consistency of human motion through RNN; the SMPL module finally predicts the human SMPL mesh model. It should be noted that B1, B2, B3, and B4 represent standard residual blocks in ResNet-50, where and have the same structure but different parameters for processing different modal data inputs (frames and events).

[0053] Figure 2 is a qualitative result display figure under the whole person setting of the EPMD dataset. The bottom two rows on the right show the reconstruction results under low light conditions, and the rest are the results under normal light conditions. Whether in bright scenes or low light scenes, the method of the present application is superior to the current state-of-the-art method EPM, showing stronger robustness and reconstruction accuracy.

[0054] Figure 3 is the qualitative result of the present application on the MMHPSD dataset as follows: the upper row is the reconstruction effect under the whole person setting, and the lower row is the result under the cross-person setting. The method EvGHMR of the present application is superior to EPM. DETAILED DESCRIPTION

[0055] The present application is further described below in conjunction with the accompanying drawings and examples.

[0056] Example 1

[0057] We propose the EvGHMR framework for realizing high-precision human three-dimensional mesh reconstruction based on event stream and image frame complementary learning. The input of the framework consists of an event stream and a grayscale image sequence. Given a grayscale image sequence G t and an event stream sequence ε t , the EvGHMR outputs the corresponding SMPL mesh sequence M t . The entire framework consists of four main modules and dense optical flow consistency loss. The main modules include: a complementary learning module (CLM), a fusion module (FM), a post-backbone network (PostBackbone), and an SMPL module.

[0058] (1) Image frame feature extractor based on large-scale visual model (LVM)

[0059] For grayscale image sequences, we adopt DINOv2 to extract features. DINOv2 can generate high-quality dense representations, especially suitable for pixel-level tasks, meeting the high accuracy requirements of human body three-dimensional reconstruction (HMR). In addition, DINOv2 performs well in multiple downstream tasks, demonstrating excellent generalization ability. It is worth noting that although DINOv2 is mainly pre-trained on RGB images, due to its self-supervised learning mechanism, it can still extract effective features with semantic meaning even when faced with grayscale images. We extract four feature maps from different stages of the ViT backbone network of DINOv2, capturing semantic information from shallow to deep layers. These feature maps are then concatenated in the channel dimension to form a unified image feature representation, denoted as f c , providing semantic support for subsequent multi-modal fusion and human mesh estimation.

[0060] (2) Complementary Learning Module (CLM)

[0061] This module aims to learn complementary features from event frames and grayscale image frames. Although f c can effectively integrate multi-scale semantic information, if directly used for human three-dimensional reconstruction tasks, it will bring a large GPU memory overhead. Therefore, we introduce a projection module E ap , which maps high-dimensional features f c to a more compact representation f ap along the channel dimension through linear transformation, reducing computational cost and facilitating subsequent network processing. Specifically, E ap is a linear convolution layer with a kernel size of 1x1, an input channel number of 1536, and an output channel number of 16, which serves to reduce dimension and compression. For event frames I t , we use E mo to adjust the channel dimension to achieve alignment and fusion with grayscale image features. Unlike E ap , the input channel of E mo is consistent with the original event frame I t , focusing on the standardization of event modality features. Then, the features of the two modalities are input into the standard block B1, respectively, to obtain the final representation where F g represents the grayscale image feature, and F e represents the event image feature, providing basic feature support for subsequent cross-modal fusion.

[0062]

[0063] Cost volume is constructed by computing the similarity or difference between corresponding pixels in image pairs. In the task of optical flow estimation, cost volume is used to measure the matching cost of pixels during cross-frame motion. Inspired by this, we propose a bidirectional interaction mechanism to align event data and image frame data in both temporal and spatial dimensions by computing local similarity through cross cost volume (CCV). This mechanism effectively enhances the feature fusion process: event frames have high temporal resolution and can capture subtle dynamic changes; while grayscale image frames contain rich spatial texture information, which helps to build semantic context. Specifically, at each spatial location (h, w), we extract a square local region with side length 2r+1 centered at the point from the grayscale image feature F g for similarity computation with the corresponding event feature.

[0064]

[0065] where K = (2r+1) 2 represents the total number of pixels contained in the square local region extracted centered at (h, w), represents the k-th neighborhood feature vector in this region. Next, we compute a cost value vector to represent the similarity between the event feature and its corresponding grayscale image local patch P (h,w) The specific calculation formula is as follows:

[0066]

[0067] Similarly, we define the reverse cost volume:

[0068]

[0069] where, is the vector from the local region in F e Stacking the cost vectors on the entire image, we get:

[0070]

[0071] Finally, we concatenate the cost volume with the corresponding modal feature in the channel dimension.

[0072]

[0073] where Γ() represents the concatenation operation.

[0074] (3) Fusion module (FM)

[0075] This module first uses E cThe expression ability of complementary features is enhanced. Then, an effective feature fusion is realized through a cross-attention-based selection mechanism. The fusion weight is learned through a convolution layer followed by Softmax normalization, so as to realize dynamic weighting of cross-modal feature contribution. The calculation process is formally represented as:

[0076]

[0077] Wherein, E c is composed of a series of convolution layers and LeakyReLU activation functions. The final fusion feature F fusion is represented as:

[0078]

[0079] Wherein, represents element-wise multiplication, and F' j represents the jth input feature.

[0080] (4) PostBackbone

[0081] The application adopts ResNet-50 (B1, B2, B3, B4) followed by a single-layer GRU (hidden dimension is 2048) as the backbone structure. On this basis, we introduce early complementary learning and multi-modal feature fusion in the B1 stage, so the post-backbone network includes B2, B3, B4 and a single-layer GRU.

[0082] (5) SMPL module

[0083] The SMPL module as a core regression component aims to infer the SMPL mesh model from F final . Specifically, F final is mapped to SMPL parameters through a multi-layer perceptron (MLP), represented as [theta t , beta t , d t ] = FC(F final ). After obtaining the estimated pose theta t , shape beta t and global translation d t , a fixed SMPL regressor can obtain the three-dimensional human mesh M t = (V t , J 3D,t ), and the calculation method is as follows:

[0084] M t = SMPL(theta t , beta t ) + d t (11)

[0085] Wherein, represents the triangle mesh vertex, represents the 3D joint position. To maintain the structure consistency and generalization ability of the pre-trained SMPL model, its parameters are kept frozen during training and inference.

[0086] (6) Dense optical flow consistency loss

[0087] Previous works introduce two kinds of flow consistency losses: image-based optical flow and mesh-based flow (human vertex motion). Both of the flows are derived from human motion. The flow consistency loss is used to constrain the consistency between the 2D optical flow and the 3D vertex flow, which helps to regularize the motion estimation task. The mesh flow is obtained by projecting two consecutive human meshes onto the image plane and calculating the motion of the corresponding vertices. However, relying only on the displacement of the corresponding vertices cannot obtain a dense mesh flow. To solve this problem, we propose a new dense flow consistency loss to enhance the consistency of the flow. For a pixel point (x, y) in the image, we find the corresponding mesh triangle by calculating the first intersection point of the ray from the camera optical center and the human mesh. Then, the barycentric coordinates of the pixel in the triangle are calculated

[0088] Then, the dense mesh flow can be calculated by the weighted sum of the vertex displacement, where the weight is the corresponding barycentric coordinate, which is specifically expressed as:

[0089]

[0090] To suppress the noise supervision from static or low motion areas, we limit the loss calculation to pixels with sufficient motion amplitude. Specifically, we define the effective area as:

[0091] Ω = {(x, y) \middle || | | O t (x, y) ||2> τ} (13)

[0092] where $\tau$ is a pre-set threshold, and $\mathbf{O}_t(x, y)$ represents the optical flow inferred from the event stream. Then, the dense flow consistency loss is defined as follows:

[0093]

[0094] We adopt five evaluation metrics to measure the accuracy of human mesh reconstruction (HMR): mean per joint position error (MPJPE), MPJPE after Procrustes alignment (PA-MPJPE), MPJPE after pelvis alignment (PEL-MPJPE), keypoint correct rate (PCKh@0.5) and per-vertex error (PVE). Specifically, PA-MPJPE measures the Euclidean distance between predicted joints and ground truth joints after Procrustes alignment (i.e., rigid alignment by scaling, rotation and translation); PEL-MPJPE only uses the translation information of the pelvis joints for alignment; PCKh@0.5 means that if the error of the predicted joint position is less than 50% of the skull length after pelvis alignment, the predicted point is correct; and PVE represents the average Euclidean distance between the corresponding vertices of the predicted SMPL mesh and the real mesh.

[0095] Table 1 summarizes the experimental data of EventHPE, EPM and the present application on the datasets MMHPSD and EPMD. From the experimental results, the embodiments achieve the best results so far.

[0096] Table 1 quantitative comparison results

[0097]

[0098]

[0099] Compared with the current most advanced method, the performance of the present application can be improved by up to 37.9%.

[0100] Although the embodiments of the present application have been disclosed as above, they are not limited to the use listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and therefore the present application is not limited to the specific details and examples shown herein without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A human mesh reconstruction method based on complementary learning, characterized in that, Comprising the following steps: Step one, constructing an EvGHMR model; the EvGHMR model comprises an image frame feature extractor, a complementary learning module, a fusion module, a post backbone network and an SMPL module; the input of the EvGHMR model comprises an event stream I t and a grayscale image sequence; the event stream I t is a sequence formed by a plurality of event image frames, and the grayscale image sequence is a sequence formed by a plurality of grayscale image frames; The image frame feature extractor is configured to extract features from the sequence of grayscale images and output an image feature representation f c The complementary learning module is configured to input the image feature representation f c and the event stream I t , learn complementary features from the event stream I t and the image feature representation f c , and output an image feature representation f c with the stitching of the alignment similarity information obtained from the event stream I t and the event stream with the stitching of the alignment similarity information obtained from the image feature representation ​ Fusion module input and First, the splicing operation is performed, and then the expression ability of the complementary features is enhanced and E c , and effective feature fusion is realized based on the selection mechanism of cross attention, and the final fusion feature F fusion is output; E c is a convolutional layer composed of a series of 3*3 convolutional layers and 5*5 convolutional LeakyReLU activation functions; The back-bone network inputs F fusion , carries out deep extraction feature extraction, and outputs deep features F final ; Where PostBone represents the posterior backbone network, FM represents the fusion module; The posterior backbone network ResNet-50 (B1, B2, B3, B4) is followed by a single-layer GRU (hidden dimension 2048) as the backbone structure. On this basis, we introduce early complementary learning and multi-modal feature fusion at the B1 stage, so the posterior backbone network contains B2, B3, B4 and a single-layer GRU The SMPL module input is a deep feature F final The SMPL parameters are then mapped, outputting a three-dimensional human mesh M t ; Step two, build the dataset, collect the event stream and grayscale image sequence of the human body and take the SMPL parameter as the label to build the training dataset; Step three, train the EvGHMR model with the training dataset to obtain the trained EvGHMR model; Step four, input the event stream and grayscale image sequence of the human body mesh to be reconstructed into the trained EvGHMR model, and obtain the corresponding three-dimensional human body mesh M t .

2. The human mesh reconstruction method based on complementary learning of claim 1, wherein, The loss function of the EvGHMR model As follows: Ω = {(x, y) \ middle || | O t (x, y) || 2 > τ} where b(x, y) is the barycentric coordinate of the pixel point (x, y) with horizontal coordinate x and vertical coordinate y, and the corresponding mesh triangle is obtained by the following method: all mesh triangles of the three-dimensional human mesh M t The intersection of the ray from the camera optical center and the mesh triangle is calculated by the algorithm, and the closest intersection point to the camera optical center is taken as the corresponding position of the pixel in the three-dimensional space. The mesh triangle where the closest intersection point to the camera optical center is located is the mesh triangle corresponding to the pixel point (x, y); denotes the jth barycentric coordinate; denotes the first barycentric coordinate, denotes the second barycentric coordinate, b v3 denotes the third barycentric coordinate; Ω denotes the effective area; |||2 denotes the L2 norm, O t (x, y) represents the optical flow estimated from the event stream, denotes the dense optical flow derived from the mesh, denotes the tth mesh triangle vertex in the input sequence, and π() denotes the projection equation.​ 3. The human mesh reconstruction method based on complementary learning of claim 1, wherein, The image frame feature extractor is a large visual model LVM; the posterior backbone network includes B2, B3, B4 of ResNet-50 and is followed by a single-layer GRU.

4. The human mesh reconstruction method based on complementary learning of claim 1, wherein, The data processing method of the complementary learning module is as follows: Firstly, the gray-scale image feature F is extracted g and the event image feature F e : E ap is a linear convolution layer with a kernel size of 1x1, an input channel number of 1536, and an output channel number of 16, which plays a role of dimension reduction compression;E mo represents a linear convolution layer, and the convolution kernel size is 1x1, represents a resnet50 standard fast corresponding to f c whose parameters are not shared; represents a resnet50 standard fast corresponding to I t whose parameters are not shared; Then at each spatial location (h, w), a square local region with side length 2r + 1 centered at (h, w) is extracted from the grayscale feature F g for similarity computation with the corresponding event feature F e ​ where K = (2r + 1) 2 denotes the total number of pixels contained in the square local region extracted with (h, w) as the center, denotes the kth neighborhood feature vector in the region, k e [1, K]; denotes a real number, C denotes the dimension; P (h,w) denotes a square local region with (h, w) as the center and 2r + 1 as the side length; Computational cost vector For representing the similarity between the event feature and its corresponding gray scale image local patch P (h,w) between the event feature and its corresponding gray scale image local patch P For event feature F e Value at region (h,w) defining a reverse cost vector represents the F e vector from the local region Stack the cost vectors on the whole image to obtain: C g→e denotes - a cost vector of the superposition, C e→g denotes a cost vector of the superposition, denotes a set of real numbers of dimension K x H x W, H denoting height and W denoting width; The cost volume is spliced with the features of the corresponding modal in the channel dimension: Where Γ() represents the splicing operation.

5. The human mesh reconstruction method based on complementary learning of claim 1, wherein, The data processing method of the fusion module is as follows: E c consists of a series of convolutional layers and LeakyReLU activation functions; Softmax denotes a normalization function, Conv 1×1,3×3,1×1 denotes a convolutional layer; denotes element-wise multiplication, F' j denotes the feature of the jth input modality; d denotes the number of input modalities, score j denotes the score of the jth input feature.

6. The human mesh reconstruction method based on complementary learning of claim 1, wherein, The data processing method of the SMPL module is as follows: F final The SMPL parameters are mapped by a multi-layer perception machine MLP, denoted as [theta t , beta t , d t ] = FC(F final ), FC() represents a multi-layer perception machine; after obtaining the estimated pose theta t , shape beta t and global translation d t , a three-dimensional human body mesh M t is obtained by a fixed SMPL regressor. M t = (V t , J 3D,t ) = SMPL(θ t , β t ) + d t (1) where, denotes a triangle mesh vertex, denotes a 3D joint position.

7. A human mesh reconstruction system based on event stream and frame complementary learning, characterized in that, The human body mesh reconstruction system based on event stream and frame complementary learning is used to run the human body mesh reconstruction method based on complementary learning according to any one of claims 1-6.

Images