Human body three-dimensional reconstruction method based on multi-modal feature fusion

Abstract

The invention provides a human body three-dimensional reconstruction method based on multi-modal feature fusion. The method comprises the following steps: taking a monocular RGB image of a scene character object and a 3D grid vertex label as an original training data set; mask shielding damage processing is carried out on the image and the vertex mark of the 3D grid; inputting the processed data into a multi-mode self-coding pre-training framework to carry out feature fusion to train a network model; downstream end-to-end fine tuning training is carried out on the pre-trained model, the weight of the model is saved, and human body grid three-dimensional reconstruction is carried out on the monocular image. According to the method, complementation among different data features is formed through a multi-modal pre-training complementation task, by utilizing information differences of monocular images of different modals and 3D grid vertexes in feature dimensions and differences between 2D images and 3D labels, so that a human body three-dimensional reconstruction task under the condition that a natural scene has a complex background and is shielded is completed.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, three-dimensional reconstruction, etc., and specifically relates to a three-dimensional reconstruction method of a human body based on multimodal feature fusion. Background Art

[0002] Reconstructing 3D human pose and shape from monocular images is an important research topic in computer vision, with wide applications in human behavior analysis, human-computer interaction, and the metaverse. However, the inherent ambiguity of 2D images to 3D space and the complex interactions between humans and the environment pose significant challenges.

[0003] With the continuous development of deep learning technology and the emergence of a large number of available human body datasets, deep learning-based reconstruction methods have achieved certain success by learning geometric features from given images and corresponding 3D annotations of human body datasets, effectively improving the accuracy of human body reconstruction and having good efficiency and performance.

[0004] However, there are still limitations in real scenes. Due to the complex outdoor environment, people in the image are often occluded or self-occluded. In addition, the existing methods of obtaining real and reliable three-dimensional data usually require complex and expensive multi-view camera arrays, motion capture systems and other equipment, which makes it difficult to obtain large quantities, and most of the existing three-dimensional annotations are inaccurate pseudo-labels generated by simulation software. These limitations have an adverse impact on data-driven methods for human three-dimensional posture and shape reconstruction, limiting the effective learning of human geometric features and resulting in a decline in the performance of existing deep learning-based methods. Summary of the invention

[0005] In view of this, the present invention proposes a method for 3D reconstruction of human body based on multimodal feature fusion. Aiming at the complex environmental background and occlusion in natural scenes, the present invention makes full use of the difference between 2D images and 3D annotations through a multimodal pre-training completion task, forms the complementarity between different data features, improves the generalization ability of the model, and completes the task of 3D reconstruction of human body in natural scenes with complex background and occlusion. The present invention shows excellent performance in a variety of common complex scenes including indoor and outdoor data, proving that the designed network improves the detection efficiency, avoids the situation that the existing large number of human body data sets are not labeled realistically and accurately or even lacking, and can be used in the task of 3D reconstruction of human body in human scenes with partial occlusion.

[0006] It obtains monocular RGB images and 3D mesh vertex annotations of scene characters as original training data sets; performs mask occlusion destruction processing on the images and 3D mesh vertex annotations; constructs a multimodal feature fusion autoencoder network; inputs the processed data into a multimodal autoencoder pre-training framework to perform feature fusion training network model; performs downstream end-to-end fine-tuning training on the pre-trained model and saves the model weights; and performs three-dimensional reconstruction of the human body mesh on the monocular image during the test phase. The present invention effectively utilizes the information difference between monocular images and 3D mesh vertices of different modalities in the feature dimension, and can achieve reconstruction of a complete 3D human body model from only a monocular RGB image even if there is partial occlusion in a natural scene, thereby improving the reconstruction accuracy and robustness of the model, and has broad application prospects and value.

[0007] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0008] A method for 3D reconstruction of the human body based on multimodal feature fusion: the monocular RGB image and 3D mesh vertex annotation of the scene character object are used as the original training data set; the image and 3D mesh vertex annotation are processed by mask occlusion destruction; the processed data is input into the multimodal autoencoder pre-training framework for feature fusion training network model; the pre-trained model is subjected to downstream end-to-end fine-tuning training and the model weights are saved, and then the human body mesh 3D reconstruction is performed on the monocular image.

[0009] It completes the task through multimodal pre-training, utilizes the information differences in feature dimensions between monocular images of different modalities and 3D mesh vertices, and forms complementarity between different data features through the differences between 2D images and 3D annotations, so as to complete the task of three-dimensional reconstruction of the human body in natural scenes with complex backgrounds and occlusions.

[0010] Preferably, the data sets used in this solution are four human data sets: Human3.6M, UP-3D, 3DPW, and MPI-INF-3DHP, which include a variety of human activity scenes both indoors and outdoors.

[0011] Furthermore, the performing mask occlusion destruction processing on the image and the 3D mesh vertex annotations is specifically: performing mask destruction processing on the 2D image and the 3D mesh vertex annotation data using a binarization method.

[0012] Furthermore, the processing of the data set also includes data enhancement processing.

[0013] Furthermore, the multimodal autoencoder pre-training framework includes a multimodal feature fusion autoencoder network, which uses ResNet-50 as an image encoder to extract local features, removes the last fully connected layer, and retains a typical feature vector with a final output dimension of 2048; in order to capture the non-local interactions between 3D mesh vertices and human joints, for the mesh branch, an encoder based on the transformer architecture is used as a mesh feature extractor; in the bottleneck layer, the two sets of feature maps are spliced ​​and fused to form a multimodal feature map containing 2D images and 3D mesh vertices for subsequent training and reasoning.

[0014] In this scheme, the image encoding based on the ResNet-50 architecture removes the last fully connected layer, and the entire network structure retains the convolutional layer, residual block and global average pooling layer. The output of the network is the feature vector generated by the global average pooling layer.

[0015] Furthermore, the self-encoding and decoding method is used to pre-train the multimodal self-encoding pre-training framework model, and the model is divided into two branches: self-encoding and decoding stages:

[0016] In the first stage, the processed data is input into the multimodal feature fusion autoencoder network, image features and mesh vertex features are output, and splicing and fusion are performed to obtain a multimodal fusion feature map embedded with global three-dimensional geometric prior information and local two-dimensional distribution of the human body;

[0017] In the second stage, the fused feature maps are input into the 2D image and 3D mesh decoders of the corresponding branches for decoding and reconstruction, and the original complete image and mesh vertices are restored in a self-supervised way. After the pre-training is completed, the optimal network parameters of the image and mesh encoders are retained for downstream end-to-end network fine-tuning training. The network objective function optimized during the pre-training process is as follows:

[0018]

[0019] Among them, θ p represents the pre-trained network parameters, represents the self-supervised loss of image reconstruction for pixel values, Represents the human body shape mesh vertex loss; The specific expressions are as follows:

[0020]

[0021] in, represents the predicted image after decoding of the image branch, H and W represent the length and width of the image size respectively, C represents the number of RGB image channels, T is the total number of images, and the predicted image X is limited by minimizing the square error of all pixels. r The pixel-level difference from the original image X; The specific expressions are as follows:

[0022]

[0023] in, Represents the predicted human mesh vertex coordinates after mesh branch decoding, N represents the number of mesh vertices, and ||·||1 represents the L1 norm. By applying L1 loss between the prediction and the true value annotation of each vertex, the difference in vertex position examples is constrained to optimize the generation of human shape mesh vertices.

[0024] Furthermore, the end-to-end human 3D reconstruction network fine-tuning training process specifically includes the following steps:

[0025] Input the image data in the original training data set into the pre-trained image encoder, and output the feature vector generated by the global average pooling layer;

[0026] The feature vector f x Attached to the vertices of the standard human SMPL mesh model, the graph convolutional neural network is used for regression prediction to obtain the preliminary human mesh vertex coordinates;

[0027] The obtained initial grid is input into the pre-trained grid encoder and further encoded into a grid feature with the same dimension as the image feature, and fused in the bottleneck layer to form a multimodal fusion feature containing the two-dimensional image of the human body and the three-dimensional geometric information. Then, a multi-layer perceptron MLP is used for regression prediction to generate a fine three-dimensional shape grid of the human body.

[0028] Furthermore, the loss function of the end-to-end training network includes the Euclidean distance loss between mesh vertices and joints and the mesh encoding space latent feature loss, as shown below:

[0029]

[0030] Where, L shape represents the Euclidean distance loss of the human body mesh vertices, and Respectively represent the 3D joint point and 2D projection joint point losses, Represents the hidden feature loss of the grid encoding space; the specific expressions of each loss function are as follows:

[0031]

[0032] in, represents the human body mesh vertex coordinates predicted by the network, N and T represent the number of mesh vertices of each sample and the total number of samples, respectively. represents the mesh feature obtained by the mesh encoder, Y represents the true value of the mesh vertex, and fy represents the true value of the mesh feature obtained by encoding the true value of the mesh vertex, ||·||1 represents the L1 norm; the loss expressions of 3D joint points and 2D projected joint points are as follows:

[0033]

[0034] in, Represents the predicted 3D joint coordinates of the human body, which are obtained by predicting the mesh vertex coordinates through the regression matrix predefined by the human mesh model Calculate and obtain; represents the 2D joint point projected from the predicted 3D joint point to the image space using the camera parameters; j 3D With j 2D They represent the corresponding 3D and 2D joint point annotation true values ​​respectively; K represents the number of joint points.

[0035] Furthermore, the monocular image is input into the optimal parameter network model saved by end-to-end training to predict the human body mesh vertices, and the 3D human body three-dimensional reconstructed mesh model is output.

[0036] Since the method and model of the present invention are generally implemented in the form of a computer program through a computer system, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for three-dimensional reconstruction of a human body based on multimodal feature fusion as described above is implemented.

[0037] And, a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for three-dimensional reconstruction of a human body based on multimodal feature fusion as described above.

[0038] Compared with the prior art, the present invention and its preferred embodiments include at least the following beneficial effects:

[0039] 1. A multimodal fusion learning pre-training framework is proposed to achieve cross-guidance between different types of human body data, form complementarity between different data features, and improve the generalization ability of the model.

[0040] 2. By building an end-to-end human 3D shape mesh estimation network and combining the feature encoding structure with the multimodal fusion compensation method, accurate human 3D mesh reconstruction and posture shape estimation tasks can be achieved even in the presence of occlusion or inaccurate 3D pseudo-labeled training data.

[0041] 3. Evaluation tests were carried out on multiple typical human datasets. Compared with existing mainstream methods, this method demonstrated superior overall performance, especially in complex outdoor scenes.

[0042] 4. The designed network framework can also be applied to human 3D posture and shape reconstruction tasks in other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0044] Figure 1 A flowchart is constructed and implemented for the embodiment of the present invention.

[0045] Figure 2 This is a partial example diagram of a UP-3D human body dataset containing multiple outdoor scenes in step S1 in an embodiment of the present invention.

[0046] Figure 3 It is the multimodal feature fusion autoencoder pre-training network diagram constructed in step S3 in the embodiment of the present invention.

[0047] Figure 4 It is the end-to-end human 3D reconstruction fine-tuning training network diagram constructed in step S5 in the embodiment of the present invention.

[0048] Figure 5 This is a visualization result diagram of reconstruction in which the test image dataset is input into the parameter optimization model for inference in step S6 in the embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0050] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] like Figure 1 As shown, this embodiment provides a scheme construction and implementation process of a human body 3D reconstruction method based on multimodal feature fusion, including the following steps:

[0053] Step S1: Obtain a monocular RGB image of a scene character object and 3D mesh vertex annotations as an original training data set;

[0054] Step S2: performing data enhancement processing such as mask occlusion destruction on the image and 3D mesh vertex annotations;

[0055] Step S3: construct a multimodal feature fusion autoencoder network;

[0056] Step S4: the processed data is input into the multimodal autoencoder pre-training framework to perform feature fusion training on the network model;

[0057] Step S5: Perform downstream end-to-end fine-tuning training on the pre-trained model and save the model weights;

[0058] Step S6: input the test image into the optimal parameter network model to predict the human body mesh vertices to obtain the reconstruction result;

[0059] As a preferred solution of this embodiment, the datasets used in step S1 are Human3.6M, UP-3D, 3DPW, and MPI-INF-3DHP. Human3.6m is a large indoor dataset with 2D and 3D annotations, containing 3.6 million video frames, including different posture activities performed by subjects such as walking, sitting, eating, and smoking. This method uses subjects S1, S5, S6, S7, and S8 for training, and subjects S9 and S11 for test evaluation; UP-3D is an outdoor dataset created by applying the SMPLify algorithm to images of a 2D human posture dataset. This method uses the training set of this dataset for training; 3DPW is a dataset containing outdoor 2D and 3D annotations. This method only uses the test set for test evaluation. MPI-INF-3DHP is a single-person 3D posture dataset collected in a multi-camera green screen studio. This method only uses the test set for test evaluation.

[0060] As a preferred solution of this embodiment, the data processing part in step S2 mainly includes data enhancement processing for image data such as rotation, scaling, random cropping and mirroring, etc. In addition, the 2D image and 3D mesh vertex annotation data are subjected to mask destruction processing using a binarization method. For example, for an RGB image X, a certain percentage of the area is first randomly reserved, and the other areas are covered with a randomized binary mask matrix m to obtain a mask image X. m , which is expressed as follows:

[0061] X m =m⊙X

[0062] In the formula, the X matrix represents the input image, m represents the binary mask matrix of the same dimension as the image, and ⊙ represents matrix multiplication. Similarly, the vertices of the 3D mesh after mask destruction are labeled Y m The expression is as follows:

[0063] Y m =(1-m)⊙Y

[0064] Where the Y matrix represents the mesh vertices, 1 represents the identity matrix of the corresponding dimension, (1-m) represents the binary mask matrix opposite to the image mask area, and ⊙ represents matrix multiplication.

[0065] As a preferred solution of this embodiment, step S3 constructs a multimodal feature fusion autoencoder network for pre-training, and uses ResNet-50 as an image encoder to extract local features. This method removes the last fully connected layer and retains a typical feature vector with a final output dimension of 2048. In order to effectively capture the non-local interactions between 3D mesh vertices and human joints, for the mesh branch, an encoder based on the transformer architecture is selected as the mesh feature extractor. Finally, the two sets of feature maps are spliced ​​and fused in the bottleneck layer to form a multimodal feature map containing 2D images and 3D mesh vertices for subsequent training and reasoning.

[0066] As a preferred solution of this embodiment, in step S4, the model is pre-trained by using a self-encoding and decoding method, which is divided into two branches, namely, self-encoding and decoding. Figure 3 As shown, the specific steps include:

[0067] In the first stage, the data processed in step S2 is input into the constructed autoencoder network, and the image feature f is output. x With the mesh vertex feature f y , splicing and fusion obtain the multimodal fusion feature map f embedded with global three-dimensional geometric prior information and local two-dimensional distribution of the human body c .

[0068] In the second stage, the fused feature maps are input into the 2D image and 3D mesh decoders of the corresponding branches for decoding and reconstruction, and the original complete image and mesh vertices are restored in a self-supervised training manner. After the pre-training is completed, the optimal network parameters of the image and mesh encoders are retained for downstream end-to-end network fine-tuning training. The network objective function optimized during the pre-training process is as follows:

[0069]

[0070] Among them, θ p represents the pre-trained network parameters, represents the self-supervised loss of image reconstruction for pixel values, Represents the vertex loss of human shape mesh. The specific expressions are as follows:

[0071]

[0072] in, represents the predicted image after decoding of the image branch, H and W represent the length and width of the image size respectively, C represents the number of RGB image channels, T is the total number of images, and the predicted image X is limited by minimizing the square error (MSE) of all pixels. r The pixel-level difference between the original image X and The specific expressions are as follows:

[0073]

[0074] in, Represents the predicted human mesh vertex coordinates after mesh branch decoding, N represents the number of mesh vertices, and ||·||1 represents the L1 norm. By applying L1 loss between the prediction and the true value annotation of each vertex, the difference in vertex position examples is constrained to optimize the generation of human shape mesh vertices.

[0075] As a preferred solution of this embodiment, step S5 applies the image encoder and mesh encoder that have learned the human body geometry prior after pre-training to the end-to-end network constructed downstream for fine-tuning training, and finally completes the output of a complete human body 3D mesh model from a monocular image. Figure 4 As shown in FIG. 1 , the end-to-end human 3D reconstruction network fine-tuning training process specifically includes the following steps:

[0076] Step S51: Input the image data obtained in step S1 into the image encoder obtained in step S4 and pre-training, remove the last fully connected layer from the image encoding based on the ResNet-50 architecture, and retain the convolutional layer, residual block and global average pooling layer in the entire network structure. The output of the network is the feature vector generated by the global average pooling layer. This embodiment uses an output f with a dimension of d=2048. x As the global feature vector of human image.

[0077] Step S52: Transform the feature vector f x Attached to the vertices of the standard human SMPL mesh model, a typical graph convolutional neural network (GCN) is used for regression prediction to obtain the preliminary human mesh vertex coordinates Y in , the graph convolution regression prediction calculation formula is as follows:

[0078]

[0079] in, It is an adjacency matrix used to represent the graph structure. Represents the input feature vector of the graph convolutional network, which is composed of image features f x Additional vertex coordinates are obtained, Represents the learnable weight matrix, and N represents the number of vertices.

[0080] Step S53: The obtained initial grid Y in The input is fed into the pre-trained grid encoder and further encoded into a grid feature f with the same dimension as the image feature y , and fused in the bottleneck layer to form a multimodal fusion feature f containing the human body two-dimensional image and three-dimensional geometric information c , and then a multi-layer perceptron (MLP) is used for regression prediction to generate a fine 3D shape mesh of the human body.

[0081] In this step, the loss function of the end-to-end training network is also constructed, including the Euclidean distance loss between mesh vertices and joints and the mesh encoding space hidden feature loss, as shown below:

[0082]

[0083] Where, L shape represents the Euclidean distance loss of the human body mesh vertices, and Respectively represent the 3D joint point and 2D projection joint point losses, Represents the grid encoding space hidden feature loss. The specific expressions of each loss function are as follows:

[0084]

[0085] in, represents the human body mesh vertex coordinates predicted by the network, N and T represent the number of mesh vertices of each sample and the total number of samples, respectively. represents the mesh feature obtained by the mesh encoder, Y represents the true value of the mesh vertex, and f y Represents the true value of the mesh feature obtained by encoding the true value of the mesh vertex. The loss expressions of 3D joint points and 2D projected joint points are as follows:

[0086]

[0087] in, Represents the predicted 3D joint coordinates of the human body, which are obtained by predicting the mesh vertex coordinates through the regression matrix predefined by the human mesh model Obtained by calculation. J represents the 2D joint point projected into the image space by the predicted 3D joint point using the camera parameters. 3D With J 2D They represent the corresponding 3D and 2D joint point annotation true values ​​respectively, and K represents the number of joint points.

[0088] As a preferred solution of this embodiment, step S6: input the tested ordinary monocular image into the optimal parameter network model trained and saved in step S5 to predict the human body mesh vertices, and output a 3D human body three-dimensional reconstruction mesh model, which can also be as follows Figure 5 As shown, it is projected onto the 2D image plane using the camera parameters.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0094] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0095] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of a human body three-dimensional reconstruction method based on multimodal feature fusion under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A method for three-dimensional reconstruction of a human body based on multimodal feature fusion, characterized in that: The monocular RGB images of scene characters and 3D mesh vertex annotations are used as the original training data set; the images and 3D mesh vertex annotations are subjected to mask occlusion destruction processing; The processed data is input into the multimodal autoencoder pre-training framework for feature fusion training of the network model; the pre-trained model is subjected to downstream end-to-end fine-tuning training and the model weights are saved, and then the human body mesh is reconstructed in three dimensions for the monocular image.

2. The method for three-dimensional reconstruction of a human body based on multimodal feature fusion according to claim 1, characterized in that: The mask occlusion destruction processing of the image and the 3D mesh vertex annotation is specifically: using a binarization method to perform mask destruction processing on the 2D image and the 3D mesh vertex annotation data.

3. The method for three-dimensional reconstruction of a human body based on multimodal feature fusion according to claim 1, characterized in that: The processing of the data set also includes data enhancement.

4. The method for 3D reconstruction of a human body based on multimodal feature fusion according to claim 1, characterized in that: The multimodal autoencoder pre-training framework includes a multimodal feature fusion autoencoder network, which uses ResNet-50 as an image encoder to extract local features, removes the last fully connected layer, and retains a typical feature vector with a final output dimension of 2048; in order to capture the non-local interactions between 3D mesh vertices and human joints, for the mesh branch, an encoder based on a transformer architecture is used as a mesh feature extractor; In the bottleneck layer, the two sets of feature maps are concatenated and fused to form a multimodal feature map containing 2D images and 3D mesh vertices for subsequent training and reasoning.

5. The method for 3D reconstruction of a human body based on multimodal feature fusion according to claim 4, characterized in that: The multimodal autoencoding pre-training framework model is pre-trained using the self-encoding and decoding method. The model is divided into two branches: self-encoding and decoding stages: In the first stage, the processed data is input into the multimodal feature fusion autoencoder network, image features and mesh vertex features are output, and splicing and fusion are performed to obtain a multimodal fusion feature map embedded with global three-dimensional geometric prior information and local two-dimensional distribution of the human body; In the second stage, the fused feature maps are input into the 2D image and 3D mesh decoders of the corresponding branches for decoding and reconstruction, and the original complete image and mesh vertices are restored in a self-supervised way. After the pre-training is completed, the optimal network parameters of the image and mesh encoders are retained for downstream end-to-end network fine-tuning training. The network objective function optimized during the pre-training process is as follows: Among them, θ p represents the pre-trained network parameters, represents the self-supervised loss of image reconstruction for pixel values, Represents the human body shape mesh vertex loss; The specific expressions are as follows: in, represents the predicted image after decoding of the image branch, H and W represent the length and width of the image size respectively, C represents the number of RGB image channels, T is the total number of images, and the predicted image X is limited by minimizing the square error of all pixels. r The pixel-level difference from the original image X; The specific expressions are as follows: in, Represents the predicted human mesh vertex coordinates after mesh branch decoding, N represents the number of mesh vertices, and ||·||1 represents the L1 norm. By applying L1 loss between the prediction and the true value annotation of each vertex, the difference in vertex position examples is constrained to optimize the generation of human shape mesh vertices.

6. The method for 3D reconstruction of a human body based on multimodal feature fusion according to claim 1, characterized in that: The end-to-end human 3D reconstruction network fine-tuning training process specifically includes the following steps: Input the image data in the original training data set into the pre-trained image encoder, and output the feature vector generated by the global average pooling layer; The feature vector f x Attached to the vertices of the standard human SMPL mesh model, the graph convolutional neural network is used for regression prediction to obtain the preliminary human mesh vertex coordinates; The obtained initial grid is input into the pre-trained grid encoder and further encoded into a grid feature with the same dimension as the image feature, and fused in the bottleneck layer to form a multimodal fusion feature containing the two-dimensional image of the human body and the three-dimensional geometric information. Then, a multi-layer perceptron MLP is used for regression prediction to generate a fine three-dimensional shape grid of the human body.

7. The method for 3D reconstruction of a human body based on multimodal feature fusion according to claim 6, characterized in that: The loss function of the end-to-end training network includes the Euclidean distance loss between mesh vertices and joint points and the latent feature loss of the mesh encoding space, as shown below: Where, L shape represents the Euclidean distance loss of the human body mesh vertices, and Respectively represent the 3D joint point and 2D projection joint point losses, Represents the hidden feature loss of the grid encoding space; the specific expressions of each loss function are as follows: in, represents the human body mesh vertex coordinates predicted by the network, N and T represent the number of mesh vertices of each sample and the total number of samples, respectively. represents the mesh feature obtained by the mesh encoder, Y represents the true value of the mesh vertex, and f y represents the true value of the mesh feature obtained by encoding the true value of the mesh vertex, ||·||1 represents the L1 norm; the loss expressions of 3D joint points and 2D projected joint points are as follows: in, Represents the predicted 3D joint coordinates of the human body, which are obtained by predicting the mesh vertex coordinates through the regression matrix predefined by the human mesh model Calculate and obtain; represents the 2D joint point projected from the predicted 3D joint point to the image space using the camera parameters; J 3D With J 2D They represent the corresponding 3D and 2D joint point annotation true values ​​respectively; K represents the number of joint points.

8. The method for 3D reconstruction of a human body based on multimodal feature fusion according to claim 1, characterized in that: The monocular image is input into the optimal parameter network model saved by end-to-end training to predict the human body mesh vertices, and the 3D human body three-dimensional reconstructed mesh model is output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for three-dimensional reconstruction of the human body based on multimodal feature fusion as described in any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for three-dimensional reconstruction of a human body based on multimodal feature fusion as described in any one of claims 1 to 8 is implemented.

Images