Grid recovery method and system based on feature fusion of vision-language large model

Through the feature fusion method of the vision-language big model, combined with image and text features, the problems of visual occlusion and ambiguity in 3D human mesh recovery are solved, the recovery accuracy and reliability are improved, and are suitable for a variety of application scenarios.

CN119007184APending Publication Date: 2024-11-22HUNAN UNIV

Patent Information

Application Number
CN202411071293.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing 3D human mesh recovery method relies on image visual features to effectively solve the problems of visual occlusion and visual ambiguity, resulting in insufficient recovery accuracy and reliability.

Method used

Using a feature fusion method based on vision-language big model, the feature interaction is performed using a linear transformation network through the combination of image features and text features, and a smooth 3D human mesh sequence is predicted through the adjacent frame aggregation module. .

Benefits of technology

It improves the robustness and prediction accuracy of the model, can restore the 3D human mesh more accurately under occlusion, enhances time consistency and smoothness, and is suitable for application scenarios such as human-computer interaction and motion capture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007184A_ABST
    Figure CN119007184A_ABST
Patent Text Reader

Abstract

The invention discloses a feature fusion grid recovery method and system based on a vision-language large model, and the method comprises the steps: carrying out the preprocessing of an original video sequence, obtaining an image semantic text and an image vector, carrying out the feature extraction of the image vector and the image semantic text, and obtaining a visual feature vector and a text feature vector; splicing the visual feature vector and the text feature vector of the same frame, and performing feature interaction based on a linear transformation network to obtain an initial fusion feature; for any frame, combining front and back frame information and based on the initial fusion feature of each frame in the aggregation network aggregation group, obtaining an inter-group aggregation feature corresponding to each frame; and iteratively updating parameters of the SMPL model by utilizing the inter-group aggregation features of each frame so as to regress a complete 3D human body grid. According to the method, the text features and the visual features are fused, the information of the front frame and the rear frame is effectively combined, and the smooth 3D human body grid sequence is comprehensively predicted for the video data with occlusion or visual ambiguity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D human body mesh restoration based on deep learning methods, and specifically relates to a mesh restoration method and system based on feature fusion of a large vision-language model. Background Art

[0002] 3D human mesh recovery technology is a research direction that has attracted much attention in the field of computer vision. At present, this research content has a wide range of application value in the fields of VR / AR, virtual try-on, fitness video games, etc. Compared with obtaining human body information with the help of traditional sensors, this method also has the characteristic of low cost. However, the current 3D human mesh recovery methods are all based on image data training. However, whether it is a data set or real-life image data, there will be occlusions, and the results are often greatly affected by occlusions.

[0003] At present, the mainstream technology of restoring 3D human body mesh using video information mainly uses deep learning algorithms. By inputting videos, the parameters of human body models such as SMPL (Skinned Multi-Person Linear) or SMPL-X are predicted. By combining the parameters with the human body model, the complete human body mesh and key points can be restored. However, this type of method currently faces the following challenges: 1) The ambiguity of predicting 3D content from 2D data. Since the same 2D joints can be observed from different perspectives, this places higher requirements on the prediction accuracy of the model. 2) Occlusion of video data. Whether it is a dataset or real-life video data, there is often occlusion, and existing methods cannot handle this problem well.

[0004] In response to the above problems, researchers have proposed a large number of technical solutions, which can be divided into two categories: 1) Introducing prior knowledge, including prior knowledge of human structure and human motion, to alleviate the inherent ambiguity problem from 2D to 3D; 2) Designing a highly robust model structure to make full use of spatiotemporal context information to predict the current frame, alleviating the problem of low prediction accuracy caused by image occlusion.

[0005] However, both methods only use visual information to predict the grid, and a single image prediction grid has ambiguity from 2D to 3D, especially in the presence of occlusion. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the current mainstream 3D human mesh recovery method only relies on image visual features and fails to solve visual occlusion and visual ambiguity, resulting in the problem that the accuracy and reliability of 3D human mesh recovery need to be improved. To this end, the technical solution of the present invention proposes a mesh recovery method and system based on feature fusion of a large visual-language model. The method starts from the two levels of image features and text features, and fully explores the temporal, spatial and semantic correlation of features. Compared with the traditional method that does not establish a scene mapping relationship, the technical solution of the present invention takes into account that the semantic information of the scene can be obtained by extracting text through a large model, which can help the model overcome the ambiguity of prediction according to different scenes; it also effectively combines the previous and next frame information through the adjacent frame aggregation module, and comprehensively predicts a smooth 3D human mesh sequence for video data with occlusion or visual ambiguity.

[0007] To this end, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a mesh recovery method based on feature fusion of a visual-language large model, comprising the following steps:

[0009] S1: Preprocessing, obtaining the original video sequence and preprocessing to obtain the image semantic text and image vector corresponding to each frame of video, wherein the image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition bounding box;

[0010] S2: feature extraction, inputting the image vector into the visual feature extractor to obtain the visual feature vector and inputting the image semantic text into the text feature extractor to obtain the text feature vector;

[0011] S3: feature fusion, concatenating the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and performing feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features;

[0012] S4: constructing an aggregation network to aggregate intra-group features. For any frame, several preceding and succeeding frames are divided into a group and initial fusion features of each frame in the group are aggregated based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame.

[0013] S5: iteratively updating the SMPL model parameters using the inter-group aggregation features of each frame, and finally inputting the updated SMPL model parameters into the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set;

[0014] The network model to be optimized in steps S2-S5 is trained through sample collection and true value labels of the samples, and a complete 3D human body mesh is obtained according to the video sequence using the trained network model.

[0015] Further preferably, the inter-group aggregation features of each frame are obtained according to the following process:

[0016] S41: dividing the current frame, and the previous and next n frames into a large group, and then dividing every m frames in the large group into a small group;

[0017] S42: Use the fully connected layer to reduce the dimension of the initial fusion features of each frame, and then perform feature splicing within the group to obtain the spliced ​​features;

[0018] S43: using a fully connected layer to reduce the dimension of the concatenated features, and then inputting the features into a softmax layer to obtain an attention value vector A, wherein the number of elements of the attention value vector A is equal to m;

[0019] S44: taking the elements of the attention value vector A as weights, multiplying them one by one with the initial fusion features in the corresponding groups, and adding the multiplication results as the inter-group aggregation features of the groups;

[0020] Among them, the inter-group aggregation features of each group are calculated according to the above process, and then the m groups are divided into a new group, and its inter-group aggregation features are regarded as the initial fusion features in S42. Aggregation is performed according to the above steps S42-S44, and the above process is repeated until the final inter-group aggregation features are obtained.

[0021] More preferably, in step S41, the current frame, and the previous and next three frames are grouped into a large group, and every three frames are divided into a small group.

[0022] Further preferably, in step S5, the process of iterating the SMPL model parameters to the updated SMPL model parameters using the inter-group aggregation features of each frame is as follows:

[0023] S51: Obtain / set initial SMPL model parameters, wherein the SMPL model parameters include posture, shape and camera parameters;

[0024] S52: Aggregate the inter-group features z t ∈R 2048 Through two linear layers, it regresses to 1024-dimensional features, R is a real number set, z t is the component aggregation feature corresponding to the t-th frame, with a dimension of 2048;

[0025] S53: concatenating the initial SMPL model parameters with the features of step S52;

[0026] S54: Use three different linear layers to regress the concatenated features to predict the difference in posture, shape and camera parameters;

[0027] S55: Add the difference value and the current posture, shape and camera parameters one by one to obtain the SMPL model parameters updated in this round;

[0028] The above steps are iteratively executed until the iteration termination condition is met to obtain updated SMPL model parameters, and each iteration uses the result of the previous iteration as the current SMPL model parameter.

[0029] Further preferably, the interactive process of obtaining the initial fusion feature by the feature interaction based on the linear transformation network is expressed as:

[0030] X=M(X ft )

[0031] M(X ft )=X ft W m

[0032] In the formula, the visual-text features corresponding to the video sequence are f t is the visual-text feature corresponding to the t-th frame of the video, T is the total number of video frames, and W m is the weight matrix, i.e., the linear network parameters to be optimized during training; M(·) is the linear transformation, the initial fusion feature corresponding to the video sequence x t is the initial fusion feature corresponding to the t-th frame video.

[0033] Further preferably, the training process takes minimization of the loss function as the training optimization goal, and the loss function is expressed as:

[0034] L=L 3D +L 2D +L SMPL

[0035] Among them, L is the loss function, L 3D is the 3D joint error, L 2D is the projected 2D joint error, L SMPL is the SMPL model parameter error, as follows:

[0036]

[0037] Where T is the total number of frames in the video sequence, X t and are the predicted 3D pose and true label of the t-th frame, z t and are the predicted 2D pose and true label of the t-th frame, β and are the predicted shape parameters and true labels, θ t and are the predicted pose parameters and true labels of the t-th frame, respectively.

[0038] Further preferably, the visual feature extractor is the ResNet50 module in the HMR model, the text feature extractor is the BERT model, and the image semantic text is obtained using the Blip2 large model.

[0039] In another aspect, the present invention provides a system based on the above-mentioned grid recovery method, comprising:

[0040] A preprocessing module is used to obtain the original video sequence and perform preprocessing to obtain the image semantic text and image vector corresponding to each frame of video, wherein the image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition bounding box;

[0041] A feature extraction module, used for inputting the image vector into a visual feature extractor to obtain a visual feature vector and inputting the image semantic text into a text feature extractor to obtain a text feature vector;

[0042] A feature fusion module is used to concatenate the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and perform feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features;

[0043] An aggregation module, for dividing a plurality of preceding and succeeding frames into a group for any frame and aggregating initial fusion features of each frame in the group based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame;

[0044] The network recovery module is used to iteratively update the SMPL model parameters using the inter-group aggregation features of each frame, and finally input the updated SMPL model parameters into the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set.

[0045] In a third aspect, the present invention provides an electronic terminal, comprising:

[0046] one or more processors;

[0047] a memory storing one or more computer programs;

[0048] The processor calls the computer program to execute:

[0049] Steps of a mesh recovery method based on feature fusion of a vision-language large model.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to execute:

[0051] Steps of a mesh recovery method based on feature fusion of a vision-language large model.

[0052] Beneficial Effects

[0053] Compared with the existing method, the advantages of the present invention are:

[0054] The present invention proposes a mesh recovery method based on feature fusion of a large visual-language model, which can simultaneously utilize the visual information and semantic information of the data, and effectively combine the previous and next frame information through the adjacent frame aggregation module, and comprehensively predict a smooth 3D human mesh sequence for video data with occlusion or visual ambiguity. Among them, for the mesh information and original data under occlusion, the traditional method does not establish a scene mapping relationship, and the present invention takes into account that the semantic information of the scene can be obtained by extracting text through a large model, that is, the image semantic text and the text feature vector, which can help the model overcome the ambiguity of prediction according to different scenes. The technical solution of the present invention deeply fuses the inter-frame visual features and text features, the semantic information provides guidance for the occluded image, and the visual information corrects the wrong semantic understanding, which can improve the robustness of the network to a certain extent. In the inter-frame fusion process, the hierarchical and progressive fusion of the previous and next frame features can further maintain the temporal consistency of the prediction results and improve the smoothness and accuracy of the model prediction results. After the model training is completed, it can be deployed in actual human-computer interaction, motion capture and other application scenarios to obtain the 3D posture and shape information of the human body. The technical solution of the present invention has the characteristics of comprehensiveness and hierarchy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A model structure diagram of a video 3D human body mesh recovery method based on interactive fusion of image features and text features in the present invention;

[0056] Figure 2 Schematic diagram of video frame preprocessing steps;

[0057] Figure 3 Schematic diagram of the extraction of visual features and text features;

[0058] Figure 4 It is the initial fusion process of visual features and text features;

[0059] Figure 5 Schematic diagram of the interactive fusion steps of temporal features;

[0060] Figure 6 Schematic diagram of the application of the video 3D human mesh recovery method based on the interactive fusion of image features and text features. DETAILED DESCRIPTION

[0061] The technical solution of the present invention aims to improve the accuracy and reliability of 3D human body mesh recovery. In particular, it solves the problem that the current mainstream 3D human body mesh recovery method only relies on image visual features and fails to solve the problem of visual occlusion and visual ambiguity. To this end, the technical solution of the present invention, on the one hand, starts from both the image features and text features, fully explores the temporal, spatial and semantic correlation of features, and integrates visual features and text features. On the other hand, it fully explores inter-frame features, effectively combines the previous and next frame information, and then comprehensively predicts a smooth 3D human body mesh sequence for video data with occlusion or visual ambiguity. To this end, the technical solution of the present invention provides a mesh recovery method based on feature fusion of a large visual-language model, comprising the following steps:

[0062] S1: Preprocessing, obtaining the original video sequence and preprocessing it to obtain the image semantic text and image vector corresponding to each frame of video, wherein the image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition bounding box;

[0063] S2: feature extraction, inputting the image vector into the visual feature extractor to obtain the visual feature vector and inputting the image semantic text into the text feature extractor to obtain the text feature vector;

[0064] S3: feature fusion, concatenating the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and performing feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features;

[0065] S4: constructing an aggregation network to perform intra-group aggregation of features. For any frame, several preceding and succeeding frames are divided into a group and initial fusion features of each frame in the group are aggregated based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame.

[0066] S5: Utilizing the inter-group aggregation features of each frame to iterate the SMPL model parameters to obtain updated SMPL model parameters, and finally inputting the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set.

[0067] The present invention will be further described below in conjunction with embodiments, but the protection scope of the present invention is not limited by the specific embodiments.

[0068] Embodiment 1:

[0069] The embodiment of the present invention provides a mesh recovery method based on feature fusion of a visual-language large model, such as Figure 1 As shown, it specifically includes: 1) a preprocessing step of a video sequence; 2) a feature extraction step; 3) a feature interaction fusion / aggregation step; 4) a 3D human body mesh and joint point recovery step; S5) a supervised training step.

[0070] Among them, the purpose of the preprocessing step of 1) video sequence is to identify the human body in the original video frame and track and crop it, and generate the semantic understanding text corresponding to the video frame, that is, to process the original video sequence containing T frames. I t is the video data of the tth frame; the output is the image semantic text describing the picture And the preprocessed image vector of the tracked human body, representing Among them C t ∈R 3×224×224 , S t is the image semantic text corresponding to the t-th frame video, C t is the preprocessed image vector of the tracked human body corresponding to the t-th frame video.

[0071] like Figure 2 As shown, the implementation process of this step in this embodiment is as follows:

[0072] S11: Human tracking, the original video sequence V is input into the human tracking submodule, which uses yolov3 as a detector and sort as a tracker to implement multi-target tracking, and returns a rectangular bounding box for human recognition for subsequent image cropping operations. Among them, human tracking is achievable by existing technologies, so it is not described in detail.

[0073] S12: Crop the video frame, that is, crop the original video frame with a rectangular bounding box, fill the cropped image with 0 pixels to make the length and width consistent, and then scale the image to C t ∈R D×H×W , D, H, W are the number of channels, height, and width of the image respectively. The image scaling in this embodiment is C t ∈R 3×224×224 .

[0074] S13: Generate semantic text, that is, use each frame I of the original video sequence V t , using the Blip2 large model to output the generated image semantic description text S t In other feasible embodiments, other feasible graph-to-text algorithms / models can also be applied to the technical ideas of the present invention.

[0075] It should be understood that the execution order of steps S11-S13 is an example of this embodiment. In other feasible embodiments, S13 can be executed simultaneously with S11 or before S11, and the present invention is not restricted in this regard.

[0076] Regarding the 2) feature extraction step, it aims to extract visual features and text features, such as Figure 3 As shown, the details are as follows:

[0077] S21: extracting visual features, that is, inputting the image vector into a visual feature extractor to obtain a visual feature vector.

[0078] HMR is an end-to-end framework for reconstructing the entire human 3D mesh from a single RGB image. This embodiment uses the feature extractor of the model to extract visual features, which can effectively speed up the model training process. The ResNet50 module in the pre-trained HMR model is used, and the input is the pre-processed image vector of the video frame. Among them C t ∈R D×H×W , output visual feature data v t ∈R 2048 , v t is the visual feature vector of the tth frame.

[0079] S22: Extract text features, that is, input the image semantic text into the text feature extractor to obtain a text feature vector.

[0080] This example uses the pre-trained BERT model to extract text features. The BERT model is a deep bidirectional Transformer model that has learned rich language representations through pre-training and fine-tuning, and can effectively capture the semantic information in the text. The input is the image semantic text generated by the semantic text generation submodule, and the output is a fixed-length text feature vector. l t ∈R 768 , l t is the text feature vector corresponding to the t-th frame of video.

[0081] Regarding the 3) feature interaction fusion / aggregation step, it aims to perform multimodal interaction fusion and temporal interaction fusion, specifically including: splicing the visual feature vector and text feature vector of the same frame to obtain visual-text features, and performing feature interaction based on a linear transformation network on the visual-text features to obtain initial fusion features; constructing an aggregation network to aggregate intra-group features to obtain inter-group aggregation features corresponding to each frame. The details are as follows:

[0082] S31: Interactive fusion of multimodal features, such as Figure 4 As shown in the figure, this step concatenates the visual features and text features and linearly transforms them into the initial fusion features through a fully connected network. x t ∈R 2048 ,as follows:

[0083] The present invention directly concatenates the visual feature vector and text feature vector of each frame to obtain the visual-text features of the video. The visual-text features corresponding to the video sequence are expressed as f t ∈R2816 , f t is the visual-text feature corresponding to the t-th frame of the video. Since the visual features and text features are in different feature spaces, they need to be connected to the linear layer for feature interaction first. The interaction process can be written as:

[0084] X=M(X ft )

[0085] M(X ft )=X ft W m ∈R T×2048

[0086] Among them, M(·) is a linear transformation, and the initial fusion feature x t ∈R 2048 , the weight matrix is ​​W m ∈R 2816 ×2048 , the weight matrix W m Parameters for subsequent network optimization.

[0087] S32: Interactive fusion of temporal features, such as Figure 5 As shown in Figure 2, this step aggregates the past frame and future frame features twice by grouping to fuse them into a time-dependent inter-group aggregate feature z t ∈R 2048 .

[0088] It should be understood that the initial fusion feature X∈R T×2048 The text semantics and visual features are integrated, but the inter-frame features are not aggregated. Therefore, the current frame is refined by integrating the adjacent time features observed from past and future frames, and better performance is achieved by strengthening the temporal correlation. For any frame, the embodiment of the present invention uses {x t±4} features, and divide these 9 frame features into 3 groups with 3 adjacent frames as a group.

[0089] First, a fully connected layer is used to reduce the features of each frame to 256 dimensions, and then the features within the group are spliced. The spliced ​​features x concat ∈R 768 , then through two fully connected layers to scale the dimension to 3, and finally through the softmax layer to explore the dependencies between the frames in the group to obtain the attention value of the softmax layer output k corresponds to the dimension, and finally the attention value is used as the weight to multiply the initial fusion feature of the group, and the results are added to obtain the inter-group aggregation feature of the group, with a dimension of 2048; for the inter-group aggregation features of the three groups, the above process is repeated again to obtain the final inter-group aggregation feature z t ∈R 2048,This aggregate feature further considers the temporal dependency based on the fusion of visual and textual features.

[0090] It should be understood that this embodiment takes 3 adjacent frames as a group and 9 frames as a large group as an example. In other feasible embodiments, according to the above technical ideas, adaptive adjustments can be made, and the present invention does not impose specific constraints on this.

[0091] Regarding the 4) 3D human mesh and joint point recovery step, the goal is to iteratively regress 85 SMPL model parameters (known parameters in the field of 3D human mesh recovery), which include 72 posture parameters, 10 shape parameters and 3 camera parameters. One iteration process is: the inter-group aggregate feature z t ∈R 2048 Through two linear layers, the 1024-dimensional features are regressed, and then the initial SMPL model parameters are spliced ​​with the features. Finally, three different linear layers are used to regress and predict the difference of posture, shape and camera parameters, and the difference and the initial parameters are added to obtain the SMPL model parameters predicted in this round. Among them, the initial parameters of the first iteration of this embodiment use the average parameters of the SMPL model human body, and each subsequent iteration uses the results of the previous iteration as the initial parameters.

[0092] Among them, after iteration, the updated SMPL model parameters are obtained, and the updated SMPL model parameters are sent to the SMPL parameterized model to regress a complete 3D human body mesh. It should be understood that the SMPL model is a pre-trained model that has been proposed, and the 3D human body mesh can be obtained by sending the SMPL model parameters output by the present invention to the SMPL parameterized model.

[0093] In addition, after obtaining the 3D human body mesh, if joint points need to be obtained, the joint point coordinates can be directly obtained through the existing weights, which is achievable with existing technology. This embodiment uses the pre-trained linear regression parameter W to calculate the 3D joint positions using the human body mesh vertices; then the 2D joint points are calculated based on the weak perspective camera model using the camera parameters and 3D joint points.

[0094] Regarding the supervised training step S5), this embodiment takes the minimization of the loss function as the training optimization goal, and optimizes the network model to be optimized in steps S2-S5. Specifically, the visual feature extraction part of this embodiment is obtained using the feature extractor of the pre-trained HMR model, and the text feature uses the pre-trained large model BLIP2 to obtain the text, and then uses the pre-trained Bert to extract features, and the SMPL parameterized model obtains the human body mesh through 85 parameter regression. None of the above needs to be optimized, and the remaining models of the remaining network need to be optimized. In other feasible embodiments, if the above-mentioned part of the network model is not pre-trained, it participates in the optimization training in this step.

[0095] This embodiment uses the following loss function formula: L = L 3D +l 2D +L SMPL

[0096] Among them, L is the loss function, L 3D is the 3D joint error, L 2D is the projected 2D joint error, L SMPL is the SMPL model parameter error, as follows:

[0097]

[0098]

[0099] Where T is the total number of frames in the video sequence, X t and are the predicted 3D pose and true label of the t-th frame, z t and are the predicted 2D pose and true label of the t-th frame, β and are the predicted shape parameters and true labels, θ t and are the predicted pose parameters and true labels of the t-th frame, respectively.

[0100] So far, the embodiment of the present invention has constructed a video 3D human body mesh recovery model based on the interactive fusion of image features and text features. Figure 6 As shown, the steps are as follows:

[0101] The camera records video data containing human bodies, tracks, crops, and scales the video frame data, and generates semantic text.

[0102] The pre-processed video frame sequence data and its corresponding semantic text are input into the constructed video 3D human mesh recovery model based on the interactive fusion of image features and text features. Each frame of data will be used as the input data of the model together with the 9 frames of data before and after it, and the 3D human mesh and 2D and 3D joint points corresponding to this frame will be obtained. Therefore, the 3D human mesh and 2D and 3D joint points corresponding to all video frames in the sequence will be obtained in the end, and a 3D human mesh and 2D and 3D joint point sequence will be output.

[0103] The downstream module receives the data and utilizes the information to visualize the output results in the display window of the user's terminal device.

[0104] Embodiment 2:

[0105] This embodiment provides a system based on the above-mentioned grid restoration method, including: a preprocessing module, a feature extraction module, a feature fusion module, an aggregation module and a grid restoration module.

[0106] The preprocessing module is used to obtain the original video sequence and perform preprocessing to obtain the image semantic text and image vector corresponding to each frame of video. The image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition boundary box;

[0107] A feature extraction module, used for inputting the image vector into a visual feature extractor to obtain a visual feature vector and inputting the image semantic text into a text feature extractor to obtain a text feature vector;

[0108] A feature fusion module is used to concatenate the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and perform feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features;

[0109] An aggregation module, for dividing a plurality of preceding and succeeding frames into a group for any frame and aggregating initial fusion features of each frame in the group based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame;

[0110] The network recovery module is used to iteratively update the SMPL model parameters using the inter-group aggregation features of each frame to obtain updated SMPL model parameters, and finally input the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set.

[0111] In some embodiments, the model can be pre-trained and then imported into the system for direct call; in some embodiments, the system also includes a training module for training the network model to be optimized in steps S2-S5 through sample collection and true value labels of the samples.

[0112] Among them, please refer to the above method content for the specific implementation process of each module, the present invention will not be repeated here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, the hardware and software equipment includes but is not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0113] For example, in some embodiments, the preprocessing module, the human body tracking submodule, the image preprocessing submodule, and the semantic text generation submodule are used for human body tracking, image cropping and scaling, and image semantic text generation, respectively.

[0114] The feature extraction module is divided into a visual feature extraction submodule and a text feature extraction submodule, which are respectively used to input the image vector into the visual feature extractor to obtain the visual feature vector and input the image semantic text into the text feature extractor to obtain the text feature vector.

[0115] Embodiment 3:

[0116] This embodiment provides an electronic terminal, including: one or more processors; and a memory storing one or more computer programs;

[0117] The processor calls the computer program to execute: the steps of a mesh recovery method based on feature fusion of a large visual-language model. For example, S1-S5 is executed; or, as in Example 1, "1) video sequence preprocessing step; 2) feature extraction step; 3) feature interaction fusion / aggregation step; 4) 3D human body mesh and joint point recovery step; S5) supervised training step", as well as actual scene application.

[0118] For the specific implementation process of each step, please refer to the description of the above method.

[0119] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0120] Embodiment 4:

[0121] The present invention provides a computer-readable storage medium storing a computer program, which is called by a processor to execute: steps of a mesh recovery method based on feature fusion of a large visual-language model. For example, S1-S5 are executed; or as in Example 1, "1) video sequence preprocessing step; 2) feature extraction step; 3) feature interaction fusion / aggregation step; 4) 3D human body mesh and joint point recovery step; S5) supervised training step" and actual scene application are executed.

[0122] For the specific implementation process of each step, please refer to the description of the above method.

[0123] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0124] Based on this understanding, the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.

[0125] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is a device for realizing the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram with reference to the instructions executed by the processor according to the method, device (system) and computer program product of the embodiment of the present application. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which realizes the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0126] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A mesh recovery method based on feature fusion of a large vision-language model, characterized by: The following steps are involved: S1: Preprocessing, obtaining the original video sequence and preprocessing to obtain the image semantic text and image vector corresponding to each frame of video, wherein the image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition bounding box; S2: feature extraction, inputting the image vector into the visual feature extractor to obtain the visual feature vector and inputting the image semantic text into the text feature extractor to obtain the text feature vector; S3: feature fusion, concatenating the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and performing feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features; S4: constructing an aggregation network to aggregate intra-group features. For any frame, several preceding and succeeding frames are divided into a group and initial fusion features of each frame in the group are aggregated based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame. S5: iteratively updating the SMPL model parameters using the inter-group aggregation features of each frame, and finally inputting the updated SMPL model parameters into the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set; The network model to be optimized in steps S2-S5 is trained through sample collection and true value labels of the samples, and a complete 3D human body mesh is obtained according to the video sequence using the trained network model.

2. The method according to claim 1, characterized in that: The inter-group aggregation features of each frame are obtained according to the following process: S41: dividing the current frame, and the previous and next n frames into a large group, and then dividing every m frames in the large group into a small group; S42: Use the fully connected layer to reduce the dimension of the initial fusion features of each frame, and then perform feature splicing within the group to obtain the spliced ​​features; S43: using a fully connected layer to reduce the dimension of the concatenated features, and then inputting the features into a softmax layer to obtain an attention value vector A, wherein the number of elements of the attention value vector A is equal to m; S44: taking the elements of the attention value vector A as weights, multiplying them one by one with the initial fusion features in the corresponding groups, and adding the multiplication results as the inter-group aggregation features of the groups; Among them, the inter-group aggregation features of each group are calculated according to the above process, and then the m groups are divided into a new group, and its inter-group aggregation features are regarded as the initial fusion features in S42. Aggregation is performed according to the above steps S42-S44, and the above process is repeated until the final inter-group aggregation features are obtained.

3. The method according to claim 2, characterized in that: In step S41, the current frame, and the previous and next three frames are grouped into a large group, and every three frames are divided into a small group.

4. The method according to claim 1, characterized in that: In step S5, the process of iterating the SMPL model parameters using the inter-group aggregation features of each frame to obtain updated SMPL model parameters is as follows: S51: Obtain / set initial SMPL model parameters, wherein the SMPL model parameters include posture, shape and camera parameters; S52: Aggregate the inter-group features z t ∈R 2048 Through two linear layers, it regresses to 1024-dimensional features, R is a real number set, z t is the component aggregation feature corresponding to the t-th frame, with a dimension of 2048; S53: concatenating the initial SMPL model parameters with the features of step S52; S54: Use three different linear layers to regress the concatenated features to predict the difference in posture, shape and camera parameters; S55: Add the difference value and the current posture, shape and camera parameters one by one to obtain the SMPL model parameters updated in this round; The above steps are iteratively executed until the iteration termination condition is met to obtain updated SMPL model parameters, and each iteration uses the result of the previous iteration as the current SMPL model parameter.

5. The method according to claim 1, characterized in that: The interactive process of obtaining the initial fusion features through feature interaction based on the linear transformation network is expressed as: X=M(X ft ) M(X ft )=X ft W m In the formula, the visual-text features corresponding to the video sequence are f t is the visual-text feature corresponding to the t-th frame of the video, T is the total number of video frames, and W m is the weight matrix, i.e., the linear network parameters to be optimized during training; M(·) is the linear transformation, the initial fusion feature corresponding to the video sequence x t is the initial fusion feature corresponding to the t-th frame video.

6. The method according to claim 1, characterized in that: The training process takes minimization of the loss function as the training optimization goal, and the loss function is expressed as: L=L 3D +L 2D +L SMPL Among them, L is the loss function, L 3D is the 3D joint error, L 2D is the projected 2D joint error, L SMPL is the SMPL model parameter error, as follows: Where T is the total number of frames in the video sequence, X t and are the predicted 3D pose and true label of the t-th frame, z t and are the predicted 2D pose and true label of the t-th frame, β and are the predicted shape parameters and true labels, θ t and are the predicted pose parameters and true labels of the t-th frame, respectively.

7. The method according to claim 1, characterized in that: The visual feature extractor is the ResNet50 module in the HMR model, the text feature extractor is the BERT model, and the image semantic text is obtained using the Blip2 large model.

8. A system based on the method according to any one of claims 1 to 7, characterized in that: include: A preprocessing module is used to obtain the original video sequence and perform preprocessing to obtain the image semantic text and image vector corresponding to each frame of video, wherein the image vector is generated after cropping, pixel filling and scaling to a uniform size according to the human body recognition bounding box; A feature extraction module, used for inputting the image vector into a visual feature extractor to obtain a visual feature vector and inputting the image semantic text into a text feature extractor to obtain a text feature vector; A feature fusion module is used to concatenate the visual feature vector and the text feature vector of the same frame to obtain visual-text features, and perform feature interaction based on a linear transformation network on the visual-text features to obtain initial fused features; An aggregation module, for dividing a plurality of preceding and succeeding frames into a group for any frame and aggregating initial fusion features of each frame in the group based on the aggregation network, thereby obtaining inter-group aggregation features corresponding to each frame; The network recovery module is used to iteratively update the SMPL model parameters using the inter-group aggregation features of each frame, and finally the updated SMPL model parameters are input into the SMPL parameterized model to regress a complete 3D human body mesh, wherein the initial SMPL model parameters are pre-set.

9. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to execute: The steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, which is called by a processor to execute: The steps of the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Robot vision man-machine interaction method and system based on text generation video

    CN113051420A

  • Three-dimensional human body reconstruction method based on time attention mechanism and related equipment

    CN115731359A

  • Human body model construction method and device, equipment and storage medium

    CN116778527A

  • Image generation method and device, electronic equipment, storage medium and program product

    CN117437317A

  • Multi-modal false news detection method

    CN117609765A

Cited By

  • Large model-based medical double-arm robot task and motion planning method and system

    CN120439319A

  • Intelligent video monitoring management method and system based on big data

    CN120976831A

  • Intelligent video monitoring management method and system based on big data

    CN120976831B