Human mesh construction method and device for large scene with occlusion, equipment and medium

Abstract

The application relates to the technical field of computer vision, in particular to a human body grid construction method and device for an occluded large scene, equipment and a medium. The method comprises the following steps: extracting a two-dimensional feature map and a two-dimensional heat map of each single-view image through a single-view feature extraction module; constructing a scene feature body based on all the two-dimensional feature maps and a three-dimensional heat map based on all the two-dimensional heat maps through a multi-view feature fusion module; determining human body parameters corresponding to the multi-view images based on the scene feature body and the three-dimensional heat map through a decoding module, and determining a human body grid based on the human body parameters. The application positions the human body in a 3D world coordinate by using the scene feature body and prior information, and combines the positioning information to regress the grid parameters to obtain the human body grid of each human body, solves the scale transformation sensitivity and the single-view depth ambiguity, and improves the accuracy of the human body grid.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and medium for constructing human body meshes in large scenes with occlusion. Background Technology

[0002] Human mesh reconstruction (HMR) technology predicts 3D human body meshes from images and is widely used in robotics, 3D animation, virtual reality, and other applications. Existing human mesh reconstruction methods are generally designed for scenes with extremely limited numbers of people (typically <15). They either reconstruct a single person's mesh from a single image or by cropping a corresponding human photograph from it, or they reconstruct a mesh for multiple people from a single image. However, single-viewpoint methods are prone to depth ambiguity and are highly sensitive to scale changes. This leads to low prediction accuracy in large-scale real-world applications where there may be large crowds and severe occlusion.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a method, device, equipment and medium for constructing human body mesh in large scenes with occlusion, in order to address the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a method for constructing human body meshes in large, occluded scenes. This method utilizes a trained human body mesh reconstruction model, which includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module. Specifically, the method for constructing human body meshes in large, occluded scenes includes:

[0006] The single-view feature extraction module extracts two-dimensional feature maps and two-dimensional heat maps for each single-view image in the multi-view images.

[0007] The multi-view feature fusion module constructs a scene feature volume based on all two-dimensional feature maps and a three-dimensional heat map based on all two-dimensional heat maps.

[0008] The decoding module determines the human body parameters of each human body in the multi-view image based on the scene feature volume and the three-dimensional heat map, and determines the human body mesh of each human body based on the human body parameters of each human body.

[0009] The method for constructing human body meshes in large, occluded scenes, wherein the construction of scene feature volumes based on all two-dimensional feature maps and the construction of three-dimensional heat maps based on all two-dimensional heat maps by the multi-view feature fusion module specifically includes:

[0010] The multi-view feature fusion module projects each two-dimensional feature map and each two-dimensional heat map onto a three-dimensional volume space to obtain each single-view feature volume and each single-view three-dimensional heat map.

[0011] The multi-view feature fusion module uses mean operation to fuse all single-view feature volumes to obtain scene feature volumes, and fuses all single-view 3D heatmaps to obtain 3D heatmaps. The 3D heatmap is used to present the likelihood distribution of the positions of the joints in the world coordinate system.

[0012] The method for constructing human body meshes in large, occluded scenes, wherein the decoding module includes a human attention mechanism module and an SMPL-X parameterized model, and the step of determining the human body parameters of each human in the multi-view images based on the scene feature volume and the 3D heatmap by the decoding module, and determining the human body mesh of each human based on the human body parameters of each human specifically includes:

[0013] Based on the scene feature body and the 3D heatmap, a human body query is constructed for each human body in the multi-view image, and a value vector and a key vector are constructed based on the scene feature body;

[0014] Each human body query, value vector, and key vector human attention mechanism module outputs human body parameters for each human body through the human attention mechanism module.

[0015] Each human body parameter is input into the SMPL-X parameterized model, and the SMPL-X parameterized model outputs the human body network for each human body.

[0016] The method for constructing human body meshes in large, occluded scenes, wherein the step of constructing human body queries for each human body in the multi-view image based on the scene feature volume and the 3D heatmap specifically includes:

[0017] Detect the human body included in the multi-view image from the three-dimensional heat map;

[0018] Human features are extracted from the scene feature body using human keypoints for each detected human body, and the human features of each human body are combined with the average human body parameters to obtain the human body query for each human body.

[0019] The method for constructing human body meshes in large, occluded scenes, wherein the step of extracting human body features from the scene feature volume using human body key points specifically involves:

[0020] Human features are extracted from the scene feature body using pelvic joint points for each detected human body.

[0021] The method for constructing human body meshes in large, occluded scenes, wherein the decoding module further includes a 3D offset prediction network, and the method further includes:

[0022] Each human body query is input into the three-dimensional offset prediction network, and the three-dimensional offset prediction network outputs the human body position offset of each human body.

[0023] The human joint positions of each human body are detected from the three-dimensional heat map, and the three-dimensional position information of each human body is determined based on the human joint positions and the human body position offset.

[0024] The method for constructing human meshes in large scenes with occlusion includes a loss function used during the training of the human mesh recovery model. This loss function includes a detection loss for locating human body queries, an SMPL-X parameter regression loss, a mesh loss for supervised regression of 3D joint and vertex coordinates in the human mesh, a human mesh orientation loss, and a 3D joint density loss. The human mesh orientation loss is used to supervise the orientation of the human mesh based on human joints, and the 3D joint density loss is used to supervise the pose of the human network based on the 3D joint density map generated from the 3D joints.

[0025] The second aspect of this application provides a human body mesh construction device for large scenes with occlusion, wherein the human body mesh construction device for large scenes with occlusion is equipped with a control module and a trained human body mesh recovery model.

[0026] The human body mesh reconstruction model includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module;

[0027] The control module is used to control the human body mesh restoration model, so as to extract two-dimensional feature maps and two-dimensional heat maps of each single-view image in the multi-view images through the single-view feature extraction module; construct scene feature volumes based on all two-dimensional feature maps and construct three-dimensional heat maps based on all two-dimensional heat maps through the multi-view feature fusion module; and determine the human body parameters of each human body in the multi-view images through the decoding module based on the scene feature volumes and the three-dimensional heat maps, and determine the human body mesh of each human body based on the human body parameters of each human body.

[0028] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the human body mesh construction method for large scenes with occlusion as described above.

[0029] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory;

[0030] The memory stores a computer-readable program that can be executed by the processor;

[0031] When the processor executes the computer-readable program, it implements the steps in the human body mesh construction method for large scenes with occlusion as described above.

[0032] Beneficial Effects: Compared with existing technologies, this application provides a method, apparatus, device, and medium for constructing human body meshes in large scenes with occlusion. The method includes extracting two-dimensional feature maps and two-dimensional heat maps from each single-view image in multi-view images using a single-view feature extraction module; constructing a scene feature volume based on all two-dimensional feature maps and a three-dimensional heat map based on all two-dimensional heat maps using a multi-view feature fusion module; determining the human body parameters of each human body in the multi-view images using the scene feature volume and the three-dimensional heat map using a decoding module; and determining the human body mesh of each human body based on the human body parameters of each human body. This application provides a three-dimensional representation of the entire scene by constructing a scene feature volume from multi-view images, provides prior information on human body joints using the three-dimensional heat map, and then locates the human body in 3D world coordinates based on the scene feature volume and the prior information, and performs mesh parameter regression based on the location information to obtain the human body mesh of each human body. This solves the problems of scale transformation sensitivity and single-view depth ambiguity, and improves the accuracy in large scenes with large crowds and severe occlusion. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating the human body mesh construction method for large scenes with occlusion, provided in this application embodiment.

[0035] Figure 2 Network structure diagram for building a human body mesh model.

[0036] Figure 3 This is a schematic diagram of a human body mesh construction device for large scenes with occlusion, provided in an embodiment of this application.

[0037] Figure 4 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0038] This application provides a method, apparatus, device, and medium for constructing human body meshes in large, occluded scenes. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0039] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application’s specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when an element is referred to as “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements present. Furthermore, “connected” or “coupled” as used herein may include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0041] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0042] Research has shown that Human Frame Reconstruction (HMR) technology predicts 3D human body meshes from images. Existing human body mesh reconstruction methods are generally designed for scenes with extremely limited numbers of people (usually <15). They either reconstruct a single person's mesh from a single image or by cropping out the corresponding human photograph from it, or they reconstruct a mesh for multiple people from a single image. However, single-viewpoint methods are prone to depth ambiguity and are highly sensitive to scale changes, which leads to low prediction accuracy in large-scale scenes with potentially large crowds and severe occlusion in the real world.

[0043] Specifically, the main problems with existing human body mesh reconstruction are:

[0044] 1. Existing single-view human body mesh restoration has not effectively solved two inherent problems of monocular vision: one is depth ambiguity, that is, 2D projection information can easily correspond to multiple 3D spatial configurations, resulting in mesh geometric distortion; the other is scale sensitivity, that is, changes in camera distance cause large changes in human body scale, which are difficult to predict accurately.

[0045] 2. Multi-person human body mesh restoration is affected by the superposition of occlusion complexity and scene scale, making it difficult to output complete and accurate mesh results. Specifically, when there are many people, mutual occlusion between people makes some parts (such as arms and torsos) invisible, and it is impossible to fill in the missing areas through context. In large scenes, changes in camera perspective exacerbate spatial position confusion, increasing the difficulty of registration.

[0046] 3. During training, simple low-dimensional parameters are used to control the orientation and posture of the human body in the 3D scene. However, low-dimensional parameters (such as Euler angles and rotation matrices) are difficult to capture the continuous changes in complex postures (such as spinal curvature and coordinated limb movements), which leads to blurred optimization gradients, difficulty in convergence, and even getting stuck in local optima. Furthermore, the prior information of joint points is not fully incorporated, which is crucial for the mesh integrity in occluded scenes (such as predicting occluded parts).

[0047] Based on this, this application fully considers multi-view prompts to provide a 3D representation of the entire scene, and utilizes prior information on human joints to construct a human mesh restoration model that can be used in large scenes with severe occlusion. The human mesh restoration model includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module. Then, the single-view feature extraction module extracts 2D feature maps and 2D heatmaps from each single-view image in the multi-view images; the multi-view feature fusion module constructs a scene feature volume based on all 2D feature maps and a 3D heatmap based on all 2D heatmaps; the decoding module determines the human parameters of each human in the multi-view images based on the scene feature volume and the 3D heatmap, and determines the human mesh of each human based on the human parameters of each human. In this way, the scene feature volume constructed from the multi-view images provides a 3D representation of the entire scene, the 3D heatmap provides prior information on human joints, and then the human is located in 3D world coordinates based on the scene feature volume and the prior information, and the mesh parameters are regressed by combining the location information to obtain the human mesh of each human. This can solve the sensitivity to scale transformation and the depth ambiguity of single-view images, and improve the accuracy of the human mesh.

[0048] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0049] This embodiment provides a method for constructing human body meshes in large, occluded scenes. It applies a trained human body mesh reconstruction model, which includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module. For example... Figure 1 and Figure 2 As shown, the method for constructing human body meshes in large, occluded scenes specifically includes:

[0050] S10. Extract the two-dimensional feature map and two-dimensional heat map of each single-view image in the multi-view image through the single-view feature extraction module.

[0051] Specifically, multi-view images are obtained by acquiring images of the same scene from different perspectives. A multi-view image may contain overlapping single-view images, or it may not contain any overlapping single-view images. Furthermore, a multi-view image may completely cover the scene, or it may cover only a portion of the scene. For example, a multi-view image may include eight single-view images with a 90° field of view and adjacent viewpoints overlapping by 45°.

[0052] Furthermore, the single-view feature extraction module includes a single-view feature extraction unit and a heatmap prediction unit. The single-view feature extraction unit and the heatmap prediction unit operate in parallel, with both taking a single-view image as input. The output of the single-view feature extraction unit is a two-dimensional feature map, and the output of the heatmap prediction unit is a two-dimensional heatmap. The single-view feature extraction unit can employ a large visual model (ViT-L), and the heatmap prediction unit can employ... Network. Correspondingly, the extraction process of the two-dimensional feature map and the two-dimensional heatmap can be represented as:

[0053] ,

[0054] ,

[0055] in, Indicates the first A single-view image, Indicates the first Two-dimensional feature map of a single-view image. Indicates the first Two-dimensional heatmap of a single-view image. This indicates the number of images per viewpoint.

[0056] This application's embodiments, by employing the ViT-L model as a single-view feature extraction unit, can capture global contextual features crucial for multi-view fusion, thereby improving the accuracy of human body meshes. Simultaneously, by adopting... As a heatmap prediction unit, the network can predict accurate human body joints to enhance spatial positioning capabilities, thereby further improving the accuracy of the human body mesh.

[0057] S20. The multi-view feature fusion module constructs a scene feature body based on all two-dimensional feature maps and a three-dimensional heat map based on all two-dimensional heat maps.

[0058] Specifically, the multi-view feature fusion module is used to fuse two-dimensional feature maps to obtain a scene feature volume and to fuse two-dimensional heatmaps to obtain a three-dimensional heatmap. The scene feature volume represents feature information in three-dimensional volume space, while the three-dimensional heatmap reflects human joint information in three-dimensional space. It presents the likelihood distribution of human joints' positions in the world coordinate system. Based on this three-dimensional heatmap, the grid positions of human joints (e.g., pelvic joints) in three-dimensional volume space can be obtained. For example, if the spatial dimensions of the three-dimensional volume space are 300×300×20, and each voxel dimension corresponds to 100 millimeters in the physical three-dimensional world, then the actual spatial dimensions of this three-dimensional volume space in the real world are 30 meters × 30 meters × 2 meters. Therefore, the scene feature volume represents the feature information of a 30-meter × 30-meter × 2-meter scene in the real world, and the three-dimensional heatmap presents the likelihood distribution of human joints' positions in this 30-meter × 30-meter × 2-meter scene.

[0059] In one embodiment, the construction of a scene feature volume based on all two-dimensional feature maps and the construction of a three-dimensional heat map based on all two-dimensional heat maps by the multi-view feature fusion module specifically include:

[0060] The multi-view feature fusion module projects each two-dimensional feature map and each two-dimensional heat map onto a three-dimensional volume space to obtain each single-view feature volume and each single-view three-dimensional heat map.

[0061] The multi-view feature fusion module uses mean operation to fuse all single-view feature volumes to obtain scene feature volumes, and fuses all single-view 3D heatmaps to obtain 3D heatmaps. The 3D heatmap is used to present the likelihood distribution of the positions of the joints in the world coordinate system.

[0062] Specifically, a single-view feature volume is a three-dimensional volume feature in three-dimensional volume space, and a single-view three-dimensional heatmap is a heatmap in three-dimensional volume space. Single-view feature volumes and two-dimensional features... Figure 1 One-to-one correspondence between single-view 3D heatmap and 2D heatmap Figure 1 One-to-one correspondence.

[0063] When projecting a two-dimensional feature map onto a three-dimensional volume space, a blank three-dimensional volume can be pre-constructed, and then perspective geometry can be used to map the coordinates of each three-dimensional voxel in the blank three-dimensional volume. Mapping to two-dimensional feature coordinates yields the corresponding two-dimensional feature points in three-dimensional voxel coordinates, thus projecting the two-dimensional feature map into three-dimensional volume space. The three-dimensional voxel coordinates... Mapping to two-dimensional feature coordinates can be represented as:

[0064] ,

[0065] in, Indicates the first Feature points in a two-dimensional feature map Indicates the first The intrinsic parameter matrix of the camera corresponding to a single-view image. Indicates the first The extrinsic parameter matrix of the camera corresponding to a single-view image. Represents a three-dimensional voxel.

[0066] like Figure 2 As shown, after projecting each two-dimensional feature map onto a three-dimensional volume space, the resulting single-view feature volumes are... Fuse them into a single 3D volume feature to obtain the scene volume feature. When fusing all single-view feature volumes, the mean value method can be used to fuse all single-view feature volumes. That is, for each three-dimensional voxel, when it corresponds to at least two single-view three-dimensional voxels, the mean value of the at least two single-view three-dimensional voxels is used as the voxel value of the three-dimensional voxel. When it corresponds to one single-view three-dimensional voxel, the single-view three-dimensional voxel is used as the voxel value of the three-dimensional voxel. When it does not correspond to a single-view three-dimensional voxel, the default value (such as 0 or 255) is used as the voxel value of the three-dimensional voxel.

[0067] Similarly, the two-dimensional heatmap is projected into the three-dimensional volume space through a projection process of the two-dimensional feature map, resulting in a single-view heatmap. Then, a three-dimensional heatmap is generated based on all the single-view heatmaps. For example, Figure 2 As shown, a 3D heatmap can be generated using a modified RootNet network or a 3D heatmap generation network such as the RootNet network. Accordingly, the 3D heatmap can be represented as:

[0068] ,

[0069] in, Represents a three-dimensional heat map. Indicates the first Two-dimensional heat map The single-view heatmap obtained by projection. .

[0070] S30. The decoding module determines the human body parameters of each human body in the multi-view image based on the scene feature body and the three-dimensional heat map, and determines the human body mesh of each human body based on the human body parameters of each human body.

[0071] Specifically, human body parameters include SMPL-X parameters in the three-dimensional world, which include pose parameters. Shape parameters and expression parameters The human body mesh is generated by the SMPL-X parametric model based on human body parameters. Specifically, the SMPL-X parameters in the 3D world represent the human attention mechanism module of the decoding module, which considers scene volume features. The decoding process utilizes a 3D heatmap to provide prior knowledge. The human body mesh is generated from the SMPL-X parametric model within the decoding module based on human body parameters.

[0072] Based on this, the decoding module includes a human attention mechanism module and an SMPL-X parameterized model. The human attention mechanism module is used to decode the scene feature volume and the 3D heatmap to obtain the human body parameters of each human body. The SMPL-X parameterized model is used to generate a human body network for each human body based on the human body parameters of each human body. Therefore, the step of determining the human body parameters of each human body in the multi-view image based on the scene feature volume and the 3D heatmap by the decoding module, and determining the human body mesh for each human body based on the human body parameters of each human body specifically includes:

[0073] Based on the scene feature body and the 3D heatmap, a human body query is constructed for each human body in the multi-view image, and a value vector and a key vector are constructed based on the scene feature body;

[0074] Each human body query, key vector, and value vector human attention mechanism module outputs human body parameters for each human body through the human attention mechanism module.

[0075] Each human body parameter is input into the SMPL-X parameterized model, and the SMPL-X parameterized model outputs the human body network for each human body.

[0076] Specifically, the human body query, key vector, and value vector serve as inputs to the human attention mechanism module. This module performs attention learning based on the human body query, key vector, and value vector to obtain human body parameters. Both the key vector and value vector are one-dimensional vectors obtained by flattening the scene feature volume. The human body query is determined based on a 3D heatmap and the scene feature volume. The 3D heatmap is used to identify the human bodies included in the multi-view images, and the scene feature volume is used to determine the human body features of each individual. Then, the human body query is constructed based on these features. In other words, the human body key points in the 3D heatmap serve as prior knowledge for constructing the human body query, improving its accuracy and thus increasing the reliability of the cross-learned human body parameters.

[0077] In one embodiment, the step of constructing the human body query for each human body in the multi-view image based on the scene feature body and the three-dimensional heatmap specifically includes:

[0078] Detect the human body included in the multi-view image from the three-dimensional heat map;

[0079] Human features are extracted from the scene feature body using human keypoints for each detected human body, and the human features of each human body are combined with the average human body parameters to obtain the human body query for each human body.

[0080] Specifically, 3D heatmaps are used to estimate the likelihood distribution of human body joints in a world coordinate system. From the 3D heatmap, each human body included in a multi-view image can be detected. Then, each human body's joint points can be used as sampling points, and human body features for each human body can be sampled from the scene volume features based on these sampling points. That is, human body features include scene volume feature points corresponding to the human body joint points. After obtaining the human body features for each human body, each human body feature can be combined with the average human body parameters to construct a human body query. The average human body parameters are the average parameters of the SMPL-X parametric model. The process of combining human body features with the average human body parameters involves either concatenating the human body features and the average human body parameters into a vector, or fusing the human body features based on the pose parameters within the human body parameters, etc.

[0081] It should be noted that when sampling human body features using human body joints, all human body joints can be used, or only some human body joints can be used. In the embodiments of this application, the focus is mainly on the human pelvis, so that human body queries for each human body can be sampled from scene volume features using pelvic joints.

[0082] Furthermore, the human attention mechanism module may consist of a self-attention layer (SA), a cross-attention layer (CA), and a feedforward layer (FFN). The key vector and value vector are the key K and value V of the cross-attention layer. The human query Q is updated through several stacked human attention mechanism modules. Subsequently, three multilayer perceptrons are introduced to regress the SMPL-X parameters of each human using the updated human query Q′ to obtain the human parameters of each human.

[0083] In one embodiment, to improve the accuracy of the human body network, this application embodiment provides prior position information for the human body network generation process through human body joints, and adds the prior position information provided by the human body joints to the human body parameters, so that SMPL-X can obtain the prior position information. Then, based on the prior position information and SMPL-X parameters, a human body mesh is constructed and the positions of the human body mesh vertices and human body joints in the world coordinate system and the camera view coordinate system are obtained.

[0084] For example, the decoding module further includes a three-dimensional offset prediction network, and the method includes:

[0085] Each human body query is input into the three-dimensional offset prediction network, and the three-dimensional offset prediction network outputs the human body position offset of each human body.

[0086] The human joint positions of each human body are detected from the three-dimensional heat map, and the three-dimensional position information of each human body is determined based on the human joint positions and the human body position offset.

[0087] Specifically, the 3D offset prediction network is used to estimate the offset of the human body position, and then combined with the human body joint positions in the 3D heatmap to determine the 3D position information of the human body joints. This 3D position information can be:

[0088] ,

[0089] in, Represents three-dimensional position information. Indicates the location of human joints. This indicates the offset of the human body position.

[0090] After obtaining the 3D position information, the human body parameters (i.e., SMPL-X parameters) and the human body parameters of the 3D position information are input into the SMPL-X layer in the SMPL-X parametric model. The human body mesh is constructed through the SMPL-X layer, and the positions of the human body mesh vertices and joints in the world coordinate system and the camera view coordinate system are obtained.

[0091] In one embodiment, to avoid orientation and pose ambiguity in occluded scenes, during the training of the human mesh reconstruction model, in addition to the existing detection loss for human location query, SMPL-X parameter regression loss, and mesh loss for supervised regression of 3D joint and vertex coordinates under human mesh, this application also employs human mesh orientation loss and 3D joint density loss. The human mesh orientation loss is used to supervise the orientation of the human mesh based on human joints, and the 3D joint density loss is used to supervise the pose of the human network based on the 3D joint density map generated from the 3D joints. This embodiment of the application, by using human mesh orientation loss to supervise the orientation of the human mesh, can solve the problem of orientation ambiguity in occluded scenes, and by using 3D joint density loss to supervise the pose of the human network, can solve the pose ambiguity in occluded scenes, thereby improving the model performance of the human mesh reconstruction model, especially for large scenes with occlusion.

[0092] Furthermore, since the global orientation in the SMPL-X parameters (controlled by a low-dimensional vector) cannot effectively supervise the orientation of the generated human body mesh, the human body mesh orientation loss in this embodiment is defined based on human body joints to achieve better supervision of the human body mesh orientation. Specifically, the left hip joint of the human body... and right hip joint It can provide the x-axis direction, and the human pelvis and spine The y-axis direction can be provided, and the orientation of the human body mesh can be obtained using the cross product of the x-axis vector and the y-axis vector. Among them, the human body grid direction It can be represented as:

[0093] .

[0094] Therefore, a human mesh orientation loss is constructed based on the human mesh orientation between the predicted human joints and the actual human joints predicted by the human mesh reconstruction model. The human mesh orientation loss can be expressed as:

[0095] ,

[0096] in, The orientation of the human body mesh between real human joints. This indicates the predicted orientation of the human body mesh between joints. This indicates the loss of orientation in the human body mesh.

[0097] Furthermore, the construction process of the three-dimensional joint density loss can be as follows: first, use a three-dimensional Gaussian kernel function to extract the density from real three-dimensional human joints. and predicting 3D human joints Both a true 3D joint density map and a predicted 3D joint density map are generated. Then, a 3D joint density loss is constructed based on the true and predicted 3D joint density maps using mean squared error loss (MSE). The 3D joint density loss is expressed as:

[0098] ,

[0099] in, This indicates the loss of three-dimensional joint density. This indicates Gaussian smoothing.

[0100] The embodiments of this application provide a stronger supervision signal for the pose of the human body mesh by calculating element by element in three-dimensional space, which can better handle the pose ambiguity problem of multi-view human body mesh reconstruction tasks under occlusion conditions.

[0101] In summary, this application provides a method for constructing human body meshes in large, occluded scenes. The method includes extracting two-dimensional feature maps and two-dimensional heatmaps from each single-view image in a multi-view image using a single-view feature extraction module; constructing scene feature volumes based on all two-dimensional feature maps and constructing three-dimensional heatmaps based on all two-dimensional heatmaps using a multi-view feature fusion module; determining human body parameters for each human in the multi-view images using the scene feature volumes and the three-dimensional heatmaps using a decoding module; and determining the human body mesh for each human based on these parameters. This application utilizes multi-view information to construct 3D feature volume representations and uses three-dimensional heatmaps to provide prior information on human joints. This addresses the sensitivity to scale changes and the depth ambiguity of single-view images. It allows for human body localization in a real 3D world coordinate system and combines localization information with mesh parameter regression, improving the accuracy of the human body mesh. Furthermore, during training, prior information on human joints is used to construct supervision regarding human orientation and the 3D scene level, unlike the simple low-dimensional parameter loss used in single-view methods to assist network training and improve model performance. Furthermore, the embodiments of this application utilize multi-view information to construct 3D feature volume representations, making the human body mesh restoration model more suitable for human body mesh pose restoration in scenarios with severe occlusion, thereby improving the generalization and application of the human body mesh restoration model.

[0102] Furthermore, to demonstrate that the human mesh reconstruction model in this application can utilize the advantages of multiple perspectives and prior knowledge of human joints to reconstruct the human mesh in scenes with severe occlusion, achieving better scene-level human mesh reconstruction results, this application compares the multi-view human mesh reconstruction method with a multi-person human mesh reconstruction method under multi-view settings and a 3D human pose estimation method for human mesh reconstruction tasks.

[0103] Distance-based multi-view fusion methods: By comparing with single-view human body mesh reconstruction methods Multi-HMR and 3DCrowdNet, AiOS, TokenHMR, a strategy is adopted to select the nearest camera prediction result for each target human body (close-range cameras may provide more accurate predictions). These methods are labeled "Multi-HMR (distance)" and "3DCrowdNet (distance)".

[0104] A multi-view fusion method based on network learning: Simultaneously, a learnable sub-network is used to fuse the multi-view SMPL-X parameter predictions of Multi-HMR. The multi-view SMPL-X prediction results are weighted and summed through a self-attention layer. This method is labeled "Multi-HMR (network learning)".

[0105] A multi-view fusion method based on averaging: This method directly uses the average value to average the weights of each view for fusion prediction, and is labeled "Multi-HMR (Averaging)". It also compares with the multi-view 3D pose estimation method VoxelPose. Human query points (based on keypoint positions predicted by VoxelPose) are sampled from the feature volume, and then SMPL-X parameters are estimated from the human query points through a regression perceptron network. This method has two variations: "VoxelSMPLX (Separate Training)" which uses only pre-trained VoxelPose and trains the regression perceptron; or "VoxelSMPLX (Joint Training)" which jointly trains VoxelPose and the perceptron.

[0106] As shown in Table 1, the method provided in this application embodiment demonstrates that it can utilize multi-view information and combine it with effective joint information to achieve better scene-level human mesh prediction results. In addition, ablation experiments were conducted on two loss functions, demonstrating that the loss functions provided in this application embodiment can effectively improve the performance of the human mesh reconstruction model. As shown in Table 2, it can be seen that the human mesh orientation loss and 3D joint density loss proposed in this application embodiment effectively utilize joint information to optimize the mesh parameter prediction of the network, thereby achieving better prediction results.

[0107] Table 1. Results of the human body mesh restoration task on the dataset.

[0108]

[0109] Table 2 Results of Loss Function Ablation Experiment

[0110]

[0111] Based on the above-described method for constructing human body meshes in large scenes with occlusion, this embodiment provides a device for constructing human body meshes in large scenes with occlusion, such as... Figure 3 As shown, the human mesh construction device for large scenes with occlusion is equipped with a control module 100 and a trained human mesh restoration model 200; the human mesh restoration model 200 includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module.

[0112] The control module 100 is used to control the human body mesh restoration model to extract two-dimensional feature maps and two-dimensional heat maps of each single-view image in the multi-view images through the single-view feature extraction module; to construct scene feature volumes based on all two-dimensional feature maps and three-dimensional heat maps based on all two-dimensional heat maps through the multi-view feature fusion module; and to determine the human body parameters of each human body in the multi-view images through the decoding module based on the scene feature volumes and the three-dimensional heat maps, and to determine the human body mesh of each human body based on the human body parameters of each human body.

[0113] Based on the above-described method for constructing human body meshes in large scenes with occlusion, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the method for constructing human body meshes in large scenes with occlusion as described in the above embodiment.

[0114] Based on the aforementioned method for constructing human body meshes in large, occluded scenes, this application also provides a terminal device, such as... Figure 4 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0115] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0116] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0117] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0118] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing human body meshes in large, occluded scenes, characterized in that, The method for constructing human meshes in large, occluded scenes using a trained human mesh reconstruction model, which includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module, specifically includes: The single-view feature extraction module extracts two-dimensional feature maps and two-dimensional heat maps for each single-view image in the multi-view images. The multi-view feature fusion module constructs a scene feature volume based on all two-dimensional feature maps and a three-dimensional heat map based on all two-dimensional heat maps. The decoding module determines the human body parameters of each human body in the multi-view image based on the scene feature body and the three-dimensional heat map, and determines the human body mesh of each human body based on the human body parameters of each human body; Specifically, the construction of a scene feature volume based on all two-dimensional feature maps and the construction of a three-dimensional heat map based on all two-dimensional heat maps by the multi-view feature fusion module include: The multi-view feature fusion module projects each two-dimensional feature map and each two-dimensional heat map onto a three-dimensional volume space to obtain each single-view feature volume and each single-view three-dimensional heat map. The multi-view feature fusion module uses mean operation to fuse all single-view feature volumes to obtain scene feature volumes, and fuses all single-view 3D heatmaps to obtain 3D heatmaps. The 3D heatmaps are used to present the likelihood distribution of the positions of the joints in the world coordinate system. The decoding module includes a human attention mechanism module and an SMPL-X parameterization model. Based on the scene feature volume and the 3D heatmap, the decoding module determines the human body parameters of each person in the multi-view image, and determines the human body mesh for each person based on these parameters. Specifically, this includes: Based on the scene feature body and the 3D heatmap, a human body query is constructed for each human body in the multi-view image, and a value vector and a key vector are constructed based on the scene feature body; Each human body query, value vector, and key vector are input into the human attention mechanism module, and the human body parameters of each human body are output through the human attention mechanism module. Each human body parameter is input into the SMPL-X parameterized model, and the SMPL-X parameterized model outputs the human body network for each human body.

2. The method for constructing human body meshes in large scenes with occlusion as described in claim 1, characterized in that, The specific steps for constructing a human body query for each human body in the multi-view image based on the scene feature volume and the 3D heatmap include: Detect the human body included in the multi-view image from the three-dimensional heat map; Human features are extracted from the scene feature body using human keypoints for each detected human body, and the human features of each human body are combined with the average human body parameters to obtain the human body query for each human body.

3. The method for constructing human body meshes in large scenes with occlusion as described in claim 2, characterized in that, The specific steps of extracting human features for each detected human body from the scene feature body using human body key points are as follows: Human features are extracted from the scene feature body using pelvic joint points for each detected human body.

4. The method for constructing human body meshes in large scenes with occlusion as described in claim 1, characterized in that, The decoding module further includes a 3D offset prediction network, and the decoding module includes a human attention mechanism module and an SMPL-X parameterized model. The method further includes: Each human body query is input into the three-dimensional offset prediction network, and the three-dimensional offset prediction network outputs the human body position offset of each human body. The human joint positions of each human body are detected from the three-dimensional heat map, and the three-dimensional position information of each human body is determined based on the human joint positions and the human body position offset.

5. The method for constructing human body meshes in large scenes with occlusion as described in claim 1, characterized in that, During the training of the human body mesh reconstruction model, the loss functions used include detection loss for locating human body queries, SMPL-X parameter regression loss, mesh loss for supervised regression of 3D joint and vertex coordinates under human body mesh, human body mesh orientation loss, and three-dimensional joint density loss. The human body mesh orientation loss is used to supervise the orientation of the human body mesh based on human body joints, and the three-dimensional joint density loss is used to supervise the pose of the human body network based on the three-dimensional joint density map generated from the three-dimensional joints.

6. A human body mesh construction device for large scenes with occlusion, characterized in that, The human body mesh construction device for large scenes with occlusion is equipped with a control module and a trained human body mesh recovery model. The human body mesh restoration model includes a single-view feature extraction module, a multi-view feature fusion module, and a decoding module; The control module is used to control the human body mesh restoration model, so as to extract two-dimensional feature maps and two-dimensional heat maps of each single-view image in the multi-view images through the single-view feature extraction module; and to construct scene feature volumes based on all two-dimensional feature maps and three-dimensional heat maps based on all two-dimensional heat maps through the multi-view feature fusion module. The decoding module determines the human body parameters of each human body in the multi-view image based on the scene feature body and the three-dimensional heat map, and determines the human body mesh of each human body based on the human body parameters of each human body; Specifically, the construction of a scene feature volume based on all two-dimensional feature maps and the construction of a three-dimensional heat map based on all two-dimensional heat maps by the multi-view feature fusion module include: The multi-view feature fusion module projects each two-dimensional feature map and each two-dimensional heat map onto a three-dimensional volume space to obtain each single-view feature volume and each single-view three-dimensional heat map. The multi-view feature fusion module uses mean operation to fuse all single-view feature volumes to obtain scene feature volumes, and fuses all single-view 3D heatmaps to obtain 3D heatmaps. The 3D heatmaps are used to present the likelihood distribution of the positions of the joints in the world coordinate system. The decoding module includes a human attention mechanism module and an SMPL-X parameterization model. Based on the scene feature volume and the 3D heatmap, the decoding module determines the human body parameters of each person in the multi-view image, and determines the human body mesh for each person based on these parameters. Specifically, this includes: Based on the scene feature body and the 3D heatmap, a human body query is constructed for each human body in the multi-view image, and a value vector and a key vector are constructed based on the scene feature body; Each human body query, value vector, and key vector are input into the human attention mechanism module, and the human body parameters of each human body are output through the human attention mechanism module. Each human body parameter is input into the SMPL-X parameterized model, and the SMPL-X parameterized model outputs the human body network for each human body.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the human body mesh construction method for large scenes with occlusion as described in any one of claims 1-5.

8. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the human body mesh construction method for large scenes with occlusion as described in any one of claims 1-5.

Images