A method, apparatus and equipment for multi-view, multi-person 3D pose estimation

Abstract

This invention relates to the field of pose estimation technology, specifically to a multi-view, multi-person 3D pose estimation method, apparatus, and device. The method includes: acquiring image information and depth information of the target using multi-view acquisition technology; segmenting the person in a single view based on the image information and extracting 2D semantic features; reconstructing the point cloud based on the depth information to obtain a 3D point cloud in a preset world coordinate system; segmenting the point cloud for different human figures based on the 3D point cloud and calculating the point cloud boundaries as human body boundaries; mapping the 2D semantic features to the 3D point cloud to obtain 3D features; and classifying different human figures and constructing 3D poses based on the 3D features and human body boundaries. This technical solution can segment the point cloud for different human figures based on the 3D point cloud, accurately define the joint positions of different figures, and achieve differentiation and matching of different figures. Furthermore, based on discrete point cloud data, it has better flexibility and fault tolerance.

Description

Technical Field

[0001] This invention relates to the field of attitude estimation technology, specifically to a multi-view, multi-person 3D attitude estimation method, apparatus, and equipment. Background Technology

[0002] The main task of 3D human pose estimation is to accurately predict the 3D structural information of the human body in three-dimensional space. This is achieved by introducing depth information based on the 2D pose estimation results. 3D human pose estimation is precise and intuitive, making it highly valuable for research. However, real-world scenes often involve multiple people and objects, leading to self-occlusion issues caused by the interaction between human joints and mutual occlusion issues arising from interaction with the environment. To minimize the strong impact of occlusion on pose estimation, a common approach is to use multiple cameras to comprehensively cover the scene, using information from other cameras to compensate for the missing information from the current camera. Therefore, effectively and rationally integrating information acquired from multiple perspectives is crucial for 3D multi-human pose estimation.

[0003] However, due to the occlusion problem, current 3D multi-human pose estimation methods lack the precision to define the joint positions of different human bodies and the precision to distinguish and match different human bodies. Furthermore, conventional 3D multi-human pose estimation methods rely on mathematical formulas for matching, which lacks flexibility and has a low fault tolerance.

[0004] Therefore, current 3D multi-person pose estimation methods lack accuracy, flexibility, and fault tolerance. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a multi-view, multi-person 3D pose estimation method, apparatus and device to solve the problems of insufficient accuracy, insufficient flexibility and low fault tolerance of existing 3D multi-person pose estimation methods.

[0006] According to a first aspect of the present invention, a multi-view, multi-person 3D pose estimation method is provided, comprising:

[0007] Multi-view acquisition technology is used to obtain image and depth information of the target being photographed;

[0008] Based on the image information, perform single-view character segmentation and extract 2D semantic features;

[0009] Based on the depth information, point cloud reconstruction is performed to obtain a 3D point cloud in a preset world coordinate system;

[0010] Based on the 3D point cloud, point cloud segmentation is performed for different human bodies, and the point cloud boundary is calculated as the human body boundary.

[0011] The 2D semantic features are mapped onto the 3D point cloud to obtain 3D features;

[0012] Based on the 3D features and the human body boundaries, different human bodies are classified and their 3D poses are constructed.

[0013] Preferably, the step of segmenting a person from a single view and extracting 2D semantic features includes:

[0014] Detection boxes for different human bodies are identified from image information from each viewpoint;

[0015] For each bounding box, a 2D heatmap is estimated using a 2D pose estimation network;

[0016] The 2D coordinates and probability map of each joint node are obtained based on the 2D heat map.

[0017] Based on the probability map, 2D semantic features are obtained using a deep convolutional neural network.

[0018] Preferably, the step of segmenting the target into point clouds of different human bodies based on the 3D point cloud and calculating the point cloud boundaries as human body boundaries includes:

[0019] The 3D point cloud is processed using the core attention mechanism of the Transformer architecture to segment the point cloud for different human bodies, and the point cloud boundary is calculated as the human body boundary.

[0020] Preferably, the step of mapping the 2D semantic features to the 3D point cloud to obtain 3D features includes:

[0021] Based on the 2D coordinates, 2D semantic features, and pre-stored camera parameter position information of each joint node in each viewpoint, the epipolar geometry principle is used to map all joint nodes to the 3D point cloud to obtain the 3D features of each viewpoint.

[0022] By fusing the joint nodes in the 3D features from all perspectives, a 3D feature is obtained that includes all joint node information at the current moment and the prior connection relationships of the joint nodes.

[0023] Preferably, the process of fusing joint nodes from 3D features across all viewpoints includes:

[0024] Using a radius-based spatial linear interpolation fusion algorithm, joint nodes of the same human body are fused from each viewpoint according to a pre-set radius threshold.

[0025] Preferably, the step of classifying different human bodies and constructing 3D poses based on the 3D features and the human body boundaries includes:

[0026] Based on the human body boundary, all joint node information is divided to obtain the joint nodes within each human body boundary;

[0027] Connect the joint nodes within each human body boundary according to the prior connection relationship to construct the 3D pose.

[0028] Preferably, the calculation of the point cloud boundary includes:

[0029] Based on the 3D point cloud segmented from the point cloud, the surface boundary of the N clusters of point clouds at a distance X from the joint node is calculated, and then the surface boundary is used as the human body boundary.

[0030] Preferably, after the 3D pose construction, the method further includes:

[0031] The constructed 3D posture is corrected based on the principle of human inverse kinematics so that the 3D posture conforms to ergonomic constraints.

[0032] According to a second aspect of the present invention, a multi-view, multi-person 3D pose estimation device is provided, comprising:

[0033] The information acquisition module is used to acquire image and depth information of the target object using multi-view acquisition technology.

[0034] The semantic feature module is used to segment people in a single view based on the image information and extract 2D semantic features.

[0035] The point cloud construction module is used to reconstruct the point cloud based on the depth information to obtain a 3D point cloud in a preset world coordinate system.

[0036] The point cloud segmentation module is used to segment the point cloud based on the 3D point cloud, with the target being different human bodies, and calculate the point cloud boundary as the human body boundary.

[0037] The feature mapping module is used to map the 2D semantic features to the 3D point cloud to obtain 3D features;

[0038] The 3D pose construction module is used to classify different human bodies and construct 3D poses based on the 3D features and the human body boundaries.

[0039] According to a third aspect of the present invention, a multi-view, multi-person 3D pose estimation device is provided, comprising:

[0040] The main controller and the memory connected to the main controller;

[0041] The memory stores program instructions;

[0042] The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0043] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0044] It is understood that the technical solution provided by this invention utilizes multi-view acquisition technology to obtain image and depth information of the target; based on the image information, it performs single-view character segmentation and extracts 2D semantic features; based on the depth information, it performs point cloud reconstruction to obtain a 3D point cloud in a preset world coordinate system; based on the 3D point cloud, it segments the target into point clouds for different human bodies and calculates the point cloud boundaries as human body boundaries; it maps the 2D semantic features to the 3D point cloud to obtain 3D features; and based on the 3D features and human body boundaries, it classifies different human bodies and constructs 3D poses. This technical solution can segment the target into point clouds for different human bodies based on the 3D point cloud, accurately define the joint positions of different characters, achieve differentiation and matching of different characters, and, based on discrete point cloud data, has better flexibility and fault tolerance.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] Figure 1 This is a schematic diagram illustrating the steps of a multi-view, multi-person 3D pose estimation method according to an exemplary embodiment;

[0048] Figure 2 This is a schematic diagram illustrating a semantic feature extraction process according to an exemplary embodiment;

[0049] Figure 3 This is a semantic segmentation flowchart illustrated according to an exemplary embodiment;

[0050] Figure 4 This is a schematic diagram illustrating a 3D pose construction process according to an exemplary embodiment;

[0051] Figure 5 This is a schematic block diagram illustrating a multi-view, multi-person 3D pose estimation device according to an exemplary embodiment. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0053] Currently, numerous methods for multi-view, multi-person 3D pose estimation have emerged. Here, we will briefly describe four of the more mainstream methods and analyze their shortcomings.

[0054] The first method is matching + triangulation reconstruction: This method first obtains the 2D poses of multiple people from different perspectives, then performs frame-to-frame matching, calculates the correlation matrix of all characters between adjacent frames, and uses the correlation coefficient to characterize the probability of whether they are the same person from different perspectives. Finally, triangulation or 3D pictorial structures (3DPs) are used to infer the 3D pose. The drawback of this method is that it relies too heavily on 2D detectors; the accuracy of the detectors directly affects the probability calculation of the subsequent correlation matrix. Furthermore, the correlation between characters is calculated rather than data-driven, making it highly random.

[0055] The second method is spatial voxelization: This method first predicts keypoint heatmaps from 2D images viewed from multiple perspectives, then projects the heatmaps into 3D space to convert them into spatial voxel features, and uses these features to locate the center points of all people in the scene. Finally, it predicts the positions of other human joints based on the center points. The drawback of this method is that the accuracy of spatial voxelization is limited by the mesh size, and the computational complexity increases cubically with the size of the space. Furthermore, complex occlusion scenes can significantly interfere with the localization of center points.

[0056] The third approach is the Transformer architecture: this method avoids complex computational pipelines, reduces computational burden, achieves real-time inference, and directly performs spatial location regression of key points. However, due to its Transformer architecture, it suffers from a common problem: it requires a large dataset for training. Since applicable datasets are limited, it's best to introduce self-supervised methods for improvement.

[0057] Example 1

[0058] Figure 1 This is a schematic diagram illustrating the steps of a multi-view, multi-person 3D pose estimation method according to an exemplary embodiment. See also... Figure 1 This paper provides a multi-view, multi-person 3D pose estimation method, including:

[0059] Step S11: Obtain image information and depth information of the target using multi-view acquisition technology;

[0060] Step S12: Based on the image information, perform single-view character segmentation and extract 2D semantic features;

[0061] Step S13: Based on the depth information, perform point cloud reconstruction to obtain a 3D point cloud in a preset world coordinate system;

[0062] Step S14: Based on the 3D point cloud, segment the point cloud for different human bodies and calculate the point cloud boundary as the human body boundary.

[0063] Step S15: Map the 2D semantic features to the 3D point cloud to obtain 3D features;

[0064] Step S16: Based on the 3D features and the human body boundaries, classify different human bodies and construct their 3D poses.

[0065] It is understood that the technical solution provided in this embodiment utilizes multi-view acquisition technology to obtain image information and depth information of the target; based on the image information, it performs single-view character segmentation and extracts 2D semantic features; based on the depth information, it performs point cloud reconstruction to obtain a 3D point cloud in a preset world coordinate system; based on the 3D point cloud, it segments the target into point clouds for different human bodies and calculates the point cloud boundaries as human body boundaries; it maps the 2D semantic features to the 3D point cloud to obtain 3D features; based on the 3D features and human body boundaries, it classifies different human bodies and constructs 3D poses. This technical solution can segment the target into point clouds for different human bodies based on the 3D point cloud, accurately define the joint positions of different characters, achieve differentiation and matching of different characters, and, based on discrete point cloud data, has better flexibility and fault tolerance.

[0066] First, single-view 2D multi-human target detection and joint semantic feature extraction are performed, as follows:

[0067] It should be noted that in step S12, the segmentation of the person in a single view and the extraction of 2D semantic features include:

[0068] Detection boxes for different human bodies are identified from image information from each viewpoint;

[0069] For each bounding box, a 2D heatmap is estimated using a 2D pose estimation network;

[0070] The 2D coordinates and probability map of each joint node are obtained based on the 2D heat map.

[0071] Based on the probability map, 2D semantic features are obtained using a deep convolutional neural network.

[0072] Preferably, the acquired image information can be an RGB image.

[0073] In practice, Figure 2 This is a schematic diagram illustrating a semantic feature extraction process according to an exemplary embodiment. See also: Figure 2 Based on some existing models, this paper achieves single-view character segmentation and semantic feature extraction from acquired RGB images. In recent years, the YOLO series of object detectors have been widely used in pose estimation backbone networks due to their lightweight design and accurate detection performance. This model can be trained from source code, and its structure can be slightly modified to achieve results closer to the ideal.

[0074] The specific approach is as follows: For each viewpoint's RGB image, a detector such as YOLO is used as the backbone network to identify bounding boxes for different human bodies. Then, for each bounding box, a 2D pose estimation network is used to estimate a 2D heatmap. The heatmap represents the coordinates of each skeletal node using a probability map, the resolution of which is often a proportionally scaled image of the original (typically 64x48). The number of skeletal nodes equals the number of joint nodes. Finally, a DCNN (Deep Convolutional Neural Network) is used to estimate the semantic information of each joint in the 2D pose. In the 2D joints and their semantics, there are representations such as head, hand-1 (labeled 1 for the hand), foot-1 (labeled 1 for the foot), etc.

[0075] Next, multi-view 3D point cloud fusion and instance segmentation based on human body categories are required, as detailed below:

[0076] In step S13, point cloud reconstruction is performed based on the depth information to obtain a 3D point cloud in a preset world coordinate system.

[0077] It should be noted that in step S14, the step of segmenting the 3D point cloud into point cloud segments for different human bodies and calculating the point cloud boundaries as human body boundaries includes:

[0078] The 3D point cloud is processed using the core attention mechanism of the Transformer architecture to segment the point cloud for different human bodies, and the point cloud boundary is calculated as the human body boundary.

[0079] In practice, for the human figure matching problem, depth information collected from multiple perspectives is used to reconstruct point clouds, creating 3D point clouds. These point clouds are then segmented to represent different human figures. The goal is to find the boundaries between different human figure models, serving as confidence intervals for human poses. This avoids matching the same person based solely on facial and shape features, instead using segmented point clouds to represent different individuals.

[0080] Figure 3 This is a semantic segmentation flowchart illustrated according to an exemplary embodiment, see [link to flowchart]. Figure 3 The specific approach is as follows: for the depth image from each viewpoint, the depth information point cloud from multiple viewpoints is mapped and transformed into a 3D point cloud in the world coordinate system. Preferably, the point cloud can be preprocessed, including point cloud fusion, outlier removal, and other operations. The most important step is how to segment human instances from the point cloud. Mainstream point cloud segmentation networks such as PointNet and DGCNN have unclear segmentation edges. If used as the point cloud segmentation model in this embodiment, it may lead to unclear human instance segmentation, and the calculated point cloud boundaries will naturally be poor. Due to the irregularity and disorder of point cloud data itself, it was previously impossible to directly process point clouds using convolutional neural networks. If you want to use deep learning to process point cloud-related tasks, it will be very inconvenient. This embodiment utilizes the Transformer architecture, mainly using its core attention mechanism to process point clouds. Because point cloud processing requires the design of an operator that is invariant in arrangement and does not depend on the connection relationship between points; the attention mechanism itself is such an operator. Therefore, using the Transformer architecture for human point cloud instance segmentation will be more effective than traditional segmentation networks. Therefore, this invention adopts a human point cloud segmentation based on the Transformer architecture.

[0081] Furthermore, 2D feature mapping and human skeleton reconstruction within the 3D point cloud boundaries are performed, as detailed below:

[0082] It should be noted that, in step S15, mapping the 2D semantic features to the 3D point cloud to obtain 3D features includes:

[0083] Based on the 2D coordinates, 2D semantic features, and pre-stored camera parameter position information of each joint node in each viewpoint, the epipolar geometry principle is used to map all joint nodes to the 3D point cloud to obtain the 3D features of each viewpoint.

[0084] By fusing the joint nodes in the 3D features from all perspectives, a 3D feature is obtained that includes all joint node information at the current moment and the prior connection relationships of the joint nodes.

[0085] It should be noted that in step S16, the classification of different human bodies and construction of 3D poses based on the 3D features and the human body boundaries include:

[0086] Based on the human body boundary, all joint node information is divided to obtain the joint nodes within each human body boundary;

[0087] Connect the joint nodes within each human body boundary according to the prior connection relationship to construct the 3D pose.

[0088] In practice, 2D features are mapped to a 3D point cloud space to obtain 3D features. Based on the 3D features and the point cloud boundaries, different human body types are classified and 3D human skeletons are reconstructed. The segmented point cloud boundaries represent the human body boundaries, while the features include the key point information at the current moment. Connecting all 3D feature points within the point cloud boundaries that have prior connectivity relationships yields the 3D pose.

[0089] It should be noted that the process of fusing joint nodes from 3D features across all perspectives includes:

[0090] Using a radius-based spatial linear interpolation fusion algorithm, joint nodes of the same human body are fused from each viewpoint according to a pre-set radius threshold.

[0091] Figure 4 This is a schematic diagram illustrating a 3D pose construction process according to an exemplary embodiment. See also: Figure 4 In practical applications, let the perspective i be denoted as V. i Character j is P j 3D key point k is G k Due to limitations in the shooting range, V i Next P j It may or may not occur; considering the impact of occlusion, P j G k Possible omissions. These will all lead to G in the final point cloud space. k The total number is less than N*M (N is P) j The number of cameras (M is the number of cameras). Ideally, the subject should not be obstructed and should appear in the view of every camera for G to be considered. k The quantity equals N*M. First, we need to consider the proximity relationships between multiple perspectives of the same person (G). k Perform fusion. Assume that G of the same person after epipolar mapping k Definitely more than different people's G k Inspired by radius-based outlier removal algorithms, a radius threshold R can be pre-set for each viewpoint P. j G k Constraining a sphere of radius R, we fuse semantically related keypoints within the sphere (e.g., using spatial linear interpolation). A well-set threshold R can yield good results. This applies to all viewpoints V. i G below k Perform radius-based spatial linear interpolation fusion to generate G′ k ,Right now:

[0092] G′ k =f(V i G k ), 0 <i≤M

[0093] Where f is the radius-based linear interpolation fusion algorithm; M is the number of cameras.

[0094] Now 3D joint G′ k Only semantic information is available, meaning only information labels for the head, neck, and hands. The next step is to combine the G′ tags from different individuals. k Separation. Using the segmented 3D point cloud described above, calculate the point cloud boundary (BPA algorithm or Poisson reconstruction), and the G′ within the boundary. k That is, P j Finally, the joints are connected according to semantic relationships (head to neck, hand to elbow, etc.) to obtain the 3D pose (3D skeleton).

[0095] It should be noted that the calculated point cloud boundary includes:

[0096] Based on the 3D point cloud segmented from the point cloud, the surface boundary of the N clusters of point clouds at a distance X from the joint node is calculated, and then the surface boundary is used as the human body boundary.

[0097] In practice, since calculating the complete point cloud boundary is computationally expensive and not all boundaries are useful, the calculation of the point cloud boundary can be optimized to only reconstruct the segmented G′. k The nearby N clusters of point clouds (N is the number of point cloud clusters to be retained for each specified node, and the distance G′ of each cluster is selected). k Find the surface boundary of the nearest n points and determine G′. k This can be used to explain G′ on the inside or outside of the curved surface. k Which P does it belong to? j .

[0098] It should be noted that after the 3D pose construction, the following steps are also included:

[0099] The constructed 3D posture is corrected based on the principle of human inverse kinematics so that the 3D posture conforms to ergonomic constraints.

[0100] After constructing the 3D pose, the resulting 3D pose can be further optimized by incorporating inverse kinematics principles for correction. Inverse kinematics can parameterize the 3D skeleton, and the skeleton parameters can be tuned using relevant formulas to make the final skeleton model more ergonomic and more intuitive and natural.

[0101] The biggest challenge in multi-view, multi-person 3D pose estimation is the multi-person matching problem. Reasonable compensation must be made for the loss of joint point information caused by shooting angles and occlusion. To address this core issue, this invention utilizes a point cloud instance segmentation algorithm to define the joint point positions of different characters using point cloud boundaries, achieving differentiated matching. This method is driven by discrete point cloud data, offering greater flexibility and fault tolerance compared to traditional matching methods that rely on mathematical formulas. Furthermore, the optimization method for point cloud boundaries can artificially reduce the required computational power, ensuring a certain level of real-time performance. This method does not require additional model training, does not rely on large datasets, and can directly use the latest object detectors as the raw input for the entire method, solving the multi-person matching problem through the aforementioned algorithms.

[0102] This invention utilizes a down-top framework, which is simple and easy to reason about. The presence of multiple viewpoints ensures that even if one viewpoint is occluded, depth information from other viewpoints can compensate. This invention uses only point cloud data for boundary constraints to obtain prior information, rather than using the point cloud boundaries as the final output, resulting in better occlusion handling. In the final step, the semantic information of 3D joints is used to directly reconstruct the 3D human skeleton, a very fast method. To output a suitable skeleton pose, a physical model is introduced for correction, making it more ergonomic.

[0103] Example 2

[0104] Figure 5 This is a schematic block diagram illustrating a multi-view, multi-person 3D pose estimation device according to an exemplary embodiment. See also: Figure 5 A multi-view, multi-person 3D pose estimation device is provided, comprising:

[0105] The information acquisition module 101 is used to acquire image information and depth information of the target being photographed using multi-view acquisition technology.

[0106] The semantic feature module 102 is used to perform single-view character segmentation based on the image information and extract 2D semantic features.

[0107] The point cloud construction module 103 is used to reconstruct the point cloud based on the depth information to obtain a 3D point cloud in a preset world coordinate system.

[0108] The point cloud segmentation module 104 is used to segment the point cloud based on the 3D point cloud, with the target being different human bodies, and calculate the point cloud boundary as the human body boundary.

[0109] Feature mapping module 105 is used to map the 2D semantic features to the 3D point cloud to obtain 3D features;

[0110] The 3D pose construction module 106 is used to classify different human bodies and construct 3D poses based on the 3D features and the human body boundaries.

[0111] It is understood that the technical solution provided in this embodiment acquires image and depth information of the target object using multi-view acquisition technology through the information acquisition module 101; performs single-view character segmentation based on the image information through the semantic feature module 102 and extracts 2D semantic features; reconstructs the point cloud based on the depth information through the point cloud module 103 to obtain a 3D point cloud in a preset world coordinate system; segments the point cloud for different human figures based on the 3D point cloud and calculates the point cloud boundaries as human body boundaries; maps the 2D semantic features to the 3D point cloud through the feature mapping module 105 to obtain 3D features; and classifies different human figures and constructs 3D poses based on the 3D features and human body boundaries through the 3D pose construction module 106. This technical solution can segment the point cloud for different human figures based on the 3D point cloud, accurately define the joint positions of different figures, and achieve differentiation and matching of different figures. Furthermore, based on discrete point cloud data, it has better flexibility and fault tolerance.

[0112] Example 3

[0113] A multi-view, multi-person 3D pose estimation device is provided, including:

[0114] The main controller and the memory connected to the main controller;

[0115] The memory stores program instructions;

[0116] The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0117] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0118] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0119] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0120] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0122] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0123] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0124] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0125] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-view, multi-person 3D pose estimation method, characterized in that, include: Multi-view acquisition technology is used to obtain image and depth information of the target being photographed; Based on the image information, perform single-view character segmentation and extract 2D semantic features; Based on the depth information, point cloud reconstruction is performed to obtain a 3D point cloud in a preset world coordinate system; Based on the 3D point cloud, point cloud segmentation is performed for different human bodies, and the point cloud boundary is calculated as the human body boundary. Mapping the 2D semantic features to the 3D point cloud to obtain 3D features includes: mapping all joint nodes to the 3D point cloud based on the 2D coordinates, 2D semantic features, and pre-stored camera parameter position information of each joint node in each viewpoint, using the epipolar geometry principle to obtain 3D features for each viewpoint; fusing the joint nodes in the 3D features of all viewpoints to obtain 3D features that include all joint node information at the current moment and the prior connectivity relationships of the joint nodes. Based on the 3D features and the human body boundaries, different human bodies are classified and their 3D poses are constructed.

2. The method according to claim 1, characterized in that, The process of segmenting a person from a single view and extracting 2D semantic features includes: Detection boxes for different human bodies are identified from image information from each viewpoint; For each bounding box, a 2D heatmap is estimated using a 2D pose estimation network; The 2D coordinates and probability map of each joint node are obtained based on the 2D heat map. Based on the probability map, 2D semantic features are obtained using a deep convolutional neural network.

3. The method according to claim 2, characterized in that, The step of segmenting the 3D point cloud into point cloud segments for different human bodies and calculating the point cloud boundaries as human body boundaries includes: The 3D point cloud is processed using the core attention mechanism of the Transformer architecture to segment the point cloud for different human bodies, and the point cloud boundary is calculated as the human body boundary.

4. The method according to claim 3, characterized in that, The process of fusing joint nodes from 3D features across all perspectives includes: Using a radius-based spatial linear interpolation fusion algorithm, joint nodes of the same human body are fused from each viewpoint according to a pre-set radius threshold.

5. The method according to claim 3, characterized in that, The step of classifying different human bodies and constructing 3D poses based on the 3D features and the human body boundaries includes: Based on the human body boundary, all joint node information is divided to obtain the joint nodes within each human body boundary; Connect the joint nodes within each human body boundary according to the prior connection relationship to construct the 3D pose.

6. The method according to claim 4, characterized in that, The calculation yields the point cloud boundary, including: Based on the 3D point cloud segmented from the point cloud, the surface boundary of the N clusters of point clouds at a distance X from the joint node is calculated, and then the surface boundary is used as the human body boundary.

7. The method according to claim 1, characterized in that, Following the 3D pose construction, the following is also included: The constructed 3D posture is corrected based on the principle of human inverse kinematics so that the 3D posture conforms to ergonomic constraints.

8. A multi-view, multi-person 3D pose estimation device, characterized in that, include: The information acquisition module is used to acquire image and depth information of the target object using multi-view acquisition technology. The semantic feature module is used to segment people in a single view based on the image information and extract 2D semantic features. The point cloud construction module is used to reconstruct the point cloud based on the depth information to obtain a 3D point cloud in a preset world coordinate system. The point cloud segmentation module is used to segment the point cloud based on the 3D point cloud, with the target being different human bodies, and calculate the point cloud boundary as the human body boundary. The feature mapping module is used to map the 2D semantic features to the 3D point cloud to obtain 3D features. This includes: mapping all joint nodes to the 3D point cloud based on the 2D coordinates, 2D semantic features, and pre-stored camera parameter position information of each joint node in each viewpoint, using the epipolar geometry principle to obtain 3D features for each viewpoint; and fusing the joint nodes in the 3D features of all viewpoints to obtain 3D features that include all joint node information at the current moment and the prior connectivity relationships of the joint nodes. The 3D pose construction module is used to classify different human bodies and construct 3D poses based on the 3D features and the human body boundaries.

9. A multi-view, multi-person 3D pose estimation device, characterized in that, include: The main controller and the memory connected to the main controller; The memory stores program instructions; The main controller is used to execute program instructions stored in the memory and perform the method as described in any one of claims 1 to 7.

Images