Multi-view-angle three-dimensional human body posture estimation method based on dynamic time sequence fusion and view angle selection

Abstract

The invention discloses a multi-view-angle three-dimensional human body posture estimation method based on dynamic time sequence fusion and view angle selection, belongs to the technical field of human body posture estimation, and aims to solve the problems of insufficient accuracy and poor robustness when a traditional method is used for processing complex scenes such as serious shielding, rapid movement and large view angle quality difference. The method comprises the following steps: firstly, constructing a space-time correlation model based on feature difference dynamic time sequence fusion, and realizing adaptive adjustment of time sequence fusion weight by introducing feature difference measurement and a motion state judgment mechanism between adjacent frames and combining a time attenuation factor and inter-frame motion consistency; secondly, designing a dynamic selection strategy based on view angle quality evaluation, and constructing a multi-view angle weight distribution model; and finally, realizing high-precision three-dimensional human body posture reconstruction through cross-view feature alignment and space-time consistency constraint.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human pose estimation, and particularly relates to a multi-view three-dimensional human pose estimation method based on dynamic time sequence fusion and view angle selection. BACKGROUND

[0002] Human pose estimation is a core research direction in the field of computer vision, aiming to locate human key points (such as shoulders, elbows, wrists, hips, knees, ankles, etc.) from images or videos through algorithms, and to construct a human skeleton model to describe the overall pose. Its technical background can be traced back to early motion capture technology based on marker point system, but traditional methods rely on expensive equipment and are difficult to extend to daily scenes. With the breakthrough of deep learning technology, human pose estimation has achieved a leap from the laboratory to the real world.

[0003] As an important research direction in the field of computer vision, human pose estimation can be divided into two-dimensional human pose estimation (2D Human Pose Estimation) and three-dimensional human pose estimation (3D Human Pose Estimation) according to its dimensionality characteristics. Two-dimensional human pose estimation focuses on accurately extracting human key point positions from two-dimensional plane images, such as head, shoulder, elbow, wrist, hip, knee and ankle, etc. By constructing the topological connection between these key points, a two-dimensional representation of the human skeleton is formed. Two-dimensional human pose estimation technology shows wide application in action recognition, human-computer interaction, virtual reality and other fields, providing a basic understanding of human pose for computers.

[0004] Compared with two-dimensional techniques, three-dimensional human pose estimation further delves into three-dimensional space, aiming to recover the precise positions of human key points in a three-dimensional coordinate system from images or video sequences. This technique breaks through the limitations of two-dimensional representation, enabling the capture of depth information of human poses such as limb extension and joint rotation angles, providing computers with more rich and accurate pose descriptions. The implementation of three-dimensional human pose estimation often requires the combination of multi-view images, depth sensor data or depth cues in monocular images, using deep learning models such as three-dimensional convolutional neural networks (3D CNNs), graph convolutional networks (GCNs) and model-based methods (such as parameterized human models) for three-dimensional pose reconstruction. Its application fields cover motion capture, sports analysis, medical rehabilitation, etc., and it has important significance for understanding human behavior, motion intention and conducting fine human motion analysis. As an important research direction in the intersection of computer vision and artificial intelligence, three-dimensional human pose estimation technology can be divided into single-view human pose estimation and multi-view human pose estimation according to the number of cameras used. Single-view methods face significant challenges in practical applications, the most prominent of which is the occlusion problem. Due to the diversity of human actions and the complexity of the scene, the human body often appears to be occluded by external objects (such as furniture, walls, etc.) or self-occluded (such as arms occluding other parts of the body) during movement. These occlusions cause key point information to be missing or blurred in the image, leading to ambiguity in pose estimation.

[0005] In contrast, multi-view human pose estimation deploys a camera array composed of multiple cameras of the same configuration, Figure 1Multi-camera array system, which captures images of the same scene from multiple angles at the same time. This method makes full use of multi-view information and can capture the details of human posture from different angles, thus obtaining more comprehensive information of the scene. Since each camera provides image data from a different angle, when occlusion occurs in a certain view, other views may still provide visible information of the key points. This redundancy helps to eliminate the ambiguity caused by occlusion. For example, in the view where the arm is occluded, other views may clearly show the position and angle of the arm. By fusing these multi-view information, the model can more accurately estimate the three-dimensional posture of the human body. In addition, multi-view methods can also provide rich depth clues, which help the model better understand the spatial structure and motion relationship of the human body, further improving the accuracy and robustness of posture estimation. In the case of extreme situations where a certain joint is invisible in multiple view images, temporal consistency needs to be used because the human posture does not change much in a very short time. The addition of temporal information in the multi-view human posture estimation method is achieved through multi-time frame fusion. The core goal of time frame fusion is to improve the accuracy, robustness and continuity of posture estimation by using the temporal information between adjacent frames. SUMMARY

[0006] The purpose of the present application is to provide a dynamic temporal fusion and view selection multi-view three-dimensional human posture estimation method that overcomes the accuracy and robustness problems of traditional methods in handling complex scenes such as severe occlusion, fast motion and large view quality difference.

[0007] The dynamic temporal fusion and view selection multi-view three-dimensional human posture estimation method of the present application comprises the following steps:

[0008] 1) Obtain multi-view human posture images, the specific content is as follows:

[0009] Use a multi-camera array to capture human posture in the scene from multiple angles. Set the number of cameras used for scene capture as N, where N is the number of view images, and name the view images as view 1, view 2, …, view N in turn;

[0010] 2) Input the N view images collected in step 1) into a two-dimensional human posture estimation network, such as ResNet, OpenPose, HRNet, etc., to extract key features from the two-dimensional images and generate heat maps or two-dimensional human joint coordinates;

[0011] 3) Use the temporal information between adjacent frames to improve the accuracy, robustness and continuity of posture estimation. Time sequence fusion can make up for the shortcomings of single frame detection by combining multi-frame information, providing more comprehensive context information to improve the accuracy of posture estimation, including the following steps:

[0012] 3.1 Feature extraction is performed on the input multi-view images to obtain feature vectors that can fully represent the pose information of the frame. Then, for the feature vectors of adjacent frames, the feature difference between them is calculated. Dynamic quantization of the feature difference between adjacent frames. The Euclidean distance is used as the measurement standard for quantizing the feature difference between adjacent frames. Given three consecutive frames 、 and , which are the feature representations of three consecutive frames, the Euclidean distance D is represented as:

[0013]

[0014]

[0015]

[0016] where n is the dimension of the feature vector; and are the feature differences between the current frame and the previous frame , and the feature differences between the previous two frames and . The change rate between and is calculated using the following formula

[0017]

[0018] where: is a very small number, when , the current frame difference is larger than the previous adjacent frame difference, and it is determined as a high dynamic scene; when , the current frame difference is reduced, and it is determined as a low dynamic scene; is larger, the dynamic change is significant. If the current scene is a low dynamic scene, multi-time sequence fusion based on time decay and inter-frame difference is performed, and the fusion weight is determined according to the difference values and size dynamic allocation of weight and time decay. The weight formula is as follows:

[0019]

[0020] where: is the current frame; is the historical frame; is the time decay coefficient; is the time step. In this way, the frame with small difference is given higher weight, and the frame with large difference is given lower weight. In the process of assigning weight, the normalized weight is used, and the fused frame image is obtained according to the time sequence fusion formula after the weight is obtained according to the feature difference. The fusion formula is as follows:

[0021]

[0022] wherein: is the fused heat map; n is the number of fused images; is the fusion weight; is the feature image of the continuous frame. In the whole fusion process, it is hoped that the frame with small feature difference is given higher fusion weight, and the frame with large difference is given smaller fusion weight.

[0023] 4) In the multi-view fusion process, the view selection is performed, and several views with better quality are selected for fusion, including the following steps:

[0024] 4.1 Sort all views of each joint node according to the confidence, and set the input confidence tensor as:

[0025]

[0026] wherein, B is the batch size, J is the number of joint nodes, and V is the number of views. X is reshaped as

[0027] According to the confidence value, all views corresponding to each joint node are sorted in descending order, and a sorted index matrix S is obtained, and the sorting formula is:

[0028]

[0029] 4.2 According to the sorted index S, the first n views with the highest joint node confidence are selected. The index of the selected view is obtained through slicing operation

[0030] 4.3 Create a Boolean mask to identify the selected view.

[0031] 4.4 Apply the mask to the original confidence tensor X, and set the confidence of the unselected view to 0, and keep the confidence of the selected view. The formula for applying the mask is:

[0032]

[0033] wherein, is the confidence tensor of the selected view, is the original confidence tensor, is the Boolean mask.​​​​​​

[0034] 4.5 The selected multi-view perspective image is put into the multi-view fusion formula based on polar geometry, and the following adaptive weight multi-view fusion formula is obtained:

[0035] ;

[0036] Wherein, is the fused heat map, indicates the adaptive weight of perspective fusion.

[0037] 5) The three-dimensional coordinates of the human body key points are calculated by triangulation method using the two-dimensional key point coordinates and the camera parameters under multi-view.

[0038] The beneficial effects of the present application are that the ambiguity of human posture estimation caused by occlusion under single view is effectively solved, and the accuracy is insufficient when dealing with complex scenes such as fast motion and large view quality difference, so that the estimation accuracy of human joint point coordinates is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is a schematic diagram of a multi-camera array system.

[0040] Figure 2 It is a workflow diagram of the present method.

[0041] Figure 3 It is a camera distribution diagram of Occlusion-Person dataset and Human3.6M dataset.

[0042] Wherein: (a) is the camera distribution diagram of Occlusion-Person dataset; (b) is the camera distribution diagram of Human3.6M dataset.

[0043] Figure 4 It is a schematic diagram of feature difference dynamic timing fusion process.

[0044] Figure 5 It is a network structure diagram of joint point confidence.

[0045] Figure 6 It is a multi-view fusion diagram based on polar geometry. DETAILED DESCRIPTION

[0046] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present application.

[0047] As Figure 2As shown, the present invention can be divided into several steps, including 2D human pose estimation, dynamic temporal fusion based on feature difference, viewpoint selection, and multi-viewpoint fusion based on epipolar constraints.

[0048] 1) Acquire multi-view human pose images

[0049] Multi-view human pose images are captured by a multi-camera array arranged in a specific order. This array primarily captures human poses from multiple perspectives within a scene. Let N be the number of cameras capturing the scene; N represents the number of viewpoint images, which are then named sequentially as Viewpoint 1, Viewpoint 2, ..., Viewpoint N. Commonly used multi-view human pose datasets include the Human3.6M dataset, the CMU Panoptic dataset, and the Occlusion-Person dataset. Figure 3 Camera distribution maps for the Occlusion-Person dataset and the Human3.6M dataset.

[0050] 2) After completing the multi-view image acquisition step, N view images covering different angles of the target human body were obtained. In order to extract key human pose representations from these two-dimensional images, these N view images were input into a pre-trained two-dimensional human pose estimation network, such as ResNet, OpenPose, HRNet, etc. Here, ResNet is mainly used. Through deep residual connections, multi-scale feature fusion, and heatmap regression mechanism, key representations are efficiently extracted from two-dimensional images and joint coordinates are generated. Its core process can be divided into four stages: feature extraction, multi-scale fusion, heatmap generation, and coordinate decoding. Key representations are extracted from two-dimensional images to generate heatmaps or two-dimensional human joint coordinates.

[0051] 3) Utilizing temporal information between adjacent frames improves the accuracy, robustness, and continuity of pose estimation. Temporal fusion, by combining information from multiple frames, can compensate for the shortcomings of single-frame detection, providing more comprehensive contextual information and thus improving the accuracy of pose estimation. Key points lost in a single frame due to occlusion (such as elbows and ankles when the human body overlaps) can have their positions inferred from the motion trajectories of preceding and following frames. This mainly involves extracting rich spatiotemporal context from multi-frame information, solving the occlusion problem in a single frame, and significantly improving the accuracy of pose estimation. For example... Figure 4 The diagram shown is a schematic of the entire dynamic temporal fusion process based on feature differences.

[0052] 3.1 First, feature extraction is performed on the input multi-view images to obtain feature vectors that can fully represent the pose information of the frame. Then, feature differences are calculated for the feature vectors of adjacent frames. The core of this process is to capture the dynamic changes in human pose between different frames. By comparing the feature vectors of adjacent frames, the evolution of human pose on the time axis is quantified, providing basic data for subsequent time-series analysis. To more accurately describe the feature differences between adjacent frames, a dynamic quantization method is adopted. Specifically, Euclidean distance is selected as the metric for quantifying the feature differences between adjacent frames. Euclidean distance can intuitively reflect the distance between two feature vectors in multi-dimensional space, thus effectively measuring the similarity or change range of human pose between different frames. For a given three consecutive frames... , and , where are the feature representations of three consecutive frames, and the Euclidean distance D is expressed as:

[0053] ;

[0054] ;

[0055] ;

[0056] Where: n is the dimension of the feature vector; and These are the current frames. With the previous frame Feature difference and the first two frames and The characteristic difference between them. Calculated using the following formula. and rate of change between

[0057] ;

[0058] in: It is an extremely small number, when When the difference between the current frame and the previous adjacent frame is greater, it is determined to be a high dynamic range scene; when When the difference between the current frames decreases, it is determined to be a low dynamic scene; When the value is large, the dynamics change significantly. If the current scene is a low-dynamic scene, multi-temporal fusion based on time decay and inter-frame differences is performed, according to the difference value. and The fusion weights are determined by dynamically allocating weights based on size and time decay. The weight formula is as follows:

[0059] ;

[0060] in: It is the current frame; It is a historical frame; It is the time decay coefficient; This refers to the time step. This approach assigns higher weights to frames with smaller differences and lower weights to frames with larger differences. Normalized weights are used in the weight allocation process. After obtaining the weights based on feature differences, the fused image of the frame is obtained according to the temporal fusion formula. The fusion formula is as follows:

[0061] ;

[0062] in, This is the fused heatmap, where n is the number of fused images. To integrate weights, The feature images are consecutive frames. Throughout the fusion process, it is desirable to assign higher fusion weights to frames with small feature differences and lower fusion weights to frames with large feature differences.

[0063] 4) Select perspectives during the multi-perspective fusion process and choose several perspectives with better quality for fusion.

[0064] 4.1 First, sort all viewpoints for each joint according to their confidence level. Figure 5 This is a schematic diagram of the network structure for keypoint confidence, given input Zhang Where B is the batch size, J is the number of keypoints, V is the number of viewpoints, and X is reshaped into... Based on the sorting index, select the top n views with the highest confidence for each keypoint. Sort the views for each keypoint in descending order of confidence to obtain the sorting index. The formula for sorting the index is:

[0065]

[0066] Then, based on the sorting index S, select the top n viewpoints with the highest keypoint confidence. The index of the selected viewpoint is obtained through slicing operations. .

[0067] Next, create a boolean mask. , used to identify the selected viewpoint.

[0068] Applying a mask to the original confidence tensor X sets the confidence of unselected views to 0, while retaining the confidence of selected views. The formula for applying the mask is:

[0069] ;

[0070] in: For the confidence tensor of selected viewpoints; The original confidence tensor; Use a Boolean mask;

[0071] Finally, the selected multi-view images are fused using epipolar geometry. Figure 6 This is a multi-view fusion diagram based on epipolar geometry, from which the following adaptive weight multi-view fusion formula is obtained:

[0072] ;

[0073] in: It is a heat map fused from multiple perspectives; It is a heatmap after time-series fusion; Represents the adaptive weights for view fusion;

[0074] 5) Using the coordinates of two-dimensional key points from multiple perspectives and camera parameters, the three-dimensional coordinates of human body key points are calculated through triangulation.

[0075] Triangulation, based on multi-view geometry, reconstructs the 3D coordinates of a point by observing it from multiple perspectives and using the inverse process of a camera projection model. It extracts the 2D coordinates of key human body points (such as shoulders, elbows, and wrists) from multi-view images, along with camera parameters including the intrinsic matrix K (focal length, principal point coordinates) and the extrinsic matrix. Construct the projection equation.

[0076] For the i-th viewpoint, the two-dimensional coordinates With three-dimensional coordinates The relationship satisfies:

[0077] ;

[0078] in: Using the scale factor, expansion yields two equations:

[0079] ;

[0080] in: and Rotation matrix The first two lines; and Translation vector The first two components.

[0081] For N viewpoints (N not less than 2), each viewpoint provides 2 equations, for a total of 2N equations. These can be solved using Direct Linear Transformation (DLT) or Singular Value Decomposition (SVD). If some viewpoints erroneously detect 2D keypoints, outliers can be removed using the Random Sample Consensus (RANSAC) algorithm to improve estimation robustness.

[0082] To address the shortcomings of traditional methods in terms of accuracy and robustness when dealing with complex scenes such as severe occlusion, rapid movement, and large differences in viewpoint quality, this invention proposes a multi-view 3D human pose estimation method with dynamic temporal fusion and viewpoint selection. This method improves the accuracy of human pose estimation to a certain extent. At the same time, the number of cameras can be flexibly adjusted according to the actual situation to achieve a balance between computational efficiency and estimation accuracy.

Claims

1. A method for multi-view 3D human pose estimation with dynamic temporal fusion and view selection, characterized in that Comprise the following steps: S1. Obtain multi-view human pose images, the specific content is as follows: Use a multi-camera array to collect human poses in a scene from multiple perspectives, set the number of cameras used for scene collection as N, the value of N is the number of perspective images, and the perspective images are named as perspective 1, perspective 2, …, perspective N in turn; S2. Input the N perspective images collected in step 1.1 into a two-dimensional human pose estimation network, such as ResNet, OpenPose, HRNet, etc., to extract key features from two-dimensional images to generate heat maps or two-dimensional human joint coordinates; S3. Use the temporal information between adjacent frames to improve the accuracy, robustness and continuity of pose estimation. Temporal fusion can make up for the shortcomings of single-frame detection by combining multi-frame information to provide more comprehensive context information, thereby improving the accuracy of pose estimation, including the following steps: S3.1 Feature extraction is performed on the input multi-view images to obtain feature vectors that can fully represent the pose information of the frames. Then, for the feature vectors of adjacent frames, the feature differences between them are calculated. Dynamic quantization of the feature differences between adjacent frames is performed. The Euclidean distance is used as the measurement standard for quantizing the feature differences between adjacent frames. Given three consecutive frames , and , which are the feature representations of the three consecutive frames, respectively. The Euclidean distance D is represented as: ； ； ； wherein: n is the dimension of the eigenvector; and are the feature difference between the current frame and the previous frame and the feature difference between the previous two frames and respectively; the rate of change between and is calculated by the following equation: ； wherein: is a very small number, when the current frame difference is greater than the previous adjacent frame difference, it is determined as a high dynamic scene; when the current frame difference is reduced, it is determined as a low dynamic scene; is larger, the dynamic changes significantly; if the current scene is a low dynamic scene, a multi-time sequence fusion based on time decay and inter-frame difference is performed, and the fusion weight is determined according to the difference value and the size of the dynamic allocation weight and the decay of time, the weight formula is as follows: ； wherein: is the current frame; is the history frame; is a temporal decay coefficient; is a time step; this is done so that frames with small differences are given higher weights and frames with large differences are given lower weights; In the process of assigning weights, normalized weights are used, and after obtaining the weights according to the feature difference, the fused image of this frame is obtained according to the temporal fusion formula, and the fusion formula is as follows: ； wherein: is the fused heat map; n is the number of fused images; is the fusion weight; is the feature image of the continuous frame; during the entire fusion process, the frame with small feature difference is given a higher fusion weight, and the frame with large difference is given a smaller fusion weight; S4. Select a few good perspectives for fusion in the multi-perspective fusion process, including the following steps: S4.1 Sort all perspectives of each joint according to confidence, set the input confidence tensor as: ； where: B is the batch size; J is the number of junctions; V is the number of views, reshaping X to ; According to the confidence value, all the view angles corresponding to each joint point are sorted in descending order to obtain a sorting index matrix The sorting index formula is: ； S4.2 According to the sorting index S, the top n view angles with the highest node confidence are selected, and the indexes of the selected view angles are obtained through a slicing operation ; S4.3 Create a Boolean mask identifying the selected view; S4.4 Apply the mask to the original confidence tensor X, set the confidence of the unselected perspective to 0, and retain the confidence of the selected perspective, the formula for applying the mask is: ； wherein: is a confidence tensor for picking over the view angle; is a raw confidence tensor; is a Boolean mask; S4.5 The selected multi-perspective perspective images are subjected to a multi-perspective fusion formula based on polar geometry, and the following multi-view fusion formula with adaptive weight is obtained: ； wherein: is a thermal map after multi-view fusion; is a thermal map after temporal fusion; represents an adaptive weight of view fusion; S5. Use the two-dimensional key point coordinates and camera parameters under multi-perspective to calculate the three-dimensional coordinates of the human key points through triangulation method.

Images