3D human body posture estimation method based on federal learning

Abstract

The invention relates to a 3D human body posture estimation method based on federated learning. The 3D human body posture estimation method comprises the following steps: S1) establishing a federated learning model; s2) obtaining a 2D human body image and cutting the 2D human body image; s3) constructing a 2D backbone network; s4) taking human joints as key points of the 2D human body image, inputting the key points into the 2D backbone network, and generating a high-dimensional feature map g theta, an intermediate heat map f theta and a combined heat map; s5) calculating the joint heat map value of each camera under the respective view angle, and taking the maximum value coordinate as the 2D coordinate of the key point; synthesizing the 2D coordinates and the projection matrix of the camera into a matrix; s6) calculating the confidence of each participant; s7) aggregating the 2D coordinate weights, and calculating 3D coordinates of the key points of the human body; and S8) connecting the 3D coordinates of the human body key points to obtain a 3D human body posture sequence. According to the method, the problem that the application in an actual scene is limited due to insufficient robustness of an existing human body posture estimation method is solved.

Description

Technical Field

[0001] The present application belongs to the technical field of image motion analysis, and specifically relates to a 3D human posture estimation method based on federated learning. Background Art

[0002] 3D human posture estimation technology aims to restore the posture information of the human body in three-dimensional space from two-dimensional images or videos. This technology can play a key role in human-computer interaction, virtual reality, motion analysis and other fields. It can provide computers with richer and more accurate scene understanding.

[0003] With the development of deep learning, 3D human pose estimation technology has also made significant progress. However, when the movements are too complex or there is occlusion, the existing technology is not robust enough, which limits its application in practical scenarios. Summary of the invention

[0004] The present application provides a 3D human pose estimation method based on federated learning, which solves the problem of insufficient robustness of the prior art when applying deep learning to human pose estimation.

[0005] A 3D human posture estimation method based on federated learning includes the following steps:

[0006] S1) establishing a federated learning model, wherein the federated learning model includes a central server and a plurality of participants consisting of cameras;

[0007] S2) obtaining 2D human body images captured by a camera at several viewing angles, and cutting the 2D human body images to a standard size;

[0008] S3) constructing a 2D backbone network, wherein the 2D backbone network sequentially includes a ResNet-152 feature extractor without a fully connected layer and a pooling layer, a transposed convolution layer, and a 1*1 convolution layer;

[0009] S4) taking the atlantoaxial joint, acromioclavicular joint, elbow joint, hip joint, knee joint and ankle joint as the key points of the 2D human body image, inputting the 2D human body image of the standard size in step S2) into the 2D backbone network, and generating a high-dimensional feature map g by the ResNet-152 feature extractor θ , the feature map g is gradually increased by using three layers of transposed convolution layers θ The spatial resolution of the intermediate heat map is generated θ ; Through a 1×1 convolution layer, the intermediate heat map f θ Transformed into joint heatmap;

[0010] S5) Based on step S4), softmax is used to calculate the joint heat map value of each camera at its respective viewing angle, and then the center of mass of all joint heat map values ​​at each viewing angle is obtained by soft-argmax method, and the coordinates of the center of mass are used as the 2D coordinates of the key point; the 2D coordinates of the human body posture at each viewing angle and the projection matrix of the camera are combined into a matrix, which is recorded as A j ;

[0011] S6) constructing a participant confidence assessment network and calculating the confidence of each participant, using the confidence as the weight of each participant; the participant confidence assessment network is composed of a ResNet-152 network as an encoder subnetwork and a convolutional network with a learnable parameter φ;

[0012] S7) the central server performs weighted aggregation on the 2D coordinates of each participant and calculates the 3D coordinates of the key points of the human body based on linear algebraic trigonometry;

[0013] S8) Connecting the 3D coordinates of the key points of the human body in each frame image from top to bottom in sequence to obtain a 3D human body posture sequence that is coherent in time period.

[0014] Furthermore, in step S2), the camera's viewing angles include a front viewing angle, a rear viewing angle, a right viewing angle, a right front viewing angle, a right rear viewing angle, and a left rear viewing angle, each viewing angle having an angular height of 1.5 m and a distance of 3 m from the target person.

[0015] Furthermore, in step S5), the formula for normalizing the joint heat map values ​​of each camera at its respective viewing angle using softmax is as follows:

[0016]

[0017] In the formula, r x , r y Represents the x,y coordinate values ​​of the joint heat map; H c,j (r) represents the joint heat map value at the r coordinate on the joint heat map; α is the adjustment parameter of the joint heat map value; r = (r x ,r y );

[0018] Furthermore, in step S5), the soft-argmax method is used to find the centroid of all joint heat map values ​​under each viewing angle, and the coordinates of the centroid are used as the 2D coordinates of the key point. The formula is as follows:

[0019]

[0020] In the formula, x c,j Represents the 2D position coordinates of the jth joint at the cth viewing angle.

[0021] Furthermore, the convolutional network in step S6) includes two convolutional layers, one global average pooling layer and three fully connected layers.

[0022] Furthermore, in step S7), the formula for calculating the 3D coordinates of the key points of the human body according to linear algebra trigonometry is as follows:

[0023]

[0024] In the formula, y j is the homogeneous vector of the 3D coordinates to be solved, represents the Hadamard product, w j is the weight of each key point corresponding to each participant, w j =(w 1,j , w 1,j ,…,w C,j , w C,j ), where C represents the number of cameras; j represents the jth key point.

[0025] The technical solution of the present invention is as follows:

[0026] Due to the adoption of the above technical solution, the beneficial effects produced by this application are as follows:

[0027] 1. This application uses a federated learning architecture to obtain a 2D human pose estimation model through local client training, which can fully adapt to the unique perspectives and surrounding environmental conditions of each camera, and help improve the robustness of the model.

[0028] 2. In the process of weight aggregation and 3D coordinate calculation, this application fully considers the confidence of each camera, thereby significantly improving the accuracy and stability of the overall pose estimation. By calculating the confidence of each camera, this application further optimizes the performance of the federated learning model, making it applicable to multi-camera scenarios, and providing algorithmic support for solving practical problems.

[0029] 2. In the process of weight aggregation and 3D coordinate calculation, this application fully considers the confidence of each camera perspective, thereby significantly improving the accuracy and stability of overall pose estimation. By calculating the confidence of each camera perspective, this application further optimizes the performance of the federated learning model, making it applicable to multi-camera scenarios, and providing algorithmic support for solving practical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1A flow chart of a 3D human pose estimation method based on federated learning provided in this application;

[0032] Figure 2 This is the overall architecture diagram of federated learning in this application. DETAILED DESCRIPTION

[0033] Based on the background technology, this application provides a 3D human posture estimation method based on federated learning, as shown in the attached Figure 1 and attached Figure 2 As shown, the following steps are included:

[0034] S1) Establishing a federated learning model, wherein the federated learning model includes a central server and a plurality of participants consisting of cameras.

[0035] In this embodiment, there are six cameras, which respectively shoot the front view, rear view, right view, right front view, right rear view, and left rear view. The angular height of each view is 1.5m and the distance from the target person is 3m. The right front view, right rear view, and left rear view are located at 45° of their respective directions. The set of central servers and client devices is a federated learning model of S = {s1, s2, s3...sn}, where n≥2, and n is the number of cameras.

[0036] It should be noted that the viewing angle in the present application is relative to the position of the target person, and the viewing angle can be changed for different recognition requirements.

[0037] S2) obtaining 2D human body images captured by a camera at several viewing angles, and cutting the 2D human body images into a standard size.

[0038] Multiple cameras can take a number of consecutive frames of photos, that is, photos of the same frame from multiple angles can be obtained at one moment. The obtained photos are calibrated to obtain their internal parameter matrix K i and the external parameter matrix [R i |t i ], used for subsequent posture estimation and 3D reconstruction. The internal parameter matrix K i Describes the internal parameters of the camera, K i It can be expressed as:

[0039]

[0040] in, is the focal length of the i-th camera in the x direction; is the focal length of the i-th camera in the y direction; is the coordinate of the principal point of the i-th camera in the x direction; is the coordinate of the principal point of the i-th camera in the y direction.

[0041] External parameter matrix [Ri |t i ] describes the position and orientation of the camera in the world coordinate system, and is used to convert world coordinates into camera coordinates. The origin of the camera coordinate system is usually located at the optical center of the camera (camera principal point), the X axis is along the horizontal direction of the camera, from left to right; the Y axis is along the vertical direction of the camera, from top to bottom; the Z axis is along the line of sight of the camera, from the camera to the object. i |t i ] can be expressed as:

[0042] In the formula, the rotation matrix R is used to describe the rotation of the camera relative to the world coordinate system, that is, the orientation of the camera; the translation vector t describes the translation of the camera relative to the world coordinate system, that is, the position of the camera; t x , t y , t z , representing the position of the camera in the x, y and z directions respectively.

[0043] In this embodiment, the picture taken by the camera is cropped, and the size after cropping is 224 pixels*224 pixels.

[0044] S3) constructing a 2D backbone network, wherein the 2D backbone network sequentially includes a ResNet-152 feature extractor without a fully connected layer and a pooling layer, a transposed convolution layer, and a 1*1 convolution layer.

[0045] S4) taking the atlantoaxial joint, acromioclavicular joint, elbow joint, hip joint, knee joint and ankle joint as the key points of the 2D human body image, inputting the 2D human body image of the standard size in step S2) into the 2D backbone network, and generating a high-dimensional feature map g by the ResNet-152 feature extractor θ , the feature map g is gradually increased by using three layers of transposed convolution layers θ The spatial resolution of the intermediate heat map is generated θ ; Through a 1×1 convolution layer, the intermediate heat map f θ Transformed into joint heatmap;

[0046] S5) Based on step S4), softmax is used to calculate the joint heat map value of each camera at its respective viewing angle, and then the center of mass of all joint heat map values ​​at each viewing angle is obtained by soft-argmax method, and the coordinates of the center of mass are used as the 2D coordinates of the key point; the 2D coordinates of the human body posture at each viewing angle and the projection matrix of the camera are combined into a matrix, which is recorded as A j . Softmax is used to calculate the joint heat map value of each camera at its own perspective for normalization. The specific formula is as follows:

[0047]

[0048] In the formula, r x , r y Represents the x,y coordinate values ​​of the joint heat map; H c,j (r) represents the joint heat map value at the r coordinate on the joint heat map; α is the adjustment parameter of the joint heat map value, r = (r x ,r y ).

[0049] The soft-argmax method is used to find the centroid of all joint heat map values ​​at each viewing angle, and the coordinates of the centroid are used as the 2D coordinates of the key points. The formula is as follows:

[0050]

[0051] In the formula, x c,j Represents the 2D position coordinates of the jth joint at the cth viewing angle.

[0052] S6) constructing a participant confidence assessment network and calculating the confidence of each participant, using the confidence as the weight of each participant; the participant confidence assessment network is composed of a ResNet-152 network as an encoder subnetwork and a convolutional network with a learnable parameter φ.

[0053] Convolutional network q with learnable parameters φ φ It includes two convolutional layers, one global average pooling layer and three fully connected layers. Input image I c After the features are extracted by ResNet-152 network, it is g θ (I c ), and then through the convolutional network q φ Input is w c,j =(q φ (g θ (I c )) j .

[0054] S7) The central server performs weighted aggregation on the 2D coordinates of each participant and calculates the 3D coordinates of the key points of the human body based on linear algebraic trigonometry.

[0055] The formula for calculating the 3D coordinates of the key points of the human body based on linear algebra trigonometry is as follows:

[0056]

[0057] In the formula, y j is the homogeneous vector of the 3D coordinates to be solved, represents the Hadamard product, w j is the weight of each key point corresponding to each participant, w j=(w 1,j , w 1,j ,…,w C,j , w C,j ), where C represents the number of cameras; j represents the jth key point.

[0058] The above formula can be solved by the differentiable singular value decomposition (SVD) algorithm: Finally, we can get U (left singular vector matrix), D (diagonal matrix containing singular values), V T (transpose of the right singular vector matrix); the last column of V is y j , for y j After normalization, the actual 3D coordinate Y can be obtained j .

[0059] S8) Connecting the 3D coordinates of the key points of the human body in each frame image at the same time from top to bottom in sequence to obtain a coherent 3D human body posture sequence in the time period. Finally, a coherent human body posture change sequence representing the entire action or movement process can be obtained.

[0060] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0061] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A 3D human pose estimation method based on federated learning, It is characterized in that The following steps are involved: S1) establishing a federated learning model, wherein the federated learning model includes a central server and a plurality of participants consisting of cameras; S2) obtaining 2D human body images captured by a camera at several viewing angles, and cutting the 2D human body images to a standard size; S3) constructing a 2D backbone network, wherein the 2D backbone network sequentially includes a ResNet-152 feature extractor without a fully connected layer and a pooling layer, a transposed convolution layer, and a 1*1 convolution layer; S4) taking the atlantoaxial joint, acromioclavicular joint, elbow joint, hip joint, knee joint and ankle joint as the key points of the 2D human body image, inputting the 2D human body image of the standard size in step S2) into the 2D backbone network, and generating a high-dimensional feature map g by the ResNet-152 feature extractor θ , the feature map g is gradually increased by using three layers of transposed convolution layers θ The spatial resolution of the intermediate heat map is generated θ ; Through a 1×1 convolution layer, the intermediate heat map f θ Transformed into joint heatmap; S5) Based on step S4), softmax is used to calculate the joint heat map value of each camera at its respective viewing angle, and the center of mass of all joint heat map values ​​at each viewing angle is obtained using the soft-argmax method, and the coordinates of the center of mass are used as the 2D coordinates of the key point; the 2D coordinates of the human body posture at each viewing angle and the projection matrix of the camera are combined into a matrix, denoted as A j ; S6) constructing a participant confidence assessment network and calculating the confidence of each participant, using the confidence as the weight of each participant; the participant confidence assessment network is composed of a ResNet-152 network as an encoder subnetwork and a convolutional network with a learnable parameter φ; S7) the central server performs weighted aggregation on the 2D coordinates of each participant and calculates the 3D coordinates of the key points of the human body based on linear algebraic trigonometry; S8) Connecting the 3D coordinates of the key points of the human body in each frame image from top to bottom in sequence to obtain a 3D human body posture sequence that is coherent in time period.

2. A 3D human body posture estimation method based on federated learning according to claim 1, It is characterized in that In step S2), the camera's viewing angles include the front viewing angle, the rear viewing angle, the right viewing angle, the right front viewing angle, the right rear viewing angle, and the left rear viewing angle. The angular height of each view is 1.5m and the distance from the target person is 3m.

3. The 3D human body posture estimation method based on federated learning according to claim 2, It is characterized in that In step S5), the formula for normalizing the joint heat map value of each camera at its viewing angle using the softmax function is as follows: In the formula, r x , r y Represents the x,y coordinate values ​​of the joint heat map; H c,j (r) represents the joint heat map value at the r coordinate on the joint heat map; α is the adjustment parameter of the joint heat map value; r = (r x ,r y ).

4. The method for 3D human posture estimation based on federated learning according to claim 3, It is characterized in that In step S5), the formula for taking the maximum value of all joint heat map values ​​under each viewing angle using the soft-argmax method is as follows: In step S5), the soft-argmax method is used to find the centroid of all joint heat map values ​​under each viewing angle, and the coordinates of the centroid are used as the 2D coordinates of the key point. The formula is as follows: In the formula, x c,j Represents the 2D position coordinates of the jth joint at the cth viewing angle.

5. The method for 3D human posture estimation based on federated learning according to claim 4, It is characterized in that The convolutional network in step S6) includes two convolutional layers, one global average pooling layer and three fully connected layers.

6. The method for 3D human posture estimation based on federated learning according to claim 5, It is characterized in that The formula for calculating the 3D coordinates of the key points of the human body according to linear algebra trigonometry in step S7) is as follows: In the formula, y j is the homogeneous vector of the 3D coordinates to be solved, represents the Hadamard product, w j is the weight of each key point corresponding to each participant, w j =(w 1,j , w 1,j ,…,w C,j , w C,j ), where C represents the number of cameras; j represents the jth key point.

Images