Multi-vision human body high-precision posture machine vision identification method based on simulation generation data training

Abstract

A multi-view-vision human body high-precision posture machine vision recognition method based on simulation generation data training is characterized by comprising the following steps that firstly, a twin system is used for generating multi-view-angle pictures and joint point labeling data; secondly, training a two-dimensional posture detection model, and accurately estimating three-dimensional coordinates of human body joint points through triangulation and a human body three-dimensional detection frame fine-grained division strategy in combination with space grids and feature projection fusion so as to determine a human body three-dimensional posture; thirdly, in order to solve the problems of joint point position jittering and limb length inconsistency in continuous detection, confidence constraint and time sequence filtering are adopted to optimize a detection result, and smoothness and stability of attitude data are ensured; and finally, generating a twinning digital human driving instruction through the detection data, realizing action simulation and multi-view data acquisition in a twinning environment, and forming a rich and diversified annotation data set. According to the invention, the efficiency and precision of three-dimensional human body posture detection are obviously improved.

Description

Technical Field

[0001] The present invention relates to a multi-view human posture recognition method. In particular, it is a technology that uses digital human data generated by a twin system as a training data set, especially for complex and occluded industrial environments, to achieve real-time and accurate recognition of human three-dimensional postures, specifically a multi-view human high-precision posture machine vision recognition method based on simulation-generated data training. Background Art

[0002] 3D human posture detection aims to reconstruct the 3D joint positions of the human body from images or videos, and is an important research direction in the field of computer vision and artificial intelligence. With the rapid development of motion capture, behavior analysis, sports rehabilitation and other fields, the application value of 3D posture detection has become increasingly prominent. This technology can not only accurately describe the spatial position of human joints, but also overcome the challenges brought by perspective changes and occlusion to a certain extent, providing more intuitive and rich information for analyzing and understanding human movements, and has broad application prospects. However, compared with 2D posture detection, 3D detection faces more technical difficulties in practical applications.

[0003] Monocular 3D human pose detection relies only on 2D images or videos from a single camera to estimate the 3D pose of the human body. In recent years, researchers have proposed a variety of technical solutions by combining deep learning, prior knowledge, and geometric constraints. Due to the lack of depth information, monocular detection faces challenges such as view limitation and occlusion in 3D reconstruction. Although the monocular method does not require multi-camera configuration and is convenient and adaptable, its accuracy is often limited by scene complexity and occlusion. In addition, the monocular method has uncertainty: the same 2D pose may correspond to multiple 3D poses, making the inference results unstable.

[0004] Multi-view 3D human posture detection uses multiple cameras to synchronously capture images from different perspectives, and uses triangulation and other methods to achieve 3D reconstruction. The advantage of the multi-view system is that it can effectively overcome the limitations of the monocular method in terms of lack of depth information and occlusion, greatly improving the accuracy of posture estimation. If the 2D posture estimation in multiple views is accurate, simple triangulation can achieve high-precision 3D posture reconstruction. However, in actual scenes, due to occlusion or noise, the 2D posture estimation may have errors, which affects the accuracy of the 3D posture.

[0005] Multi-view deep learning uses multi-view input to overcome occlusion and view limitations, and enhances the model's understanding and inference capabilities of three-dimensional space through data from different viewpoints. Mainstream datasets such as Human3.6M and Panoptic Studio provide multi-view images and human 3D joint annotations, which are widely used for model training. However, these datasets are generally collected in laboratory environments with a single background and stable lighting, resulting in insufficient generalization capabilities of the model in actual complex industrial scenarios. In addition, the annotation cost of multi-view datasets is high, especially the accurate 3D joint and skeleton annotations. At the same time, errors in data synchronization and camera calibration may further affect the annotation accuracy, limiting model performance.

[0006] In industrial scenarios, accuracy, real-time and stability are important indicators for 3D human posture detection. However, existing technologies mainly extract 3D posture from offline pictures or videos, making it difficult to achieve detection based on real-time camera input. At the same time, since the detection processes are independent of each other, the generated joint positions are prone to jitter, resulting in unstable limb lengths, which makes it difficult to meet the reliability and stability requirements of industrial scenarios.

[0007] In summary, improving the accuracy, stability and real-time performance of 3D human posture detection in industrial scenarios has become a key issue that needs to be solved urgently. Developing new methods to meet these challenges is of great significance to promoting the industrial application of 3D human posture detection technology. Summary of the invention

[0008] The purpose of this invention is to address the shortcomings of current 3D human posture detection in terms of accuracy, stability and real-time performance, and propose a multi-view human high-precision posture machine vision recognition method based on simulation-generated data training. It aims to solve the challenges currently faced in real-time 3D posture detection of multiple people in specific scenarios.

[0009] The technical solution of the present invention is:

[0010] A multi-view human body high-precision posture machine vision recognition method based on simulation-generated data training, characterized by:

[0011] First, the training data required for deep learning is generated through the twin system, including joint point coordinates, occlusion labels, camera parameters and multi-view images. Joint point coordinates refer to some key parts or feature points of the human body, which are used to represent the posture of the human body, including pixel coordinates and three-dimensional world coordinates. These points usually correspond to the joints or prominent parts of the human skeleton. By connecting these points, the skeleton structure of the human body can be constructed. The occlusion label represents the visibility of a joint point. The visibility of the joint point is represented by a number between 0 and 1, where 0 represents completely invisible and 1 represents completely visible. Multi-view images are human posture images taken by different cameras at the same time. The camera parameters represent the position of the camera in the world coordinate system and the mapping relationship from the three-dimensional world to the two-dimensional image. These parameters calibrate the corresponding relationship of the perspectives of different images. This multi-view posture data is mainly generated for specific scenes. In order to improve the detection capability of specific scenes but not limited to a certain scene, because the twin system can easily realize the background switching of different scenes, it can generate scene-specific data for different scenes.

[0012] Second, establish a two-dimensional posture detection module. According to the pixel coordinates in the human posture data, train a two-dimensional joint posture detection model to generate a heat map of each joint point. Since the human body is usually in a limited space, the approximate volume range of the human body in this space can be estimated based on the three-dimensional world coordinates of specific joint points, which can be represented by a rectangular stereo frame. According to the pixel coordinates of the specific joints obtained after detection, the actual three-dimensional world coordinates of these joints are obtained by triangulation. The human body detection stereo frame is estimated through these joint points as the input of the subsequent three-dimensional posture detection module.

[0013] Third, the human body space is divided into regular rectangular grids within the range of the human body detection stereo frame. These grid points can roughly represent the space where the human body is located, that is, there is a corresponding relationship between the joint points and the grid points, so the features of the generated grid points can ultimately restore the human body posture information. In order to generate the multi-view features of the spatial grid, the spatial grid is projected to each view through the camera parameters, and the heat map features of each view are fused to form a spatial feature with the position information of the human body joint points. The feature of each grid point represents the probability that the joint point is located here. Then, the posture detection model is used to detect the three-dimensional world coordinates of the human body from the spatial grid after feature projection fusion. This model projects the features of the spatial grid to the three coordinate planes specified by the world coordinate system respectively, thereby obtaining the two-dimensional plane features of the human body posture, and then detects the two-dimensional coordinates of the joint points in each coordinate plane from the two-dimensional plane features, and finally combines all two-dimensional coordinates to determine the three-dimensional posture of the human body.

[0014] Fourth, based on the above-mentioned human body three-dimensional detection module, the three-dimensional coordinates of the human body joints are obtained. However, since each detection is an independent process, for continuous image input, there will still be relative jitter in the joint position between frames, and the joint line, that is, the limb length will not be completely consistent. Therefore, the three-dimensional data of the joints is further optimized. Specifically, first, the limb length of the human body is determined based on the confidence of the detected joints, and the limb length with the highest confidence is used as the optimization target to achieve relative consistency of limb length at different times; secondly, for the problem of joint position jitter, the joints are smoothed and filtered from the time dimension to ensure smooth changes in the joint position at adjacent moments. Finally, more accurate and stable human posture detection data is obtained.

[0015] Fifth, the driving instructions required to control the twin digital human are generated through human posture detection data. Since driving the digital human requires determining the position of the human body and the rotation process between the limbs, the driving instructions mainly include the three-dimensional world coordinates of the joints and the quaternions of the limb rotations. The three-dimensional coordinates determine the positions of the digital human at different times, and the quaternions ensure the rotation of the digital human's limbs between various positions. While driving the digital human, a virtual camera is added to capture multi-view images of the digital human, and through the automatic annotation script, the posture annotation data can be automatically generated and exported, leaving an interface for the subsequent expansion of other scene data sets.

[0016] It can be seen that the present invention can realize real-time detection of three-dimensional human posture and generation of three-dimensional multi-view data sets for specific scenes, effectively overcoming the problems of low efficiency and data set limitation of existing three-dimensional posture detection.

[0017] The beneficial effects of the present invention are:

[0018] The present invention solves the challenges faced in real-time detection of three-dimensional human posture under multiple perspectives. In view of the problems of insufficient efficiency and accuracy of existing three-dimensional human posture detection, insufficient accuracy of data set annotation, and difficulty in expansion, a human posture machine vision recognition method based on data generated by a twin system is proposed. This method can train a three-dimensional human body detection model through multi-perspective human posture data in a twin environment to achieve real-time detection of three-dimensional human posture, and further optimize and process the detection data after the detection is completed, and finally form relevant data to drive the digital human, and further generate new multi-perspective annotation information to ensure that the model is capable of posture detection tasks in different environments and backgrounds. This method has the following significant advantages:

[0019] First, this method uses the twin system to generate a multi-view human posture dataset, which can control background, lighting and other conditions in a virtual environment to generate diverse training data. This method not only improves the efficiency of data generation, but also enables the model to adapt to complex situations in different scenes, thereby enhancing the generalization ability of the model in the actual environment. Compared with the traditional dataset collection method, the flexibility and diversity of the twin system effectively compensate for the problems of single background and lack of variation in real data, and the dataset can provide accurate human posture and the inaccuracy of the training data's own posture joints, providing high-quality data support for 3D human posture detection in complex scenes.

[0020] Second, this method uses multi-view information and spatial grid division strategies to effectively improve the accuracy and efficiency of 3D posture estimation, thereby achieving real-time detection of human 3D posture. For the detection of continuous images, this method optimizes the detection data in the time dimension, reduces the data jitter problem of joint points with the help of limb length constraints and time smoothing filters, and makes the joint point position and limb length more stable. This can better meet the accuracy and real-time requirements of detection and ensure stable and accurate detection data.

[0021] Third, by using the twin digital human to generate various human postures and movements, combined with virtual cameras and automatic annotation scripts, multi-view annotation data can be quickly generated. Compared with manual annotation, the twin system's digital human data generation method is more efficient and adaptable, and can provide a variety of human motion postures. In addition, the twin system's human joints are highly accurately annotated and do not rely on sensors, which is conducive to improving detection accuracy and reducing costs, solving the problem of lack of human posture data in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The distribution of human joints.

[0023] Figure 2 Schematic diagram of voxel meshing.

[0024] Figure 3 Schematic diagram of mesh feature reprojection.

[0025] Figure 4 Data optimization flow chart.

[0026] Figure 5 A holistic approach to 3D pose recognition. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] like Figure 1-5 shown.

[0029] A multi-view human body high-precision machine vision recognition method based on simulation generated data training. The overall process is as follows Figure 5 shown.

[0030] The specific steps include:

[0031] Step 1: Generate multi-view human posture data through the twin system (simulation). This includes the 3D coordinates of 14 human joints, pixel coordinates, occlusion labels of joints, original images from multiple perspectives, camera distortion coefficients, intrinsic matrix, rotation matrix and translation matrix relative to the world coordinate system. The specific joint positions and names are as follows: Figure 1 As shown in . By setting a world coordinate origin in the twin environment and binding all required joint points to the corresponding positions of the digital human, the precise coordinates of all joint points in the world coordinate system are obtained. Various parameters of the camera can also be directly calculated by the twin system, such as Figure 1 shown.

[0032] In the second step, the original images of each view are detected according to the trained two-dimensional posture detection model, so as to obtain the heat map features and pixel coordinates of the joint points of the images of each view. In order to make the estimation of the stereo frame as simple as possible, a stereo detection frame containing the human body is obtained with the hip joint as the reference. Specifically, the three-dimensional coordinates of the hip joint are calculated by the triangulation principle. The corner points of the stereo frame are used as the coordinate origin of this stereo space, so that the coordinate system of the frame is parallel to the world coordinate system, and the size of this stereo space is fixed to 3000mm×3000mm×2000mm, which is enough to contain the human body. Then, the calculated hip joint is used as the center point of this stereo space, and a stereo frame containing one person is obtained. For multiple people, the corresponding positions of the hip joints in different views of the same human body can be obtained through polar constraints, so as to match the hip joints of the same person and prevent errors in the triangulation process. Although only the hip joint points of the human body are matched, in fact, there is no need to match other human joint points in the future, because the stereo frame associates all joint points of the same person and separates different human bodies. A two-dimensional posture detection module is obtained.

[0033] Step 3: In order to obtain the final spatial grid, the entire space needs to be further refined. Let L, W and H represent the number of grid points in each axis direction, and the entire discretized space is represented as {S L×W×H},like Figure 2 In general, when the space size is fixed, the larger the values ​​of L, W and H are, the finer the space granularity and the higher the accuracy, but the amount of calculation will also increase. Through experiments, we finally set L, W and H to 100, 100 and 100 respectively to achieve a balance between calculation efficiency and accuracy. There are a total of 100×100×100=10 6grid points. Therefore, the horizontal distance between grid points is The vertical distance is

[0034] Then, multi-view feature fusion is performed on each human posture detection space grid. Based on the above-divided space grid, by projecting all grid points to all camera viewpoints, and sampling and fusing the heat map values ​​of the corresponding positions of all viewpoints, the obtained value is the feature value of the grid point. The heat map value of the X and Y pixel positions of the two-dimensional heat map in the view angle υ is represented by M υ ∈R K×X×Y , where K represents the total number of heat maps. For each grid point S L×W×H , and its projection in the viewing angle υ is expressed as The heat map value at a point is represented as Then calculate the average value of the heat map under all viewing angles as the eigenvalue of the grid point: Where N is the number of cameras. This feature actually represents the joint point appearing in S L×W×H probability.

[0035] After determining the features of the mesh, the coordinate position of each joint point needs to be restored from it. The mesh features are represented as V. In order to reduce the computational cost and meet the real-time requirements, the feature volume is reprojected to each reference plane of the world coordinate system, namely the xy plane, xz plane and yz plane of the world coordinate system, and the mesh features are converted into 2D plane features. xy ∈R K×L×W ,P xz ∈R K×L×H ,P yz ∈R K×W×H Respectively represent the reprojected feature maps corresponding to these three planes, and use the maximum pooling method for feature projection, such as Figure 3 shown.

[0036] After the projection is completed, the three 2D plane features are used as a batch to locate the coordinates of all related nodes from the 2D convolutional neural network. The 2D convolutional neural network will generate a P t The heat map feature estimation of the joint points with the same shape (t∈{xy,xz,yz}) is expressed as In order to reduce the quantization error caused by discrete space, it is necessary to calculate The center of mass of is not the maximum value, which is used to locate the joint point. Specifically, the joint point position is estimated The calculation method is:

[0037]

[0038] The training phase also requires a loss function to supervise the training process. This loss is achieved by calculating the error between the three-dimensional coordinates of the real human joint points and the predicted joint point coordinates:

[0039]

[0040] In fact, each coordinate point estimated by a plane feature contains two coordinate values, that is, xy, yz and xz appear twice in each direction, but their values ​​are not exactly the same. In order to solve this problem, a confidence fusion network is also needed to balance the weights between different values. This confidence fusion network consists of a convolutional layer, a pooling layer and a fully connected layer. The weight generated by this network is represented as W (t) ∈R K , and use the softmax function to normalize and obtain the final 3D coordinate estimation in a pairwise manner Specifically, for the joint point k, the final coordinate value estimation calculation method is:

[0041]

[0042] in Indicated in The first part of the 2D coordinate estimate is the x coordinate value, and the rest has the same meaning as Similarly, let J represent the true 3D coordinates, and also use loss supervision to train this confidence fusion network:

[0043]

[0044] Therefore, the overall training goal of this multi-view fusion prediction network is:

[0045] L=L hm +L c (14)

[0046] The three-dimensional posture detection module is obtained through the above operations.

[0047] In the fourth step, the joint point data is further adjusted on the model output results through the data optimization module. In order to meet the limb length constraint, a priori value of the limb length is required. Since the limb length of each person is different, we propose to dynamically determine the limb length of each different detection object through the detection confidence. The above-mentioned three-dimensional posture detection module can output the three-dimensional coordinates and the joint point confidence at the same time. All data will be imported into the dynamic database for storage. The limb length is represented by joint pairs as Bones = {(i, j)}, such as Figure 1As shown in (3,4) and (4,5), it represents the left arm, and the other limbs are similar. The limb confidence is set to the average confidence of the two joints, and the limb joint data with an average confidence greater than 0.85 and a single confidence not less than 0.8 are selected. Finally, the limb length l is calculated by weighted average confidence. ij The optimized joint point coordinates are expressed as P = {p1, p2, ..., p k}, the coordinates of each joint point are p i =(x i ,y i ,z i ). For the joint point k, in order to keep the adjustment result close to the initial estimate, the optimization objective function is:

[0048]

[0049] At the same time, the limb length should be limited to a certain range to avoid excessive changes in limb length. The optimization objective function is:

[0050]

[0051] The overall objective function of the final data optimization is:

[0052] L=L pose +L bone (17)

[0053] At the same time, real-time posture estimation will also bring errors in the time dimension, that is, the problem of discontinuous changes in joints between frames. We average the position information of the joints within a time window and regard it as a smooth estimate of the current position. In this way, high-frequency jitter in a short period of time is filtered out and the trend and low-frequency information of the data are retained. Set the sliding window size to w=5, place the data of the current frame in the middle of the window, and the calculation formula for the smooth position of the joints at time t is:

[0054]

[0055] By optimizing the data processing based on the detection confidence and time dimension, more accurate and stable 3D posture data of human body detection is finally obtained. The overall flow chart is as follows: Figure 4 shown.

[0056] In step 5, the three-dimensional posture detection data generated only contains the position coordinates of each joint point, and cannot directly drive the digital human. In order to solve the rotation problem of the digital human's limbs, it is also necessary to calculate the three-dimensional coordinate data to generate quaternions, and control the rotation of the limbs through the interpolation calculation of the quaternion. The two form the driving instructions of the final twin digital human, and set up a virtual camera in the twin environment, so that multi-view images and specific limb movements in different scenes can be collected, and more multi-view human posture standard master data can be generated through automatic annotation scripts to meet the training requirements of different scenes and different movements.

[0057] The present invention can be applied to real-time detection of multi-person postures in various scenarios. By detecting and collecting the motion data of characters in various scenarios, a data basis can be provided for subsequent operations such as character motion simulation or generation of new data sets.

[0058] The present invention does not involve parts that are the same as the prior art or can be implemented by using the prior art.

Claims

1. A multi-view human body high-precision machine vision recognition method based on simulation-generated data training, characterized by: It includes the following steps: First, the training data required for deep learning is generated through the twin system, including joint point coordinates, occlusion labels, camera parameters and multi-view images; these training data eventually form multi-view pose data, which is mainly generated for specific scenes. In order to improve the detection capability of specific scenes but not limited to a certain scene, the twin system is used to conveniently switch the background of different scenes, so it is possible to generate scene-specific data for different scenes; Second, establish a two-dimensional posture detection module; The 3D posture detection module can train a 2D joint posture detection model based on the pixel coordinates in the human posture data to generate a heat map of each joint point; since the human body is usually in a limited space, the approximate volume range of the human body in this space can be estimated based on the 3D world coordinates of specific joint points and represented by a rectangular stereo frame; the actual 3D world coordinates of these joint points are obtained by triangulation based on the pixel coordinates of the specific joints obtained after detection, and the human body detection stereo frame is estimated through these joint points as the input of the subsequent 3D posture detection module; Third, establish a human body three-dimensional posture detection module; divide the human body space into regular rectangular grids within the range of the human body detection stereo frame; these grid points can roughly represent the space where the human body is located, that is, there is a corresponding relationship between the joint points and the grid points, so the features of the generated grid points can ultimately restore the human body posture information; in order to generate multi-view features of the spatial grid, the spatial grid is projected to each view through the camera parameters, and the heat map features of each view are fused to form a spatial feature with the position information of the human body joints. The feature of each grid point represents the probability that the joint point is located here; then the posture detection model is used to detect the three-dimensional world coordinates of the human body from the spatial grid after feature projection fusion; this model projects the features of the spatial grid to the three coordinate planes specified by the world coordinate system respectively, thereby obtaining the two-dimensional plane features of the human body posture, and then detects the two-dimensional coordinates of the joint points in each coordinate plane from the two-dimensional plane features, and finally combines all the two-dimensional coordinates to determine the three-dimensional posture detection module of the human body; Fourth, establish a data optimization module; Based on the above-mentioned human body 3D posture detection module, the 3D coordinates of the human body joints are obtained. However, since each detection is an independent process, for continuous image input, there will still be relative jitters in the joint positions between frames, and the joint line, that is, the limb length will not be completely consistent. Therefore, the 3D data of the joints is further optimized and processed to finally obtain more accurate and stable human body posture detection data. Fifth, the driving instructions needed to control the twin digital human are generated through human posture detection data; because driving the digital human requires determining the position of the human body and the rotation process between the limbs, the driving instructions mainly include the three-dimensional world coordinates of the joints and the quaternion of the limb rotation; the three-dimensional coordinates determine the positions of the digital human at different times, and the quaternion ensures the rotation of the digital human limbs between various positions. While driving the digital human, a virtual camera is added to collect multi-view images of the digital human, and through the automatic annotation script, the posture annotation data can be automatically generated and exported, leaving an interface for the subsequent expansion of other scene data sets.

2. The method according to claim 1, characterized in that: The joint point coordinates refer to some key parts or feature points of the human body, which are used to represent the posture of the human body, including pixel coordinates and three-dimensional world coordinates. These points usually correspond to the joints or prominent parts of the human skeleton. By connecting these points, the skeleton structure of the human body can be constructed. The occlusion label represents the visibility of a joint point, and the visibility of the joint point is represented by a number between 0 and 1, where 0 represents completely invisible and 1 represents completely visible. Multi-view images are human posture images taken by different cameras at the same time. The camera parameters represent the position of the camera in the world coordinate system and the mapping relationship between the three-dimensional world and the two-dimensional image. These parameters calibrate the corresponding relationship between the perspectives of different images.

3. The method according to claim 1, characterized in that: When optimizing the three-dimensional data of the joint points: first, the length of the human limbs is determined based on the confidence of the detected joint points, and the limb length with the highest confidence is used as the optimization target to achieve relatively consistent limb lengths at different times; secondly, for the problem of joint point position jitter, the joint points are smoothed and filtered from the time dimension to ensure smooth changes in the joint point positions at adjacent moments, and finally more accurate and stable human posture detection data is obtained.

4. The method according to claim 1, characterized in that: When dividing the human body space into regular rectangular grids, first, further refine the overall space; set L, W and H to represent the number of grid points in each axis direction, and represent the entire discretized space as {S L×W×H }, when the space size is fixed, the larger the values ​​of L, W and H are, the finer the space granularity is and the higher the accuracy is, but the amount of calculation will also increase. Through experiments, L, W and H are finally set to 100, 100 and 100 respectively to achieve a balance between calculation efficiency and accuracy. There are a total of 100×100×100=10 6 grid points; therefore the horizontal distance between grid points is The vertical distance is Then, multi-view feature fusion is performed on each human posture detection space grid. According to the above-mentioned divided space grid, all grid points are projected to all camera viewpoints, and the heat map values ​​of the corresponding positions of all viewpoints are sampled and fused. The obtained value is the feature value of the grid point. The heat map values ​​of the X and Y pixel positions of the two-dimensional heat map in the viewing angle υ are represented as M υ ∈R K ×X×Y , where K represents the total number of heat maps, for each grid point S L×W×H , and its projection in the viewing angle υ is represented as P υ L×W×H ; P υ L×W×H The heat map value at a point is represented as Then calculate the average value of the heat map under all viewing angles as the eigenvalue of the grid point: Where N is the number of cameras; This feature actually represents the joint point appearing in S L×W×H probability; After determining the features of the mesh, the coordinate position of each joint point needs to be recovered from it; The mesh feature is represented as V. In order to reduce the computational cost and meet the real-time requirements, the feature volume is reprojected to each reference plane of the world coordinate system, namely, the xy plane, xz plane and yz plane of the world coordinate system, and the mesh feature is converted into a 2D plane feature; Using P xy ∈R K×L×W ,P xz ∈R K ×L×H ,P yz ∈R K×W×H Represent the reprojected feature maps corresponding to these three planes respectively, and use the maximum pooling method for feature projection; After the projection is completed, the three 2D plane features are used as a batch to locate the coordinates of all related nodes from the 2D convolutional neural network; the 2D convolutional neural network will generate a P t The heat map feature estimation of the joint points with the same shape (t∈{xy,xz,yz}) is expressed as In order to reduce the quantization error caused by discrete space, it is necessary to calculate The center of mass of the joint point is not the maximum value; the joint point position is estimated The calculation method is: The training phase also requires a loss function to supervise the training process. This loss is achieved by calculating the error between the three-dimensional coordinates of the real human joint points and the predicted joint point coordinates: Since the coordinate points of each plane feature estimate contain two coordinate values, that is, the xy, yz and xz coordinates appear twice in each direction, but their values ​​are not exactly the same, in order to solve this problem, a confidence fusion network is needed to balance the weights between different values; this confidence fusion network consists of a convolutional layer, a pooling layer and a fully connected layer; the weight generated by this network is represented as W (t) ∈R K , and use the softmax function to normalize and obtain the final 3D coordinate estimation in a pairwise manner For joint point k, the final coordinate value is estimated and calculated as follows: in Indicated in The first part of the 2D coordinate estimate is the x coordinate value, and the rest has the same meaning as Similarly; use J to represent the true 3D coordinates, and also use loss supervision to train this confidence fusion network: Therefore, the overall training goal of this multi-view fusion prediction network is: L=L hm +L c (5) 5. The method according to claim 1 or 3, characterized in that: When optimizing the three-dimensional data of joints, it is necessary to obtain a priori value of limb length first; since the limb length of each person is different, the limb length of each different detection object is dynamically determined through the detection confidence; the three-dimensional posture detection module can output the three-dimensional coordinates and the joint confidence at the same time, and all data will be imported into the dynamic database for storage; the limb length is represented by a joint pair as Bones = {(i, j)}; the limb confidence is set to the average confidence of the two joints, and the limb joint data with an average confidence greater than 0.85 and a single confidence not less than 0.8 are selected, and finally the limb length l is calculated by weighted calculation of the average confidence. ij ; The optimized joint point coordinates are expressed as P = {p1, p2, ..., p k }, the coordinates of each joint point are p i =(x i ,y i ,z i ); For the joint point k, in order to keep the adjustment result close to the initial estimate, the optimization objective function is: At the same time, the limb length should be limited to a certain range to avoid excessive changes in limb length. The optimization objective function is: The overall objective function of the final data optimization is: L=L pose +L bone (8) At the same time, real-time posture estimation will also bring about errors in the time dimension, that is, the problem of discontinuous changes in joints between frames. To this end, the position information of the joints within a time window is averaged and regarded as a smooth estimate of the current position. In this way, high-frequency jitter in a short period of time is filtered out, and the trend and low-frequency information of the data are retained. Set the sliding window size to w=5, place the data of the current frame in the middle of the window, and calculate the smooth position of the joint point at time t using the following formula: By optimizing the data based on detection confidence and time dimension, more accurate and stable three-dimensional posture data of human body detection is finally obtained.

Images