Human joint heat map triangulation method and system based on time domain prediction

Abstract

This invention provides a method and system for human joint point heatmap triangulation based on temporal prediction, relating to the fields of computer vision and computer graphics technology. The method includes: Step S1: Starting a sub-camera process for each camera, synchronously running a 2D human pose detection network using a process-level parallelization framework, and outputting the pixel coordinates of human joint points; Step S2: Predicting the 3D positions of each joint point in the current frame using the 3D positions and velocities of the human joint points based on linear Kalman filtering; Step S3: Establishing a 3D joint point confidence heatmap based on the temporal prediction points, and calculating the final positions of the joint points in the current frame. This invention can be used for real-time, label-free, high-precision single-person human tracking, effectively improving human tracking accuracy in real-time scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics technology, specifically to a thermal model of human joints based on temporal domain prediction. Figure 3 Keratinization methods and systems. Background Technology

[0002] 3D human motion tracking is a common problem in the field of computer vision, with numerous applications, such as motion analysis, human-computer interaction, pose recognition, games, and online video conferencing. Currently, commonly used methods are divided into two categories: single-view human tracking and multi-view human tracking.

[0003] Currently, the latest single-view human tracking methods fall into two categories. One relatively direct method is to use convolutional neural networks to directly calculate 3D coordinates from 2D images. However, since different human poses tend to produce identical or similar projections onto the camera plane, and single-view human tracking lacks depth information and struggles to address occlusion issues, this approach falls under the category of ill-posed problems and is quite challenging.

[0004] Another approach is to first use a 2D pose detection network to detect the 2D coordinates of key points on the human body, and then use a deep neural network to convert the 2D coordinates into 3D coordinates, thereby achieving human body tracking. This method is simpler and more flexible than the previous one, as it can be trained directly using motion capture data, and the 2D pose detection network can be replaced after training. However, this approach is still limited by a single viewpoint, the depth is unclear, and it is difficult to solve the occlusion problem. Furthermore, a small error in the 2D coordinates of a key point can lead to a large error after conversion to 3D coordinates.

[0005] Most high-precision multi-view human tracking systems (such as OptiTrack) currently require the wearing of markers. Marker-based human tracking is often used as ground truth in dataset creation due to its higher accuracy. However, its drawbacks are also significant: firstly, it requires higher hardware specifications, typically limiting its use to laboratory environments; secondly, wearing markers can cause slight discomfort to the tracked individual, and in some scenarios, may even hinder their free movement.

[0006] Label-free multi-view human tracking directly utilizes multiple RGB cameras as input to achieve human tracking, naturally offering the advantage of being unaffected by occlusion compared to single-view tracking. Some methods optimize the parameters of the human model to make its 2D projection on each camera plane similar to the input for human tracking. Other methods employ a step-by-step strategy: first, a pre-trained pose detection network detects the 2D coordinates of human keypoints on the camera plane from each camera's viewpoint; then, based on the camera's intrinsic and extrinsic parameters and the keypoint coordinates calculated in the previous step, triangulation and other methods are used to calculate the 3D coordinates of each human keypoint. This method is more intuitive, and the two steps are relatively independent; replacing the pose detection network does not affect triangulation, resulting in good scalability.

[0007] Existing unmarked single-person multi-camera motion tracking systems do not consider real-time requirements, only using synchronized offline video as input, making them unusable in real-time scenarios. Furthermore, their configuration and use often have high requirements for the environment or hardware, making them difficult to use in ordinary users' homes.

[0008] In summary, among existing technologies, the "volumetric triangulation approach" proposed by Samsung's team is computationally slow, requires a multi-layered network design, and is difficult to apply to real-time scenarios. It also incurs high costs, requiring the pre-training of multiple neural network models.

[0009] The “spatiotemporally constrained triangulation approach” proposed by the University of Washington team is slow in computation. It transforms triangulation calculation into an optimization problem, and it continuously iterates the calculation parameters, which is not possible in real time and is also costly.

[0010] Traditional linear triangulation methods have the lowest computational accuracy and exhibit significant jitter. Summary of the Invention

[0011] To address the shortcomings of existing technologies, this invention provides a time-domain prediction-based thermal model of human joints. Figure 3 Keratinization methods and systems.

[0012] The present invention provides a time-domain prediction-based thermodynamics of human joints. Figure 3 Keratinization method and system, the scheme of which is as follows:

[0013] Firstly, it provides a time-domain prediction-based thermal model of human joints. Figure 3 Keratinization method, the method comprising:

[0014] Step S1: Start a sub-camera process for each camera, use a process-level parallelization framework to run the 2D human pose detection network synchronously, and output the pixel coordinates of human joints on the camera plane.

[0015] Step S2: Based on linear Kalman filtering, predict the 3D position of each joint in the current frame using the 3D position and velocity of the human body joints;

[0016] Step S3: Based on the temporal prediction points, establish a 3D joint confidence heatmap and calculate the final position of the joint in the current frame.

[0017] Preferably, step S1 includes:

[0018] Sub-camera process: At the beginning of system startup, the main process detects the number of cameras and starts a sub-camera process for each camera and binds it to that camera;

[0019] Data receiving process: After starting the corresponding sub-camera process for each camera, the main thread starts an additional data receiving process to receive and integrate the running results of the current frame of all sub-cameras;

[0020] Data processing process: After the main process starts the data receiving process, an additional data processing process is started to perform triangulation calculations.

[0021] Preferably, the sub-camera process includes: in each frame, the sub-camera process independently runs a 2D human pose detection network using the RGB input of the current frame of the camera to calculate the pixel coordinates of human key points in the current frame;

[0022] After acquiring the multi-process lock, the pixel coordinates are sent to the data receiving process, and the process enters a sleep state, waiting to be woken up by the data receiving process. Once woken up, the process executes the input of the next frame.

[0023] When its corresponding camera is turned off or the last frame of the video has been read, the sub-camera process sends a special data structure, namely poison pil1, to the data receiving process to indicate that the process has ended, and then exits the process.

[0024] Preferably, a dual-buffering mechanism is introduced between the sub-camera process and the data receiving process to parallelize the writing and reading of the attitude detection network's running results. When the sub-camera process writes pixel coordinates into one buffer, the data receiving process can read data from the other buffer.

[0025] Preferably, the data receiving process includes: after all the results of the sub-camera processes have been received in each frame, the data receiving process forwards the integrated results to the data processing process and wakes up all the dormant sub-camera processes to process the input of the next frame;

[0026] If a poison pill is received from a sub-camera process, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process exits.

[0027] Preferably, the data processing process includes: after receiving the integrated sub-camera process results in each frame, the data processing process performs triangulation calculation to obtain the triangulation calculation result;

[0028] The process exits when it receives poison pill data from the data receiving process.

[0029] Preferably, step S2 includes:

[0030] Step S2.1: Use the prediction matrix and add noise to make a preliminary prediction of the 3D position of the human joints in the current frame;

[0031] Step S2.2: Use the 3D positions of human joints calculated by the linear triangulation method as observations to further refine the best predicted values ​​of each joint position.

[0032] Preferably, step S3 includes:

[0033] Step S3.1: Divide the 3D space into a voxel mesh with fixed resolution and fixed size;

[0034] Step S3.2: Reproject each voxel mesh back to each camera plane, calculate its confidence value, and generate a 3D confidence heatmap;

[0035] Step S3.3: Calculate the final location of the key point from the 3D heat map.

[0036] Secondly, it provides a time-domain prediction-based thermal model of human joints. Figure 3 Keratinization system, the system comprising:

[0037] Module M1: Enables a sub-camera process for each camera, uses a process-level parallelization framework to synchronously run a 2D human pose detection network, and outputs the pixel coordinates of human joints on the camera plane.

[0038] Module M2: Based on linear Kalman filtering, it predicts the 3D position of each joint in the current frame using the 3D position and velocity of human joints.

[0039] Module M3: Based on temporal prediction points, it establishes a 3D joint confidence heatmap and calculates the final position of the joint in the current frame.

[0040] Preferably, the module M1 includes:

[0041] Sub-camera process: At the beginning of system startup, the main process detects the number of cameras and starts a sub-camera process for each camera and binds it to that camera;

[0042] Data receiving process: After starting the corresponding sub-camera process for each camera, the main thread starts an additional data receiving process to receive and integrate the running results of the current frame of all sub-cameras;

[0043] Data processing process: After the main process starts the data receiving process, an additional data processing process is started to perform triangulation calculations;

[0044] The sub-camera process includes: in each frame, the sub-camera process independently runs a 2D human pose detection network using the RGB input of the current frame of the camera to calculate the pixel coordinates of human key points in the current frame.

[0045] After acquiring the multi-process lock, the pixel coordinates are sent to the data receiving process, and the process enters a sleep state, waiting to be woken up by the data receiving process. Once woken up, the process executes the input of the next frame.

[0046] When its corresponding camera is turned off or the last frame of the video has been read, the sub-camera process sends a special data structure, namely poison pill, to the data receiving process to indicate that the process has ended, and then exits the process.

[0047] A dual-buffering mechanism is introduced between the sub-camera process and the data receiving process to parallelize the writing and reading of the attitude detection network's running results. When the sub-camera process writes pixel coordinates into one buffer, the data receiving process can read data from the other buffer.

[0048] The data receiving process includes: after receiving the results of all sub-camera processes in each frame, the data receiving process forwards the integrated results to the data processing process and wakes up all dormant sub-camera processes to process the input of the next frame;

[0049] If a poison pill data is received from a sub-camera process, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process exits.

[0050] The data processing process includes: after receiving the integrated sub-camera process results in each frame, the data processing process performs triangulation calculation to obtain the triangulation calculation result;

[0051] The process exits when it receives poiSon pill data sent by the data receiving process;

[0052] Step S2 includes:

[0053] Step S2.1: Use the prediction matrix and add noise to make a preliminary prediction of the 3D position of the human joints in the current frame;

[0054] Step S2.2: Use the 3D positions of human joints calculated by the linear triangulation method as observations to further refine the best predicted values ​​of each joint position;

[0055] Step S3 includes:

[0056] Step S3.1: Divide the 3D space into a voxel mesh with fixed resolution and fixed size;

[0057] Step S3.2: Reproject each voxel mesh back to each camera plane, calculate its confidence value, and generate a 3D confidence heatmap;

[0058] Step S3.3: Calculate the final location of the key point from the 3D heat map.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. This invention utilizes time-domain information to perform real-time, unmarked tracking of a single person using multiple RGB cameras, effectively improving tracking accuracy in real-time scenarios and offering faster computation speed, making it applicable to real-time scenarios.

[0061] 2. This invention has low cost, does not require pre-trained network models, and can run on both CPU and GPU.

[0062] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0063] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0064] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0065] Figure 2 This is a flowchart of temporal prediction of human key point locations based on linear Kalman filtering;

[0066] Figure 3 For heat Figure 3 A schematic diagram of the spatial positions of 17 joints in the human body after keratinization calculation;

[0067] Figure 4 A process-level parallelization framework for human pose detection networks;

[0068] Figure 5 A schematic diagram of the dual-caching scheme used in the process-level parallelization framework. Detailed Implementation

[0069] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0070] This invention provides a time-domain prediction-based thermal model of human joints. Figure 3 Keratinization method, refer to Figure 1 and Figure 2 As shown, the method specifically includes:

[0071] Step S1: Start a sub-camera process for each camera, and use a process-level parallelization framework to run the 2D human pose detection network synchronously, and output the pixel coordinates of human joints on the camera plane.

[0072] Step S1 includes the sub-camera process, the data receiving process, and the data processing process.

[0073] The sub-camera process works as follows: At the beginning of system startup, the main process detects the number of cameras and starts a sub-camera process for each camera and binds it to that camera; in each frame, the sub-camera process independently runs a 2D human pose detection network using the RGB input of the current frame of that camera to calculate the pixel coordinates of the human key points in the current frame; after acquiring the multi-process lock, it sends the pixel coordinates to the data receiving process and enters a sleep state, waiting for the data receiving process to wake it up. Once woken up, it executes the input of the next frame.

[0074] When its corresponding camera is turned off or the last frame of the video has been read, the sub-camera process sends a special data structure, namely a poison pill, to the data receiving process to indicate that the process has ended, and then exits the process.

[0075] Data receiving process: After starting the corresponding sub-camera process for each camera, the main thread starts an additional data receiving process to receive and integrate the current frame results of all sub-cameras.

[0076] After receiving the results from all sub-camera processes in each frame, the data receiving process forwards the integrated results to the data processing process and wakes up all dormant sub-camera processes to process the input for the next frame. If a poison pill is received from a sub-camera process, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process exits.

[0077] A dual-buffering mechanism is introduced between the sub-camera process and the data receiving process to parallelize the writing and reading of the attitude detection network's results. When the sub-camera process writes pixel coordinates to one buffer, the data receiving process can read data from the other buffer.

[0078] Data processing process: After the main process starts the data receiving process, an additional data processing process is started to perform triangulation calculations.

[0079] The data processing process performs triangulation calculations after receiving the integrated results of the sub-camera process in each frame to obtain the triangulation calculation result; the process exits when it receives poison pill data sent by the data receiving process.

[0080] Step S2: Based on linear Kalman filtering, predict the 3D position of each joint in the current frame using the 3D position and velocity of human joints.

[0081] Step S2 includes:

[0082] Step S2.1: Use the prediction matrix and add noise to make a preliminary prediction of the 3D position of the human joints in the current frame;

[0083] Step S2.2: Use the 3D positions of human joints calculated by the linear triangulation method as observations to further refine the best predicted values ​​of each joint position.

[0084] Step S3: Based on the temporal prediction points, establish a 3D joint confidence heatmap and calculate the final position of the joint in the current frame.

[0085] Step S3 includes:

[0086] Step S3.1: Divide the 3D space into a voxel mesh with fixed resolution and fixed size;

[0087] Step S3.2: Reproject each voxel mesh back to each camera plane, calculate its confidence value, and generate a 3D confidence heatmap;

[0088] Step S3.3: Calculate the final location of the key point from the 3D heat map.

[0089] This invention also provides a time-domain prediction-based thermal model of human joints. Figure 3 Keratinization system, the human joint thermodynamics based on time-domain prediction Figure 3 The keratinization system can perform the aforementioned time-domain predicted human joint thermodynamics. Figure 3 The process steps of the keratinization method are implemented, meaning that those skilled in the art can apply the time-domain predicted human joint thermal analysis. Figure 3 The keratinization method can be understood as the human joint thermal analysis based on time-domain prediction. Figure 3 A preferred embodiment of a keratinization system. This system specifically includes:

[0090] Module M1: Enables a sub-camera process for each camera, uses a process-level parallelization framework to synchronously run a 2D human pose detection network, and outputs the pixel coordinates of human joints.

[0091] The M1 module includes a sub-camera process, a data receiving process, and a data processing process.

[0092] The sub-camera process works as follows: At the beginning of system startup, the main process detects the number of cameras and starts a sub-camera process for each camera and binds it to that camera; in each frame, the sub-camera process independently runs a 2D human pose detection network using the RGB input of the current frame of that camera to calculate the pixel coordinates of the human key points in the current frame; after acquiring the multi-process lock, it sends the pixel coordinates to the data receiving process and enters a sleep state, waiting for the data receiving process to wake it up. Once woken up, it executes the input of the next frame.

[0093] When its corresponding camera is turned off or the last frame of the video has been read, the sub-camera process sends a special data structure, namely a poison pill, to the data receiving process to indicate that the process has ended, and then exits the process.

[0094] Data receiving process: After starting the corresponding sub-camera process for each camera, the main thread starts an additional data receiving process to receive and integrate the current frame results of all sub-cameras.

[0095] After receiving the results from all sub-camera processes in each frame, the data receiving process forwards the integrated results to the data processing process and wakes up all dormant sub-camera processes to process the input for the next frame. If a poison pill is received from a sub-camera process, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process exits.

[0096] A dual-buffering mechanism is introduced between the sub-camera process and the data receiving process to parallelize the writing and reading of the attitude detection network's results. When the sub-camera process writes pixel coordinates to one buffer, the data receiving process can read data from the other buffer.

[0097] Data processing process: After the main process starts the data receiving process, an additional data processing process is started to perform triangulation calculations.

[0098] The data processing process performs triangulation calculations after receiving the integrated results of the sub-camera process in each frame to obtain the triangulation calculation result; the process exits when it receives poison pill data sent by the data receiving process.

[0099] Module M2: Based on linear Kalman filtering, it predicts the 3D position of each joint in the current frame using the position and velocity of the human joints.

[0100] Module M2 includes:

[0101] Module M2.1: Uses the prediction matrix and adds noise to make a preliminary prediction of the human joint position in the current frame;

[0102] Module M2.2: The optimal predicted values ​​of each joint position are further corrected by using the human joint position calculated by the linear triangulation method as the observation value.

[0103] Module M3: Based on temporal prediction points, it establishes a 3D joint confidence heatmap and calculates the final position of the joint in the current frame.

[0104] This module M3 includes:

[0105] Module M3.1: Divides 3D space into voxel meshes of fixed resolution and fixed size;

[0106] Module M3.2: Reproject each voxel mesh back to each camera plane, calculate its confidence value, and generate a 3D confidence heatmap;

[0107] Module M3.3: Calculates the final location of the key point from the 3D heat map.

[0108] The present invention will now be described in more detail.

[0109] Example 1:

[0110] In this embodiment, the positions of key human body points in the current frame are initially estimated based on time-domain information. The real-time position changes of different key human body points and the observation of the positions of human body joints can be regarded as a dynamic system. When the dynamic system has a lot of random noise, or the obtained observation values ​​have large errors, Kalman filtering can be used to make relatively accurate predictions of the current hidden state and future state of the system. The linear Kalman filter used in this embodiment is more suitable for processing dynamic systems that change over time and has better performance. It is often used in radar tracking systems, control systems, computer graphics, and other fields.

[0111] The movement of key points on the human body can exhibit irregular motion over a long period. However, in the real-time scenario of this embodiment, when the frame rate is high, the movement of a key point between adjacent frames can be considered as near-regular motion. Under this premise, the higher the camera frame rate, the closer the motion trajectory between adjacent frames is to linear motion. Furthermore, when it is predictable that a tracked key point will randomly change direction or accelerate / decelerate, prior knowledge can be incorporated into the trajectory prediction by adding processing noise in the prediction phase of the Kalman filter. In other words, the weight of the predicted value is reduced in the correction phase of the Kalman filter.

[0112] The Kalman filter described in this embodiment consumes less memory, requiring only the state of the previous time point to predict the state of the current time point, and the calculation is relatively fast. For human keypoints, in a given frame, the linear triangulated position calculated in the previous section can be considered as the observed position of the keypoint. Since the 2D observations output by the neural network in each camera plane contain errors, the 3D observations calculated using these observations have significant errors and uncertainties. When assuming that the keypoints move at approximately uniform linear speeds between frames, the predicted points calculated based on temporal information also have errors. The Kalman filter can integrate the erroneous observations and predictions, thereby calculating a predicted value that is closer to the true position than the observed and predicted values.

[0113] The Kalman filter described in this embodiment assumes that all hidden state variables follow a Gaussian distribution model and are randomly distributed, with each variable i having a mean μ. i and variance The relationships between the various dimensions of the hidden state can be represented by a covariance matrix ∑, where each element of the covariance matrix ∑ ij This represents the relationship between the i-th state variable and the j-th state variable. For a given keypoint j, assume its hidden state at time t is determined by its current position p. t With velocity v t Composition, that is

[0114]

[0115] Where, x t y t z t This indicates the position coordinates of the joint point; This indicates the components of the current frame velocity of the joint on each axis; Represents the velocity vector (same as v) t The covariance matrix P corresponding to this hidden state. t for:

[0116]

[0117] Based on the hidden state of the previous frame in this embodiment, the prediction matrix F is used. t Calculate the estimated value of the hidden state for the current frame.

[0118]

[0119] in, This represents the estimated value of the hidden state in the current frame. This represents the estimated value for the current frame position. This represents the estimated value of the current frame rate; assuming that in this embodiment the keypoints move at a constant linear speed between adjacent frames, then we have:

[0120] p t =p t-l +v t-1 Δt

[0121] v t =v t-1

[0122] Therefore, the prediction matrix F in this embodiment can be obtained. t for

[0123]

[0124] Where Δt represents the time difference between frames; and the covariance matrix in this embodiment is updated as follows:

[0125]

[0126]

[0127] Since this embodiment assumes that key points move at a constant linear velocity between adjacent frames at high frame rates, the actual velocity direction and magnitude of the key points' movement between adjacent frames will change. Therefore, the temporal prediction of key points based on constant linear velocity in this embodiment contains errors. In Kalman filtering, random noise is added after the prediction phase to reduce the impact of this error. Each state in the original prediction may move to some new states, assuming the covariance of the noise is Q. t Then for a certain key point x t-1 Its predicted position in the new state of the current frame follows a covariance of Q. t The Gaussian distribution. By adding Q... t Therefore, the complete prediction formula for the current frame state in this embodiment can be obtained as follows:

[0128]

[0129]

[0130] In this embodiment, the best prediction of the current frame state is equal to the prediction value calculated from the best prediction of the previous frame plus the correction for known external influences; the uncertainty of the current frame state is the prediction value based on the uncertainty of the previous frame plus the correction from environmental noise.

[0131] In each frame of this embodiment, the result of linear triangulation can be used as an observation to further refine the optimal prediction. Let H represent the observation (i.e., linear triangulation calculation) operation, then for the current hidden state in this embodiment... have

[0132]

[0133]

[0134] For the optimal prediction point calculated above in this embodiment The mean and covariance of the Gaussian distribution centered at point are calculated by projecting it onto each camera plane and then performing linear triangulation.

[0135]

[0136] ∑ cxpcctcd =HP t H T

[0137] The linear triangulation calculation described in this embodiment inherently contains errors. These errors can originate from the errors in the intrinsic and extrinsic parameters after camera calibration, or from the errors in the keypoint pixel coordinates calculated by the pose detection network. Combining these two errors can be considered as observation noise in the Kalman filter. Regarding the linear triangulation calculation itself, if the calculated result is... (i.e., the observed value is) If the distribution of the true values ​​is such that the distribution of the true values ​​conforms to the following: A Gaussian distribution centered at [0, 1]. Using R... t This represents the covariance matrix among the three dimensions of the coordinates calculated using linear triangulation. For any hidden state [z1, z2, z3]... T This corresponds to two probabilities: if the hidden state is [z1, z2, z3] T If the true value is true, then the observed value is obtained based on that true value. The probability that the hidden state falls within the best prediction value calculated in the preceding chapters; The probability is calculated from the product of the two probabilities, centered at Gaussian distributions. The probability that the hidden state is the true value is maximized when the product of the two probabilities is maximized. Considering that both probability distributions follow a Gaussian distribution, when the two centers are respectively... and When Gaussian distributions with covariance matrices of ∑1 and ∑2 are multiplied together, the center of the new Gaussian distribution is obtained. The covariance matrix ∑′ is respectively

[0138] K = ∑1(∑1 + ∑2) -1

[0139]

[0140] ∑′=∑1-K∑1

[0141] Substituting the predicted and observed values ​​(the result of linear triangulation) into the aforementioned equation and rearranging, we obtain:

[0142]

[0143]

[0144]

[0145] in, Indicates Kalman gain; This represents the final estimated value of the hidden state in the current frame; R represents the covariance matrix corresponding to the final hidden state. t This represents the covariance matrix among the three dimensions of the observed values. In this embodiment, based on linear Kalman filtering, the position of key human body points in the current frame is predicted temporally using the temporal information from the previous frame. This is used as an example of the thermodynamic-based method in Embodiment 2. Figure 3 The baseline input for angularity calculation.

[0146] Example 2:

[0147] Reference Figure 3 As shown, the purpose of this embodiment is to integrate the human keypoint information output by each 2D human pose detection network from multiple viewpoints into 3D information. This embodiment uses the concept of spatial voting to generate a 3D confidence heatmap for each human keypoint. The entire process consists of three steps: first, the 3D space is divided into voxel grids of fixed resolution and fixed size; next, each voxel grid is reprojected back onto each camera plane, its confidence value is calculated, and a 3D confidence heatmap is generated; finally, the final position of the keypoint is calculated from the 3D heatmap.

[0148] For each human body keypoint j, a 3D bounding box of size L×L×L is first constructed centered on the temporal prediction point calculated in Example 1, where the z-axis of the bounding box is perpendicular to the ground. Then, the bounding box is decomposed into R×R×R voxels at a fixed resolution R. Each voxel uses its center's world coordinates as its position. For each viewpoint c, its 3D coordinates are calculated using the camera projection matrix P. c Projected back to camera plane

[0149]

[0150] In this embodiment, for each voxel's projection point, the certainty value of the voxel's position in the current viewpoint can be extracted from the 2D heatmap. This operation is divided into two cases depending on the output of the 2D neural network:

[0151] When the 2D heatmap is the direct output of a neural network (e.g., OpenPose), it has a lower resolution compared to the input image. Therefore, upsampling is needed on the low-resolution 2D heatmap to obtain the confidence level of the projected points at the original resolution. In this embodiment, bilinear interpolation is used for upsampling. If the heatmap output by the 2D neural network corresponding to camera c is H... c Let {·} denote the bilinear interpolation operation, then the confidence level of the voxel in this camera is:

[0152]

[0153] The heatmap interpolation sampling method described in this embodiment has the advantage of better utilizing the output of the 2D human pose detection network to obtain more accurate triangulation results. However, it has the disadvantage of requiring bilinear interpolation of J×R×R×R points on C planes in each frame, resulting in a large computational load that is difficult to parallelize using a GPU, easily becoming a performance bottleneck in scenarios with high real-time requirements. Furthermore, when the resolution of the neural network output is too low, the bilinear interpolation upsampling results also have significant errors. When the pose detection network cannot directly output a 2D heatmap, this embodiment uses the keypoint coordinates output by the pose detection network as the center and establishes a Gaussian distribution map based on the distance from the projection point to the center, using the standard deviation σ. For a projection point on the camera plane c... If the keypoint coordinates output by the neural network are The degree of certainty that the point lies on camera plane c is:

[0154]

[0155] Compared to interpolation sampling methods, the Gaussian heatmap method is faster in computation and easier to parallelize using GPUs. Furthermore, it can provide different standard deviations σ for different key points to generate different movement ranges. For real-time considerations, this embodiment primarily employs the Gaussian heatmap method.

[0156] In this embodiment, after calculating the confidence level of each voxel projected onto each camera plane, two methods for integrating the confidence levels of each camera plane are implemented: directly adding the confidence levels of each camera plane and taking the average of the confidence levels of each camera plane. After integrating the confidence levels of the voxel mesh for each keypoint, a 3D heatmap cube space is generated around the temporal predicted value for each keypoint. In this embodiment, at the real-time frame rate, the actual position of the same keypoint between two adjacent frames falls near the temporal predicted value; that is, the position with the maximum confidence level in the 3D heatmap space is taken as the 3D position after spatial information correction. When extracting the maximum value from the 3D heatmap space, this embodiment uses the centroid of the 3D heatmap space as the final position of the keypoint, that is, multiplying the spatial position of each voxel by its corresponding confidence level, adding them together, and then averaging the results, which is the centroid of the voxel confidence level. This method can theoretically obtain points at any position in the 3D heatmap space, thus achieving higher accuracy.

[0157] In this embodiment, projecting the 2D heatmap onto the 3D voxel mesh provides better robustness compared to the linear triangulation method. The additional spatial information provided by the heatmap can be used to calculate the 3D coordinates of each key point while minimizing real-time performance loss.

[0158] Example 3

[0159] This embodiment describes a parallel operation scheme for a process-level human pose detection network, such as... Figure 4 As shown, it parallelizes network operation, data reception, and data processing in three stages, thereby improving the efficiency of the 2D pose detection module and ultimately increasing the frame rate of multiple 2D pose detection operations running simultaneously to approximately 20 frames per second. The process-level parallelization scheme in this embodiment includes three sub-processes: a sub-camera process, a data reception process, and a data processing process.

[0160] The sub-camera process, in this embodiment, involves the main process detecting the number of cameras and starting a sub-camera process for each camera at system startup, binding it to that camera. In each frame, the sub-camera process independently runs a network similar to the 2D human pose detection network in the section on the current frame, using the RGB input of that camera, to calculate the pixel coordinates of the human keypoints in the current frame. After acquiring a multi-process lock, it sends the pixel coordinates to the data receiving process and enters a sleep state, waiting to be awakened by the data receiving process. Upon awakening, it executes the input for the next frame. When its corresponding camera is off or the last frame of the video has been read, the sub-camera process sends a special data structure (i.e., a "poison pill") to the data receiving process to indicate the process's termination, and then exits the process.

[0161] In this embodiment, after starting a corresponding sub-camera process for each camera, the main thread will additionally start a data receiving process to receive and integrate the results of the current frame from all sub-cameras. After receiving the results from all sub-camera processes in each frame, the data receiving process forwards the integrated results to the data processing process and wakes up all dormant sub-camera processes to process the input for the next frame. If a poison pill data is received from a sub-camera process, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process exits.

[0162] In this embodiment, after the main process starts the data receiving process, an additional data processing process is started to perform triangulation calculations. The data processing process performs triangulation calculations after receiving the integrated results from the sub-camera process in each frame, and exits the process when it receives poison pill data sent by the data receiving process.

[0163] In the process-level parallelization scheme described in this embodiment, the time-consuming attitude detection network is processed in parallel, and the data reception, triangulation calculation and other operations are further parallelized. This effectively improves CPU utilization while avoiding deadlock, and is the basic framework for the real-time operation of this invention.

[0164] In this embodiment, to further improve parallelism, increase memory utilization, and reduce the latency between output and input for each frame, a double-buffering mechanism is introduced between the sub-camera process and the data receiving process, enabling parallelization of writing and reading the attitude detection network's results. The double-buffering mechanism between a sub-camera process and the data receiving process is as follows: Figure 5 As shown.

[0165] While the sub-camera process writes pixel coordinates to one buffer, the data receiving process can read data from the other buffer. This dual-buffering mechanism avoids data loss when the triangulation module's processing speed is slower than the pose detection network. Compared to a single-buffering mechanism, it improves data throughput and reduces the impact of bottlenecks.

[0166] Compared with existing technologies, the method proposed in this invention innovatively introduces time-domain information, providing a new method for high-precision human joint tracking in real-time scenarios, and has low cost consumption. It is a very novel and valuable method.

[0167] This invention provides a time-domain prediction-based thermal model of human joints. Figure 3 The triangulation method and system combine a human joint time-domain prediction method based on linear Kalman filtering with a human joint triangulation method based on spatial heatmaps, and provide a process-level parallel operation framework to achieve real-time label-free high-precision single-person human tracking. This effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0168] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0169] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for triangulation of human joint heat map based on time domain prediction, characterized in that, The method comprises the following steps: Step S1: starting a sub-camera process for each camera, synchronously running a 2D human pose detection network by using a process-level parallelization framework, and outputting pixel coordinates of human joint points on a camera plane; Step S2: predicting 3D positions of joint points in a current frame based on a linear Kalman filter and a 3D position and a speed of joint points in a previous frame; The step S2 comprises: Step S2.1: making a preliminary prediction of the 3D position of the human joint points in the current frame by using a prediction matrix and adding noise; Step S2.2: further correcting the best prediction value of the position of each joint point based on the pixel coordinates obtained in step S1 and the 3D position of the human joint points calculated by using a linear triangulation method; Step S3: establishing a 3D joint confidence heat map based on a time domain prediction point, and calculating a final position of the joint points in the current frame; The step S1 comprises: A sub-camera process: at the beginning of system startup, the main process detects the number of cameras, starts a sub-camera process for each camera, and binds the sub-camera process to the camera; A data receiving process: after starting the corresponding sub-camera process for each camera, the main thread additionally starts a data receiving process to receive and integrate the running results of all sub-camera processes in the current frame; A data processing process: after starting the data receiving process, the main process additionally starts a data processing process to perform triangulation calculation; The sub-camera process comprises: in each frame, the sub-camera process independently runs the 2D human pose detection network by using the RGB input of the current frame of the camera, and calculates the pixel coordinates of the human joint points in the current frame; After obtaining a multi-process lock, the pixel coordinates are sent to the data receiving process, and the sub-camera process enters a sleep state and waits to be awakened by the data receiving process, and after being awakened, the sub-camera process executes the input of the next frame; When the corresponding camera is closed or the last frame of the video has been read, the sub-camera process sends a special data structure, i.e., a poison pill, to the data receiving process to indicate the end of the process, and then exits the process; A double-buffering mechanism is introduced between the sub-camera process and the data receiving process, so that the writing and reading of the running results of the 2D human pose detection network are parallelized, and when the sub-camera process writes the pixel coordinates into a certain buffer, the data receiving process reads the data in another buffer; The data receiving process comprises: after receiving all the results of the sub-camera processes in each frame, the data receiving process forwards the integrated results to the data processing process, and awakens all the sleeping sub-camera processes to process the input of the next frame; If the data receiving process receives the poison pill data sent by a sub-camera process, all the sub-camera processes that have not ended are terminated, another poison pill is sent to the data processing process, and then the process is exited.

2. The human joint heat map triangulation method based on time domain prediction according to claim 1, wherein, The data processing process comprises: after receiving the integrated results of the sub-camera processes in each frame, the data processing process performs triangulation calculation to obtain the triangulation calculation results; When the data receiving process sends the poison pill data, the data processing process is exited.

3. The human joint heat map triangulation method based on time domain prediction according to claim 1, wherein, The step S3 comprises: Step S3.1: dividing the 3D space into a voxel grid with a fixed resolution and a fixed size; Step S3.2: Projecting back to each camera plane for each voxel grid, calculating its confidence value and generating 3D joint confidence heat map; Step S3.3: Calculating the final position of the joint from the 3D joint confidence heat map.

4. A human joint heat map triangulation system based on time domain prediction, characterized in that, Comprise: Module M1: Start a sub-camera process for each camera, synchronize running 2D human pose detection network using process-level parallelization framework, output human joint pixel coordinates in camera plane; Module M2: Predict the 3D position of each joint in the current frame based on the previous frame human joint 3D position and velocity using linear Kalman filter; The module M2 comprises: Module M2.1: Make a preliminary prediction of the 3D position of the human joint in the current frame using the prediction matrix and adding noise; Module M2.2: Based on the pixel coordinates obtained by module M1, the 3D position of the human joint calculated by the linear triangulation method is used as the observation value to further correct the best prediction value of each joint position; Module M3: Based on the time domain prediction point, establish a 3D joint confidence heat map, and calculate the final position of the joint in the current frame; The module M1 comprises: Sub-camera process: At the beginning of system startup, the main process detects the number of cameras and starts a sub-camera process for each camera and binds to the camera; Data receiving process: After starting the corresponding sub-camera process for each camera, the main thread additionally starts a data receiving process to receive and integrate the running results of all sub-camera current frames; Data processing process: After starting the data receiving process, the main process additionally starts a data processing process to perform triangulation calculation; The sub-camera process comprises: in each frame, the sub-camera process independently runs the 2D human pose detection network using the RGB input of the current frame of the camera to calculate the pixel coordinates of the human joint in the current frame; After obtaining the multi-process lock, the pixel coordinates are sent to the data receiving process, and the sleep state is entered, waiting to be awakened by the data receiving process, and after being awakened, the next frame of input is executed; When the corresponding camera is closed or the last frame of the video has been read, the sub-camera process sends a special data structure, i.e. poison pill, to the data receiving process to indicate the end of the process, and then exits the process; The double buffering mechanism is introduced between the sub-camera process and the data receiving process, so that the writing and reading of the 2D human pose detection network running results are parallelized, and when the sub-camera process writes the pixel coordinates to a certain buffer, the data receiving process reads the data in another buffer; The data receiving process comprises: after receiving all the sub-camera process results in each frame, the data receiving process forwards the integrated results to the data processing process, and wakes up all the sleeping sub-camera processes to process the next frame of input; If a poison pill data sent by a sub-camera process is received, all unfinished sub-camera processes are terminated, another poison pill is sent to the data processing process, and then the process is exited; The data processing process comprises: the data processing process performs triangulation calculation after receiving the integrated sub-camera process results in each frame, and obtains the triangulation calculation results; The process exits when it receives the poison pill data sent by the data receiving process; The module M3 comprises: Module M3.1: divide the 3D space into a fixed-resolution, fixed-size voxel grid; Module M3.2: re-project each voxel grid back to each camera plane, calculate its certainty value and generate a 3D joint certainty heat map; Step S3.3: calculate the final position of the joint from the 3D joint certainty heat map.

Images