A real-time human perception method based on neural network

Abstract

The application discloses a real-time human perception method based on a neural network, which comprises the following steps: extracting the position, scale and body segmentation result of a human key point in an image; solving the position information and scale value of the human key point in a three-dimensional space; constructing a human skeleton model according to the position of the human key point, uniformly sampling in the scale range of the key point around the skeleton model, and obtaining a human reconstructed point cloud set; projecting the reconstructed point cloud into the body segmentation result, and filtering out the point cloud corresponding to the projection result outside the body segmentation result; selecting a reference frame view angle and a target frame view angle, calculating the intersection of the observable point sets under the reference frame and the target frame view angles, counting the number of point clouds in the intersection, uniformly sampling on the epipolar line by using an epipolar search method, comparing the similarity between the sampling point projection result and the reference point projection result, and selecting the most similar sampling point as the final optimization result; and the application can realize more stable and accurate perception effect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer collaboration, specifically to a real-time human perception method based on neural networks. Background Technology

[0002] Human-robot collaboration refers to a new industrial model where robots perform highly precise and repetitive tasks, while workers assist them in creative work. To achieve the desired functions of human-robot collaboration, the safety of such systems must first be addressed.

[0003] Accurate and complete scene perception technology is a prerequisite for realizing safe human-machine collaboration. Existing human-machine collaboration systems can be divided into human-machine collaboration systems based on the type of sensors used, namely those relying on visual cameras, those relying on visual-depth cameras, and those relying on depth sensors.

[0004] Compared to visual-depth cameras and depth sensors, visual cameras offer advantages such as low cost, simple deployment, and high versatility, enabling the deployment of perception systems at a lower cost. Scene perception using visual cameras remains a hot research topic in computer vision; therefore, human-computer collaborative scene perception technology based on visual cameras has high research value and broad application prospects.

[0005] Compared to the widely studied and emphasized tasks of target recognition and scene reconstruction, the task of 3D human perception has not received sufficient attention. Current human-computer collaboration systems and solutions represent the human body as a skeletal model composed of key points, and design interaction and planning technologies based on this model. Such designs pose significant security risks in practical applications.

[0006] With the rise of deep learning, many computer vision tasks have achieved state-of-the-art results using neural networks, such as human surface reconstruction using visual cameras. Convolutional neural networks extract image features by performing convolution operations on image matrices. By stacking multiple layers of networks performing convolution calculations together to form a deep network model, they can achieve excellent performance in many tasks. However, current technologies struggle to obtain accurate human perception results in complex, unstructured human-computer collaboration scenarios.

[0007] The above background information is provided only to assist in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed before the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0008] To address the technical problem of inaccurate human perception results, the present invention aims to provide a real-time human perception method based on neural networks.

[0009] The technical problem of this invention is solved by the following technical solution:

[0010] A real-time human perception method based on neural networks includes the following steps:

[0011] S1. Real-time acquisition of images of the human-machine collaborative assembly area; extraction of pixel coordinates, scale values, and confidence scores of human key points, as well as human segmentation results; statistical analysis of the confidence scores of human key point detection, and calculation of the position coordinates and scale values ​​of human key points in three-dimensional space.

[0012] S2. Construct a human skeleton model, perform uniform random sampling, and construct a dense human reconstruction point cloud set; project the dense human reconstruction point cloud set onto the human segmentation results from each viewpoint.

[0013] S3. Select the reference frame view and the target frame view, calculate the intersection of the observable point sets under the reference frame and the target frame view, and count the number of point clouds in the intersection.

[0014] S4. Determine whether there is a sufficient number of point clouds that can be observed simultaneously by the reference frame and the target frame. If the number of point clouds observed simultaneously is less than or equal to the threshold, repeat step S3; if it is greater than the threshold, execute step S5.

[0015] S5. Draw epipolar lines from the observation optical center from the perspective of the reference frame and sample within the neighborhood of the epipolar lines; calculate the similarity of pixel distribution within the local window at the projection position of the sampling point in the perspective of the reference frame and the target frame; select the sampling point with the highest similarity evaluation on each epipolar line. If the similarity evaluation of the sampling point is higher than the threshold, add it to the reconstruction result; otherwise, discard it.

[0016] S6. After traversing all perspectives, output the results of human density perception.

[0017] In some embodiments of the present invention, step S1 involves using a multi-view synchronous camera to acquire images of the human-machine collaborative assembly area in real time, and the multiple cameras used in the multi-view synchronous camera need to receive the same external trigger signal to achieve synchronization.

[0018] In some embodiments of the present invention, step S1 uses a convolutional neural network to extract the pixel coordinates, scale values, and confidence scores of human body key points from various viewpoints, as well as the human body segmentation results; the main structure of the convolutional neural network includes a backbone network, an encoder, and multiple decoders; the convolutional neural network is trained in a supervised manner or a pre-trained model; the format for extracting human body key points from various viewpoints using the convolutional neural network is selected as either the upper body key point format or the full body key point format; the human body segmentation result is a human body mask with the same resolution as the input image, where the human body portion is 1 and the non-human body portion is 0.

[0019] In some embodiments of the present invention, step S1 calculates the confidence level of human keypoint detection from each viewpoint. The confidence level of the human keypoints from each viewpoint is then used to determine the reliability of the detection result. If the confidence level of the human keypoints is greater than 0.7, the detection result is accurate and can proceed to the next step of calculation; otherwise, it is not considered. For fewer than two human keypoints with a confidence level higher than the threshold, no further reconstruction is performed. For at least two human keypoints with a confidence level higher than the threshold, the next step of calculation can proceed. When calculating the position coordinates and scale values ​​of the human keypoints in three-dimensional space, the confidence level of the human keypoint detection is used to optimize the calculation result.

[0020] (W·A)X=0

[0021] Where X is the 3D homography coordinate of the keypoint, W is the detection confidence of the keypoint at the corresponding viewpoint, and A is the DLT matrix calculated based on camera parameters and keypoint detection results.

[0022] In some embodiments of the present invention, in step S2, the position coordinates and scale values ​​of the human body key points in three-dimensional space calculated in step S1 are used to construct a human skeleton model; the uniform random sampling specifically refers to: uniformly and randomly sampling 4000 points in a spherical region with the key point scale as the radius at the location of the human body key points, and uniformly and randomly sampling 4000 points in a cylindrical region with the key point scale as the radius in the skeleton region between key points as cylinders.

[0023] In some embodiments of the present invention, in step S2, the dense reconstructed point cloud set of the human body is projected onto the human body segmentation result under each viewpoint, and the projection position is determined to be within the human body segmentation result by whether the response value of the projection position is 1; otherwise, it is filtered out.

[0024] In some embodiments of the present invention, in step S4, it is determined whether there is a sufficient number of point clouds that can be observed simultaneously by the reference frame and the target frame. The determination is based on the following: if the number of jointly observed point clouds is less than or equal to a preset threshold, it is considered that there is no reliable joint observation relationship between the current reference frame and the target frame, and step S3 is repeated until two viewpoints with sufficient common observation points are found; if the number of jointly observed point clouds is greater than the preset threshold, it is considered that there is a reliable joint observation relationship between the current reference frame and the target frame, and reliable correspondence can be found between the images of the corresponding viewpoints to optimize the reconstruction results, and step S5 is executed.

[0025] In some embodiments of the present invention, step S5 uses the point cloud information calculated in step S3 to draw an epipolar line from the observation optical center of the reference frame viewpoint to each observation point, and performs uniform sampling at equal intervals in the neighborhood of the observation point on the epipolar line; the observation optical center of the reference frame viewpoint refers to the translation component of the extrinsic parameters of the corresponding camera; five points are uniformly sampled at 2mm intervals in the direction near the observation point and in the direction away from the optical center.

[0026] In some embodiments of the present invention, the criterion for determining whether a sampling point is used in step S5 is whether the similarity evaluation is greater than 1; by projecting the sampling points obtained in step S4 onto the target frame view, the similarity between the pixel distribution in the local window at the projection position of the sampling point in the reference frame and the pixel distribution in the local window at the projection position in the target frame is calculated; the similarity S is calculated as follows:

[0027]

[0028] Where A and B are 3×3 local windows near the projection positions on the reference frame and the target frame, respectively, and i and j are the row and column indices of the local windows.

[0029] In some embodiments of the present invention, step S6 determines whether all viewpoints have been traversed. If the traversal has been completed, the reconstruction process ends and the human body density perception result is output; otherwise, step S3 is executed.

[0030] The present invention has the following beneficial effects:

[0031] The real-time human perception method based on neural networks proposed in this invention decouples the human perception task into two sub-tasks: human sparse feature estimation and dense reconstruction, by adopting a human perception framework based on a visual camera. This can address the stability problem of existing human perception methods in complex and unstructured scenarios, thereby achieving a more stable and accurate perception effect.

[0032] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0033] Figure 1 This is a flowchart of a real-time human perception method based on neural networks according to an embodiment of the present invention;

[0034] Figure 2 This is a diagram of the convolutional neural network structure used in the embodiments of the present invention;

[0035] Figure 3a This is a schematic diagram illustrating the principle of similarity evaluation used in the embodiments of the present invention;

[0036] Figure 3b This is an embodiment of the present invention. Figure 3a Enlarged view of the area within the dashed box;

[0037] Figure 4a This is a schematic diagram of the human body's outer contour reconstructed from posture 1 in an embodiment of the present invention;

[0038] Figure 4b This is a schematic diagram of the human body's outer contour reconstructed from posture 2 in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0040] Current designs do not consider accurate external boundary information of the human body, posing significant safety risks in practical applications. How to obtain accurate external boundary information of the human body and use it for collision-free trajectory planning and interaction of robots remains a gap in the field of human-robot collaboration. Experimental results from existing technologies rely on the size and diversity of the dataset, making it difficult to obtain accurate results in complex, unstructured human-robot collaboration scenarios.

[0041] Recently, data densification for perception has become an emerging research direction in the field of scene perception. Existing dense perception methods aim to obtain dense representations of scenes. However, due to the difficulty of labeling training data, dense perception methods lack generalization ability for complex scenes. On the other hand, sparse perception methods aim to obtain sparse representations of scenes. These representations have advantages such as stability, robustness, and ease of acquisition. How to densify sparse perception results into dense perception results has become the focus of research for many scene perception methods and has broad research and application prospects.

[0042] The purpose of this invention is to improve the human body perception method in existing human-computer collaboration systems. It proposes a real-time human body perception method based on neural networks. During the perception process, the method outputs sparse features such as human body key points and human body segmentation results through convolutional neural networks. It constructs a dense point cloud representation of the human body through sampling, and optimizes the dense reconstruction results by combining image information and human body segmentation results, and outputs accurate three-dimensional human body contour information.

[0043] The following embodiments of the present invention provide a real-time human perception method based on neural networks. Employing a human perception framework based on a visual camera, the dense human perception task is decoupled into two sub-tasks: sparse human perception and dense reconstruction. This addresses the stability issues of existing human perception methods in complex, unstructured scenes. The neural network-based sparse human perception, leveraging abundant data resources and deep learning techniques, demonstrates stability and accuracy surpassing dense perception algorithms. Combined with dense reconstruction methods, it can obtain even more stable and accurate human perception results. The method includes the following steps:

[0044] S1. Use a multi-view synchronous camera to acquire real-time images of the human-machine collaborative assembly area;

[0045] S2. Use a convolutional neural network to extract human key points and human segmentation results from various perspectives. The key point detection output of this network includes the pixel coordinates, scale value and confidence of human key points in the current frame image. The key point format can be defined in different ways according to the actual task requirements. The human segmentation output is a human mask with the same resolution as the input image, with 1 for human parts and 0 for non-human parts.

[0046] S3. Calculate the confidence of keypoint detection from each viewpoint. For viewpoints with less than 2 keypoints whose confidence is higher than the threshold, no further reconstruction is performed. For viewpoints with more than 2 keypoints whose confidence is higher than the threshold, use the DLT algorithm to calculate the position coordinates and scale values ​​of the keypoints in 3D space.

[0047] S4. Construct a human skeleton model using the human key points calculated in step S3. Perform uniform random sampling within a region around the skeleton model with the scale of the key points as the radius to obtain a set of human reconstructed point clouds.

[0048] S5. Project the reconstructed point cloud onto each viewpoint and determine whether the projection position is within the human body segmentation result. If the projection result is outside the human body segmentation result, it is regarded as an outlier and filtered out.

[0049] S6. Select two viewpoints as the reference frame and the target frame, calculate the intersection of the observable point sets under the reference viewpoint and the target viewpoint, and count the number of points in the intersection.

[0050] S7. Determine if there is a sufficient number of point clouds that can be observed simultaneously by the reference frame and the target frame. If the number of point clouds observed together is less than or equal to the threshold, repeat step S6 until two viewpoints with enough points observed together are found. If the number of point clouds observed together is greater than the threshold, proceed to step S8.

[0051] S8. Using the point cloud information calculated in step S6, draw an epipolar line from the observation optical center of the reference frame to each observation point, and perform uniform sampling at equal intervals in the neighborhood of the observation point on the epipolar line.

[0052] S9. Project the sampling points obtained in step S7 onto the target frame viewpoint, and calculate the similarity between the pixel distribution in the local window at the projection position of the sampling points in the reference frame and the pixel distribution in the local window at the projection position in the target frame.

[0053] S10. Select the sampling point with the highest similarity evaluation on each epipolar line, and determine whether the similarity evaluation of these sampling points exceeds the threshold. If the similarity evaluation of the point is higher than the threshold, add it to the reconstruction result; if the similarity evaluation of the point is lower than the threshold, discard it.

[0054] S11. Determine whether all viewpoints have been traversed. If traversal has been completed, proceed to step S12; otherwise, proceed to step S6.

[0055] S12. End the reconstruction process and output the results of human density perception.

[0056] Example 1

[0057] refer to Figure 1 The positioning method of this invention includes the following steps S1 to S12:

[0058] S1. Real-time image acquisition of the human-machine collaborative area is achieved using multi-view synchronous cameras. Preparatory work before image acquisition includes: calibrating the intrinsic parameters of each camera individually using Zhang's calibration method and a checkerboard pattern, and then jointly calibrating the extrinsic parameters using a checkerboard pattern to obtain the position and orientation information of all cameras in the reference coordinate system. During operation, all cameras need to receive signals from the same trigger source to achieve hard synchronization.

[0059] S2. Use a convolutional neural network to extract human body key points and obtain human body segmentation results, refer to... Figure 2The convolutional neural network described in this embodiment of the invention mainly includes a backbone network, an encoder, and two decoders. The backbone network extracts image features by first slicing and recombining the image to obtain a lower-resolution input, reducing the complexity of subsequent calculations. Then, feature maps of different resolutions are obtained through a 3x3 convolutional layer, three residual modules, and a pyramid pooling module and input into the encoder. The residual module contains two cascaded 3x3 convolutional layers and a parallel connection layer for feature fusion. The pyramid pooling module contains parallel 3x3, 5x5, and 7x7 max pooling layers and a concatenation layer for combining multi-scale features. The encoder receives features of different resolutions output from the three residual modules of the backbone network. The low-resolution feature map is upsampled after passing through a 3x3 convolutional layer and concatenated with the higher-resolution feature map from the backbone network. Each convolutional layer is followed by a batch normalization layer and a SiLU activation function. Finally, the encoder outputs three feature maps of different resolutions for that frame of image. The keypoint decoder receives three feature maps at different scales from the encoder, outputs detection results at different scales after passing through six depthwise separable convolutional layers, and finally performs an upsampling operation on the low-resolution result and performs difference fusion with the higher-resolution result to obtain the extracted human keypoint location information, scale values, and confidence results. The segmentation decoder receives the highest-resolution feature map from the encoder, and then outputs the human segmentation result after passing through two consecutive sets of 3*3 convolutional layers and an upsampling module. The segmentation result is a human mask with the same resolution as the input image, where the response value of the human part is 1 and the response value of the non-human part is 0. The format of the human keypoints output by the convolutional neural network can be changed according to the actual task requirements. For example, the format of the upper body keypoints can be selected for collaborative assembly tasks, and the format of the whole body keypoints can be selected for collaborative transportation tasks.

[0060] S3. Calculate the confidence score of human keypoint detection from each viewpoint. For viewpoints with fewer than two keypoints and a confidence score higher than the preset threshold of 0.7, they are considered unreliable for observation and therefore not further reconstructed. Detection failures may be due to the corresponding keypoint not being in the image or motion blur. For viewpoints with at least two keypoints and a confidence score higher than the preset threshold of 0.7, these keypoints are considered reliably observed. A weighted DLT (Direct Linear Transform) algorithm is used to calculate the position coordinates and scale values ​​of the human keypoints in the current frame in 3D space. The improvement lies in using the confidence score of keypoint detection to optimize the calculation results.

[0061] (W·A)X=0

[0062] Where X is the 3D homography coordinate of the keypoint, W is the detection confidence of the keypoint at the corresponding viewpoint, and A is the DLT matrix calculated based on camera parameters and keypoint detection results.

[0063] S4. Construct a human skeleton model using the human key points calculated in step S3. Use the BPP algorithm to perform uniform random sampling in the area around the skeleton model with the key point scale as the radius. Randomly and uniformly sample 4000 points in a spherical area with the key point scale as the radius at the human key point location. Treat the skeleton area between key points as a cylinder and randomly and uniformly sample 4000 points in a cylindrical area with the key point scale as the radius to obtain a dense human reconstruction point cloud set.

[0064] S5. Based on the assumption that the projection positions of points on the real human body contour should be within the human body segmentation result from each viewpoint, the reconstructed point cloud is projected onto each viewpoint, and it is determined whether the projection position is within the human body segmentation result. The criterion for this determination is whether the response value of the corresponding projection position is 1. All points whose projection positions are outside the human body segmentation result are considered outliers and are filtered out.

[0065] S6. Select two viewpoints as the reference frame and the target frame, use the HPR algorithm to calculate the observable point set under each viewpoint, calculate the common point set under the reference viewpoint and the target viewpoint, that is, the intersection of the observable point sets of the reference viewpoint and the target viewpoint, and count the number of points in the common point set.

[0066] S7. Determine if there are enough point clouds that can be simultaneously observed by the reference frame and the target frame. The determination is based on the following: if the number of jointly observed point clouds is less than or equal to a preset threshold of 50, it is considered that there is no reliable joint observation relationship between the current reference frame and the target frame. Repeat step S6 until two viewpoints with enough jointly observed points are found. If the number of jointly observed point clouds is greater than the preset threshold of 50, it is considered that there is a reliable joint observation relationship between the current reference frame and the target frame. Reliable correspondences can be found between the images of the corresponding viewpoints to optimize the reconstruction results. Proceed to step S8.

[0067] S8. Using the point cloud information calculated in step S6, draw an epipolar line from the observation optical center of the reference frame to each observable point. The observation optical center of the reference frame refers to the translation component of the extrinsic parameters obtained by the camera in step S1. Sample 5 points at equal intervals of 2 mm in the direction of the epipolar line near and away from the optical center, respectively, near the observable point.

[0068] S9. Project the sampling points obtained in step S7 onto the target frame view to perform block matching search, referencing... Figure 3a and Figure 3b D c This represents the reconstructed points that meet the conditions after filtering in step S5. D represents the k-th sampling point obtained in step S8. G Represents points on the boundary of a real human body, where Figure 3b yes Figure 3a A magnified view of the dashed box area. The similarity between the pixel distribution within a local window at the projected location of the sampling point in the reference frame and the pixel distribution within a local window at the projected location in the target frame is calculated. The similarity S is calculated as follows:

[0069]

[0070] Where A and B are 3×3 local windows near the projection position in the reference frame and the target frame, respectively, and i and j are the row and column indices of the local window.

[0071] S10. Select the point with the highest similarity evaluation of the sampling points on each epipolar line. If the similarity evaluation of the point is higher than the preset threshold 1, it is considered that the point has a high pixel correlation with the corresponding position in the reference frame and the target frame. It can be considered that the point is near the actual human body boundary and is added to the reconstruction result. If the similarity evaluation of the point is lower than the threshold, it is considered that the point is far away from the actual human body boundary and is discarded.

[0072] S11. Determine whether all viewpoints have been traversed. If traversal has been completed, proceed to step S12; otherwise, proceed to step S6.

[0073] S12. End the reconstruction process and output the human density perception results. (Reference) Figure 4a and Figure 4b The reconstructed human body obtained by this method has a more accurate outer contour and does not suffer from severe distortion. This method can achieve a real-time update rate of 7Hz, while the best-performing existing method, the pixel-aligned implicit function (PIFu), can only achieve a update rate of 0.33Hz.

[0074] The real-time human perception method based on neural networks proposed in the above embodiments of the present invention includes: extracting the location, scale, and human segmentation results of human key points from multi-view images; selecting a definition method for human key points according to actual needs; combining camera parameters and image human key point estimation results to calculate the location information and scale values ​​of human key points in three-dimensional space; constructing a human skeleton model based on the location of human key points, uniformly sampling within the scale range of key points around the skeleton model to obtain a human reconstruction point cloud; projecting the reconstructed point cloud onto the human segmentation results, filtering out point clouds whose corresponding projection results are outside the human segmentation results; extracting part of the surface point cloud based on the camera co-view relationship, uniformly sampling along the epipolar line using the epipolar line search method, comparing the similarity between the projection results of the sampled points and the projection results of the reference points, and selecting the sampled point with the most similar result as the final optimized result. The above embodiments of the present invention can achieve a more stable and accurate perception effect.

[0075] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A real-time human perception method based on neural networks, characterized in that, Includes the following steps: S1. Real-time acquisition of images of the human-machine collaborative assembly area; Extract the pixel coordinates, scale values, and confidence scores of key human body points, as well as the human body segmentation results; The confidence level of human keypoint detection is statistically analyzed, and the position coordinates and scale values ​​of human keypoints in three-dimensional space are calculated. S2. Construct a human skeleton model, perform uniform random sampling, and construct a dense human reconstruction point cloud set; project the dense human reconstruction point cloud set onto the human segmentation results from each viewpoint. S3. Select the reference frame view and the target frame view, calculate the intersection of the observable point sets under the reference frame and the target frame view, and count the number of point clouds in the intersection. S4. Determine whether there is a sufficient number of point clouds that can be observed simultaneously by the reference frame and the target frame. If the number of point clouds observed simultaneously is less than or equal to the threshold, repeat step S3; if it is greater than the threshold, execute step S5. S5. Draw out the epipolar line from the observation optical center from the perspective of the reference frame, and sample in the neighborhood of the epipolar line; Calculate the similarity of pixel distribution within the local window at the projection position of the sampling point in the reference frame and the target frame view; Select the sampling point with the highest similarity evaluation on each epipolar line. If the similarity evaluation of the sampling point is higher than the threshold, it is added to the reconstruction result; otherwise, it is discarded. S6. After traversing all perspectives, output the results of human density perception.

2. The real-time human perception method based on neural networks as described in claim 1, characterized in that, Step S1 involves using a multi-view synchronous camera to acquire images of the human-machine collaborative assembly area in real time, and the multiple cameras used in the multi-view synchronous camera must receive the same external trigger signal to achieve synchronization.

3. The real-time human perception method based on neural networks as described in claim 1, characterized in that, Step S1 uses a convolutional neural network to extract the pixel coordinates, scale values, confidence scores, and human segmentation results of key human points from various viewpoints; the main structure of the convolutional neural network includes a backbone network, an encoder, and multiple decoders; the convolutional neural network is trained in a supervised manner or a pre-trained model; the format for extracting key human points from various viewpoints using the convolutional neural network is selected as either the upper body key point format or the full body key point format. The human body segmentation result is a human body mask with the same resolution as the input image, where the human body portion is 1 and the non-human body portion is 0.

4. The real-time human perception method based on neural networks as described in claim 1, characterized in that, Step S1: Calculate the confidence score of human keypoint detection from each viewpoint. Determine the reliability of the detection result by checking if the confidence score of each human keypoint is greater than 0.

7. If the confidence score is greater than 0.7, the detection result is accurate and can proceed to the next step of calculation; otherwise, it is not considered. For fewer than two human keypoints with a confidence score higher than the threshold, no further reconstruction is performed. For at least two human keypoints with a confidence score higher than the threshold, the next step of calculation can proceed. When calculating the position coordinates and scale values ​​of human keypoints in three-dimensional space, the confidence score of human keypoint detection is used to optimize the calculation result. (W·A)X=0 Where X is the 3D homography coordinate of the keypoint, W is the detection confidence of the keypoint at the corresponding viewpoint, and A is the DLT matrix calculated based on camera parameters and keypoint detection results.

5. The real-time human perception method based on neural networks as described in claim 1 or 4, characterized in that, In step S2, the position coordinates and scale values ​​of the human body key points in three-dimensional space calculated in step S1 are used to construct a human skeleton model; the uniform random sampling is specifically as follows: 4000 points are uniformly and randomly sampled in a spherical region with the key point scale as the radius at the location of the human body key points, and the skeleton region between the key points is regarded as a cylinder, and 4000 points are uniformly and randomly sampled in a cylindrical region with the key point scale as the radius.

6. The real-time human perception method based on neural networks as described in claim 1, characterized in that, In step S2, the dense reconstructed point cloud set of the human body is projected onto the human body segmentation results from each viewpoint. The projection position is determined to be within the human body segmentation results by whether the response value of the projection position is 1; otherwise, it is filtered out.

7. The real-time human perception method based on neural networks as described in claim 1, characterized in that, In step S4, it is determined whether there is a sufficient number of point clouds that can be observed simultaneously by the reference frame and the target frame. The determination is based on the following criteria: if the number of jointly observed point clouds is less than or equal to a preset threshold, it is considered that there is no reliable joint observation relationship between the current reference frame and the target frame, and step S3 is repeated until two viewpoints with sufficient common observation points are found; if the number of jointly observed point clouds is greater than the preset threshold, it is considered that there is a reliable joint observation relationship between the current reference frame and the target frame, and reliable correspondence can be found between the images of the corresponding viewpoints to optimize the reconstruction results, and step S5 is executed.

8. The real-time human perception method based on neural networks as described in claim 1, characterized in that, Step S5 uses the point cloud information calculated in step S3 to draw an epipolar line from the observation optical center of the reference frame view to each observation point, and performs uniform sampling at equal intervals in the neighborhood of the observation point on the epipolar line; the observation optical center of the reference frame view refers to the translation component of the extrinsic parameters of the corresponding camera; five points are uniformly sampled at 2mm intervals in the direction closer to the optical center and further away from the optical center near the observation point.

9. The real-time human perception method based on neural networks as described in claim 1, characterized in that, In step S5, the criterion for determining whether a sampling point is used is whether the similarity evaluation is greater than 1. By projecting the sampling points obtained in step S4 onto the target frame view, the similarity between the pixel distribution within the local window at the projection position of the sampling point in the reference frame and the pixel distribution within the local window at the projection position in the target frame is calculated. The similarity S is calculated as follows: Where A and B are 3×3 local windows near the projection positions on the reference frame and the target frame, respectively, and i and j are the row and column indices of the local windows.

10. The real-time human perception method based on neural networks as described in claim 1, characterized in that, Step S6 determines whether all viewpoints have been traversed. If the traversal has been completed, the reconstruction process ends and the human dense perception result is output; otherwise, step S3 is executed.

Images