A human posture estimation and spatial positioning method based on cross-view information fusion

Abstract

The present invention discloses a cross-perspective information fusion method for human posture estimation and spatial positioning, which combines a machine vision image acquisition system with a high-precision servo platform, replaces a fixed image acquisition lens with an electronically controlled servo zoom lens, and realizes multi-eye synchronous image acquisition of the same athlete in a large stadium through two or more sets of pan-tilt zoom servo motion platforms, thereby broadening the application field of human posture estimation technology and motion analysis. The present invention uses a dual-perspective information fusion method to solve the problem of the inevitable loss of human key point position information due to occlusion of the target limb in a single-perspective image acquisition method. By constructing an incomplete autoencoder, the output of a multi-granularity cross-layer cascade network is subjected to multi-perspective information fusion, thereby realizing mutual correction and completion of human key point position information under different perspectives.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and computer control, and specifically relates to technical fields such as visual servo tracking technology, high-precision servo control, human posture estimation and cross-view information fusion, and is a cross-view information fusion and space positioning method. Background Art

[0002] The patent application with publication number CN113283373A discloses a method for enhancing the detection of limb motion parameters by a depth camera. This method uses a non-contact depth camera for data collection and does not interfere with the subject. However, due to the limitation of the field of view of a single depth camera, it cannot avoid the influence of the subject's own limb occlusion. Secondly, this method uses the FC fully connected layer direct regression method, which has the problem of low accuracy.

[0003] The patent application with publication number CN113205595A discloses a method for constructing a 3D human posture estimation model and its application. The method processes the temporal information of a two-dimensional skeleton and utilizes spatial features and temporal information of different scales, but cannot solve the problem of single-frame image information loss caused by occlusion.

[0004] Using a variable-focus multi-eye tracking system for image acquisition can effectively solve the contradiction between the field of view and resolution of traditional machine vision systems, expand the image acquisition range, and reduce the impact of limb occlusion on the person being measured. Summary of the invention

[0005] In view of the defects of the prior art, the present invention provides a method for human posture estimation and spatial positioning by cross-view information fusion, so as to solve the problems of small application scope, low detection accuracy and poor portability of the prior art.

[0006] In order to achieve the above object, in a first aspect, the present invention provides a multi-view information fusion posture estimation method, comprising the following steps:

[0007] S1. Camera pose calibration: Select, including but not limited to, 4 marker points within the use range, use a total station to measure the spatial positions of the marker points, and use the PNP calibration method to calculate the camera pose parameters.

[0008] S2. Construct a multi-granularity cross-layer cascade network, making full use of multi-granularity feature fusion technology and hierarchical context learning ideas. The network is mainly divided into feature encoding stage, multi-level feature fusion stage and feature decoding stage;

[0009] The input image size is unified to 512×512 through affine transformation, and the image pixel values ​​are normalized according to the mean and variance of the COCO dataset;

[0010] The feature extraction backbone network is mainly composed of three CSP cross-layer connections. The CSP cross-layer connection performs channel splicing on the input and the output after passing through multiple feature extraction modules. The channel splicing uses convolution with a convolution kernel size of 1 and a step size of 1 to fuse channel information.

[0011] The multi-granularity feature fusion module is mainly divided into two main branches. Each main branch consists of a 1×1 convolutional layer with a cross-layer jump connection. The outputs of different branches are added together to obtain the final module output.

[0012] The main function of the detection front end is to perform corresponding decoding operations on the feature map of the feature encoding network, and obtain the human body position distribution heat map, bounding box size, bias regression value, key point position distribution heat map, key point bias value and key point discrete value respectively;

[0013] S3, separation coding multi-view information fusion;

[0014] The input of the separation encoding network is the images taken at the same time but from different perspectives, denoted as I 0 and I 1 The two images are respectively processed through feature extraction modules with the same structure and shared weights. The feature extraction module encodes the deep implicit features of the image. The implicit features are fused with the implicit features of other perspectives through the feature transfer channel through the incomplete autoencoder.

[0015] In a second aspect, the present invention provides a variable-focus multi-eye tracking image acquisition method, comprising:

[0016] The quantity includes but is not limited to two sets of ultra-high-definition servo motor systems, which respectively control the two-dimensional pointing mechanism to rotate in the horizontal and vertical directions;

[0017] The ultra-high-definition servo motor system is placed on the side of the target sports field, and uses network cables to achieve data transmission with the processing terminal;

[0018] The target position in the image is determined by using a temporal two-way comparison and selection strategy of the target detection and tracking candidate frames. After the system enters the tracking mode, the intersection and union ratio of the target detection candidate frame and the target tracking candidate frame with the target position of the previous frame is automatically calculated, and the candidate frame that meets the threshold requirement is selected as the target position in the image.

[0019] The ultra-high-definition servo motor system is a closed-loop control system. After obtaining the position of the target in the image, the PID control method is used to achieve the position and posture control of the gimbal.

[0020] The target three-dimensional spatial position is calculated based on the high-precision servo tracking pan / tilt encoder value and the zoom lens encoder feedback value, and automatic focusing is achieved.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) The present invention provides a method for human posture estimation and spatial positioning by cross-view information fusion. The fixed image acquisition lens is replaced with an electronically controlled servo zoom lens. Two or more pan-tilt zoom servo motion platforms are used to achieve multi-eye synchronous image acquisition of the same moving target in a large sports field, thus broadening the application fields of human posture estimation technology and motion analysis. This technology can also be used to solve the spatial positioning problem of large-space virtual reality, which is urgently needed, and open up a new solution for large-space dynamic positioning measurement.

[0023] (2) The present invention provides a method for cross-view information fusion of human posture estimation and spatial positioning, which improves the translation equivariance of convolutional neural networks, thereby improving the convolutional network's ability to encode the target spatial position, and fully utilizes image feature information and context information of different granularities to obtain a more accurate heat map of the distribution of key points of the human body.

[0024] (3) The present invention provides a method for human posture estimation and spatial positioning by cross-view information fusion. It uses cameras installed at different positions to form a cross-field of view, and realizes the mutual complementation and correction of the heat map of the position distribution of key points of the human body between different perspectives. To a certain extent, it can effectively solve the occlusion problem between limbs.

[0025] (4) The present invention provides a method for human posture estimation and spatial positioning by fusing cross-view information, thereby realizing the automated measurement of the spatial position of a moving target. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a hierarchical design of a system architecture provided in Example 1 of the present invention;

[0027] Figure 2 A schematic diagram of the follow-up tracking shooting provided by Embodiment 1 of the present invention;

[0028] Figure 3 A flow chart of a target detection and tracking candidate box temporal bidirectional comparison and selection strategy provided in Example 1 of the present invention;

[0029] Figure 4 A schematic diagram of a high-precision servo tracking pan-tilt platform provided in Embodiment 1 of the present invention;

[0030] Figure 5 A schematic diagram of a multi-granularity cross-layer cascade network architecture provided in Example 1 of the present invention;

[0031] Figure 6 Schematic diagram of the network structure of the separation encoding type different perspective feature fusion provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the described embodiments are intended only to facilitate the understanding of the present invention and do not have any limiting effect on it. The accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. The structures shown in the accompanying drawings are part of the actual structure. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] Embodiment 1,

[0034] A cross-view information fusion method for human posture estimation and spatial positioning, suitable for large-scene moving target image acquisition and spatial position measurement. The schematic diagram of the hierarchical design of the system architecture is shown in the figure. Figure 1 As shown, the following steps are included:

[0035] S1, Schematic diagram of variable-focus multi-eye tracking acquisition Figure 2 As shown, the ultra-high-definition servo motor system drives the camera to rotate following the moving target. In order to further illustrate the follow-up tracking image acquisition method provided by the present invention, target detection and tracking are used as an example for explanation below;

[0036] Specifically, YOLO v4 and SiamDW are used as detectors and trackers respectively, and the target detection and tracking candidate box temporal bidirectional comparison selection strategy is as follows: Figure 3 As shown; the high-precision servo tracking system automatically returns to the starting position. The structural diagram of the high-precision servo tracking system is as follows Figure 4 As shown, the starting position is also the initial position of the moving target. The system calls the YOLOv4 target detection algorithm to detect moving people in the first two frames, stores the position of the moving target in the image, and uses the target bounding box to initialize the tracker 1;

[0037] When the prediction confidence of tracker 1 is greater than 0.8 but the intersection-over-union ratio with the target box of the previous frame is less than 0.3, the intersection-over-union ratio of the detector predicted bounding box and the target position stored in the previous frame is calculated. When the intersection-over-union ratio is greater than 0.3, the detection result is used to initialize the tracker;

[0038] When the prediction confidence of tracker 1 is less than 0.8, it can be considered that the tracker has lost the target. At this time, tracker 2 is initialized using the target position stored in the previous frame, and the intersection and union ratio of the predicted bounding box of tracker 2 and the predicted bounding box of the detector is calculated. The candidate box that meets the intersection and union ratio requirements is used to initialize tracker 1;

[0039] After continuous tracking for 20 frames, the tracker is corrected using the detector’s predicted bounding box.

[0040] Image data is written into shared memory in real time and distributed to the display process, target detection process, tracking process and image storage process.

[0041] S2. Encode the image features through the feature extraction network to generate a heat map of the key points of the human body. The position distribution of each key point of the human body is obtained through the heat map, and the classification combination of different individual key points is obtained by combining the bounding box of the human body.

[0042] Specifically, the feature extraction backbone network consists of three CSP cross-layer connections. The first two cross-layer connections only use 7×7 modules to reduce the amount of calculation and information loss. Downsampling operations are performed between cross-layer connection submodules. The pooling layer will cause a lot of information loss and affect the translation equivariance of the convolutional network. Therefore, all downsampling stages in the multi-granularity cascade network do not perform pooling, but use strided convolution with a convolution kernel size of 3 and a step size of 2 for downsampling.

[0043] Preferably, in order to avoid the loss of information such as corners, contours and boundaries caused by single-line propagation and optimize the gradient vanishing problem, the CSP residual cross-layer connection model is used to achieve the fusion of semantic information at different levels, and the convolution with a convolution kernel size of 1 and a step size of 1 is used for channel information fusion. Since there is a certain semantic gap between features at different levels, in order to avoid the problem of model non-convergence caused by simple addition operations, a cross-layer jump connection is formed by a 1×1 convolution layer, and the outputs of different branches are added to obtain the final module output. This process can be formalized as: Among them, f scale0 express Figure 5 The basic module in f scale1 Expressed as Figure 4 The 7×7 module in , x is the input feature map of the module, is the output feature map.

[0044] The main function of the detection front end is to perform corresponding decoding operations on the feature map of the feature encoding network, and obtain the human body position distribution heat map, bounding box size, bias regression value, key point position distribution heat map, key point bias value and key point discrete value, and key point feature extraction network. Through the human body heat map and bounding box size, different categories of human key points can be clustered according to the distance from the key point to the center of the human body. Due to the downsampling operation in the feature extraction network, there are rounding and scaling operations in the label production process. In order to ensure that the output result can be correctly mapped back to the original input image size, the bias and discrete value of the key point need to be predicted at the detection front end.

[0045] In order to solve the problem of sample imbalance, Focal Loss is used as the loss function of the heat map during model training. Focal Loss is expressed as Where N is the number of key points in the feature map, α and β are hyperparameters determined experimentally. According to the experimental results, α is set to 2 and β is set to 4. is the predicted value of each point in a channel feature map in the heat map. Before the loss calculation, the heat map is processed by Sigmod and the values ​​of all points are limited to (1e -4 ,1-1e -4 ). xyc Indicates the corresponding point value of the label heat map. (1-H xyc ) β The term is the weight of each point whose value is not 1. This weight term can effectively alleviate the problem of uneven positive and negative samples. The reason is that there is only one target point for each point and it presents a Gaussian distribution. The target point is actually the point with the largest response value in the Gaussian distribution, and the rest of the points will be used as negative samples. xyc =1, it means that the position is the target point, and the value of the position in the corresponding network generated heat map is The closer it is to 1, the smaller the calculated loss is. The further away from the weight item 1 The bigger.

[0046] It should be noted that when making label heat maps for training, overlapping key points are inevitable. Selecting the maximum value of the confidence of overlapping key points in the heat map instead of the average value can effectively keep the peak points clear and achieve the best effect.

[0047] The multi-granularity cascade network downsamples the image. In order to remap the target points obtained in the feature map back to the original image, the offset is regressed for each target point. The offset is trained using SmoothL1Loss and is expressed as in The value predicted by the network.

[0048] Furthermore, in order to fuse the key point information under different perspectives and obtain the high-precision key point position of the human body without giving up a certain amount of fast processing capability, a separation encoding dual-view information fusion network is used. The schematic diagram of the network result is shown in Figure 6 The input of the separation encoding network is the images taken at the same time but from different perspectives, denoted as I 0 and I 1 The two images are respectively passed through feature extraction modules with the same structure and shared weights. The feature extraction module encodes the deep implicit features of the image. The implicit features are fused with the implicit features of the other view through the feature transfer channel through the incomplete autoencoder. The heat map obtained by encoding the image under the set view u is expressed as Use convolutional layers H with kernel sizes of 1×w and 1×h respectively. out Sparse coding is performed to obtain The feature vector of H is then pooled using adaptive average pooling layers of size 1×w and 1×h. out Sparse coding is performed to obtain and The feature vector of the convolutional layer and the feature vector of the pooling layer are fused in the following way: and Among them, f is an incomplete autoencoder composed of fully connected layers, and the number of neurons in the hidden layer is initially set to one sixteenth of the number of input neurons. The autoencoder can be divided into two parts: an encoder represented by the function h = f(x) and a decoder r = g(x) for feature reconstruction. An incomplete autoencoder means an autoencoder whose hidden layer h has a dimension smaller than the input dimension.

[0049] The incomplete autoencoder generates the code through the hidden layer h. In the training process of dual-view information fusion, the principal subspace of view information migration can be learned. According to the feature vectors obtained by the two incomplete autoencoders, the fusion feature map of the heat map at view u relative to view v can be obtained, which is expressed as Similarly, the global confidence map of the heat map at perspective v relative to perspective u is Where H and W represent the height and width of the input image respectively, and R represents the downsampling factor of the feature extraction module.

[0050] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for human posture estimation and spatial positioning based on cross-view information fusion, Features: The steps include: 1) Design quantity including but not limited to two sets of high-precision two-dimensional turntables, which together with motion controllers, motor drivers, and host computers form a high-precision servo control system, and fix the high frame rate camera on the turntable bracket; 2) Real-time detection and tracking of moving targets. The high-precision servo control system drives the turntable to follow the moving target to ensure that the moving target is in the center of the image. The image data of the moving person and the turntable posture data are stored simultaneously in multiple camera positions in a time sequence; 3) Use a multi-granularity cross-layer cascade network to perform separate encoding on images at the same time but from different perspectives to obtain deep features of the image. The multi-granularity cross-layer cascade network includes a feature extraction module, an autoencoder and a detection front end. The detection front end is used to perform corresponding decoding operations on the feature map of the feature encoding network to obtain a human body position distribution heat map, a bounding box size, a bias regression value, a key point position distribution heat map, a key point bias value and a key point discrete value. The separate encoding group is composed of two feature extraction modules with the same structure and shared weights. The feature extraction module encodes the image The deep implicit features of the implicit features are fused with the implicit features of another perspective through the feature transfer channel through an incomplete autoencoder. The autoencoder can be divided into an encoder represented by a function h=f(x) and a decoder r=g(x) for feature reconstruction. The incomplete autoencoder means an autoencoder whose dimension of the hidden layer h is smaller than that of the input. The autoencoder generates codes through the hidden layer h. In the training process of dual-perspective information fusion, the principal element subspace of the perspective information migration is learned. The feature vectors in the row direction and column direction obtained by the two incomplete autoencoders are v respectively. h and v w , we can get the fusion feature map of the heat map at the perspective u relative to the perspective v, and the expression is The global confidence map of the heat map at view v relative to view u is Where H and W represent the height and width of the input image, respectively, and R represents the downsampling factor of the feature extraction module; 4) The human body postures at each perspective are combined with the camera posture data to obtain the three-dimensional posture of the moving target.

2. A method for human posture estimation and spatial positioning by cross-view information fusion as claimed in claim 1, It is characterized in that The high-precision servo control system uses ultra-high-precision servo motors to control the rotation of the two-dimensional pointing mechanism in the horizontal and vertical directions respectively.

3. The method for human posture estimation and spatial positioning by cross-view information fusion as claimed in claim 1, It is characterized in that The fixed image acquisition lens is replaced with an electronically controlled servo zoom lens to form a pan-tilt zoom machine vision system, referred to as PTZ (Pan Tilt Zoom), which solves the contradiction between the field of view and resolution of the traditional binocular machine vision system.

4. The method for human posture estimation and spatial positioning by cross-view information fusion as claimed in claim 1, It is characterized in that A deep neural network is used for target detection and target tracking, and it is trained on self-made and public datasets to achieve detection and tracking of targets without coordination of athletes.

5. The method for human posture estimation and spatial positioning by cross-view information fusion as claimed in claim 1, It is characterized in that A multi-granularity cross-layer cascade network is used to extract features from the image. The multi-granularity cross-layer cascade network is a bottom-up detection mode. The distribution heat map of the key points of the target human body in the image is obtained by encoding and decoding the input image. The multi-granularity cross-layer cascade network consists of a feature encoding stage, a multi-level feature fusion stage, and a feature decoding stage. The feature extraction backbone network consists of three CSP cross-layer connections. The first two cross-layer connections only use 7×7 modules, which can reduce the amount of calculation while reducing information loss. Downsampling operations are performed between cross-layer connection sub-modules. The pooling layer will cause a large amount of information loss and affect the translation and equivariance of the convolutional network. Therefore, all downsampling stages in the multi-granularity cascade network do not perform pooling. Instead, strided convolution with a convolution kernel size of 3 and a step size of 2 is used for downsampling.

6. The method for human posture estimation and spatial positioning by cross-view information fusion as claimed in claim 1, It is characterized in that The information of each perspective is fused using the separation encoding method. The images at different perspectives are respectively passed through a multi-granularity cross-layer cascade network to obtain the deep implicit features of the image. The deep implicit features of the image at the perspective u are expressed as Use convolutional layers H with kernel sizes of 1×w and 1×h respectively. out Sparse coding is performed to obtain and The feature vector of H is then pooled using adaptive average pooling layers of size 1×w and 1×h. out Sparse coding is performed to obtain and The feature vector of the fusion is divided into and Among them, f is an incomplete autoencoder composed of fully connected layers, and the number of neurons in the hidden layer is set to one sixteenth of the number of input neurons.

Images