Human Key Point Detection Method, Device and Medium Based on Orthogonal Matching Pursuit

Abstract

The present invention relates to a human key point detection method, device and medium based on orthogonal matching pursuit. The method includes: inputting a to-be-detected image containing a person into a trained target detector to obtain a cropped image of a single person; setting the annotation coordinates of human key points; scaling the cropped image to a set size and inputting it into a trained feature encoding backbone network to obtain a sensing matrix; and performing sparse recovery on the annotation coordinates of human key points by using the orthogonal matching pursuit algorithm according to the sensing matrix to obtain the positions of human key points of the person in the to-be-detected image. Compared with the prior art, the present invention reduces the computational burden while ensuring relatively high computational accuracy and has high detection efficiency.

Description

Technical Field

[0001] The present invention relates to a human key point detection technology, and particularly to a human key point detection method, device and medium based on orthogonal matching pursuit. Background Technique

[0002] With the booming development of fields such as driverless, human-computer interaction, video surveillance, and behavior recognition, the upstream of these researches, such as human pose estimation, becomes particularly important. In existing pose detection and other tasks, it is first necessary to extract features from an image. Then, the feature information is input into a prediction network for pose and other predictions. To facilitate the completion of pose detection and other tasks, it is generally necessary to determine the positions of each key point of the human body. In the prior art, the human body is often regarded as a whole to determine the positions of its key points. Existing human pose estimation methods can be divided into Heatmap-based methods and regression-based methods. Among them, Heatmap-based methods can also be divided into top-down methods and bottom-up methods. Since Heatmap-based methods are based on dense estimation methods with sub-pixel level accuracy and are very popular, but have a large amount of computation; regression-based methods have the characteristic of small computation, but the accuracy is generally inferior to Heatmap-based methods. Summary of the Invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a human key point detection method, device and medium based on orthogonal matching pursuit, which reduces the computational burden while ensuring high computational accuracy and has high detection efficiency.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A human key point detection method based on orthogonal matching pursuit, comprising:

[0006] Input the to-be-detected image containing a person into a trained object detector to obtain a cropped image of a single person;

[0007] Set the annotation coordinates of human key points;

[0008] Scale the cropped image to a set size and input it into a trained feature encoding backbone network to obtain a sensing matrix;

[0009] According to the sensing matrix, use the orthogonal matching pursuit algorithm to perform sparse recovery on the annotation coordinates of human key points to obtain the positions of human key points of the person in the to-be-detected image;

[0010] Based on the fact that human body key points are sparse, according to the compressive sensing theory of signal processing: if a signal is sparse and compressible, the signal can be sampled at a sampling rate much lower than the Nyquist limit. When the sampled data covers the main information in the original signal, the original signal can be accurately reconstructed with high probability. Use the target detector to segment the cropped image of a single person, unify the size and then input it into the feature encoding backbone network to obtain the sensing matrix. Use the orthogonal matching pursuit algorithm to sparsely recover the coordinates of human body key points, realizing compressive sensing, with small computational complexity, reducing the computational burden while ensuring high computational accuracy, and high detection efficiency.

[0011] Furthermore, the sensing matrix is denoted as G M×N , and the expression is

[0012]

[0013] where, g j is the j-th column of G M×N , g ij is the element in the i-th row and j-th column of G M×N , i = 1, 2,..., M, j = 1, 2,..., N, M is the number of selected key support points, M ≥ K, and N is the total number of pixels of the cropped image.

[0014] Furthermore, the orthogonal matching pursuit algorithm includes:

[0015] Set the sparsity K, the initial residual signal r 0 and the initial support set Λ 0 , r 0 = {X, Y}, {X, Y} are the coordinates of key point annotations, K is the number of human body key points;

[0016] Loop through the residual update step K times, and the residual update step includes:

[0017] 1) Calculate the column vector k max that is most relevant to the current residual signal:

[0018]

[0019] where, Λ D = [1, 2,..., N], r k-1 is the residual signal after looping k - 1 times, g k is the k-th column of G M×N , k = 1, 2,..., K;

[0020] 2) Update the support set:

[0021] Λ k = Λ k-1∪k max

[0022] Among them, Λ k and Λ k-1 are the support sets after k - loop and (k - 1)-loop respectively;

[0023] 3) Sparse decomposition of the coordinates of the key point annotations:

[0024]

[0025] Among them, q k is the sparse solution after k - loop, is the current support set, and p is the coordinates of the key point annotations;

[0026] 4) Update the residual:

[0027]

[0028] Obtain the positions of the human key points

[0029]

[0030] Among them, q K is the sparse solution after K - loop, that is, the sparse representation of the human key points.

[0031] Furthermore, the training process of the target detector and the feature encoding backbone network includes:

[0032] Train the target detector and the feature encoding backbone network separately;

[0033] After the separate training is completed, jointly train the target detector and the feature encoding backbone network. The two networks promote each other to improve the detection accuracy.

[0034] Furthermore, during the process of separately training the feature encoding backbone network, data augmentation is performed on the input of the feature encoding backbone network to improve the robustness of the network.

[0035] Furthermore, the data augmentation process includes one or more of random flipping, random scaling, and random rotation.

[0036] Furthermore, the target detector is a convolutional neural network.

[0037] Furthermore, the convolutional neural network is Faster RCNN.

[0038] A human key point detection device based on orthogonal matching pursuit, comprising a memory and a processor, wherein the memory stores a computer program, and the processor can execute the detection method by invoking the program instructions.

[0039] A computer-readable storage medium, comprising a computer program, and the computer program can be executed by a processor to perform the detection method.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention inputs the image to be detected containing a person into a trained target detector to obtain a cropped image of a single person, sets the annotation coordinates of the human key points, scales the cropped image to a set size, and inputs it into a trained feature encoding backbone network to obtain a sensing matrix. According to the sensing matrix, the orthogonal matching pursuit algorithm is used to sparsely recover the annotation coordinates of the human key points, and the positions of the human key points of the person in the image to be detected are obtained. Based on the fact that the human key points are sparse, the present invention uses the target detector to segment the cropped image of a single person, unifies the size and then inputs it into the feature encoding backbone network to obtain a sensing matrix, and uses the orthogonal matching pursuit algorithm to sparsely recover the human key point coordinates, realizing compressive sensing, with small computational complexity, reducing the computational burden while ensuring high computational accuracy, and high detection efficiency;

[0042] (2) The present invention trains the target detector and the feature encoding backbone network separately. After the separate training is completed, the target detector and the feature encoding backbone network are jointly trained. The two networks promote each other to improve the detection accuracy;

[0043] (3) During the process of separately training the feature encoding backbone network, data augmentation is performed on the input of the feature encoding backbone network to improve the robustness of the network. Description of the Drawings

[0044] Figure 1 It is a structural schematic diagram of the present invention. Detailed Embodiments

[0045] The present invention will be described in detail below with reference to the drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and gives detailed implementation manners and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0046] Embodiment 1

[0047] A human key point detection method based on orthogonal matching pursuit, as Figure 1 , includes:

[0048] Input the image to be detected containing people into the trained object detector to obtain the cropped image of a single person;

[0049] Set the GroundTruth of the human key point annotation coordinates;

[0050] Scale the cropped image to the set size and input it into the trained feature encoding backbone network to obtain the sensing matrix;

[0051] According to the sensing matrix, use the orthogonal matching pursuit algorithm to sparsely recover the human key point annotation coordinates and obtain the positions of the human key points of the people in the image to be detected;

[0052] Based on the fact that the human key points are sparse, according to the compressive sensing theory of signal processing: if the signal is sparse and compressible, the signal can be sampled at a sampling rate much lower than the Nyquist limit. When the sampled data covers the main information in the original signal, the original signal can be accurately reconstructed with high probability. Use the object detector to segment the cropped image of a single person, input it into the feature encoding backbone network after unifying the size to obtain the sensing matrix, and use the orthogonal matching pursuit algorithm to sparsely recover the human key point coordinates to achieve compressive sensing, with small computational complexity, reducing the computational burden while ensuring high computational accuracy and high detection efficiency.

[0053] The sensing matrix is denoted as G M×N , and the expression is

[0054]

[0055] where, g j is the j-th column of G M×N , and g ij is the element in the i-th row and j-th column of G M×N , i = 1, 2,..., M, j = 1, 2,..., N, M is the number of selected key support points, M ≥ K, and N is the total number of pixels of the cropped image.

[0056] The orthogonal matching pursuit algorithm includes:

[0057] Set the sparsity K, the initial residual signal r 0 and the initial support set Λ 0 , r 0 = {X, Y}, {X, Y} are the key point annotation coordinates, K is the number of human key points;

[0058] Loop the residual update step K times, and the residual update step includes:

[0059] 1) Select the column vector in the sensing matrix that is most correlated with the residual signal, i.e., take the inner product of the two, and select the column vector corresponding to the atom with the largest absolute value to add to the support set. The column vector \(k\) that is most correlated with the current residual signal max is calculated as follows:

[0060]

[0061] where \(\Lambda\) D = [1, 2, ..., N], \(r\) k-1 is the residual signal after \(k - 1\) loops, \(g\) k is the \(k\)-th column of \(G\) M×N , \(k = 1, 2, ..., K\), and "\\" means removal;

[0062] 2) Update the support set:

[0063] \(\Lambda\) k = \(\Lambda\) k-1 \(\cup k\) max

[0064] where \(\Lambda\) k and \(\Lambda\) k-1 are the support sets after \(k\) loops and \(k - 1\) loops respectively, and "\(\cup\)" means addition;

[0065] 3) Perform sparse decomposition on the key point annotation coordinates:

[0066]

[0067] where \(q\) k is the sparse solution after \(k\) loops, is the current support set, and \(p\) is the key point annotation coordinate;

[0068] 4) Update the residual:

[0069]

[0070] Obtain the positions of human key points

[0071]

[0072] where \(q\) K is the sparse solution after \(K\) loops, i.e., the sparse representation of human key points.

[0073] Finally, obtain a support set of \(K\) columns through the orthogonal matching pursuit algorithm, and the human key points are linearly represented by the column vectors of the support set.

[0074] Since the column vectors selected by the orthogonal matching pursuit algorithm are orthogonal to the residual, it avoids repeatedly selecting the same column vector.

[0075] For the COCO dataset, K = 17, and for the CrowdPose dataset, K = 14. The number of iterations of the orthogonal matching pursuit algorithm usually depends on the number of human key points K required. However, the visible human key points are not fixed in practice. Therefore, the upper limit of the residual can be used to terminate the iteration, or the iteration can be determined whether to terminate according to the scores of the obtained key points. If the score is lower than the preset threshold, the iteration is terminated.

[0076] The training process of the object detector and the feature encoding backbone network includes:

[0077] Train the object detector and the feature encoding backbone network separately;

[0078] After the separate training is completed, jointly train the object detector and the feature encoding backbone network. The two networks promote each other to improve the detection accuracy.

[0079] During the process of separately training the feature encoding backbone network, data augmentation is performed on the input of the feature encoding backbone network to improve the robustness of the network.

[0080] The data augmentation process includes one or more of random flipping, random scaling, and random rotation.

[0081] The object detector is a convolutional neural network.

[0082] The convolutional neural network is Faster RCNN.

[0083] As Figure 1 shown, first use the object detector to detect a single individual in the image to be detected, obtain 3 cropped images of single individuals, then scale the cropped images to the set size uniformly, and input them into the feature encoding backbone network to obtain the sensing matrix. Then, according to the sensing matrix, use the orthogonal matching pursuit algorithm to sparsely recover the annotated coordinates of the human key points, and obtain the positions of the human key points of the people in the image to be detected.

[0084] Example 2

[0085] A human key point detection device based on orthogonal matching pursuit includes a memory and a processor. The memory stores a computer program, and the processor can execute the detection method of Example 1 by calling the program instructions.

[0086] Example 3

[0087] A computer-readable storage medium includes a computer program, and the computer program can be executed by a processor to implement the detection method of Example 1.

[0088] Embodiment 1, Embodiment 2 and Embodiment 3 propose a human key point detection method, device and medium based on orthogonal matching pursuit. Based on the fact that human key points are sparse, according to the compressive sensing theory of signal processing, the cropped image of a single person is segmented by an object detector, and after unifying the size, it is input into the feature encoding backbone network to obtain a sensing matrix. The orthogonal matching pursuit algorithm is used to sparsely recover the coordinates of human key points, realizing compressive sensing. The amount of calculation is small, the calculation burden is reduced while ensuring high calculation accuracy, and the detection efficiency is high.

[0089] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A human keypoint detection method based on orthogonal matching pursuit, characterized in that, it includes: Input the image to be detected containing a person into a trained object detector to obtain a cropped image of a single person; Set the annotation coordinates of human keypoints; Scale the cropped image to a set size and input it into a trained feature encoding backbone network to obtain a sensing matrix; According to the sensing matrix, use the orthogonal matching pursuit algorithm to perform sparse recovery on the annotation coordinates of human keypoints to obtain the positions of human keypoints of the person in the image to be detected; Among them, the perception matrix is denoted as G M×N , and the expression is where, g j is the j-th column of G M×N , and g ij is the element at the i-th row and j-th column of G M×N , where i = 1, 2,..., M, j = 1, 2,..., N, M is the number of selected key support points, M ≥ K, and N is the total number of pixels of the cropped image; The orthogonal matching pursuit algorithm includes: Set the sparsity K, the initial residual signal r 0 and the initial support set Λ 0 , r 0 = {X, Y}, where {X, Y} are the key point annotation coordinates, K is the number of human key points; Performing the residual update step K times in a loop, and the residual update step includes: 1) Calculate the column vector k that is most correlated with the current residual signal max : where, Λ D = [1, 2, ..., N], r k-1 is the residual signal after k - 1 iterations, g k is the k-th column of G M×N , k = 1, 2, ..., K; 2) Update the support set: Λ k = Λ k-1 ∪k max Among them, Λ k and Λ k-1 are the support sets after k loops and k - 1 loops respectively; 3) Perform sparse decomposition on the keypoint annotation coordinates: where q k is the sparse solution after k iterations, is the current support set, and p is the key point annotation coordinate; 4) Update the residual: Obtain the positions of human key points where q K is the sparse solution after K iterations.

2. A human keypoint detection method based on orthogonal matching pursuit according to claim 1, characterized in that, the training processes of the object detector and the feature encoding backbone network include: Individually train the object detector and the feature encoding backbone network respectively; After the individual training is completed, jointly train the object detector and the feature encoding backbone network.

3. A human keypoint detection method based on orthogonal matching pursuit according to claim 2, characterized in that, During the process of individually training the feature encoding backbone network, data augmentation is performed on the input of the feature encoding backbone network.

4. A human keypoint detection method based on orthogonal matching pursuit according to claim 3, characterized in that, the data augmentation process includes one or more of random flipping, random scaling, and random rotation.

5. A human keypoint detection method based on orthogonal matching pursuit according to claim 1, characterized in that, the object detector is a convolutional neural network.

6. A human keypoint detection method based on orthogonal matching pursuit according to claim 5, characterized in that, the convolutional neural network is Faster RCNN.

7. A human keypoint detection device based on orthogonal matching pursuit, characterized in that, it includes a memory and a processor, the memory stores a computer program, and the processor can execute the detection method according to any one of claims 1 to 6 by calling the program.

8. A computer-readable storage medium, characterized in that, it includes a computer program, and the computer program can be executed by a processor to implement the detection method according to any one of claims 1 - 6.

Images