Pose tracking method, apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose a pose tracking method, device, electronic equipment and computer readable medium. A specific embodiment of the method comprises: performing frame-by-frame detection on a pedestrian video to obtain a human body bounding box set; inputting the human body bounding box set into a joint node confidence network to obtain a human body pose information set; performing matching processing on each human body pose information set in the human body pose information set to obtain a matching result set; generating a human body optical flow box set; inputting the human body optical flow box set into the joint node confidence network to obtain an optical flow human body pose information set and an optical flow joint node confidence information set; and filtering the optical flow human body pose information set based on the optical flow joint node confidence information set to obtain a target human body pose information set. The embodiment can improve the accuracy of a multi-person pose tracking task.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer technology, and in particular, to a pose tracking method and apparatus, an electronic device, and a computer readable medium. BACKGROUND

[0002] The multi-person pose tracking task in the field of computer vision refers to detecting the pose of each pedestrian in each frame of a video by processing the input video, and then calculating and analyzing the appearance features, position, motion state, and other information of the target to correctly record the continuous pose trajectory of each person over time. Currently, when performing the multi-person pose tracking task, in addition to correctly identifying the position of the joint node of each pedestrian, it is also necessary to filter the unavailable joint nodes caused by occlusion and other reasons. When filtering the joint nodes, the commonly used method is to filter the joint nodes according to the position probability.

[0003] However, when filtering the joint nodes in the above manner, the following technical problems often exist:

[0004] First, the joint nodes corresponding to the position probability with a low value caused by motion blur are incorrectly filtered, which leads to inaccurate tracking of some pedestrians and low accuracy of the multi-person pose tracking task.

[0005] Second, in the scene of human body interlaced occlusion, the joint nodes corresponding to the position probability with a high value caused by incorrect labeling on other people cannot be filtered, which leads to inaccurate tracking of some pedestrians and low accuracy of the multi-person pose tracking task. SUMMARY

[0006] The summary of the present disclosure is intended to introduce the concepts in a simplified form, which will be described in detail in the specific embodiments section below. The summary of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of the present disclosure propose a pose tracking method, apparatus, electronic device, and computer readable medium to solve one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of the present disclosure provide a pose tracking method, comprising: performing frame-by-frame detection on a pedestrian video to obtain a set of human bounding box groups, wherein the human bounding box groups in the set of human bounding box groups correspond to the pedestrian video frames included in the pedestrian video; inputting the set of human bounding box groups into a joint node confidence network to obtain a set of human pose information groups; performing matching processing on each human pose information group in the set of human pose information groups according to the set of human pose information groups to generate a set of matching results, to obtain a set of matching result groups, wherein the matching result in the matching result group includes at least one video frame number of the human bounding box corresponding to the matching result; in response to determining that there is a matching result in the set of matching result groups that satisfies a preset video frame number condition, generating a set of human optical flow box groups according to each matching result group corresponding to the matching result that satisfies the preset video frame number condition, wherein the preset video frame number condition is that the at least one video frame number included in the matching result does not contain a next video frame number, and the next video frame number is the video frame number of the next frame video of the video frame number corresponding to the matching result in the at least one video frame number; inputting the set of human optical flow box groups into the joint node confidence network to obtain a set of optical flow human pose information groups and a set of optical flow joint node confidence information groups; and filtering the set of optical flow human pose information groups based on the set of optical flow joint node confidence information groups to obtain a set of target human pose information groups.

[0009] In a second aspect, some embodiments of the present disclosure provide a pose tracking device, the device comprising: a detection unit configured to perform frame-by-frame detection on a pedestrian video to obtain a set of human bounding box groups, wherein a human bounding box group in the set of human bounding box groups corresponds to a pedestrian video frame included in the pedestrian video; a first input unit configured to input the set of human bounding box groups into a joint node confidence network to obtain a set of human pose information groups; a matching unit configured to perform matching processing on each human pose information group in the set of human pose information groups according to the set of human pose information groups to generate a set of matching results, wherein a matching result in the matching result group includes at least one video frame number of a human bounding box corresponding to the matching result; a generation unit configured to, in response to determining that there is a matching result in the set of matching results that satisfies a preset video frame number condition, generate a set of human optical flow box groups according to each matching result group corresponding to the matching result that satisfies the preset video frame number condition, wherein the preset video frame number condition is that the at least one video frame number included in the matching result does not contain a next video frame number, and the next video frame number is a video frame number of a next frame video of a video frame number corresponding to the matching result in the at least one video frame number; a second input unit configured to input the set of human optical flow box groups into the joint node confidence network to obtain a set of optical flow human pose information groups and a set of optical flow joint node confidence information groups; and a filtering unit configured to filter the set of optical flow human pose information groups based on the set of optical flow joint node confidence information groups to obtain a set of target human pose information groups.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; and a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any implementation manner of the first aspect.

[0012] The above various embodiments of the present disclosure have the following beneficial effects: through the pose tracking method of some embodiments of the present disclosure, the accuracy of the multi-person pose tracking task can be improved. Specifically, the reason why the accuracy of the multi-person pose tracking task is low is that the position probability with a low corresponding value caused by motion blur is mistakenly filtered out, resulting in that some pedestrians cannot be accurately tracked, and thus the accuracy of the multi-person pose tracking task is low. Based on this, the pose tracking method of some embodiments of the present disclosure first performs frame-by-frame detection on the pedestrian video to obtain a human body bounding box set. In this way, the position of the pedestrian in the pedestrian video can be preliminarily obtained. Then, the human body bounding box set is input into a joint node confidence network to obtain a human body pose information set. In this way, the human body pose information set representing the human body pose can be obtained to facilitate tracking of the human body pose. Secondly, according to the human body pose information set, each human body pose information set in the human body pose information set is matched to generate a matching result set, and a matching result set is obtained. In this way, the matching result corresponding to each human body pose information can be obtained. Then, in response to determining that there is a matching result in the matching result set that satisfies a preset video frame number condition, a human body optical flow bounding box set is generated according to each matching result corresponding to the matching result that satisfies the preset video frame number condition. In this way, a human body optical flow bounding box set representing the predicted human body pose can be obtained. After that, the human body optical flow bounding box set is input into the joint node confidence network to obtain an optical flow human body pose information set and an optical flow joint node confidence information set. In this way, an optical flow human body pose information set representing the human body pose of the human body optical flow bounding box in the human body optical flow bounding box set can be obtained. Finally, based on the optical flow joint node confidence information set, the optical flow human body pose information set is filtered to obtain a target human body pose information set. In this way, the filtering of the optical flow human body pose information set can be realized, and thus a target human body pose information set with a high corresponding confidence can be obtained. Because the optical flow human body pose information set is filtered using the optical flow joint node confidence information set, the target human body pose information set obtained has a high corresponding confidence. For the joint node that is imaged blurred due to motion, when the corresponding confidence is high, the joint node is also retained, thereby avoiding mistakenly filtering out the joint node corresponding to the position probability with a low corresponding value caused by motion blur, and thus the joint node of the pedestrian can be accurately tracked, and thus the accuracy of the multi-person pose tracking task can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings. In the drawings like reference numerals designate like elements throughout the various figures. It should be understood that the drawings are schematic and elements and features are not necessarily drawn to scale.

[0014] Figure 1is a flowchart of some embodiments of a pose tracking method according to the present disclosure;

[0015] Figure 2 is a network diagram of a joint position confidence network of a pose tracking method according to the present disclosure;

[0016] Figure 3 is a structural diagram of some embodiments of a pose tracking apparatus according to the present disclosure;

[0017] Figure 4 is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings of the present disclosure and the embodiments are only for illustrative purposes and should not be used to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] It should be noted that the terms "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the terms "one", "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only used for illustrative purposes, and are not used to limit the scope of the messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0024] Figure 1 Flow 100 shows some embodiments of a pose tracking method according to the present disclosure. The pose tracking method comprises the following steps:

[0025] Step 101, frame-by-frame detection is performed on the pedestrian video to obtain a set of human body bounding box groups.

[0026] In some embodiments, the execution subject (e.g., a computing device) of the pose tracking method can perform frame-by-frame detection on the pedestrian video to obtain a set of human bounding box groups. The human bounding box groups in the set of human bounding box groups correspond to the pedestrian video frames included in the pedestrian video. The human bounding box in the human bounding box group corresponds to a pedestrian appearing in the pedestrian video frame. The pedestrian video can be a video in which at least one pedestrian appears in the recording scene. In practice, the execution subject can use an HTC (Hybrid Task Cascade) detector to perform frame-by-frame detection on the pedestrian video to obtain the set of human bounding box groups. Thus, the position of the pedestrian in the pedestrian video can be preliminarily obtained.

[0027] It should be noted that the computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made herein. It should be understood that any number of computing devices can be provided according to the implementation needs.

[0028] In step 102, the set of human bounding box groups is input into a joint confidence network to obtain a set of human pose information.

[0029] In some embodiments, the execution subject can input the set of human bounding box groups into the joint confidence network to obtain a set of human pose information and a set of joint confidence information. The joint confidence network can be a neural network that takes the set of human bounding box groups as input and outputs the set of human pose information and the set of joint confidence information. For example, the joint confidence network can be an Hourglass Network. Thus, the set of human pose information representing the human pose can be obtained to facilitate tracking of the human pose.

[0030] As Figure 2In some optional implementations of some embodiments, the joint confidence network includes a backbone network, a joint prediction branch, and a joint availability branch, the joint availability branch includes a residual network and at least one classifier. The backbone network can be a High-Resoultion Net (HRNet). The joint prediction branch can be a neural network that takes the feature map set output by the backbone network as input, and outputs a joint position probability information set and a joint position coordinate information set. In practice, the joint prediction branch can perform a series of transposed convolutions on the feature map set output by the backbone network. The loss function used to train the joint prediction branch can be as follows:

[0031]

[0032] where L represents the loss function of the joint prediction branch. K represents the number of preset joint types. For example, K can take the value 15. W represents the length of the heat map. H represents the width of the heat map. The heat map is a heat map generated by the joint prediction branch during operation. k represents the kth joint. i represents the ith column of the heat map. j represents the jth row of the heat map. represents the value of the ith column and the jth row of the heat map matrix of the kth joint. represents the value of the ith column and the jth row of the second-order Gaussian distribution label matrix of the kth joint. The second-order Gaussian distribution label matrix is obtained by converting each preset joint real label. and have the same size. k represents the preset joint real label of the kth joint. The preset joint real label can be a pre-set joint real label. The value of the preset joint real label is 0 or 1.

[0033] The joint availability branch can be a neural network that takes the feature map set output by the backbone network as input, and outputs a joint availability probability information set. The focal loss function can be used as the loss function for training the joint availability branch, which can be as follows:

[0034] FL(p)=-α p ×(1-p) γ ×log(p)。

[0035] p=(p avl ) y ×(1-p avl ) 1-y .

[0036] α p= a y x (1 - a) 1-y .

[0037] wherein, y represents a preset node true label. The preset node true label is 0 or 1. p represents a node classification probability. p avl represents a node availability probability. Wherein, the node availability probability can be the probability of the classifier output included in the node availability branch. As shown in the above formula, when the preset node true label y is 1, the node classification probability p is p avl . When the preset node true label y is 0, the node classification probability p is 1-p avl . FL(p) represents the Focal Floss function. a p is a weight factor corresponding to the node classification probability p, and g is a focus parameter. Here, the values of other parameters can adopt the default parameters of the focal loss function. For example, the value of g is 2. The value of a is 0.25.

[0038] In practice, first, the human body bounding boxes included in the human body bounding box group set can be cropped and scaled. In practice, the execution subject can crop the human body bounding boxes included in the human body bounding box group set from the corresponding pedestrian video frame, and scale each human body bounding box cropped to a fixed size (for example, 384x288). Then, when processing the human body bounding box group set after cropping and rescaling, the first step, the execution subject can input the human body bounding box group set after cropping and rescaling to the backbone network to obtain a feature map group set. Here, each high-resolution feature output by the last exchange unit of the fourth stage of the HRNet can be taken as the feature map output by the backbone network. The second step, the execution subject can input the feature map group set to the node prediction branch and the node availability branch respectively to obtain a node position probability information group set, a node position coordinate information group set and a node availability probability information group set. The third step, the execution subject can generate a node confidence information group set according to the node position probability information group set and the node availability probability information group set. In practice, for the node position probability information and the node availability probability information, the product of the node position probability and the node availability probability corresponding to the same node can be determined as the node confidence to obtain the node confidence information. The fourth step, the execution subject can determine the human body posture represented by each feature map in the feature map group set as human body posture information to obtain a human body posture information group set. The fifth step, the execution subject outputs the node confidence information group set and the human body posture information group set.

[0039] Step 103, according to the human body posture information group set, each human body posture information group in the human body posture information group set is matched to generate a matching result group, and a matching result group set is obtained.

[0040] In some embodiments, the execution entity may perform matching processing on each human body posture information group in the human body posture information group set based on the human body posture information group set to generate a matching result group and obtain a matching result group set. The matching result in the matching result group includes at least one video frame number of a human body bounding box corresponding to the matching result. The video frame number may be the frame number of each pedestrian video frame in the pedestrian video. In practice, the Hungarian algorithm may be used to perform matching processing on each human body posture information group in the human body posture information group set. Thus, a matching result corresponding to each human body posture information may be obtained.

[0041] In some optional implementations of some embodiments, first, the execution subject may perform the following operations on the human body posture information groups corresponding to two adjacent pedestrian video frames in the human body posture information group set:

[0042] In the first step, a human body posture information group corresponding to a previous pedestrian video frame in two adjacent pedestrian video frames is determined as a first human body posture information group.

[0043] In the second step, the human body posture information group corresponding to the latter pedestrian video frame of two adjacent pedestrian video frames is determined as the second human body posture information group.

[0044] The third step is to determine the distance between each first human posture information in the first human posture information group and each second human posture information in the second human posture information group to obtain a distance set. The distance may be an IoU (Intersection over Union) distance.

[0045] In a fourth step, the first human posture information in the first human posture information group and the second human posture information in the second human posture information group are allocated and processed based on the distance set to obtain adjacent video frame matching results. In practice, the first human posture information in the first human posture information group and the second human posture information in the second human posture information group can be allocated and processed based on the distance set using a Hungarian algorithm to obtain adjacent video frame matching results.

[0046] Then, a matching result set can be generated based on the matching results of each adjacent video frame. In practice, the video frame numbers corresponding to each second human posture information that has been successfully matched can be combined into a matching result, thereby obtaining a matching result set. In this way, inter-frame matching of human postures can be achieved.

[0047] In response to determining that there is a matching result in the matching result group set satisfying the preset video frame number condition, the execution subject generates a human body optical flow frame group set according to each matching result group corresponding to the matching result satisfying the preset video frame number condition.

[0048] In some embodiments, the execution subject can generate a human body optical flow frame group set according to each matching result group corresponding to the matching result satisfying the preset video frame number condition in response to determining that there is a matching result in the matching result group set satisfying the preset video frame number condition. The preset video frame number condition is that the matching result does not include the next video frame number in at least one video frame number. The next video frame number is the video frame number of the next frame video of the video frame number corresponding to the matching result in the at least one video frame number. In practice, the human body optical flow frame group set can be generated in various ways. Thus, the human body optical flow frame group set representing the predicted human body posture can be obtained.

[0049] In some optional implementations of some embodiments, in the first step, the execution subject can determine a set of to-be-processed video frames according to each matching result group corresponding to the matching result satisfying the preset video frame number condition. In practice, the video frames corresponding to each matching result group corresponding to the matching result satisfying the preset video frame number condition can be determined as the to-be-processed video frames. In the second step, the following sub-steps can be performed according to each pair of adjacent two to-be-processed video frames in the set of to-be-processed video frames:

[0050] In the first sub-step, an optical flow map matrix is generated according to the adjacent two to-be-processed video frames. The optical flow map matrix includes a set of pixel offset amounts. In practice, the optical flow map matrix can be generated using the optical flow method according to the adjacent two to-be-processed video frames.

[0051] In the second sub-step, a human body posture information group corresponding to the adjacent two to-be-processed video frames is selected from the human body posture information group set as a first human body posture information group, and a first human body posture information group set is obtained.

[0052] In the third sub-step, a first human body posture information corresponding to the matching result satisfying the preset video frame number condition is selected from the first human body posture information group set as a second human body posture information, and at least one second human body posture information is obtained.

[0053] In the fourth sub-step, for each second human body posture information in the at least one second human body posture information, a pixel offset amount corresponding to a pixel located in a range corresponding to the second human body posture information is selected from the set of pixel offset amounts as a target pixel offset amount, and a set of target pixel offset amounts corresponding to the second human body posture information is obtained. The pixel can be a pixel after offset in two pixels corresponding to the pixel offset amount.

[0054] A fifth sub-step, generating an optical flow mask map according to the set of target pixel offsets. In practice, the set of target pixel offsets can be constructed into a matrix to obtain the optical flow mask map.

[0055] A sixth sub-step, determining a minimum bounding rectangle of the optical flow mask map as a human optical flow frame. The human optical flow frame corresponds to a later frame of the two adjacent frames of the video to be processed. Thus, the human optical flow frame representing the prediction of the later frame of the video to be processed can be obtained.

[0056] Step 105, inputting the set of human optical flow frames into the joint node confidence network to obtain a set of optical flow human pose information and a set of optical flow joint node confidence information.

[0057] In some embodiments, the execution subject can input the set of human optical flow frames into the joint node confidence network to obtain a set of optical flow human pose information and a set of optical flow joint node confidence information. Thus, the set of optical flow human pose information representing the human pose of the human optical flow frame in the set of human optical flow frames can be obtained.

[0058] Step 106, filtering the set of optical flow human pose information based on the set of optical flow joint node confidence information to obtain a set of target human pose information.

[0059] In some embodiments, the execution subject can filter the set of optical flow human pose information based on the set of optical flow joint node confidence information to obtain a set of target human pose information. In practice, a joint node confidence threshold condition can be set, and in response to the presence of joint node confidence information in the set of optical flow joint node confidence information satisfying the set joint node confidence threshold condition, the corresponding optical flow human pose information is deleted. For example, the joint node confidence threshold condition can be that the optical flow joint node confidence included in the optical flow joint node confidence information is less than a preset confidence threshold. Thus, the filtering of the set of optical flow human pose information can be realized, so as to obtain the set of target human pose information with higher confidence.

[0060] In some optional implementations of some embodiments, firstly, for each optical flow joint confidence information included in the set of optical flow joint confidence information, the execution subject can generate an average optical flow joint confidence according to the optical flow joint confidence information. In practice, the average of the individual optical flow joint confidence included in the optical flow joint confidence information can be determined as the average optical flow joint confidence. Secondly, the average optical flow joint confidence less than the preset optical flow threshold can be selected from the obtained average optical flow joint confidence as the first to-be-filtered confidence. Finally, the optical flow human pose information corresponding to the obtained first to-be-filtered confidence can be deleted from the set of optical flow human pose information. In this way, the optical flow human pose information corresponding to a higher confidence can be obtained.

[0061] Optionally, firstly, the execution subject can determine the video frame number corresponding to each optical flow human pose information in the set of optical flow human pose information after the deletion processing as a target video frame number, and obtain a set of target video frame numbers. Secondly, the human pose information set corresponding to the video frame number within the set of target video frame numbers can be selected from the set of human pose information as a contrast human pose information set, and a set of contrast human pose information sets is obtained. Then, the set of pose overlap degrees can be generated according to the set of optical flow human pose information after the deletion processing and the set of contrast human pose information sets. In practice, the pose overlap degree can be generated by using the following formula:

[0062]

[0063] Wherein, δ represents a function of converting a Boolean type result into 0 and 1. When the condition in the bracket is met, δ converts the Boolean type result into 1. When the condition in the bracket is not met, δ converts the Boolean type result into 0. p and q respectively represent two pedestrians corresponding to the optical flow human pose information and the contrast human pose information. p，q represents the IoU value (Intersection over Union) of the pedestrian p and the pedestrian q. The IoU value can be determined in advance. k represents the kth joint. represents the Euclidean distance between the kth joint of the pedestrian p and the kth joint of the pedestrian q. 2 represents the scale factor of the pedestrian p and the pedestrian q. The scale factor can be determined by the square root of the sum of the areas of the human body bounding boxes corresponding to the optical flow human pose information and the contrast human pose information, respectively. represents the normalization factor of the kth joint. The normalization factor can be determined in advance. pk represents the confidence of the kth joint of the pedestrian p. qk represents the confidence of the kth joint of the pedestrian q. ε is a preset parameter.

[0064] Then, the pose overlap degree greater than the preset pose overlap degree threshold can be selected from the pose overlap degree set as the second to-be-filtered confidence. Finally, the optical flow human pose information corresponding to the second to-be-filtered confidence can be deleted from the optical flow human pose information set.

[0065] The above is one of the inventive points of the embodiments of the present disclosure, which solves the second technical problem mentioned in the background. The second technical problem is that in the scene of human body interlaced occlusion, the joint nodes corresponding to the position probability with a higher value due to the error labeling to others cannot be filtered, resulting in that some pedestrians cannot be accurately tracked, and the accuracy of the multi-person pose tracking task is low. The factors leading to the low accuracy of the multi-person pose tracking task are as follows: in the scene of human body interlaced occlusion, due to the higher value of the position probability corresponding to the error labeling to others, the IoU measurement method is used to determine the intersection over union, and thus the similarity is low and cannot be filtered; in addition, the joint nodes corresponding to the two boxes are different, which also leads to low similarity and cannot be filtered. If the above factors are solved, the target positioning information and the target classification information of the target can be put into actual production and use. In order to achieve this effect, the present disclosure filters the joint nodes by using the above-mentioned pose overlap degree. Therefore, by using the above-mentioned pose overlap degree, the overlapped optical flow human pose information can be effectively filtered. Thus, the accurate tracking of some pedestrians can be realized, and the accuracy of the multi-person pose tracking task can be improved.

[0066] Optionally, first, the execution subject can determine the matching results in the matching result set that satisfy a preset single video frame number condition as target matching results, and obtain a target matching result set. The preset single video frame number condition can be that the matching result includes only one video frame number. Second, the human body posture information corresponding to the target matching results included in the target matching result set can be selected from the human body posture information set as to-be-integrated human body posture information, and a to-be-integrated human body posture information set is obtained. Then, the to-be-integrated human body posture information set can be input into a preset pedestrian feature model to obtain a to-be-integrated pedestrian feature information set. The preset pedestrian feature model can be a neural network model that takes the to-be-integrated human body posture information set as input and takes the to-be-integrated pedestrian feature information set as output. The preset pedestrian feature model can include a backbone network, a pedestrian feature extraction module, and a classifier. The backbone network can be an HRNet. The pedestrian feature extraction module can include a convolution layer, an average pooling layer, and a batch normalization layer. The classifier can include one fully connected layer and a Softmax layer. It should be noted that the classifier is only used for training of the preset pedestrian feature model, and the classifier is not used when the preset pedestrian feature model is actually applied. Subsequently, pedestrian feature similarity information of each to-be-integrated pedestrian feature information in the to-be-integrated pedestrian feature information set can be generated according to the to-be-integrated pedestrian feature information set, and a pedestrian feature similarity information set is obtained. In practice, the similarity between each to-be-integrated pedestrian feature information and other to-be-integrated pedestrian feature information in the to-be-integrated pedestrian feature information set can be determined, and a pedestrian feature similarity information set is obtained. The formula for determining the similarity is as follows:

[0067]

[0068] wherein p and q respectively represent two pedestrians corresponding to two to-be-integrated pedestrian feature information. S(p, q) represents the pedestrian feature similarity. D represents the dimension of the feature vector represented by the to-be-integrated pedestrian feature information. k represents the kth joint node. represents the feature vector of the kth joint node of the pedestrian p. represents the feature vector of the kth joint node of the pedestrian q.

[0069] Secondly, pedestrian feature similarity information in the pedestrian feature similarity information set that meets a preset similarity condition can be determined as target similarity information, and a target similarity information set is obtained. The preset similarity condition can be that the pedestrian feature similarity in the pedestrian feature similarity information is greater than a preset pedestrian feature similarity threshold. Then, the matching result group set can be updated according to the target similarity information set. In practice, the video frame number corresponding to each target similarity information in the target similarity information set can be recorded in the corresponding matching result. Finally, for each target similarity information in the target similarity information set, the pedestrian feature similarity information corresponding to the target similarity information can be deleted from the pedestrian feature similarity information set. Thus, the human body posture information with only one video frame number in the matching result can be matched again, so that the human body posture information that is not successfully matched due to the performance of the detector is successfully matched, thereby improving the accuracy of the multi-person posture tracking task.

[0070] The above various embodiments of the present disclosure have the following beneficial effects: through the pose tracking method of some embodiments of the present disclosure, the accuracy of the multi-person pose tracking task can be improved. Specifically, the reason why the accuracy of the multi-person pose tracking task is low is that the position probability with a relatively low value caused by motion blur is mistakenly filtered out, resulting in that some pedestrians cannot be accurately tracked, and thus the accuracy of the multi-person pose tracking task is low. Based on this, the pose tracking method of some embodiments of the present disclosure first performs frame-by-frame detection on the pedestrian video to obtain a human body bounding box set. In this way, the position of the pedestrian in the pedestrian video can be preliminarily obtained. Then, the human body bounding box set is input into the joint node confidence network to obtain a human body pose information set. In this way, the human body pose information set representing the human body pose can be obtained to facilitate tracking of the human body pose. Secondly, according to the human body pose information set, each human body pose information set in the human body pose information set is matched to generate a matching result set, and a matching result set is obtained. In this way, the matching result corresponding to each human body pose information can be obtained. Then, in response to determining that there is a matching result in the matching result set that satisfies a preset video frame number condition, a human body optical flow bounding box set is generated according to each matching result corresponding to the matching result that satisfies the preset video frame number condition. In this way, the human body optical flow bounding box set representing the predicted human body pose can be obtained. After that, the human body optical flow bounding box set is input into the joint node confidence network to obtain an optical flow human body pose information set and an optical flow joint node confidence information set. In this way, the optical flow human body pose information set representing the human body pose of the human body optical flow bounding box in the human body optical flow bounding box set can be obtained. Finally, based on the optical flow joint node confidence information set, the optical flow human body pose information set is filtered to obtain a target human body pose information set. In this way, the filtering of the optical flow human body pose information set can be realized, so that the target human body pose information set with a relatively high confidence can be obtained. Because the optical flow human body pose information set is filtered using the optical flow joint node confidence information set, the target human body pose information set obtained has a relatively high confidence. For the joint node that is imaged blurred due to motion, when the confidence is relatively high, the joint node is also retained, so that the joint node corresponding to the position probability with a relatively low value caused by motion blur is not mistakenly filtered out, and thus the joint node of the pedestrian can be accurately tracked, and thus the accuracy of the multi-person pose tracking task can be improved.

[0071] With reference to the foregoing Figure 3 , as an implementation of the method shown in the above figures, the present disclosure provides some embodiments of a pose tracking device, which device embodiments correspond to the method embodiments shown in Figure 3 , and the device can be specifically applied to various electronic devices.

[0072] As Figure 3As shown, the pose tracking apparatus 300 of some embodiments includes a detection unit 301, a first input unit 302, a matching unit 303, a generation unit 304, a second input unit 305, and a filtering unit 306. The detection unit 301 is configured to perform frame-by-frame detection on the pedestrian video to obtain a set of human bounding box groups, wherein the human bounding box groups in the set of human bounding box groups correspond to the pedestrian video frames included in the pedestrian video. The first input unit 302 is configured to input the set of human bounding box groups into a joint node confidence network to obtain a set of human pose information groups. The matching unit 303 is configured to perform matching processing on each human pose information group in the set of human pose information groups according to the set of human pose information groups to generate a set of matching results, to obtain a set of matching result groups, wherein the matching result in the matching result group includes at least one video frame number of the human bounding box corresponding to the matching result. The generation unit 304 is configured to, in response to determining that there is a matching result in the set of matching result groups that satisfies a preset video frame number condition, generate a set of human optical flow box groups according to each matching result group corresponding to the matching result that satisfies the preset video frame number condition, wherein the preset video frame number condition is that the at least one video frame number included in the matching result does not include a next video frame number, and the next video frame number is a video frame number of a next frame video of the video frame number corresponding to the matching result in the at least one video frame number. The second input unit 305 is configured to input the set of human optical flow box groups into the joint node confidence network to obtain a set of optical flow human pose information groups and a set of optical flow joint node confidence information groups. The filtering unit 306 is configured to filter the set of optical flow human pose information groups based on the set of optical flow joint node confidence information groups to obtain a set of target human pose information groups.

[0073] It can be understood that the units described in the apparatus 300 correspond to the respective steps in the method described with reference to Figure 3 Thus, the operations, features, and advantages described above for the method also apply to the apparatus 300 and the units included therein, which will not be described here again.

[0074] Reference is made below to Figure 4 which shows a structural schematic diagram of an electronic device (e.g., a computing device) 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0075] As Figure 4As shown, the electronic device 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0076] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device 400 to communicate wirelessly or wired with other devices to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and more or fewer devices can alternatively be implemented. Figure 4 Each block shown in the flowcharts can represent a device, or multiple devices, as necessary.

[0077] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 409, or installed from the storage devices 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0078] Note that the computer readable medium in some embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus or device, or that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a computer readable program code propagated in or on a carrier medium, in which the computer readable program code is embodied. Such propagated computer readable program code can take many forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any suitable medium, including but not limited to, wire, cable, wireless, RF, infrared or any suitable combination of the foregoing.

[0079] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0080] The computer readable medium can be included in the electronic device, or can exist separately from the electronic device. The computer readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform frame-by-frame detection on a pedestrian video to obtain a set of human bounding box groups, wherein the human bounding box groups in the set of human bounding box groups correspond to pedestrian video frames included in the pedestrian video; input the set of human bounding box groups into a joint node confidence network to obtain a set of human pose information groups; perform matching processing on each human pose information group in the set of human pose information groups according to the set of human pose information groups to generate a set of matching results, to obtain a set of sets of matching results, wherein a matching result in the set of matching results includes at least one video frame number of a human bounding box corresponding to the matching result; in response to determining that there is a matching result in the set of sets of matching results that satisfies a preset video frame number condition, the preset video frame number condition being that the at least one video frame number included in the matching result does not include a next video frame number, the next video frame number being a video frame number of a next frame of video of the video frame number corresponding to the matching result in the at least one video frame number, generate a set of human optical flow bounding boxes according to each set of matching results corresponding to the matching result that satisfies the preset video frame number condition; input the set of human optical flow bounding boxes into the joint node confidence network to obtain a set of optical flow human pose information groups and a set of optical flow joint node confidence information groups; and filter the set of optical flow human pose information groups based on the set of optical flow joint node confidence information groups to obtain a set of target human pose information groups.

[0081] Computer program code for carrying out operations of some embodiments of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0082] The computer program product of the first aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a system as described above. The computer readable storage medium can include one or more of: a magnetic disk; a magnetic disk drive; a magnetic tape; a magneto-optical drive; a solid state drive; a semiconductor drive; a flash drive; an optical drive; a holographic drive; a holographic medium; a memory stick; a floppy disk; a flexible disk; a hard disk; a hard disk drive; a holographic storage medium; a RAM drive; a ROM drive; a flash drive; an optical drive; a holographic drive; a magnetic drive; a removable memory port; a bay adapter; a memory PC card slot; a PC card socket; and a disk cartridge comprising a computer readable storage medium.

[0083] The units described in some embodiments of the present disclosure can be implemented by means of software, or by means of hardware. The units described can also be implemented by means of a processor, for example, a processor can be described as comprising a detection unit, a first input unit, a matching unit, a generation unit, a second input unit, and a filtering unit. In some cases, the names of the units do not constitute a limitation on the units themselves, for example, the first input unit can also be described as "a unit that inputs the set of human body bounding boxes into the joint confidence network to obtain the set of human pose information".

[0084] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0085] The above description is merely exemplary of some of the most preferred embodiments of the present disclosure and of the application of the principles thereof. It is to be understood that the scope of the protection is not limited to the specific methods, compositions, devices, and implementations described herein, but only by the scope of the appended claims, wherein reference to an "consisting of" limits the scope of a claim to the specified embodiments.

Claims

1. A posture tracking method, comprising: Performing frame-by-frame detection on the pedestrian video to obtain a human bounding box group set, wherein the human bounding box groups in the human bounding box group set correspond to pedestrian video frames included in the pedestrian video; Inputting the human body bounding box set into a joint point confidence network to obtain a human body posture information set; According to the human body posture information group set, matching processing is performed on each human body posture information group in the human body posture information group set to generate a matching result group, thereby obtaining a matching result group set, wherein the matching result in the matching result group includes at least one video frame number of a human body bounding box corresponding to the matching result; In response to determining that a matching result that satisfies a preset video frame number condition exists in the matching result group set, generating a human body optical flow frame group set according to each matching result group corresponding to the matching result that satisfies the preset video frame number condition, wherein the preset video frame number condition is that at least one video frame number included in the matching result does not include a next video frame number, and the next video frame number is a video frame number of a video frame next to the video frame number corresponding to the matching result in the at least one video frame number; Inputting the human body optical flow frame set into the joint point confidence network to obtain an optical flow human body posture information set and an optical flow joint point confidence information set; Based on the optical flow joint point confidence information set, the optical flow human posture information set is filtered to obtain a target human posture information set.

2. The method according to claim 1, wherein Generating a human optical flow frame set includes: Determining a set of video frames to be processed according to each matching result group corresponding to the matching results that meet the preset video frame number condition; According to each pair of two adjacent video frames to be processed in the set of video frames to be processed, the following steps are performed: Generate an optical flow map matrix according to the two adjacent frames of video to be processed, wherein the optical flow map matrix includes a pixel offset set; Selecting the human body posture information groups corresponding to the two adjacent frames of video frames to be processed from the human body posture information group set as the first human body posture information group to obtain a first human body posture information group set; Selecting first human body posture information corresponding to a matching result that satisfies the preset video frame number condition from the first human body posture information set as second human body posture information, and obtaining at least one second human body posture information; For each second human posture information in the at least one second human posture information, selecting a pixel offset of a corresponding pixel within a range corresponding to the second human posture information from the pixel offset set as a target pixel offset, to obtain a target pixel offset set corresponding to the second human posture information; Generating an optical flow mask map according to the target pixel offset set; The minimum circumscribed rectangle of the optical flow mask image is determined as the human body optical flow frame.

3. The method according to claim 1, wherein The filtering of the optical flow human posture information set includes: For each optical flow joint point confidence information included in the optical flow joint point confidence information set, generating an average optical flow joint point confidence according to the optical flow joint point confidence information; Selecting an average optical flow joint point confidence less than a preset optical flow threshold from the obtained average optical flow joint point confidences as a first confidence to be filtered; The optical flow human posture information corresponding to the obtained first confidence level to be filtered is deleted from the optical flow human posture information group set.

4. The method according to claim 1, wherein The joint point confidence network includes a backbone network, a joint point prediction branch and a joint point availability branch, and the joint point availability branch includes a residual network and at least one classifier.

5. The method according to claim 1, wherein The method further comprises: Determine the matching results that meet the preset single video frame number condition in the matching result group as target matching results, and obtain a target matching result set; Selecting human body posture information corresponding to the target matching result included in the target matching result set from the human body posture information set as human body posture information to be integrated, to obtain a human body posture information set to be integrated; Inputting the human body posture information set to be integrated into a preset pedestrian feature model to obtain the pedestrian feature information set to be integrated; generating pedestrian feature similarity information of each piece of pedestrian feature information to be integrated in the pedestrian feature information set to be integrated according to the pedestrian feature information set to be integrated, to obtain a pedestrian feature similarity information set; Determining pedestrian feature similarity information that meets a preset similarity condition in the pedestrian feature similarity information set as target similarity information to obtain a target similarity information set; updating the matching result set according to the target similarity information set; For each target similarity information in the target similarity information set, the pedestrian feature similarity information corresponding to the target similarity information is deleted from the pedestrian feature similarity information set.

6. The method according to any one of claims 1 to 5, wherein: The matching process for each human body posture information group in the human body posture information group set includes: For the human body posture information groups corresponding to two adjacent pedestrian video frames in the human body posture information group set, the following operations are performed: Determine the human body posture information group corresponding to the previous pedestrian video frame in two adjacent pedestrian video frames as the first human body posture information group; Determine the human body posture information group corresponding to the latter pedestrian video frame in two adjacent pedestrian video frames as the second human body posture information group; determining a distance between each piece of first human body posture information in the first human body posture information group and each piece of second human body posture information in the second human body posture information group to obtain a distance set; performing allocation processing on the first human body posture information in the first human body posture information group and the second human body posture information in the second human body posture information group according to the distance set to obtain adjacent video frame matching results; A matching result set is generated based on the obtained matching results of each adjacent video frame.

7. A posture tracking device comprising: a detection unit configured to perform frame-by-frame detection on the pedestrian video to obtain a set of human bounding box groups, wherein the human bounding box groups in the set of human bounding box groups correspond to pedestrian video frames included in the pedestrian video; A first input unit is configured to input the human body bounding box set into a joint point confidence network to obtain a human body posture information set; a matching unit configured to perform matching processing on each human body posture information group in the human body posture information group set according to the human body posture information group set to generate a matching result group, wherein the matching result in the matching result group includes at least one video frame number of a human body bounding box corresponding to the matching result; a generating unit configured to, in response to determining that a matching result that satisfies a preset video frame number condition exists in the matching result set, generate a set of human optical flow frame groups according to each matching result group corresponding to the matching result that satisfies the preset video frame number condition, wherein the preset video frame number condition is that at least one video frame number included in the matching result does not include a next video frame number, and the next video frame number is a video frame number of a video frame next to the video frame number corresponding to the matching result in the at least one video frame number; A second input unit is configured to input the human optical flow frame set into the joint point confidence network to obtain an optical flow human posture information set and an optical flow joint point confidence information set; The filtering unit is configured to filter the optical flow human posture information set based on the optical flow joint point confidence information set to obtain a target human posture information set.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images