Human pose estimation methods, devices, equipment and storage media

Abstract

This application provides a method, apparatus, device, and storage medium for human pose estimation, relating to the field of computer vision technology. The method includes: acquiring the position coordinates of key points of a human body in a target image sequence; performing anomaly detection on consecutive frames of the target image sequence based on the local data density of the position coordinates to obtain anomaly points in the target image sequence; performing anomaly verification on the anomaly points based on the feature correlation between adjacent frames in the consecutive frames to obtain anomaly verification results; and predicting and correcting the position coordinates based on the anomaly verification results to obtain the target human pose. By utilizing local data density for significant anomaly detection during single-frame prediction, significant jitter generated in human pose estimation can be effectively and specifically reduced, improving the visual quality of the output results. Secondary anomaly verification using feature correlation avoids over-smoothing in scenarios with rapid key point movement, enabling rapid response.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, device, and storage medium for human pose estimation. Background Technology

[0002] Human pose estimation is fundamental for computers to understand human movements and behaviors, and it has broad application prospects in fields such as intelligent healthcare, security monitoring, autonomous driving, human-computer interaction, and entertainment. It has received widespread attention and research in the field of computer vision. This task involves locating the position of a specific body part or joint within a given image or video sequence and estimating its coordinates.

[0003] Currently, most human pose estimation methods exhibit significant jitter and poor stability in their output results. For example, when predicting a series of consecutive frames in a video, factors such as noise, occlusion, background, and lighting can cause slight displacements in the prediction results of adjacent frames, resulting in fluctuations in the final synthesized consecutive frame output. In cases of severe noise and occlusion, single-frame prediction results may also exhibit large-amplitude pose anomalies, causing significant swaying and jitter in the final output. These jitters not only increase the error but also greatly reduce the user's visual experience. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a human pose estimation method, apparatus, device and storage medium, which detects anomalies in continuous frame images by local data density, verifies anomalies based on the correlation of features of adjacent frames, and makes corresponding prediction corrections according to different anomaly verification results. This can effectively and specifically reduce the significant jitter generated in human pose estimation, improve the visual quality of the output results and solve the above-mentioned technical problem of "significant jitter and poor stability".

[0005] In a first aspect, embodiments of this application provide a human pose estimation method, the method comprising: acquiring the position coordinates of key points of a human body in a target image sequence; performing anomaly detection on consecutive frame images in the target image sequence based on the local data density of the position coordinates to obtain anomaly points in the target image sequence; performing anomaly verification on the anomaly points based on the feature correlation between adjacent frame images in the consecutive frame images to obtain anomaly verification results; and performing prediction correction on the position coordinates according to the anomaly verification results to obtain the target human pose.

[0006] In the above implementation process, by utilizing local data density for significant anomaly detection during single-frame prediction, significant jitter generated in human pose estimation can be effectively and specifically reduced, improving the visual quality of the output results. Secondary anomaly verification using feature similarity avoids over-smoothing in fast-moving keypoint scenarios, enabling rapid response.

[0007] Optionally, the step of performing anomaly detection on consecutive frames of the target image sequence based on the local data density of the position coordinates to obtain anomaly points in the target image sequence includes: calculating the k nearest neighbor distance of a human key point in consecutive frames and the average reachable distance from the k nearest neighbor points to the human key point; wherein, the k nearest neighbor points are the k points closest to the human key point; determining the local data density of the human key point based on the k nearest neighbor distance and the average reachable distance; if the local data density of the human key point is much smaller than the local data density of the k nearest neighbor points, then the human key point is identified as an anomaly point.

[0008] In the above implementation process, anomaly detection is performed by utilizing local data density and defined anomaly indicators. When the local data density of a point differs significantly from the density of other points, it indicates that the point is out of the group, i.e., there is an anomaly. By plotting a key point at different times on the coordinates, we can find where the key point is out of the group / mutates. This method does not depend on hyperparameter settings and has good robustness and generalization performance across datasets, further improving the visual quality of the output results.

[0009] Optionally, after determining the local data density of the human body keypoint based on the nearest neighbor distance and the average reachable distance, the method further includes: if the local data density of the human body keypoint is not much less than the local data density of the k nearest neighbor points, the human body keypoint is determined as a normal point; the predicted coordinates of the normal point calculated by the Kalman filter model and the output coordinates of the human body keypoint in the current frame are weighted together to obtain the corrected coordinates of the normal point in the current frame; the coordinates of the target human body keypoint in the target human body pose are determined based on the corrected coordinates and the output coordinates of the normal point in the previous frame by the human body pose estimation model.

[0010] In the above implementation process, the anomaly detection results can detect not only abnormal points in human body key points, but also normal points in human body key points. For normal points detected by anomaly detection, ordinary smoothing is directly performed based on the correlation of features between adjacent frames and the output coordinates of the previous frame, which is good and conducive to outputting more accurate and stable human body posture.

[0011] Optionally, the anomaly verification result includes: a detected anomaly result and a motion mutation anomaly result; the step of performing anomaly verification on the anomaly point based on the feature correlation between adjacent frames in the continuous frame images to obtain the anomaly verification result includes: calculating the feature correlation between the current frame and the previous frame in the continuous frame images; comparing the feature correlation with a preset threshold; if the feature correlation is greater than the preset threshold, then determining the anomaly verification result of the anomaly point as a detected anomaly result; if the feature correlation is less than the preset threshold, then determining the anomaly verification result of the anomaly point as a motion mutation anomaly result.

[0012] In the above implementation process, by setting the feature correlation threshold as a hyperparameter in the model, excessive smoothing in the case of rapid human keypoint movement is avoided, which enables rapid response and improves the stability and accuracy of human pose estimation. At the same time, the correlation measurement is easier to generalize than point-to-point error, which improves the robustness of the model.

[0013] Optionally, the anomaly verification result includes: anomaly detection result and motion mutation anomaly result; the step of predicting and correcting the position coordinates based on the anomaly verification result to obtain the target human pose includes: if the anomaly verification result is anomaly detection result, then the predicted coordinates of the human keypoints calculated by the Kalman filter model are used as the corrected coordinates of the human keypoints in the current frame; if the anomaly verification result is motion mutation anomaly result, then the predicted coordinates of the human keypoints calculated by the Kalman filter model and the output coordinates of the human keypoints in the current frame by the human pose estimation model are weighted to obtain the corrected coordinates of the human keypoints in the current frame; the target human keypoint coordinates in the target human pose are determined based on the corrected coordinates and the output coordinates of the human keypoints in the previous frame by the human pose estimation model.

[0014] In the above implementation process, based on the two different prediction and correction methods, the corrected coordinate prediction result of the current frame can be obtained. Then, the final output coordinates of the current frame are obtained by weighting the coordinates with the output coordinates of the previous frame based on the correlation of features between adjacent frames. For detected anomalies, a stronger smoothing effect is applied; for rapid motion abrupt changes, a normal smoothing effect is applied, which follows the anomalies better and is conducive to outputting more accurate and stable human posture.

[0015] Optionally, determining the target human key point coordinates in the target human pose based on the corrected coordinates and the output coordinates of the human pose estimation model for the human key points in the previous frame includes: weighting the corrected coordinates of the human key points in the current frame with the output coordinates of the human pose estimation model for the human key points in the previous frame based on the feature correlation to obtain the target human key point coordinates.

[0016] In the above implementation process, weighting is performed based on feature correlation. That is, different weight coefficients are assigned to the current frame and the previous frame according to the different correlations between two adjacent frames, which improves the visual quality of the output results.

[0017] Optionally, the feature correlation between adjacent frame images is determined by at least one of Euclidean distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, Hamming distance, cosine similarity, Pearson correlation coefficient, and KL divergence.

[0018] In the above implementation process, various methods that can characterize feature correlation are used to measure the relationship between two adjacent frames, which improves the versatility of the human pose estimation method.

[0019] Secondly, embodiments of this application provide a human pose estimation device, the device comprising: an acquisition module, configured to acquire the position coordinates of key points of a human body in a target image sequence; an anomaly detection module, configured to perform anomaly detection on consecutive frame images in the target image sequence based on the local data density of the position coordinates, to obtain anomaly points in the target image sequence; an anomaly verification module, configured to perform anomaly verification on the anomaly points based on the feature correlation between adjacent frame images in the consecutive frame images, to obtain anomaly verification results; and a pose estimation module, configured to predict and correct the position coordinates according to the anomaly verification results, to obtain the pose of the target human body.

[0020] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the above-described method.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A flowchart illustrating a human pose estimation method provided in an embodiment of this application;

[0025] Figure 2 A flowchart illustrating a human pose estimation method provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the functional modules of the human posture estimation device provided in the embodiments of this application;

[0027] Figure 4 A block diagram of an electronic device that provides a human posture estimation device for embodiments of this application.

[0028] Icons: 210 - Acquisition module; 220 - Anomaly detection module; 230 - Anomaly verification module; 240 - Attitude estimation module; 300 - Electronic device; 311 - Memory; 312 - Memory controller; 313 - Processor; 314 - Peripheral interface; 315 - Input / output unit; 316 - Display unit. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] The inventors of this application have noted that, when predicting a series of frames in a video, to avoid the influence of factors such as noise, occlusion, background, and lighting, most existing human pose estimation methods smooth the prediction results of consecutive frames through temporal convolution. These methods can only address small fluctuations caused by errors in the prediction results of adjacent frames, and perform poorly when faced with significant anomalies in single-frame predictions (e.g., an anomaly in a single human keypoint, such as the nose, in a single frame of a video sequence). Another part of the existing technology detects abnormal frames by calculating the point-to-point error (PE) between corresponding human keypoints in adjacent frames, and performs additional processing on frames with PE exceeding a certain threshold. This hard-thresholding method lacks robustness and generalization, is overly dependent on hyperparameters, and cannot adapt to fast-moving scenes. In view of this, the embodiments of this application provide a human pose estimation method as described below:

[0032] Please see Figure 1 , Figure 1 The flowchart of a human pose estimation method provided in this application embodiment may include steps 100, 120, 140 and 160.

[0033] Step 100: Obtain the position coordinates of key points of the human body in the target image sequence;

[0034] Step 120: Based on the local data density of the position coordinates, perform anomaly detection on consecutive frames in the target image sequence to obtain anomaly points in the target image sequence;

[0035] Step 140: Based on the feature correlation between adjacent frames in consecutive frames, perform anomaly verification on outliers and obtain anomaly verification results;

[0036] Step 160: Based on the anomaly verification results, predict and correct the position coordinates to obtain the target human pose.

[0037] For example, the target image sequence can be two-dimensional (2D) or three-dimensional (3D) human images or videos captured in real time by a mobile shooting device (usually a camera) or communication transmission component. The video can be a series of captured images (called frames) displayed at a given frequency, and the images can be single video frames stopped at specific frames in a sequence. For example, it could be a real-time 2D RGB monocular image sequence. Human keypoints can be all points to be detected in all image or video frames input to the entire human pose estimation task. They can also be N joint localizations generated by human keypoint detection for each human body appearing in each frame of the target image sequence. For example, they could be more than thirty human keypoints detected in a real-time 2D RGB monocular image sequence, such as the nose, left shoulder, right shoulder, right elbow, right elbow, left hip, right hip, left knee, right knee, left ankle, right ankle, etc.

[0038] Real-time 2D RGB monocular image sequences are input into the inference part of a pre-trained human pose estimation convolutional neural network model. Image features are extracted from each input image frame, and human keypoints are detected, identifying N human keypoints. These keypoints are then mapped to N tuples of coordinates. Optionally, the human pose estimation algorithm employs the MoveNet algorithm to obtain the position coordinates of human keypoints from the image sequence. ResNet is used as a local convolutional network, and the ResNet network is laterally connected to a feature pyramid network to process image patches in keyframes, outputting high-resolution key feature maps rich in semantic information. Four prediction heads are used to predict the position coordinates of human keypoints in the key feature maps, roughly locating the range of human keypoints. Post-processing weighted refinement of human keypoint coordinates outputs a set of human keypoint position coordinates closest to the center of the image patch.

[0039] After obtaining the position coordinates of key human body points, the following steps are taken: First, anomaly detection is performed on consecutive frames in the target image sequence using local data density. Specifically, for each key human body point, the local data density of the current position coordinates is calculated based on the historical position coordinates of the previous T frames and the current position coordinates to determine if an anomaly is detected. This anomaly could be a sudden change caused by rapid motion or a detection anomaly, requiring secondary verification. Next, the anomaly points detected are validated using the feature correlation between adjacent frames. Specifically, shallow image features are extracted, and the feature correlation between the current frame and the previous frame is calculated. The magnitude of the feature correlation determines whether the difference between the current frame and the previous frame is a sudden change caused by rapid motion or a detection anomaly. Finally, the position coordinates of key human body points are predicted and corrected based on the corresponding anomaly results. Specifically, if the features are similar before and after, but the latter is anomalous, it indicates that the latter does indeed have a detection anomaly, and a stronger smoothing is applied. If the features are dissimilar, but the latter is anomalous, it may be due to rapid motion, and a normal smoothing is applied, which follows the pattern well. After smoothing, the final estimated human pose is obtained.

[0040] By utilizing local data density for significant anomaly detection in single-frame prediction, significant jitter in human pose estimation can be effectively and specifically reduced, improving the visual quality of the output. Secondary anomaly verification using feature similarity avoids over-smoothing in fast-moving keypoint scenarios, enabling rapid response.

[0041] In one embodiment, step 120 may include steps 121, 122 and 123.

[0042] Step 121: Calculate the k nearest neighbor distances of human body keypoints in consecutive frame images and the average reachable distances from the k nearest neighbor points to the human body keypoints; where the k nearest neighbor points are the k points closest to the human body keypoints.

[0043] Step 122: Determine the local data density of human body key points based on the k-nearest neighbor distance and average reachable distance;

[0044] Step 123: If the local data density of a human body key point is much smaller than the local data density of the k nearest neighbor points, then the human body key point is identified as an anomaly.

[0045] For example, local data density can be a local outlier factor (LOF), which determines whether a point p is an outlier by comparing the density of each point p with that of its neighboring points. The lower the density of point p, the more likely it is to be identified as an outlier.

[0046] like Figure 2As shown, after step 100, for each keypoint, the local data density of the current position coordinates can be calculated based on the historical position coordinates of the previous T frames and the current position coordinates to determine whether it is abnormal; if abnormal, a secondary verification of significant anomalies in a single frame is performed; otherwise, no anomaly verification is performed, and the output is directly smoothed. Optionally, the anomaly detection process can be:

[0047] Consider a sample set consisting of a human keypoint p in the current frame and the corresponding human keypoints p in the previous T frames, with a sample size of T+1. Define the k-nearest neighbor distance d of point p. k (p) represents the Euclidean distance between all sample points except p and the k-th nearest sample point to p. The k-nearest neighbor distance d of point p. k The smaller the value of (p), the denser the data around point p. Define the directed reachable distance d from point o to point p. r (o,p)=max{d k (o),||op||}, that is, when the distance from point o to point p is less than the k-nearest neighbor distance d of point o. k When (o), its reachable distance d is considered to be... r (o,p) equals d k (o).

[0048] Define the local data density ld of point p k (p) is the reciprocal of the average reachable distance from point p to its k nearest neighbors o, i.e.:

[0049]

[0050] Among them, the local data density ld k The larger the value of (p), the more compact the data around point p. By comparing the local data density of point p with that of other sample points in its neighborhood, it can be determined whether point p is an anomaly. The anomaly index is defined as:

[0051]

[0052] When Λ(p) is much greater than 1, it means that the local data density of point p is significantly less than (much less than) the data density of other sample points in its neighborhood, and the key point p of the human body can be considered an anomaly.

[0053] Anomaly detection is performed by utilizing local data density and defined anomaly indicators. When the local data density of a point differs significantly from that of other points, it indicates that the point is out of the group, i.e., there is an anomaly. By plotting a key point at different times on the coordinate system, we can find where the key point is out of the group / mutates. This method does not depend on hyperparameter settings and has good robustness and generalization performance across datasets, further improving the visual quality of the output results.

[0054] In one embodiment, please continue reading Figure 2 After step 122, the method may further include steps 130, 131 and 132.

[0055] Step 130: If the local data density of a human body key point is not much less than the local data density of the k nearest neighbor points, the human body key point is determined as a normal point.

[0056] Step 131: Weight the predicted coordinates of the normal points calculated by the Kalman filter model and the output coordinates of the human body key points in the current frame by the human pose estimation model to obtain the corrected coordinates of the normal points in the current frame.

[0057] Step 132: Determine the coordinates of key points of the target human body in the target human body pose based on the corrected coordinates and the output coordinates of the normal points in the previous frame of the human body pose estimation model.

[0058] For example, "not much less than" can include two cases: greater than and slightly less than. Therefore, for anomaly detection, there are two results: normal and abnormal. The defined anomaly index is the average of the ratios, so there are several specific cases: when the ratio is close to 1, it means that the density of human keypoint p is close to that of other points (slightly less or slightly greater), and it can be regarded as a normal point; when the ratio is less than 1, it means that the density of human keypoint p is greater than that of other points, and human keypoint p is a dense point, which can also be regarded as a normal point; when the ratio is much greater than 1, it means that the density of human keypoint p is much less than that of other points, and all of these are regarded as abnormal, which can be specifically ordinary abnormal points and significant abnormal points.

[0059] If the point is a normal point (i.e., there are no significant anomalies), no further anomaly verification is needed. Ordinary smoothing can be performed directly using the Kalman filter model formula, weighting the actual measured value and the model's predicted value to smooth out ordinary jitter. If the point is an anomaly (i.e., there are significant anomalies), indicating that the actual measured value is inaccurate, then only the predicted value is used for weighting. Specifically, the predicted coordinates of the normal point calculated by the Kalman filter model are weighted with the output coordinates of the human pose estimation model for the current frame's human keypoints to obtain the corrected coordinates of the normal point in the current frame. The corrected coordinates are then further weighted with the output coordinates of the normal point in the previous frame to obtain the final coordinates of the human keypoints in the human pose. For example, the N coordinate tuple outputs of the human pose estimation convolutional neural network model are smoothed to obtain the corrected coordinate prediction results, which are then weighted with the output coordinates of the previous frame based on the correlation of features between adjacent frames to obtain the final output coordinates of the current frame.

[0060] In addition to detecting abnormal points in human body key points, the anomaly detection results can also detect normal points in human body key points. For normal points detected by anomaly detection, ordinary smoothing is directly performed based on the correlation of features between adjacent frames and the output coordinates of the previous frame, which has a good tracking effect and is conducive to outputting more accurate and stable human posture.

[0061] In one embodiment, the anomaly verification result includes: anomaly detection result and motion mutation anomaly result; step 140 may include: step 141, step 142, step 143 and step 144.

[0062] Step 141: Calculate the feature correlation between the current frame and the previous frame in a series of frames;

[0063] Step 142: Compare the feature correlation with the preset threshold;

[0064] Step 143: If the feature correlation is greater than the preset threshold, then the anomaly verification result of the anomaly point is determined as the anomaly detection result;

[0065] Step 144: If the feature correlation is less than the preset threshold, the abnormal verification result of the abnormal point is determined to be the abnormal result of motion mutation.

[0066] For example, the anomaly verification result can include two cases: anomalies caused by the detection itself and sudden anomalies caused by rapid motion. The feature correlation between the current frame and the previous frame is calculated, and it is determined whether it is less than a preset threshold. If the correlation is greater than the preset threshold, it means that the difference between the two frames is small, and there is a detection anomaly in the current frame. If the correlation is less than the preset threshold, it means that the difference between the two frames is large, and the significant change in the single frame coordinates is caused by the difference in the image itself (i.e., sudden motion) rather than by the detection anomaly.

[0067] Optionally, cosine similarity is used to represent the feature correlation between adjacent frames. To better represent image texture information, a human pose estimation convolutional neural network model is used to extract shallow image features from the input target image sequence. Let the image features of the current frame be f(I t The feature of the previous frame is f(I). t-1 ), where I is a sequence of images, then its feature cosine similarity can be expressed as:

[0068]

[0069] The feature correlation between adjacent frames ranges from 0 to 1. A value of 1 indicates correlation, and a value of 0 indicates no correlation. Generally: 0.8-1.0 indicates extremely strong correlation; 0.6-0.8 indicates strong correlation; 0.4-0.6 indicates moderate correlation; 0.2-0.4 indicates weak correlation; and 0.0-0.2 indicates extremely weak correlation or no correlation. The preset threshold can be set to any value within the range of 0-1 according to the actual situation. Optionally, if the preset threshold is set to 0.3, then when the feature correlation is greater than 0.3, it is considered that the difference between adjacent frame images is small, and the current frame has an anomaly. When the feature correlation is less than 0.3, it is considered that the difference between adjacent frame images is large, and the current frame has a sudden change anomaly.

[0070] By setting a feature correlation threshold as a hyperparameter in the model, excessive smoothing in scenarios with rapid human keypoint movement is avoided, enabling rapid response and improving the stability and accuracy of human pose estimation. Simultaneously, the correlation metric is more easily generalized than point-to-point error, enhancing the model's robustness. In one embodiment, the anomaly verification result includes: detected anomaly results and motion mutation anomaly results; step 160 may include: steps 161, 162, and 163.

[0071] Step 161: If the anomaly verification result is an anomaly detection result, then the predicted coordinates of the human body key points calculated by the Kalman filter model are used as the corrected coordinates of the human body key points in the current frame.

[0072] Step 162: If the anomaly verification result is a motion mutation anomaly result, then the predicted coordinates of the human key points calculated by the Kalman filter model and the output coordinates of the human key points in the current frame by the human pose estimation model are weighted to obtain the corrected coordinates of the human key points in the current frame.

[0073] Step 163: Determine the coordinates of the target human key points in the target human pose based on the corrected coordinates and the output coordinates of the human pose estimation model for the human key points in the previous frame.

[0074] For example, assuming that at time t, a frame (the current frame) is input into the processing system corresponding to the human pose estimation method of this applicant, the human pose estimation model first outputs a coordinate, and then the Kalman filter model corrects it to obtain the corrected coordinate of the current frame. Among them, the quantity with the subscript t is regarded as the current frame, and the quantity with the subscript t-1 is regarded as the previous frame. Specifically, (1) for detecting anomalies: the predicted coordinates of the Kalman filter model are directly used as the corrected coordinates of the current frame. Assume that the coordinates of the human key point p in the sample space are (x p ,y p ), the speed of motion or movement is (Δx) p ,Δy p The state variables of the discrete linear system are X = (xp ,y p ,Δx p ,Δy p The system model can be represented as:

[0075] X t =AX t-1 +w t-1

[0076] Z t =HX t-1 +v t-1

[0077] Where Z represents the state observations, A and H are the state transition matrix and observation matrix, respectively, and w and v are the process noise and measurement noise, with covariances Q and R, respectively. Due to a significant anomaly in the current frame, this step... The final output state is based solely on the human body key points of the previous frame. Predict the correction state of the current frame The first two items in the state This refers to the corrected coordinates of the key points of the human body in the current frame.

[0078] (2) For motion abrupt changes: The predicted coordinates of the Kalman filter model are weighted with the output coordinates of the human pose estimation model to obtain the corrected coordinates for the current frame. The output state of the human pose estimation model is used as the state observation Z. k .pass Predict the current frame state, and simultaneously determine the prediction error covariance matrix as P. t - =AP t-1 A T +Q; then calculate the weighted average of the predicted coordinates of the Kalman filter model and the output coordinates of the human pose estimation model as K = P t - H T HP t - H T +R) -1 And the weighted average is used to obtain the corrected state of the current frame. The first two items in the state This refers to the corrected coordinates of the human body keypoints in the current frame; where the covariance matrix is ​​maintained as P. t =(I-KH)P t - .

[0079] Based on the two different prediction and correction methods described above, the corrected coordinate prediction result for the current frame can be obtained. Then, the final output coordinates of the current frame are obtained by weighting the coordinates with the output coordinates of the previous frame based on the correlation of features between adjacent frames. For detected anomalies, a stronger smoothing effect is applied; for rapid motion abrupt changes, a normal smoothing effect is applied, which follows the anomalies better and contributes to outputting more accurate and stable human posture.

[0080] In one embodiment, step 163 may include: step 1631.

[0081] Step 1631: Based on feature correlation, the corrected coordinates of the human body key points in the current frame are weighted with the output coordinates of the human body key points in the previous frame by the human pose estimation model to obtain the coordinates of the target human body key points.

[0082] For example, the current frame refers to the frame currently being processed (i.e., the newly acquired frame); the previous frame refers to the video frame at the moment preceding the current frame. For instance, if the time sequence number of the current frame is t, then the time sequence number of the previous frame is t-1. In this embodiment, feature correlation is represented by the cosine similarity of features between adjacent frames. Momentum updates are performed on the corrected coordinates based on the cosine similarity of features between adjacent frames, that is, the corrected coordinates of the current frame are weighted with the output coordinates of the previous frame to obtain the final output coordinates of the current frame. A higher cosine similarity between features between adjacent frames indicates a smaller change in the position of human key points; therefore, the weight of the human key point position coordinates of the previous frame is increased in the momentum update, and vice versa. The final formula for calculating the target human key point coordinates can be:

[0083]

[0084] in, The corrected coordinates for the current frame. The coordinates are the output coordinates of the previous frame. It can be seen that the weighting coefficients of the current frame coordinates and the previous frame coordinates are related to the cosine similarity. The larger the cosine similarity, the larger the cosine similarity, and the greater the weight of the previous frame in the formula. Weighting based on feature correlation—that is, assigning different weighting coefficients to the current frame and the previous frame according to the different correlations between adjacent frames—improves the visual quality of the output result.

[0085] In one embodiment, the feature correlation between adjacent frame images is determined by at least one of Euclidean distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, Hamming distance, cosine similarity, Pearson correlation coefficient, and KL divergence.

[0086] For example, the representation methods for feature correlation include, but are not limited to, Euclidean distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, Hamming distance, cosine similarity, Pearson correlation coefficient, and KL divergence. These representation methods yield a numerical correlation result, using numbers to measure the relationship between two adjacent frames. This can also be directional; for example, 1 represents positive correlation, -1 represents negative correlation, and 0 represents no correlation. This allows for the measurement of the strength of the relationship between variables in two adjacent frames, with the correlation being weaker the closer to 0. Employing multiple methods to represent feature correlation and measure the relationship between two adjacent frames improves the versatility of human pose estimation methods.

[0087] Please see Figure 3 , Figure 3 This is a schematic diagram of a human posture estimation device provided in an embodiment of the present application. The module may include: an acquisition module 210, an anomaly detection module 220, an anomaly verification module 230, and a posture estimation module 240.

[0088] The acquisition module 210 is used to acquire the position coordinates of key human body points in the target image sequence;

[0089] Anomaly detection module 220 is used to perform anomaly detection on consecutive frames of images in a target image sequence based on local data density of position coordinates, and to obtain anomaly points in the target image sequence;

[0090] Anomaly verification module 230 is used to perform anomaly verification on anomaly points based on the feature correlation between adjacent frame images in continuous frame images, and obtain anomaly verification results.

[0091] The pose estimation module 240 is used to predict and correct the position coordinates based on the anomaly verification results in order to obtain the pose of the target human body.

[0092] Optionally, the anomaly detection module 220 can be used for:

[0093] Calculate the k-nearest neighbor distances of human keypoints in consecutive frames of images and the average reachable distances from the k nearest neighbor points to the human keypoints; where the k nearest neighbor points are the k points closest to the human keypoints.

[0094] The local data density of human body key points is determined based on the nearest neighbor distance and the average reachable distance;

[0095] If the local data density of a key point on the human body is much smaller than the local data density of its k nearest neighbors, then the key point on the human body is identified as an anomaly.

[0096] Optionally, the anomaly detection module 220 can be used for:

[0097] If the local data density of a human body key point is not much less than the local data density of the k nearest neighbor points, the human body key point is determined as a normal point.

[0098] The predicted coordinates of normal points calculated by the Kalman filter model and the output coordinates of human key points in the current frame by the human pose estimation model are weighted to obtain the corrected coordinates of normal points in the current frame.

[0099] Based on the corrected coordinates and the output coordinates of the normal points in the previous frame by the human pose estimation model, the coordinates of the key points of the target human body in the target human pose are determined.

[0100] Optionally, the anomaly verification module 230 can be used for:

[0101] Based on the feature correlation between adjacent frames in consecutive image frames, anomaly verification is performed on outliers to obtain anomaly verification results, including:

[0102] Calculate the feature correlation between the current frame and the previous frame in a series of frames;

[0103] The feature correlation is compared with a preset threshold;

[0104] If the feature correlation is greater than the preset threshold, the anomaly verification result of the anomaly point is determined as the anomaly detection result;

[0105] If the feature correlation is less than the preset threshold, the anomaly verification result of the anomaly point is determined to be a motion mutation anomaly result.

[0106] Optionally, the attitude estimation module 240 can be used for:

[0107] Based on the anomaly verification results, the position coordinates are predicted and corrected to obtain the target human pose, including:

[0108] If the anomaly verification result is an anomaly detection result, the predicted coordinates of the human body key points calculated by the Kalman filter model will be used as the corrected coordinates of the human body key points in the current frame.

[0109] If the abnormal verification result is a motion mutation abnormal result, the predicted coordinates of human key points calculated by the Kalman filter model and the output coordinates of human key points in the current frame by the human pose estimation model are weighted to obtain the corrected coordinates of human key points in the current frame.

[0110] Based on the corrected coordinates and the output coordinates of the human body key points in the previous frame by the human body pose estimation model, the coordinates of the target human body key points in the target human body pose are determined.

[0111] Optionally, the attitude estimation module 240 can be used for:

[0112] Based on feature correlation, the corrected coordinates of human key points in the current frame are weighted with the output coordinates of human key points in the previous frame by the human pose estimation model to obtain the coordinates of the target human key points.

[0113] Optionally, the feature correlation between adjacent frame images is determined by at least one of Euclidean distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, Hamming distance, cosine similarity, Pearson correlation coefficient, and KL divergence.

[0114] Please see Figure 4 , Figure 4 This is a block diagram of an electronic device. The electronic device 300 may include a memory 311, a memory controller 312, a processor 313, a peripheral interface 314, an input / output unit 315, and a display unit 316. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 300. For example, the electronic device 300 may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0115] The aforementioned memory 311, memory controller 312, processor 313, peripheral interface 314, input / output unit 315, and display unit 316 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 313 is used to execute executable modules stored in the memory.

[0116] The memory 311 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 311 stores programs, and the processor 313 executes these programs upon receiving execution instructions. The methods executed by the electronic device 300, as defined in any embodiment of this application, can be applied to or implemented by the processor 313.

[0117] The aforementioned processor 313 may be an integrated circuit chip with signal processing capabilities. The processor 313 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0118] The peripheral interface 314 described above couples various input / output devices to the processor 313 and the memory 311. In some embodiments, the peripheral interface 314, the processor 313, and the memory controller 312 can be implemented in a single chip. In other instances, they can be implemented by separate chips.

[0119] The input / output unit 315 described above is used to provide user input data. The input / output unit 315 may be, but is not limited to, a mouse and keyboard.

[0120] The aforementioned display unit 316 provides an interactive interface (e.g., a user interface) for the user to reference between the electronic device 300 and the user. In this embodiment, the display unit 316 may be a liquid crystal display (LCD) or a touch screen display. The LCD or touch screen display can show the process of the processor executing the program.

[0121] The electronic device 300 in this embodiment can be used to perform the various steps in the various methods provided in the embodiments of this application.

[0122] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0123] The computer program product of the above-described method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps in the above-described method embodiments. For details, please refer to the above-described method embodiments, which will not be repeated here.

[0124] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. The functional modules in the embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] It should be noted that if the function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A human pose estimation method, characterized by, The method comprises: obtaining position coordinates of human body key points in a target image sequence; based on the local data density of the position coordinates, performing anomaly detection on the continuous frame images in the target image sequence to obtain abnormal points in the target image sequence; based on the feature correlation between adjacent frame images in the continuous frame images, performing anomaly verification on the abnormal points to obtain an anomaly verification result; according to the anomaly verification result, performing prediction correction on the position coordinates to obtain a target human body posture; the method comprises: calculating the k-nearest neighbor distance of the human body key points in the continuous frame images and the average reachable distance of the k-nearest neighbors to the human body key points; wherein the k-nearest neighbors are the k points closest to the human body key points; determining the local data density of the human body key points according to the k-nearest neighbor distance and the average reachable distance; if the local data density of the human body key points is much smaller than the local data density of the k-nearest neighbors, the human body key points are determined as abnormal points; after determining the local data density of the human body key points according to the k-nearest neighbor distance and the average reachable distance, the method further comprises: if the local data density of the human body key points is not much smaller than the local data density of the k-nearest neighbors, the human body key points are determined as normal points; weighting the predicted coordinates of the normal points calculated by the Kalman filter model and the output coordinates of the current frame human body key points by the human body posture estimation model to obtain the corrected coordinates of the current frame normal points; determining the target human body key point coordinates in the target human body posture according to the corrected coordinates and the output coordinates of the previous frame normal points by the human body posture estimation model; wherein the anomaly verification result comprises: detection anomaly result and motion mutation anomaly result; the method comprises: calculating the feature correlation between the current frame and the previous frame in the continuous frame images; comparing the feature correlation with a preset threshold; if the feature correlation is greater than the preset threshold, the anomaly verification result of the abnormal point is determined as the detection anomaly result; if the feature correlation is less than the preset threshold, the anomaly verification result of the abnormal point is determined as the motion mutation anomaly result; the method comprises: if the anomaly verification result is the detection anomaly result, the predicted coordinates of the human body key points calculated by the Kalman filter model are taken as the corrected coordinates of the current frame human body key points; if the anomaly verification result is the motion mutation anomaly result, the predicted coordinates of the human body key points calculated by the Kalman filter model and the output coordinates of the current frame human body key points by the human body posture estimation model are weighted to obtain the corrected coordinates of the current frame human body key points. According to the modified coordinates of the current frame of human body key points and the output coordinates of the previous frame of human body key points by the human body posture estimation model, target human body key point coordinates in a target human body posture are determined.

2. The method of claim 1, wherein, The determining of the target human body key point coordinates in the target human body posture according to the modified coordinates of the current frame of human body key points and the output coordinates of the previous frame of human body key points by the human body posture estimation model comprises: The modified coordinates of the current frame of human body key points and the output coordinates of the previous frame of human body key points by the human body posture estimation model are weighted based on the feature correlation to obtain the target human body key point coordinates.

3. The method according to any of claims 1-2, characterized by, The feature correlation between the adjacent frame images is determined by at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Mahalanobis distance, Hamming distance, cosine similarity, Pearson correlation coefficient, and KL divergence. The device comprises:

4. A human pose estimation apparatus characterized by comprising: An acquisition module is configured to acquire position coordinates of human body key points in a target image sequence. An anomaly detection module is configured to perform anomaly detection on consecutive frame images in the target image sequence based on local data density of the position coordinates to obtain an anomaly point in the target image sequence. An anomaly verification module is configured to perform anomaly verification on the anomaly point based on feature correlation between adjacent frame images in the consecutive frame images to obtain an anomaly verification result. A posture estimation module is configured to predict and correct the position coordinates according to the anomaly verification result to obtain a target human body posture. The anomaly detection module is specifically configured to: calculate a k-nearest neighbor distance of a human body key point in a consecutive frame image and an average reachable distance of k nearest neighbors to the human body key point; wherein the k nearest neighbors are the k points closest to the human body key point; determine local data density of the human body key point according to the k-nearest neighbor distance and the average reachable distance; if the local data density of the human body key point is much smaller than the local data density of the k nearest neighbors, determine the human body key point as an anomaly point; if the local data density of the human body key point is not much smaller than the local data density of the k nearest neighbors, determine the human body key point as a normal point; weight a predicted coordinate of the normal point calculated by a Kalman filter model and an output coordinate of the current frame of human body key points by a human body posture estimation model to obtain a modified coordinate of the current frame of normal points; and determine target human body key point coordinates in a target human body posture according to the modified coordinates and output coordinates of the previous frame of normal points by the human body posture estimation model. The anomaly verification result comprises a detection anomaly result and a motion mutation anomaly result. The anomaly verification module is specifically configured to: calculate a feature correlation between a current frame and a previous frame in consecutive frame images; compare the feature correlation with a preset threshold; if the feature correlation is greater than the preset threshold, determine that the anomaly verification result of the anomaly point is a detection anomaly result; and if the feature correlation is less than the preset threshold, determine that the anomaly verification result of the anomaly point is a motion mutation anomaly result. ​ The posture estimation module is specifically configured to: if the abnormality verification result is a detection abnormality result, taking the predicted coordinates of the human body key points calculated by the Kalman filtering model as the corrected coordinates of the human body key points of the current frame; if the abnormality verification result is a motion mutation abnormality result, weighting the predicted coordinates of the human body key points calculated by the Kalman filtering model and the output coordinates of the human body key points of the current frame output by the human body posture estimation model to obtain the corrected coordinates of the human body key points of the current frame; and determining the target human body key point coordinates in the target human body posture according to the corrected coordinates and the output coordinates of the human body key points of the previous frame output by the human body posture estimation model.

5. An electronic device, comprising: The method comprises the following steps: A processor, a memory, wherein the memory stores machine readable instructions executable by the processor, and when the electronic device is running, the machine readable instructions are executed by the processor to perform the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the method according to any one of claims 1 to 3.

Images