A behavior recognition method, device, apparatus and storage medium

Abstract

This application discloses a behavior recognition method, apparatus, device, and storage medium, relating to the field of computer vision. The method includes: detecting key points in a target video using a preset detection method; filtering the key points based on preset filtering rules to obtain discarded key points and filtered key points; establishing a linear model based on the discarded key points and the filtered key points, and using the linear model to perform regression prediction on the discarded key points to obtain regressed key points; correcting the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; and performing corresponding analysis and recognition operations based on the corrected key points. This application improves the accuracy and robustness of behavior recognition by filtering to obtain discarded key points and establishing a linear model to correct them, even when key point detection is inaccurate.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a behavior recognition method, apparatus, device, and storage medium. Background Technology

[0002] In real life, various scenarios present diverse application requirements for behavior recognition algorithms, such as recognizing falls. Skeleton-based algorithms are a common behavior recognition solution. This method typically begins with a feature extraction network to extract the coordinates of key human points in video images. Commonly used skeletal key point extraction feature networks include OpenPose (environment setup) and AlphaPose (human pose recognition). Then, graph neural networks or convolutional networks are used to learn the positional relationships of different key points, ultimately recognizing the human's actions. This is a two-stage behavior recognition method, usually comprising a feature extraction network and a behavior classification network. Its advantage lies in the fact that skeleton feature extraction can filter out background information irrelevant to the human body from RGB (Red-Green-Blue, the three primary colors) images. However, a significant problem arises in practical applications: the accuracy of the second-stage behavior classification network heavily depends on the first-stage feature extraction network. When the key human points extracted by the feature extraction network are inaccurate or incomplete, the behavior classification network often fails to accurately recognize the human's actions. However, real-world scenarios often involve incomplete images or occlusion issues. The entire human action can be affected by target occlusion or the poor robustness of the feature extraction network to complex scenes, leading to inaccurate feature point extraction.

[0003] Keypoints can be considered as feature values ​​of data. For the problems of missing or drifting keypoints, a common engineering solution is to handle missing feature values, such as discarding data or using the average feature value as a substitute. Discarding data means that when keypoint data is missing, it is sometimes discarded in practice. The average feature value substitution method means that when a keypoint is missing from the keypoint data, the average value of that keypoint across all data is used to replace the missing value of that skeletal keypoint. However, this method has the following drawbacks: due to the complexity and diversity of real-world application scenarios, there will be a large amount of incomplete skeletal data. Discarding data with missing feature points will greatly reduce data utilization during recognition model training, limiting the model's generalization ability; during recognition model inference, if the regressed keypoints in the application scenario are inaccurate, the model will fail. The average feature value substitution method ignores the positional relationship between the missing keypoint and other keypoints, leading to inaccurate keypoint data and ultimately affecting the performance of the recognition model.

[0004] Another practical method is to regress missing keypoints using a borderline curve fitting algorithm between consecutive frames. This method utilizes the relationship between consecutive frames and an adjacent edge curve fitting algorithm to calculate the keypoint coordinates and confidence level for keypoints with a confidence level below a certain value. The drawbacks of this method are: it requires a large amount of information from consecutive frames, and the algorithm cannot regress keypoints from a single captured image, limiting its application scope. Borderline curve fitting must ensure that the keypoints are within the image area; it cannot regress keypoints outside the image area. If adjacent points in consecutive frames are outside the image region, the algorithm fails. Therefore, this algorithm cannot solve the problem of keypoint regression and correction for incomplete human figures. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a behavior recognition method, apparatus, device, and storage medium that can improve the accuracy and robustness of behavior recognition. The specific solution is as follows:

[0006] Firstly, this application discloses a behavior recognition method, including:

[0007] The target video is detected using a preset detection method to obtain key points;

[0008] The key points are filtered based on preset filtering rules to obtain discarded key points and filtered key points.

[0009] A linear model is established based on the discarded key points and the filtered key points, and the linear model is used to perform regression prediction on the discarded key points to obtain the regressed key points.

[0010] The discarded key points are corrected according to the preset correction rules and the key points after regression to obtain the corrected key points;

[0011] Based on the corrected key points, corresponding analysis and identification operations are performed.

[0012] Optionally, the step of using a preset detection method to detect key points in the target video includes:

[0013] A preset detection method is used to detect each frame of the target video to obtain a preset number of key points, the coordinates of the key points, and the confidence level of the key points in each frame.

[0014] Optionally, the step of filtering the key points based on preset filtering rules to obtain discarded key points and filtered key points includes:

[0015] Compare the confidence scores of all the key points with a preset confidence threshold;

[0016] The key points whose confidence level is greater than the preset confidence threshold are determined as the filtered key points.

[0017] The key points whose confidence level is less than the preset confidence threshold are identified as the discarded key points.

[0018] Optionally, after establishing a linear model based on the discarded key points and the filtered key points, the method further includes:

[0019] Obtain standard keypoint data to obtain a standard dataset;

[0020] Normalize all data in the standard dataset to obtain a normalized dataset;

[0021] Based on the linear model and the normalized dataset, a preset parameter solution operation is performed so that the target formula can be obtained after calculating the unknown parameters in the linear model.

[0022] Optionally, the step of using the linear model to perform regression prediction on the discarded key points to obtain the regressed key points includes:

[0023] Based on the target formula and the discarded key points, the target key points are calculated to obtain the target key points;

[0024] The coordinate scale corresponding to the coordinates of the target key points is changed to the original coordinate scale to obtain the coordinates of the corresponding regressed key points.

[0025] Optionally, after correcting the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points, the method further includes:

[0026] Determine whether the regression coordinate values ​​corresponding to the key points after regression exceed the preset pixel range;

[0027] If the regression coordinate value exceeds the preset pixel range, the coordinates of the corrected key point are corrected a second time by using a preset centering correction rule to obtain the centered coordinates.

[0028] The confidence level of the discarded key points is corrected using a pre-set confidence correction formula to obtain the target confidence level.

[0029] Optionally, the analysis and identification operation based on the corrected key points includes:

[0030] The filtered key points and the corrected key points are combined to form a target key point set;

[0031] Generate a target matrix based on the coordinates corresponding to the target key point set and the confidence level;

[0032] The target matrix is ​​input into a preset recognition network for training, so that the trained recognition network can be used to perform corresponding analysis and recognition operations on the video.

[0033] Secondly, this application discloses a behavior recognition device, comprising:

[0034] The key point detection module is used to detect key points in the target video using a preset detection method.

[0035] The key point filtering module is used to filter the key points based on preset filtering rules to obtain discarded key points and filtered key points.

[0036] The model building module is used to build a linear model based on the discarded key points and the filtered key points.

[0037] The key point regression module is used to perform regression prediction on the discarded key points using the linear model to obtain the regressed key points.

[0038] The key point correction module is used to correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points.

[0039] The analysis and recognition module is used to perform corresponding analysis and recognition operations based on the corrected key points.

[0040] Thirdly, this application discloses an electronic device, including:

[0041] Memory, used to store computer programs;

[0042] A processor is configured to execute the computer program to implement the steps of the behavior recognition method disclosed above.

[0043] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the behavior recognition method disclosed above.

[0044] As can be seen, this application provides a behavior recognition method, including: detecting a target video using a preset detection method to obtain key points; filtering the key points based on preset filtering rules to obtain discarded key points and filtered key points; establishing a linear model based on the discarded key points and the filtered key points, and using the linear model to perform regression prediction on the discarded key points to obtain regressed key points; correcting the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; and performing corresponding analysis and recognition operations based on the corrected key points. Therefore, this application obtains discarded key points through filtering, establishes a linear model based on the discarded key points to correct them, that is, obtains accurate corrected key points corresponding to inaccurately detected discarded key points, and then performs analysis and recognition operations based on the corrected key points, thereby improving the accuracy and robustness of behavior recognition. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart of a behavior recognition method disclosed in this application;

[0047] Figure 2 This is a flowchart of a specific behavior recognition method disclosed in this application;

[0048] Figure 3 This is a flowchart of a specific behavior recognition method disclosed in this application;

[0049] Figure 4 This is a flowchart of a specific behavior recognition method disclosed in this application;

[0050] Figure 5 A schematic diagram of the behavior recognition device provided in this application;

[0051] Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Currently, the accuracy of the second-stage behavior classification network heavily relies on the first-stage feature extraction network. When the key points extracted by the feature extraction network are inaccurate or incomplete, the behavior classification network often fails to accurately recognize human actions. However, real-world scenarios often involve incomplete or occluded images, and the entire human action can be affected by target occlusion or the poor robustness of the feature extraction network to complex scenes, leading to inaccurate feature point extraction. Therefore, this application provides a behavior recognition method that can improve the accuracy and robustness of behavior recognition.

[0054] This invention discloses a behavior recognition method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0055] Step S11: Use a preset detection method to detect the target video to obtain key points.

[0056] In this embodiment, a preset detection method is used to detect key points in the target video. Specifically, the preset detection method is used to detect each frame of the target video to obtain a preset number of key points, the coordinates of the key points, and the confidence level of the key points in each frame. The preset detection method can employ commonly used industry-standard methods. In practical applications, multi-user key point detection algorithms are used, including bottom-up methods such as OpenPose and AlphaPose, or top-down methods such as Faster R-CNN (object detection algorithm) plus HRNet (neural network). By inputting each frame of the video frame sequence into the algorithm, the preset number of key point coordinates and their corresponding confidence levels for each target can be regressed. For example, the coordinates of 17 key points for each portrait and the confidence level of each key point can be regressed. The correspondence between the key point number and the position can be set as follows: 1-nose, 2-left eye, 3-right eye, 4-left ear, 5-right ear, 6-left shoulder, 7-right shoulder, 8-left elbow, 9-right elbow, 10-left wrist, 11-right wrist, 12-left hip, 13-right hip, 14-left knee, 15-right knee, 16-left ankle, 17-right ankle.

[0057] It should be noted that, in addition to AlphaPose, OpenPose, and Faster R-CNN plus HRNet, there are other keypoint extraction networks that can complete the entire technical solution process.

[0058] Step S12: Filter the key points based on preset filtering rules to obtain discarded key points and filtered key points.

[0059] In this embodiment, after detecting key points in the target video using a preset detection method, the key points are filtered based on preset filtering rules to obtain discarded key points and filtered key points. It is understood that among the detected key points, some are accurately detected, while others are inaccurately detected. Therefore, a filtering operation is needed to determine the inaccurately detected key points. The discarded key points are those that do not meet the current filtering conditions, and the filtered key points are the key points that are retained and meet the current filtering conditions. In one specific implementation, filtering is performed based on the confidence level of the detected key points.

[0060] Step S13: Establish a linear model based on the discarded key points and the filtered key points, and use the linear model to perform regression prediction on the discarded key points to obtain the regressed key points.

[0061] In this embodiment, the key points are filtered based on preset filtering rules to obtain discarded key points and filtered key points. A linear model is then established based on the discarded and filtered key points, and this linear model is used to perform regression prediction on the discarded key points to obtain the regressed key points. It can be understood that the linear model assumes a linear relationship between each discarded key point and the known set of key points. Therefore, a linear model is established based on each discarded and filtered key point, and then the linear model is used to perform regression prediction on the discarded key points to obtain the regressed key points. That is, it predicts the accurate regressed key points corresponding to the inaccurately detected discarded key points.

[0062] Step S14: Correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points.

[0063] In this embodiment, the linear model is used to perform regression prediction on the discarded keypoints. After obtaining the regressed keypoints, the discarded keypoints are corrected according to a preset correction rule and the regressed keypoints to obtain corrected keypoints. It is understood that the regressed keypoints obtained by using the linear model to perform regression prediction on the discarded keypoints are only calculated preset keypoints, not actual existing keypoints. Therefore, the coordinates of the new regressed keypoints need to replace the original coordinates of the discarded keypoints to obtain replaced coordinates. Then, the discarded keypoints are corrected according to the preset correction rule and the replaced coordinates to obtain corrected keypoints. The coordinates of the corrected keypoints are weighted coordinates obtained by summing the original coordinates of the discarded keypoints and the predicted regressed keypoint coordinates. It should be noted that the corrected keypoints are the keypoints that meet the filtering conditions in this case.

[0064] Step S15: Perform corresponding analysis and identification operations based on the corrected key points.

[0065] In this embodiment, after correcting the discarded keypoints according to preset correction rules and the regressed keypoints, corresponding analysis and recognition operations are performed based on the corrected keypoints. It can be understood that the corrected keypoints and the original filtered keypoints form a target keypoint set, which is the set of keypoints obtained under accurate detection conditions. Then, the trained recognition network is used to perform corresponding analysis and recognition operations on the target keypoint set obtained in the video.

[0066] Understandably, when using keypoints for behavior recognition, if the confidence level of the reconstructed human keypoints is low, the coordinates of these keypoints are often inaccurate, leading to a decrease in behavior recognition accuracy. Real-world application scenarios are diverse, making inaccurate keypoints a common occurrence. Therefore, if... Figure 2 As shown, firstly, low-confidence keypoints (i.e., discarded keypoints) are filtered. The detected human keypoints are filtered based on their confidence levels using a threshold. Keypoints with low confidence are considered as discarded keypoints that need correction. Then, linear regression modeling is performed on the discarded keypoints. The model parameters are solved using known, accurate, and complete human keypoint data to regress the positions of low-confidence keypoints. Finally, all the regressed keypoints are normalized and corrected to obtain accurate and complete human keypoint data. Ultimately, this improves the accuracy of the human behavior recognition model when the human body is occluded or incomplete. In other words, it improves the accuracy and robustness of behavior recognition even when keypoint detection is inaccurate.

[0067] As can be seen, this application provides a behavior recognition method, including: detecting target videos using a preset detection method to obtain key points; filtering the key points based on preset filtering rules to obtain discarded key points and filtered key points; establishing a linear model based on the discarded key points and the filtered key points, and using the linear model to perform regression prediction on the discarded key points to obtain regressed key points; correcting the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; and performing corresponding analysis and recognition operations based on the corrected key points. Therefore, this application obtains discarded key points through filtering, establishes a linear model based on the discarded key points to correct them, that is, obtains accurate corrected key points corresponding to inaccurately detected discarded key points, and then performs analysis and recognition operations based on the corrected key points, thereby improving the accuracy and robustness of behavior recognition even when key point detection is inaccurate.

[0068] See Figure 3 As shown, this embodiment of the invention discloses a behavior recognition method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0069] Step S21: Use a preset detection method to detect the target video to obtain key points.

[0070] Step S22: Compare the confidence scores of all the key points with a preset confidence threshold to obtain the key points to be discarded and the key points after filtering.

[0071] In this embodiment, the confidence scores of all keypoints are compared with a preset confidence threshold to obtain discarded keypoints and filtered keypoints. Keypoints with confidence scores greater than the preset confidence threshold are determined as filtered keypoints; keypoints with confidence scores less than the preset confidence threshold are determined as discarded keypoints. It can be understood that keypoints detected by the keypoint detection model are filtered according to their confidence scores; those below the confidence threshold are filtered out. In practical applications, for example, the filtering threshold is set to 0.6. The filtering threshold can be adjusted according to the specific scenario, and the keypoints that are filtered out are determined as discarded keypoints.

[0072] Specifically, the key points before filtering are:

[0073] P A ={p1,p2,…,p 17}, (Formula 1);

[0074] Keypoints below the threshold (i.e., discarded keypoints) are:

[0075] P L ={p l1 ,pl2 ,…,p lM}, (Formula 2);

[0076] Where M < 8, meaning that at most 8 values ​​can be discarded, and P L ∈P A ;

[0077] Key points after filtering:

[0078] P R ={p r1 ,p r2 ,…,p rN}, (Formula 3);

[0079] Where P R =P A -P L N = 17 - M.

[0080] Step S23: Establish a linear model based on the discarded key points and the filtered key points.

[0081] In this embodiment, a linear model is established based on the discarded key points and the filtered key points. The model assumes that there is a linear relationship between each discarded key point and the known set of key points, and a linear model is established for each discarded key point and the filtered key points.

[0082] The specific formula is as follows:

[0083]

[0084]

[0085] …

[0086]

[0087] Where p R =[p r1 ,p r2 ,…,p rN ] T p R Let x1, x2, ..., x be an N-dimensional column vector. M ∈R N R N It is an N-dimensional real vector space.

[0088] Based on the above relationships, the matrix can be obtained:

[0089] p L =Xp R (Formula 7); p L =[p l1 ,p l2,…,p lM ] T p L Let X be an M-dimensional column vector; X = [x1, x2, ..., x...] M ] T X is an M*N matrix, and T represents the transpose.

[0090] Step S24: Obtain standard keypoint data to obtain a standard dataset.

[0091] In this embodiment, standard keypoint data is acquired to obtain a standard dataset. It is understood that after the linear model is established, the aforementioned parameter matrix is ​​needed to solve for the parameters in each linear model formula, i.e., to obtain X in Formula 7. First, the parameters in Formula 4 are solved, and the solution process is as follows: data collection and filtering; data normalization and partitioning; and parameter solving. Specifically, the data collection and filtering involves collecting complete and accurate human keypoint data. Keypoint data from public datasets such as COCO (Common Object in Context) can be used, or a keypoint extraction network can be used. Simple, complete, and normal human data from various scenarios are selected and input into the network. After manual screening, data with accurate coordinate extraction and containing all keypoints are selected. To make the parameters more robust, multiple scenario data can be selected, and the amount of data should be as large as possible while meeting hardware limitations. Finally, the dataset D0∈R is obtained. S×17 , where S is the number of data records used to extract parameters after filtering, generally S>1000.

[0092] Step S25: Normalize all data in the standard dataset to obtain a normalized dataset.

[0093] In this embodiment, all data in the standard dataset is normalized to obtain a normalized dataset. It can be understood that the data normalization and partitioning operations are as follows:

[0094] Perform min-max normalization on the keypoint coordinates for all d0∈D0 and d0∈R. 17 :

[0095]

[0096] Finally, we obtain the normalized data D∈R S×17 .

[0097] Extract the location d of the unknown key point in Formula 4 from the dataset. l1 ∈R S×1 Extract the known key point position D from Formula 4. R ∈R S×N , where dl1 and D R They all belong to the column space of D.

[0098] Step S26: Perform a preset parameter solution operation based on the linear model and the normalized dataset, so as to obtain the target formula after calculating the unknown parameters in the linear model.

[0099] In this embodiment, a preset parameter solution operation is performed based on the linear model and the normalized dataset to obtain the target formula after calculating the unknown parameters in the linear model. Specifically, according to modeling formula 4 and the acquired data, the following exists:

[0100]

[0101] Assume the loss function is L2 squared:

[0102]

[0103] Substituting Equation 9 into Equation 10 and expanding the loss function, the final simplified result is:

[0104]

[0105] L is a convex function with respect to x1, and the optimal solution can be obtained directly by taking its derivative.

[0106] make We can obtain:

[0107]

[0108] Following the above procedure, other parameters can be calculated:

[0109]

[0110]

[0111] This yields the parameter matrix X, which is the final known linear model.

[0112] Step S27: Calculate the target key points based on the target formula and the discarded key points.

[0113] In this embodiment, the target key points are obtained based on the target formula and the discarded key points. Specifically, after obtaining the solved parameters, the target key points corresponding to the discarded key points generated by threshold filtering can be directly calculated, and M target key points P are calculated according to Formula 7. L ′, thus completing the linear regression prediction from known points to unknown points.

[0114] Step S28: Change the coordinate scale corresponding to the coordinates of the target key points to the original coordinate scale to obtain the coordinates of the corresponding regressed key points.

[0115] In this embodiment, after obtaining the target key points based on the target formula and the discarded key point calculation, the coordinate scale corresponding to the coordinates of the target key points is changed to the original coordinate scale to obtain the coordinates of the corresponding regressed key points. Specifically, the regressed key point coordinates are transformed back to the original coordinate scale:

[0116]

[0117] Where P L It is the key point that is being filtered. These are the key points after regression following the scaling transformation.

[0118] Step S29: Correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points.

[0119] In this embodiment, the discarded key points are corrected according to preset correction rules and the regressed key points to obtain corrected key points. After scaling transformation, the regressed coordinates are obtained. The coordinates have been transformed back to the original coordinates P. L At this scale. In practical applications, to obtain more accurate predictions, a prediction correction formula can be used to correct low-confidence keypoints. This correction is achieved by using the coordinates regressed by the keypoint detection network and the coordinates predicted by linear regression.

[0120]

[0121] in P represents the transformed linear regression prediction coordinates. L The low-confidence coordinate values ​​are filtered out. λ is an adjustable real parameter, ranging from 0 to 1. The larger the λ, the greater the weight of the linear regression prediction value in the result.

[0122] Understandably, the coordinate correction process involves weighting, normalizing, and centering the low-confidence coordinates and the modeling regression coordinates to obtain the corrected keypoint coordinates. The confidence correction method involves linear interpolation using weighting parameters and a filtering threshold to obtain the corrected keypoint confidence value.

[0123] Step S210: Perform corresponding analysis and identification operations based on the corrected key points.

[0124] For details regarding steps S21 and S210, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0125] As can be seen, this embodiment of the application detects key points in a target video using a preset detection method; compares the confidence scores of all key points with a preset confidence threshold to obtain discarded key points and filtered key points; establishes a linear model based on the discarded key points and the filtered key points; acquires standard key point data to obtain a standard dataset; normalizes all data in the standard dataset to obtain a normalized dataset; performs a preset parameter solving operation based on the linear model and the normalized dataset to obtain a target formula after calculating the unknown parameters in the linear model; and calculates based on the target formula and the discarded key points. The process involves obtaining target key points; changing the coordinate scale corresponding to the target key points to the original coordinate scale to obtain the coordinates of the corresponding regressed key points; correcting the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; in this embodiment, the linear relationship between key points is utilized, and through linear modeling and predicted coordinate point regression, the coordinates of key points with low confidence are more accurately corrected. Then, based on the corrected key points, corresponding analysis and recognition operations are performed, improving the accuracy and robustness of behavior recognition. Through preset correction rules, the behavior recognition algorithm based on human key points has more accurate input data and obtains better indicators.

[0126] See Figure 4 As shown, this embodiment of the invention discloses a behavior recognition method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0127] Step S31: Use a preset detection method to detect the target video to obtain key points.

[0128] Step S32: Filter the key points based on preset filtering rules to obtain discarded key points and filtered key points.

[0129] Step S33: Establish a linear model based on the discarded key points and the filtered key points, and use the linear model to perform regression prediction on the discarded key points to obtain the regressed key points.

[0130] Step S34: Correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points.

[0131] Step S35: Determine whether the regression coordinate values ​​corresponding to the key points after regression exceed the preset pixel range.

[0132] In this embodiment, the discarded keypoints are corrected according to the preset correction rules and the regressed keypoints. After obtaining the corrected keypoints, it is determined whether the regression coordinate values ​​corresponding to the regressed keypoints exceed the preset pixel range. Since the coordinate values ​​calculated by the scale transformation of Formula 15 may exceed the pixel range, this modeling method can regress and calculate low-confidence coordinate values ​​outside the pixels, thereby completing the regression of keypoints outside the image.

[0133] Step S36: If the regression coordinate value exceeds the preset pixel range, then the coordinates of the corrected key points are corrected a second time by using a preset centering correction rule to obtain the centered coordinates.

[0134] In this embodiment, if the regressed coordinate values ​​exceed the preset pixel range, the coordinates of the corrected key points are subjected to secondary coordinate correction using a preset centering correction rule to obtain the centered coordinates. Similarly, There may also be coordinate points outside the image. To maintain relative coordinate relationships, these can be directly used as input for subsequent algorithms. To perform centering correction, all pixels are repositioned to the center of the image. The specific formula is as follows:

[0135]

[0136] Where λ center This refers to the proportion of the image center to the total pixel count, P. max This refers to the maximum pixel value. It is the coordinate before centralization. These are the coordinates after centralization.

[0137] Step S37: Correct the confidence of the discarded key points using a pre-set confidence correction formula to obtain the target confidence.

[0138] In this embodiment, a pre-set confidence correction formula is used to correct the confidence of the corrected keypoints to obtain the target confidence level. It is understood that, in addition to correcting the coordinate regression for low-confidence points, confidence correction is also needed for these keypoints. Since the corrected keypoint coordinates incorporate known keypoint coordinate information, these corrected keypoints have higher confidence. The confidence correction formula used in practice is as follows:

[0139]

[0140] Where θ th This is the keypoint confidence filtering threshold, and λ is the coordinate correction parameter in Formula 16. It is the confidence level of the corrected coordinates, which obviously has The larger the value of λ, the greater the weight of the regression coordinates, and the higher the confidence value of the correction.

[0141] Step S38: Perform corresponding analysis and identification operations based on the corrected key points.

[0142] In this embodiment, corresponding analysis and identification operations are performed based on the corrected key points. (Regression to corrected key points) And the remaining key points P after threshold filtering R All human body key points were reconstructed. It should be noted that the filtered key points and the corrected key points are combined to form a target key point set. Then, a target matrix is ​​generated based on the coordinates and confidence scores corresponding to the target key point set. Finally, the target matrix is ​​input into a preset recognition network for training, so that the trained recognition network can be used to perform corresponding analysis and recognition operations on the video.

[0143] Specifically, all the reconstructed human keypoints are labeled according to behavior type and fed into the recognition network for training. For example, in practical applications, the recognition network POSEC3D can be used, or graph networks or other basic network structures can be employed. Taking POSEC3D as an example, 17 keypoints need to be mapped into a 17-channel heatmap, and 48 consecutive frames are extracted as network input. POSEC3D will fuse the spatiotemporal information of the keypoints. When generating the heatmap, each channel of the input data is generated by a keypoint coordinate P and a keypoint confidence θ. A two-dimensional Gaussian sparse matrix is ​​generated with coordinate P as the center point and confidence θ as the peak value. For high-confidence keypoint coordinates P and confidence θ, the output value of the keypoint regression network is used, while for low-confidence keypoint coordinates and confidence, a corrected value is used. and The initial learning rate was set to 0.01, and the parameters were optimized using the stochastic gradient descent algorithm. The training was conducted on all data for 240 epochs, and the parameters could be adjusted according to the data size and scenario.

[0144] During inference, the pre-trained parameters of the recognition network are loaded. The parameter matrix X for the correction stage can also be calculated offline even with limited computational resources. Then, during the inference stage, formulas 7 and 15 are used directly to complete the regression correction of the keypoint coordinates. In a real-world scene with significant occlusion, the results of different schemes under different λ values ​​are shown in Table 1:

[0145] Table 1

[0146]

[0147] Adding a regression correction algorithm can significantly improve the recall of keypoint behavior recognition algorithms while maintaining similar accuracy. The correction parameter λ should not be set too high or too low; different values ​​can be selected based on different scenarios to significantly improve model metrics. It should be noted that, in addition to POSEC3D mentioned in the paper, there are other keypoint-based behavior recognition networks, including but not limited to ST-GCN, that can also complete the entire technical solution process.

[0148] For details regarding steps S31 to S34, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0149] As can be seen, this application embodiment utilizes a preset detection method to detect target videos to obtain key points; filters the key points based on preset filtering rules to obtain discarded key points and filtered key points; establishes a linear model based on the discarded key points and the filtered key points, and uses the linear model to perform regression prediction on the discarded key points to obtain regressed key points; corrects the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; determines whether the regression coordinate values ​​corresponding to the regressed key points exceed a preset pixel range; if the regression coordinate values ​​exceed the preset pixel range, performs secondary coordinate correction on the coordinates of the corrected key points using a preset centering correction rule to obtain centered coordinates; corrects the confidence of the discarded key points using a preset confidence correction formula to obtain target confidence; and performs corresponding analysis and recognition operations based on the corrected key points, thereby improving the accuracy and robustness of behavior recognition.

[0150] See Figure 5 As shown in the embodiments, this application also discloses a behavior recognition device, including:

[0151] The key point detection module 11 is used to detect key points in the target video using a preset detection method.

[0152] The key point filtering module 12 is used to filter the key points based on preset filtering rules to obtain discarded key points and filtered key points.

[0153] Model building module 13 is used to build a linear model based on the discarded key points and the filtered key points;

[0154] Key point regression module 14 is used to perform regression prediction on the discarded key points using the linear model to obtain the regressed key points;

[0155] The key point correction module 15 is used to correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points.

[0156] The analysis and identification module 16 is used to perform corresponding analysis and identification operations based on the corrected key points.

[0157] As can be seen, this application includes: detecting key points in a target video using a preset detection method; filtering the key points based on preset filtering rules to obtain discarded key points and filtered key points; establishing a linear model based on the discarded key points and the filtered key points, and using the linear model to perform regression prediction on the discarded key points to obtain regressed key points; correcting the discarded key points according to preset correction rules and the regressed key points to obtain corrected key points; and performing corresponding analysis and recognition operations based on the corrected key points. Therefore, this application obtains discarded key points through filtering, establishes a linear model based on the discarded key points to correct them, that is, obtains accurate corrected key points corresponding to inaccurately detected discarded key points, and then performs analysis and recognition operations based on the corrected key points, thereby improving the accuracy and robustness of behavior recognition even when key point detection is inaccurate.

[0158] In some specific embodiments, the key point detection module 11 specifically includes:

[0159] The key point detection unit is used to detect each frame of the target video using a preset detection method to obtain a preset number of key points, the coordinates of the key points, and the confidence level of the key points on each video frame.

[0160] In some specific embodiments, the key point filtering module 12 specifically includes:

[0161] A confidence comparison unit is used to compare the confidence of all the key points with a preset confidence threshold.

[0162] The filtered key point determination unit is used to determine the key points corresponding to the confidence levels of the key points that are greater than the preset confidence threshold as the filtered key points.

[0163] The discard key point determination unit is used to determine the key points corresponding to the confidence levels of the key points that are less than the preset confidence threshold as the discard key points.

[0164] In some specific embodiments, the model building module 13 specifically includes:

[0165] A linear model building unit is used to build a linear model based on the discarded key points and the filtered key points.

[0166] The standard keypoint data acquisition unit is used to acquire standard keypoint data to obtain a standard dataset;

[0167] The normalization unit is used to perform a normalization operation on all data in the standard dataset to obtain a normalized dataset.

[0168] The parameter solving unit is used to perform a preset parameter solving operation based on the linear model and the normalized dataset, so as to obtain the target formula after calculating the unknown parameters in the linear model.

[0169] In some specific embodiments, the key point regression module 14 specifically includes:

[0170] The target key point acquisition unit is used to calculate the target key points based on the target formula and the discarded key points.

[0171] The coordinate scale changing unit is used to change the coordinate scale corresponding to the coordinates of the target key point to the original coordinate scale in order to obtain the coordinates of the corresponding regressed key point.

[0172] In some specific embodiments, the key point correction module 15 specifically includes:

[0173] A pixel range determination unit is used to determine whether the regression coordinate values ​​corresponding to the key points after regression exceed a preset pixel range.

[0174] The centralization correction unit is used to perform secondary coordinate correction on the coordinates of the corrected key points according to the preset centralization correction rule if the regression coordinate value exceeds the preset pixel range, so as to obtain the centralized coordinates.

[0175] The confidence correction unit is used to correct the confidence of the discarded key points using a preset confidence correction formula to obtain the target confidence.

[0176] In some specific embodiments, the analysis and identification module 16 specifically includes:

[0177] An analysis and identification unit is used to perform corresponding analysis and identification operations based on the corrected key points;

[0178] The target key point set composition unit is used to compose the filtered key points and the corrected key points into a target key point set.

[0179] A matrix generation unit is used to generate a target matrix based on the coordinates corresponding to the target key point set and the confidence level.

[0180] The training unit is used to input the target matrix into a preset recognition network for training, so that the trained recognition network can be used to perform corresponding analysis and recognition operations on the video.

[0181] Furthermore, embodiments of this application also provide an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0182] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the behavior recognition method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0183] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0184] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0185] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the behavior recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0186] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the behavior recognition method steps disclosed in any of the foregoing embodiments.

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0188] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0189] The above provides a detailed description of the behavior recognition method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A behavior recognition method, characterized in that, include: The target video is detected using a preset detection method to obtain key points; The key points are filtered based on preset filtering rules to obtain discarded key points and filtered key points; wherein, the filtered key points are the key points that conform to the preset filtering rules. A linear model is established based on the discarded key points and the filtered key points, and the linear model is used to perform regression prediction on the discarded key points to obtain the regressed key points. The discarded key points are corrected according to the preset correction rules and the key points after regression to obtain the corrected key points; Based on the corrected key points, corresponding analysis and identification operations are performed.

2. The behavior recognition method according to claim 1, characterized in that, The step of using a preset detection method to detect key points in the target video includes: A preset detection method is used to detect each frame of the target video to obtain a preset number of key points, the coordinates of the key points, and the confidence level of the key points in each frame.

3. The behavior recognition method according to claim 2, characterized in that, The filtering of the key points based on preset filtering rules to obtain discarded key points and filtered key points includes: Compare the confidence scores of all the key points with a preset confidence threshold; The key points whose confidence level is greater than the preset confidence threshold are determined as the filtered key points. The key points whose confidence level is less than the preset confidence threshold are identified as the discarded key points.

4. The behavior recognition method according to claim 2, characterized in that, After establishing a linear model based on the discarded key points and the filtered key points, the method further includes: Obtain standard keypoint data to obtain a standard dataset; Normalize all data in the standard dataset to obtain a normalized dataset; Based on the linear model and the normalized dataset, a preset parameter solution operation is performed so that the target formula can be obtained after calculating the unknown parameters in the linear model.

5. The behavior recognition method according to claim 4, characterized in that, The step of using the linear model to perform regression prediction on the discarded key points to obtain the regressed key points includes: Based on the target formula and the discarded key points, the target key points are calculated to obtain the target key points; The coordinate scale corresponding to the coordinates of the target key points is changed to the original coordinate scale to obtain the coordinates of the corresponding regressed key points.

6. The behavior recognition method according to claim 5, characterized in that, After correcting the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points, the method further includes: Determine whether the regression coordinate values ​​corresponding to the key points after regression exceed the preset pixel range; If the regression coordinate value exceeds the preset pixel range, the coordinates of the corrected key point are corrected a second time by using a preset centering correction rule to obtain the centered coordinates. The confidence level of the discarded key points is corrected using a pre-set confidence correction formula to obtain the target confidence level.

7. The behavior recognition method according to any one of claims 2 to 6, characterized in that, The analysis and identification operation based on the corrected key points includes: The filtered key points and the corrected key points are combined to form a target key point set; Generate a target matrix based on the coordinates corresponding to the target key point set and the confidence level; The target matrix is ​​input into a preset recognition network for training, so that the trained recognition network can be used to perform corresponding analysis and recognition operations on the video.

8. A behavior recognition device, characterized in that, include: The key point detection module is used to detect key points in the target video using a preset detection method. A key point filtering module is used to filter the key points based on preset filtering rules to obtain discarded key points and filtered key points; wherein, the filtered key points are the key points that conform to the preset filtering rules. The model building module is used to build a linear model based on the discarded key points and the filtered key points. The key point regression module is used to perform regression prediction on the discarded key points using the linear model to obtain the regressed key points. The key point correction module is used to correct the discarded key points according to the preset correction rules and the regressed key points to obtain the corrected key points. The analysis and recognition module is used to perform corresponding analysis and recognition operations based on the corrected key points.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the behavior recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the behavior recognition method as described in any one of claims 1 to 7.

Images