A behavior recognition method, device and storage medium
By extracting optical flow direction and amplitude periodicity consistency features, combined with BP neural network and Adaboost algorithm, the shortcomings of deep learning and traditional methods in behavior recognition are solved, and efficient and accurate behavior recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2022-12-16
- Publication Date
- 2026-08-04
AI Technical Summary
Existing deep learning methods cannot effectively describe motion information in time series in behavior analysis, and they suffer from high computational costs and low interpretability. Traditional methods rely on input features, and if the features are insufficient, the recognition accuracy is limited, with high rates of missed detections and false detections.
We employ a feature extraction method based on the periodicity of optical flow direction and the periodicity of optical flow amplitude. Combined with a BP neural network to train a behavior recognition model, we calculate feature values using optical flow direction angle and amplitude value, adjust weights using the Adaboost algorithm, and integrate multiple weak classifiers to improve recognition accuracy.
By calculating the periodic consistency of optical flow features, the accuracy of behavior recognition is improved, and the false negative and false positive rates are reduced, making it suitable for real-time video processing.
Smart Images
Figure CN116912540B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and more particularly to a behavior recognition method, apparatus, and storage medium. Background Technology
[0002] In related technologies, methods for analyzing behavior can be divided into deep learning methods and traditional learning methods.
[0003] Deep learning is an end-to-end machine learning approach. After raw data is input into the model, the model automatically extracts features, trains itself, and outputs the learning results. Common methods include autoencoders and convolutional neural networks. Deep learning models are primarily built for images and cannot effectively describe motion information with temporal sequences; their learning ability for video data is limited. Therefore, when using deep learning for behavior analysis, further improvements to the network structure are needed, such as building 3D convolutional neural networks or two-stream convolutional neural networks to extract spatiotemporal features. However, deepening the network structure further increases the algorithm's complexity and demands higher computational power. Furthermore, because deep learning methods automatically learn features, they also suffer from low interpretability and the inability to correct learning results.
[0004] Traditional methods for behavior analysis are based on low-level images. These methods utilize image information such as motion trajectories, geometric features, and texture features, and achieve behavior analysis by manually constructing input features. A representative method involves obtaining candidate motion regions using a Gaussian optical flow model, then extracting optical flow histograms from the target regions and using an SVM model for training and classification. Traditional methods primarily rely on optical flow feature information, with common approaches including thresholding and histogram methods. However, the recognition performance of these methods is directly related to the input features. Insufficient input features will result in limited accuracy in fighting behavior recognition, with high rates of missed detections and false detections. Summary of the Invention
[0005] In view of this, the main objective of the present invention is to provide a behavior recognition method, apparatus and storage medium.
[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:
[0007] This invention provides a behavior recognition method, the method comprising:
[0008] Foreground targets are extracted from the target video to determine the motion region of each frame in the target video;
[0009] The motion region is subjected to feature extraction using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude.
[0010] The feature results corresponding to each frame of the image are identified using a preset behavior recognition model to obtain the recognition result; the recognition result indicates whether the target video has the target behavior.
[0011] In the above scheme, the method further includes: training the behavior recognition model; the training of the behavior recognition model includes:
[0012] Obtain a sample set; the sample set includes: at least one data sample, each data sample including: a feature value vector of each frame image and a label corresponding to each frame image;
[0013] Based on the sample set, the BP neural network is trained as a base learner to obtain a first weak classifier.
[0014] The weights of each data sample are adjusted based on a preset algorithm to obtain at least one optimized sample set;
[0015] Train a BP neural network model for each of the at least one optimized sample sets to obtain at least one second weak classifier;
[0016] Based on the classification errors of the first weak classifier and at least one second weak classifier, weights are assigned to the first weak classifier and at least one second weak classifier. The weighted first weak classifier and at least one second weak classifier are then integrated to obtain the behavior recognition model.
[0017] In the above scheme, obtaining the sample set includes:
[0018] Foreground targets are extracted from each sample video to determine the motion region of each frame in the sample video;
[0019] The motion region of the sample is extracted using a preset feature extraction method to obtain the feature results, which are then used as data samples.
[0020] In the above scheme, the preset feature extraction method is used to extract features from the corresponding motion region to obtain feature results, including:
[0021] Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector;
[0022] Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle;
[0023] The optical flow direction period consistency is calculated based on the optical flow direction angle to obtain a first characteristic value; and / or, the optical flow amplitude period consistency is calculated based on the optical flow amplitude value to obtain a second characteristic value;
[0024] The corresponding motion region may include: the motion region extracted from any frame of the target video, and the sample motion region extracted from any frame of the sample video.
[0025] In the above scheme, the step of calculating the periodic consistency of the optical flow direction based on the optical flow direction angle to obtain the first characteristic value includes:
[0026] A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval;
[0027] A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval;
[0028] Calculate the first similarity result between the second matrix and the first matrix for each second image interval;
[0029] The maximum value in the first similarity results corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image.
[0030] In the above scheme, calculating the first similarity result between the second matrix and the first matrix includes:
[0031] The optical flow direction angle set corresponding to the i-th frame image in the second matrix and the optical flow direction angle set corresponding to the i-th frame image in the first matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0032] The average of the X optical flow amplitude similarity values is taken as the first similarity result.
[0033] In the above scheme, the step of calculating the optical flow amplitude periodicity consistency based on the optical flow amplitude value to obtain the second characteristic value includes:
[0034] A third matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame; the third matrix includes: the set of optical flow amplitude values corresponding to each frame in the first image interval.
[0035] A fourth matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the fourth matrix includes: a set of optical flow amplitude values corresponding to each frame image in the second image interval;
[0036] Calculate the second similarity result between the fourth matrix and the third matrix for each second image interval;
[0037] The maximum value among the second similarity results corresponding to the at least one second image interval is determined as the second feature value of the j-th frame image.
[0038] In the above scheme, calculating the second similarity result between the fourth matrix and the third matrix includes:
[0039] The optical flow direction angle set corresponding to the i-th frame image in the fourth matrix and the optical flow direction angle set corresponding to the i-th frame image in the third matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0040] The average of the X optical flow amplitude similarity values is used as the second similarity result.
[0041] In the above scheme, the step of extracting features from the motion region to obtain feature results further includes:
[0042] Extract at least one feature from the motion region, including optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy, to obtain the feature value of at least one feature.
[0043] The method in the above scheme further includes:
[0044] For the x-th feature of the j-th frame image, the feature values of the x-th feature from the jZ-th frame image to the j-th frame image are calculated using at least one of the following: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation, to obtain the feature set of the x-th feature; Z is greater than or equal to 1.
[0045] This invention provides a behavior recognition device, the device comprising:
[0046] The first processing module is used to extract foreground targets from the target video and determine the motion region of each frame in the target video;
[0047] The second processing module is used to extract features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude.
[0048] The recognition module is used to identify the feature results corresponding to each frame of the image using a preset behavior recognition model, and obtain the recognition result; the recognition result indicates whether the target video has the target behavior.
[0049] In the above scheme, the device further includes a training module for training the behavior recognition model; training the behavior recognition model includes:
[0050] Obtain a sample set; the sample set includes: at least one data sample, each data sample including: a feature value vector of each frame image and a label corresponding to each frame image;
[0051] Based on the sample set, the BP neural network is trained as a base learner to obtain a first weak classifier.
[0052] The weights of each data sample are adjusted based on a preset algorithm to obtain at least one optimized sample set;
[0053] Train a BP neural network model for each of the at least one optimized sample sets to obtain at least one second weak classifier;
[0054] Based on the classification errors of the first weak classifier and at least one second weak classifier, weights are assigned to the first weak classifier and at least one second weak classifier. The weighted first weak classifier and at least one second weak classifier are then integrated to obtain the behavior recognition model.
[0055] In the above scheme, the training module is used to extract foreground targets for each sample video and determine the sample motion region of each frame image in the sample video;
[0056] The motion region of the sample is extracted using a preset feature extraction method to obtain the feature results, which are then used as data samples.
[0057] In the above scheme, the preset feature extraction method is used to extract features from the corresponding motion region to obtain feature results, including:
[0058] Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector;
[0059] Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle;
[0060] The optical flow direction period consistency is calculated based on the optical flow direction angle to obtain a first characteristic value; and / or, the optical flow amplitude period consistency is calculated based on the optical flow amplitude value to obtain a second characteristic value;
[0061] The corresponding motion region may include: the motion region extracted from any frame of the target video, and the sample motion region extracted from any frame of the sample video.
[0062] In the above scheme, the step of calculating the periodic consistency of the optical flow direction based on the optical flow direction angle to obtain the first characteristic value includes:
[0063] A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval;
[0064] A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval;
[0065] Calculate the first similarity result between the second matrix and the first matrix for each second image interval;
[0066] The maximum value in the first similarity results corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image.
[0067] In the above scheme, calculating the first similarity result between the second matrix and the first matrix includes:
[0068] The optical flow direction angle set corresponding to the i-th frame image in the second matrix and the optical flow direction angle set corresponding to the i-th frame image in the first matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0069] The average of the X optical flow amplitude similarity values is taken as the first similarity result.
[0070] In the above scheme, the step of calculating the optical flow amplitude periodicity consistency based on the optical flow amplitude value to obtain the second characteristic value includes:
[0071] A third matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame; the third matrix includes: the set of optical flow amplitude values corresponding to each frame in the first image interval.
[0072] A fourth matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the fourth matrix includes: a set of optical flow amplitude values corresponding to each frame image in the second image interval;
[0073] Calculate the second similarity result between the fourth matrix and the third matrix for each second image interval;
[0074] The maximum value among the second similarity results corresponding to the at least one second image interval is determined as the second feature value of the j-th frame image.
[0075] In the above scheme, calculating the second similarity result between the fourth matrix and the third matrix includes:
[0076] The optical flow direction angle set corresponding to the i-th frame image in the fourth matrix and the optical flow direction angle set corresponding to the i-th frame image in the third matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0077] The average of the X optical flow amplitude similarity values is used as the second similarity result.
[0078] In the above scheme, the second processing module is further used to extract at least one feature from the optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy of the moving region, and obtain the feature value of at least one feature.
[0079] In the above scheme, the second processing module is further used to calculate at least one of the following for the x-th feature of the j-th frame image: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation, for the x-th feature of the j-th frame image; Z is greater than or equal to 1.
[0080] This invention provides a behavior recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the methods described above.
[0081] This invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.
[0082] This invention provides a behavior recognition method, apparatus, and storage medium. The method includes: extracting foreground targets from a target video to determine the motion region of each frame in the target video; extracting features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is an optical flow direction periodicity consistency feature value, and the second feature value is an optical flow amplitude periodicity consistency feature value; and the method uses a preset behavior recognition model to identify the feature results corresponding to each frame to obtain a recognition result; the recognition result characterizes whether the target video has a target behavior. Thus, by using the two features of optical flow direction periodicity consistency and optical flow amplitude periodicity consistency, the difference in motion periodicity consistency between target behavior and normal behavior is distinguished, effectively improving the accuracy of the model's recognition results. Attached Figure Description
[0083] Figure 1 This is a flowchart illustrating a behavior recognition method provided in an embodiment of the present invention;
[0084] Figure 2 A flowchart illustrating a behavior recognition method provided in an application embodiment of the present invention;
[0085] Figure 3 This is a schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present invention;
[0086] Figure 4 This is a schematic diagram of another behavior recognition device provided in an embodiment of the present invention. Detailed Implementation
[0087] The present invention will be further described in detail below with reference to the embodiments.
[0088] Figure 1 This is a flowchart illustrating a behavior recognition method provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the method can be applied to a server, and the method includes:
[0089] Step 101: Extract foreground targets from the target video to determine the motion region of each frame in the target video;
[0090] Step 102: Use a preset feature extraction method to extract features from the motion region to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude;
[0091] Step 103: Use a preset behavior recognition model to identify the feature results corresponding to each frame of the image to obtain the recognition result; the recognition result indicates whether the target video has the target behavior;
[0092] The behavior recognition model is trained based on a sample set, and each data sample in the sample set includes at least: a feature value vector and a label; the feature value vector includes at least a first feature value and a second feature value.
[0093] In some embodiments, the method further includes: training the behavior recognition model. Training the behavior recognition model includes:
[0094] Obtain a sample set; the sample set includes: at least one data sample, each data sample including: a feature value vector of each frame image and a label corresponding to each frame image; the label represents normal behavior or target behavior;
[0095] Using a back propagation neural network (BPNN) as the base learner and the feature value vector of each frame of the image as the data sample, a first weak classifier is trained.
[0096] The weights of each data sample are adjusted based on a preset algorithm to obtain at least one optimized sample set;
[0097] Train a BP neural network model for each of the at least one optimized sample sets to obtain at least one second weak classifier;
[0098] Based on the classification errors of the first weak classifier and at least one second weak classifier, weights are assigned to the first weak classifier and at least one second weak classifier. The weighted first weak classifier and at least one second weak classifier are then integrated to obtain the behavior recognition model.
[0099] Here, the preset algorithm is used to adjust the weights of data samples, such as the AdaBoost algorithm. That is, adjusting the weight of each data sample based on the preset algorithm can include: based on the AdaBoost algorithm, decreasing the weight of correctly classified samples and increasing the weight of incorrectly classified samples.
[0100] The method provided in this invention involves multiple weight adjustments, each resulting in a different optimized sample set. Specifically, after training each second weak classifier, the weights of the used optimized sample set are adjusted, and a second weak classifier is trained again based on the adjusted optimized sample set. This process is repeated to obtain multiple second weak classifiers.
[0101] In some embodiments, obtaining the sample set includes:
[0102] Foreground targets are extracted from each sample video to determine the motion region of each frame in the sample video;
[0103] The motion region of the sample is extracted using a preset feature extraction method to obtain the feature results, which are then used as data samples.
[0104] Here, foreground target extraction can be performed using target detection algorithms, such as the YOLOv3 algorithm and the YOLOv5 algorithm.
[0105] For example, in this embodiment of the invention, the YOLOv5 algorithm is used to extract foreground targets from sample videos and target videos. By balancing detection performance and inference speed, the deep learning network structure is optimized to obtain the anchor boxes of foreground moving targets in each frame of the sample and target videos, i.e., the motion region of each frame. This allows for the identification of motion regions in the video, reduces static background interference and subsequent optical flow extraction, and improves model computational efficiency.
[0106] In practical applications, video analysis reveals that normal behaviors such as walking and running typically exhibit periodic repetitive characteristics, while abnormal behaviors such as fighting display chaotic and disordered patterns. This characteristic can be manifested through changes in the direction and amplitude of optical flow. Therefore, this invention proposes two features—periodic consistency of optical flow direction and periodic consistency of optical flow amplitude—to improve the accuracy of behavior recognition.
[0107] Based on this, in some embodiments, the step of using a preset feature extraction method to extract features from the corresponding motion region to obtain feature results includes:
[0108] Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector;
[0109] Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle;
[0110] Based on the optical flow direction angle, the periodicity of the optical flow direction is calculated to obtain the first characteristic value; and / or,
[0111] The second characteristic value is obtained by calculating the periodic consistency of optical flow amplitude based on the optical flow amplitude value.
[0112] The corresponding motion region may include: the motion region extracted from any frame of the target video, and the sample motion region extracted from any frame of the sample video.
[0113] Specifically, optical flow features can be extracted using any algorithm for solving optical flow, such as the Lucas-Kanade algorithm and the Horn-Schunck algorithm. Among them, the Lucas-Kanade algorithm has less computation and higher computational efficiency compared to the Horn-Schunck algorithm. Therefore, the Lucas-Kanade algorithm is preferred here, as it helps to meet the requirements of processing real-time video.
[0114] Suppose we use the Lucas-Kanade algorithm to extract optical flow features, and obtain the optical flow vector calculated at time t0 for pixel i in image I, denoted as (u i0 ,v i0 By converting it to polar coordinates, the amplitude value ρ of the optical flow can be obtained. i0 and optical flow direction angle θ i0 The calculation formulas are as follows:
[0115]
[0116]
[0117] In some embodiments, the step of calculating the optical flow direction period consistency based on the optical flow direction angle to obtain the first characteristic value includes:
[0118] A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval;
[0119] A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval;
[0120] Calculate the first similarity result between the second matrix and the first matrix for each second image interval;
[0121] The maximum value in the first similarity result corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image, namely the optical flow direction periodicity consistency feature value.
[0122] Here, the image interval before the j-th frame image can specifically refer to the interval formed by the Y-frame images before the j-th frame image.
[0123] Specifically, suppose the input data contains P image frames, and the j-th frame (which can also be understood as t) jIf there are a sufficient number of images (greater than Y) before the image at time Y, then the second image interval can refer to the image interval formed by X consecutive images within the Y-frame image interval.
[0124] If the number of images before the j-th frame is less than Y but greater than X, then the second image interval can be an image interval formed by X consecutive frames from the first frame to the (j-1)-th frame.
[0125] In addition, if Y is greater than X, multiple second image intervals can be obtained, such as the image interval formed by the first frame to the Xth frame, the image interval formed by the second frame to the (X+1)th frame, and so on.
[0126] The following is a specific example. Suppose the input data contains image frames P = 300, Y = 60, X = 30, and the j-th frame is the 70th frame. Then the first image interval is [t]. 70 ,t 99 ];
[0127] Within a time span of 60 frames forward from the current image (i.e., the 70th frame), with intervals of 30 frames, at least one second image interval is determined. The determined second image interval may include: [t] 10 ,t 29 ]、[t 11 ,t 30 ]……、[t 39 ,t 68 ]、[t 40 ,t 69 ].
[0128] It should be noted that the value of Y can be set according to the actual scenario, and no specific value is limited here.
[0129] The calculation of the first similarity result between the second matrix and the first matrix includes:
[0130] The optical flow direction angle set corresponding to the i-th frame image in the second matrix and the optical flow direction angle set corresponding to the i-th frame image in the first matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0131] The average of the X optical flow amplitude similarity values is taken as the first similarity result.
[0132] In practical applications, the optical flow amplitude similarity between each frame of the video and other images can be pre-calculated and recorded in the first similarity matrix. When the first similarity result needs to be calculated, the required optical flow amplitude similarity value can be directly extracted from the first similarity matrix, thereby improving computational efficiency.
[0133] The following section details the calculation method for the periodic consistency of optical flow direction.
[0134] Assume the input data contains P image frames; N optical flow vectors are extracted from image I; and the set of optical flow direction angles of the image at time t0 is A0 = [θ]. 10 ,θ 20 ,…θ N0 The set of optical flow direction angles of the image at time t1 is A1 = [θ]. 11 ,θ 21 ,…θ N1 The similarity of the optical flow directions at two moments in,
[0135] The optical flow direction similarity matrix (i.e., the first similarity matrix) between each time step and its forward time step can be calculated in advance. To identify the periodic characteristics of normal behavior.
[0136] In practical applications, similar segments are searched within a time span of 60 frames forward from the current moment, with an interval length of 30 frames (i.e., X is 30). (If the number of forward frames is less than or equal to 30, no calculation is performed; if the number of forward frames is greater than 30 but less than 60, the calculation starts from the first frame). The second image interval [t0, t...] is used. 29 ] and the first image interval [t j ,t j+29 For example, the formula for calculating the similarity of the optical flow directions of the two is: After calculating the similarity with the second matrix of all second image intervals, the maximum value of the first similarity result calculated in all second image intervals is taken as the periodic consistency value of the optical flow direction of the current j-th frame image, denoted as t. j The value at time should be SA j .
[0137] In some embodiments, the step of calculating the optical flow amplitude periodicity consistency based on the optical flow amplitude value to obtain the second characteristic value includes:
[0138] A third matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame; the third matrix includes: the set of optical flow amplitude values corresponding to each frame in the first image interval.
[0139] A fourth matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the fourth matrix includes: a set of optical flow amplitude values corresponding to each frame image in the second image interval;
[0140] Calculate the second similarity result between the fourth matrix and the third matrix for each second image interval;
[0141] The maximum value in the second similarity results corresponding to the at least one second image interval is determined as the second feature value of the j-th frame image, namely the optical flow amplitude periodicity consistency feature value.
[0142] The calculation of the second similarity result between the fourth and third matrices includes:
[0143] The optical flow direction angle set corresponding to the i-th frame image in the fourth matrix and the optical flow direction angle set corresponding to the i-th frame image in the third matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0144] The average of the X optical flow amplitude similarity values is used as the second similarity result.
[0145] The following section details the calculation method for the periodic consistency of optical flow amplitude.
[0146] The calculation method for optical flow amplitude periodicity consistency is similar to that for optical flow direction periodicity consistency. Here, it is assumed that the input data contains P image frames, and the set of optical flow amplitude values at time t0 is denoted as L0 = [ρ]. 10 , ρ 20 , ...ρ N0 Let L1 = [ρ] be the set of optical flow amplitude values of the image at time t1. 11 ,ρ 21 ,...ρ N1 The method for calculating the similarity of optical flow amplitude values corresponding to the images at times t0 and t1 is as follows: in,
[0147] The optical flow amplitude similarity matrix (i.e., the second similarity matrix) between each time step and its preceding time step can be calculated in advance. To identify the periodic characteristics of normal behavior.
[0148] In practical applications, similar segments are searched within a time span of 60 frames forward from the current moment, with an interval length of 30 frames (i.e., X is 30). (If the number of forward frames is less than or equal to 30, no calculation is performed; if the number of forward frames is greater than 30 but less than 60, the calculation starts from the first frame). The second image interval [t0, t...] is used. 29 ] and the first image interval [t j ,t j+29 For example, the formula for calculating the similarity of the optical flow directions of the two is: After calculating the similarity with the fourth matrix of all second image intervals, the maximum value of the calculated second similarity result among all second image intervals is taken as the optical flow amplitude periodicity consistency value of the current j-frame image. Finally, t... j The periodic uniformity of optical flow amplitude at any given time is SL. j .
[0149] In some embodiments, the step of extracting features from the motion region to obtain feature results further includes:
[0150] Extract at least one feature from the motion region, including optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy, to obtain the feature value of at least one feature.
[0151] Thus, based on the two characteristics of optical flow direction periodicity and optical flow amplitude periodicity, the behavior is identified by combining at least one of optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy, in order to distinguish whether it is normal behavior or special behavior (such as fighting behavior).
[0152] In practical applications, to enhance the regularity of features in different scenarios and reduce noise interference, the method of this invention proposes to process the feature values of each frame again before inputting the features into the model, based on the principle of video stream continuity. That is, the feature values of the t-th frame in the video stream are processed again. j-9 Frame to t j The feature values of the frame are integrated and accumulated to obtain the t-th frame. j The feature values of the frame.
[0153] Based on this, in some embodiments, the method further includes:
[0154] For the x-th feature of the j-th frame image, the feature values of the x-th feature from the jZ-th frame image to the j-th frame image are calculated using at least one of the following: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation, to obtain the feature set of the x-th feature; Z is greater than or equal to 1.
[0155] That is, the feature results include: a feature set of at least one feature; at least one feature includes: optical flow direction periodicity consistency feature, optical flow amplitude periodicity consistency feature, optical flow direction entropy, direction entropy based on amplitude weighting, and corner average kinetic energy.
[0156] Each feature set includes: the feature value of the feature, and at least one of the following: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation.
[0157] Here, Z can be set based on requirements, such as Z=10. That is, according to the indicators commonly used in statistics to measure the distribution characteristics of data, the mean, variance, upper quartile, median, lower quartile, and mean absolute deviation of each feature for 10 consecutive frames are selected for calculation, which is the input feature set for each feature.
[0158] Regarding the calculation methods for optical flow directional entropy, amplitude-weighted directional entropy, and corner average kinetic energy, any appropriate method can be used, or the method described below can be used.
[0159] The optical flow histogram is created by dividing the motion direction angle between [0°, 360°) into n equal parts (in the method provided in this embodiment, the value is to be 18), then the size of each interval is... (Take a 20° angle) and record the number of optical flows n falling into each interval after dividing at time t0. i0 ,i=1,2,...,18.
[0160] Optical flow direction entropy: The optical flow direction entropy can be calculated based on the statistical data obtained from each interval. Let E be the optical flow direction entropy at time t0. d0 , in, The greater the entropy of the optical flow direction, the more chaotic and disordered the direction of motion.
[0161] Amplitude-weighted directional entropy: This index considers the amplitude of optical flow in each frame based on the optical flow directional entropy, using an appropriate amplitude of optical flow as the weight for optical flow statistics. Taking time t0 as an example, the statistical value of optical flow amplitude falling into each directional interval is denoted as H. i0 , Where N0 is the optical flow characteristic number at time t0, ρ j0 Let a(F) be the amplitude of the i-th optical flow. j0 Let be the interval into which the i-th optical flow falls, and δ be the Kronecker delta function, where δ is 1 when both are the same and 0 when they are different. For H... i0 Normalization yields Then the amplitude-weighted directional entropy at time t0 is:
[0162] Corner average kinetic energy: Corner average kinetic energy is mainly used to further distinguish irregular behaviors such as talking while walking from target behaviors (such as fighting), thereby compensating for the shortcomings of other indicators in identifying such behaviors. The corner average kinetic energy at time t0 is denoted as E. k0 ,
[0163] The method of this invention improves existing behavior recognition models by using deep learning methods only for foreground target extraction, making the key intermediate result of target recognition verifiable. Then, in the behavior recognition model construction stage, it proposes the periodic consistency features of optical flow motion direction and optical flow motion amplitude, and uses the Adaboost algorithm to jointly improve the recognition effect for normal behavior and target behavior (such as fighting behavior).
[0164] Figure 2 A flowchart illustrating a behavior recognition method provided in an application embodiment of the present invention; as shown below. Figure 2 As shown, the method can identify normal behavior and fighting behavior (an example of a target behavior), and the method includes:
[0165] Step 201: Video data preprocessing;
[0166] Here, step 201 specifically includes:
[0167] Step 2011: Use an object detection algorithm (such as YOLOv5) to extract foreground objects from the input video data and obtain the extraction results. By balancing the detection effect and inference speed, optimize the deep learning network structure to obtain the anchor boxes of foreground moving objects in each frame of the video, i.e., determine the motion region. The purpose of step 2011 is to identify motion regions in the video, reduce static background interference and subsequent optical flow extraction, and improve the model's computational efficiency.
[0168] Step 2012: Extract optical flow features within the identified motion region;
[0169] Here, algorithms such as the Lucas-Kanade algorithm and the Horn-Schunck algorithm can be used to solve optical flow. Compared with the Horn-Schunck algorithm, the Lucas-Kanade algorithm has a smaller computational load and is therefore more efficient, which helps to meet the requirements of real-time video processing. Assuming the Lucas-Kanade algorithm is used, optical flow features are extracted based on the Lucas-Kanade algorithm; the specific method is described in [details omitted]. Figure 1 The method is explained in the diagram and will not be repeated here.
[0170] Step 202: Construct a fighting behavior recognition model;
[0171] Here, step 202 specifically includes:
[0172] Step 2021: Construct the input feature set;
[0173] First, based on the optical flow vectors extracted within the motion region, the periodicity of the optical flow direction and the periodicity of the optical flow amplitude are calculated. Methods for calculating the periodicity of the optical flow direction and the periodicity of the optical flow amplitude have been described in [the original text]. Figure 1 The method described is explained in the text and will not be repeated here. This involves observing video to identify normal behaviors such as walking and running, which typically exhibit periodic repetitive characteristics, while fighting behaviors display chaotic and disorderly behavior. This characteristic can be represented by changes in the direction and amplitude of optical flow. Using the periodic consistency of optical flow direction and amplitude as two features for fighting behavior recognition can improve the accuracy of the recognition.
[0174] Then, the characteristic values of optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy are calculated. The calculation results of optical flow direction periodicity characteristics and optical flow amplitude periodicity characteristics are combined with optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy to jointly identify fighting behavior.
[0175] Finally, based on the statistically commonly used metrics for measuring data distribution characteristics, the mean, variance, upper quartile, median, lower quartile, and mean absolute deviation of each feature for 10 consecutive frames are calculated to form the final input feature set of the model.
[0176] Here, the method of this embodiment of the invention, in order to enhance the regularity of features in different scenarios and reduce noise interference, processes the feature values of each frame again before inputting the features into the model, based on the principle of video stream continuity; that is, it processes the feature values of the t-th frame in the video stream. j-9 Frame to t j The feature values of the frame are integrated and accumulated to obtain the t-th frame. j The feature values of the frame.
[0177] Step 2022: Based on the input feature set, first use a BP neural network as the base learner, and use the feature value vector of each frame as a data sample to train a weak classifier.
[0178] Step 2023: Adjust the weights of each data sample based on the Adaboost algorithm. Decrease the weight of correctly classified samples and increase the weight of misclassified samples. Under the new data sample distribution, retrain the BP neural network model to obtain a second weak classifier. Adjust the sample weights again based on the classification results, and so on to obtain multiple weak classifiers.
[0179] Step 2024: Assign weights to each classifier based on their classification errors and integrate them to obtain a fighting behavior recognition model with strong classification capabilities.
[0180] Step 203: For the video to be identified, foreground targets are extracted to determine the motion region of each frame in the video; features are extracted from the motion region to obtain feature results; the fighting behavior recognition model is used to identify the feature results to obtain the behavior recognition result.
[0181] The feature results include at least one of the following: optical flow direction periodicity consistency feature, optical flow amplitude periodicity consistency feature, optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy. The extraction method is the same as the extraction method used to construct the input feature set described above, and will not be repeated here.
[0182] Figure 3 This is a schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present invention; as shown below. Figure 3 As shown, the device is applied to a server, and the device includes:
[0183] The first processing module is used to extract foreground targets from the target video and determine the motion region of each frame in the target video;
[0184] The second processing module is used to extract features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude.
[0185] The recognition module is used to identify the feature results corresponding to each frame of the image using a preset behavior recognition model, and obtain the recognition result; the recognition result indicates whether the target video has the target behavior.
[0186] In some embodiments, the apparatus further includes a training module for training the behavior recognition model; training the behavior recognition model includes:
[0187] Obtain a sample set; the sample set includes: at least one data sample, each data sample including: a feature value vector of each frame image and a label corresponding to each frame image;
[0188] Based on the sample set, the BP neural network is trained as a base learner to obtain a first weak classifier.
[0189] The weights of each data sample are adjusted based on a preset algorithm to obtain at least one optimized sample set;
[0190] Train a BP neural network model for each of the at least one optimized sample sets to obtain at least one second weak classifier;
[0191] Based on the classification errors of the first weak classifier and at least one second weak classifier, weights are assigned to the first weak classifier and at least one second weak classifier. The weighted first weak classifier and at least one second weak classifier are then integrated to obtain the behavior recognition model.
[0192] In some embodiments, the training module is configured to extract foreground targets for each sample video and determine the sample motion region of each frame in the sample video;
[0193] The motion region of the sample is extracted using a preset feature extraction method to obtain the feature results, which are then used as data samples.
[0194] In some embodiments, the preset feature extraction method is used to extract features from the corresponding motion region to obtain feature results, including:
[0195] Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector;
[0196] Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle;
[0197] The optical flow direction period consistency is calculated based on the optical flow direction angle to obtain a first characteristic value; and / or, the optical flow amplitude period consistency is calculated based on the optical flow amplitude value to obtain a second characteristic value;
[0198] The corresponding motion region may include: the motion region extracted from any frame of the target video, and the sample motion region extracted from any frame of the sample video.
[0199] In some embodiments, the step of calculating the optical flow direction period consistency based on the optical flow direction angle to obtain the first characteristic value includes:
[0200] A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval;
[0201] A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval;
[0202] Calculate the first similarity result between the second matrix and the first matrix for each second image interval;
[0203] The maximum value in the first similarity results corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image.
[0204] In some embodiments, calculating the first similarity result between the second matrix and the first matrix includes:
[0205] The optical flow direction angle set corresponding to the i-th frame image in the second matrix and the optical flow direction angle set corresponding to the i-th frame image in the first matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0206] The average of the X optical flow amplitude similarity values is taken as the first similarity result.
[0207] In some embodiments, the step of calculating the optical flow amplitude periodicity consistency based on the optical flow amplitude value to obtain the second characteristic value includes:
[0208] A third matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame; the third matrix includes: the set of optical flow amplitude values corresponding to each frame in the first image interval.
[0209] A fourth matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the fourth matrix includes: a set of optical flow amplitude values corresponding to each frame image in the second image interval;
[0210] Calculate the second similarity result between the fourth matrix and the third matrix for each second image interval;
[0211] The maximum value among the second similarity results corresponding to the at least one second image interval is determined as the second feature value of the j-th frame image.
[0212] In some embodiments, calculating the second similarity result between the fourth matrix and the third matrix includes:
[0213] The optical flow direction angle set corresponding to the i-th frame image in the fourth matrix and the optical flow direction angle set corresponding to the i-th frame image in the third matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values.
[0214] The average of the X optical flow amplitude similarity values is used as the second similarity result.
[0215] In some embodiments, the second processing module is further configured to extract at least one feature from the motion region, including optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy, to obtain a feature value of at least one feature.
[0216] In some embodiments, the second processing module is further configured to, for the x-th feature of the j-th frame image, calculate at least one of the following for the feature values of the x-th feature from the j-th frame Z images to the j-th frame image: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation, to obtain the feature set of the x-th feature; Z is greater than or equal to 1.
[0217] It should be noted that the behavior recognition device provided in the above embodiments is only illustrated by the division of the above program modules when implementing the corresponding behavior recognition method. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the server can be divided into different program modules to complete all or part of the processing described above. In addition, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.
[0218] Figure 4 This is a schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the behavior recognition device 40 includes: a processor 401 and a memory 402 for storing computer programs that can run on the processor;
[0219] When the processor 401 runs the computer program, it performs the following actions: extracting foreground targets from the target video to determine the motion region of each frame in the target video; extracting features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least: optical flow direction consistency feature values and optical flow amplitude consistency feature values; and using a preset behavior recognition model to identify the feature results corresponding to each frame to obtain a recognition result; the recognition result characterizes whether the target video has a target behavior. Specifically, the behavior recognition device can also perform the following actions: Figure 1 The method shown is the same as Figure 1 The behavior recognition method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0220] In practical applications, the behavior recognition device 40 may further include at least one network interface 403. The various components of the behavior recognition device 40 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4All buses are labeled as bus system 404. The number of processors 401 can be at least one. Network interface 403 is used for wired or wireless communication between the behavior recognition device 40 and other devices.
[0221] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the behavior recognition device 40.
[0222] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 402. Processor 401 reads the information in memory 402 and combines its hardware to complete the steps of the aforementioned method.
[0223] In an exemplary embodiment, the behavior recognition device 40 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.
[0224] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the following actions: extracting foreground targets from a target video to determine the motion region of each frame in the target video; extracting features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least a first feature value and a second feature value; the first feature value is an optical flow direction periodicity consistency feature value, and the second feature value is an optical flow amplitude periodicity consistency feature value; and using a preset behavior recognition model to identify the feature results corresponding to each frame to obtain a recognition result; the recognition result characterizes whether the target video has a target behavior. Specifically, the computer program can also perform the following actions: Figure 1 The method shown is the same as Figure 1 The behavior recognition method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0225] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0226] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0227] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0228] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0229] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0230] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0231] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.
[0232] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A behavior recognition method, characterized in that, The method includes: Foreground targets are extracted from the target video to determine the motion region of each frame in the target video; The motion region is subjected to feature extraction using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude. A preset behavior recognition model is used to identify the feature results corresponding to each frame of the image to obtain a recognition result; the recognition result indicates whether the target video has a target behavior. The method further includes: training the behavior recognition model; The training to obtain the behavior recognition model includes: acquiring a sample set; The acquisition of the sample set includes: Foreground targets are extracted from each sample video to determine the motion region of each frame in the sample video; Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector; Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle; The optical flow direction period consistency is calculated based on the optical flow direction angle to obtain a first feature value; and / or, the optical flow amplitude period consistency is calculated based on the optical flow amplitude value to obtain a second feature value; the first feature value and / or the second feature value are used as data samples; The corresponding motion region may include: a motion region extracted from any frame of the target video, and a sample motion region extracted from any frame of the sample video; A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval; A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval; Calculate the first similarity result between the second matrix and the first matrix for each second image interval; The maximum value in the first similarity results corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image.
2. The method according to claim 1, characterized in that, The training to obtain the behavior recognition model further includes: The sample set includes: at least one data sample, and each data sample includes: a feature vector of each frame image and a label corresponding to each frame image; The first weak classifier is obtained by training the sample set and the BP neural network as the base learner. The weights of each data sample are adjusted based on a preset algorithm to obtain at least one optimized sample set; Train a BP neural network model for each of the at least one optimized sample sets to obtain at least one second weak classifier; Based on the classification errors of the first weak classifier and at least one second weak classifier, weights are assigned to the first weak classifier and at least one second weak classifier. The weighted first weak classifier and at least one second weak classifier are then integrated to obtain the behavior recognition model.
3. The method according to claim 1, characterized in that, Calculating the first similarity result between the second matrix and the first matrix includes: The optical flow direction angle set corresponding to the i-th frame image in the second matrix and the optical flow direction angle set corresponding to the i-th frame image in the first matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values. The average of the X optical flow amplitude similarity values is taken as the first similarity result.
4. The method according to claim 1, characterized in that, The step of calculating the optical flow amplitude periodicity consistency based on the optical flow amplitude value to obtain the second characteristic value includes: A third matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame; the third matrix includes: the set of optical flow amplitude values corresponding to each frame in the first image interval. A fourth matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the fourth matrix includes: a set of optical flow amplitude values corresponding to each frame image in the second image interval; Calculate the second similarity result between the fourth matrix and the third matrix for each second image interval; The maximum value among the second similarity results corresponding to the at least one second image interval is determined as the second feature value of the j-th frame image.
5. The method according to claim 4, characterized in that, Calculate the second similarity result between the fourth matrix and the third matrix, including: The optical flow direction angle set corresponding to the i-th frame image in the fourth matrix and the optical flow direction angle set corresponding to the i-th frame image in the third matrix are compared to calculate the optical flow amplitude similarity, resulting in X optical flow amplitude similarity values. The average of the X optical flow amplitude similarity values is used as the second similarity result.
6. The method according to claim 1, characterized in that, The step of extracting features from the motion region to obtain feature results also includes: Extract at least one feature from the motion region, including optical flow direction entropy, amplitude-weighted direction entropy, and corner average kinetic energy, to obtain the feature value of at least one feature.
7. The method according to claim 1 or 4, characterized in that, The method further includes: For the x-th feature of the j-th frame image, the feature values of the x-th feature from the jZ-th frame image to the j-th frame image are calculated using at least one of the following: mean, variance, upper quartile, median, lower quartile, and mean absolute deviation, to obtain the feature set of the x-th feature; Z is greater than or equal to 1.
8. A behavior recognition device, characterized in that, The device includes: The first processing module is used to extract foreground targets from the target video and determine the motion region of each frame in the target video; The second processing module is used to extract features from the motion region using a preset feature extraction method to obtain feature results; the feature results include at least: a first feature value and a second feature value; the first feature value is the periodic consistency feature value of the optical flow direction, and the second feature value is the periodic consistency feature value of the optical flow amplitude. The recognition mode is used to identify the feature results corresponding to each frame of the image using a preset behavior recognition model, and obtain the recognition result; the recognition result indicates whether the target video has the target behavior. The device is also used to: train the behavior recognition model; The training to obtain the behavior recognition model includes: acquiring a sample set; The acquisition of the sample set includes: Foreground targets are extracted from each sample video to determine the motion region of each frame in the sample video; Optical flow features are extracted within the corresponding motion region to obtain the optical flow vector; Based on the optical flow vector, determine the optical flow amplitude value and the optical flow direction angle; The optical flow direction period consistency is calculated based on the optical flow direction angle to obtain a first feature value; and / or, the optical flow amplitude period consistency is calculated based on the optical flow amplitude value to obtain a second feature value; the first feature value and / or the second feature value are used as data samples; The corresponding motion region may include: a motion region extracted from any frame of the target video, and a sample motion region extracted from any frame of the sample video; A first matrix is determined corresponding to the first image interval, wherein the first image interval is an image interval formed by X consecutive frames starting with the j-th frame image; the first matrix includes: the set of optical flow direction angles corresponding to each frame image in the first image interval; A second matrix is determined for each second image interval in at least one second image interval, wherein the second image interval is an image interval formed by X consecutive frames of images within the image interval preceding the j-th frame image; the second matrix includes: a set of optical flow direction angles corresponding to each frame image in the second image interval; Calculate the first similarity result between the second matrix and the first matrix for each second image interval; The maximum value in the first similarity results corresponding to the at least one second image interval is determined as the first feature value of the j-th frame image.
9. A behavior recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.