Human behavior recognition method and system based on key frame extraction

Abstract

The present invention provides a method and system for human behavior recognition based on key frame extraction, the method comprising: obtaining three-dimensional position information and three-dimensional angle information of each joint point, constructing a position information set and an angle information set of each frame image; constructing a multidimensional data set of each frame image; dividing the image in the target monitoring video into K initial clusters based on the frame sequence, defining a left boundary set and a right boundary set, and iteratively traversing the left boundary set and the right boundary set based on an improved KKF algorithm to obtain K final clusters; obtaining a feature mean set of each final cluster, calculating a feature distance, screening out a minimum feature distance from multiple feature distances under the same final cluster, screening out a key frame image under each final cluster according to the minimum feature distance, and constructing a human behavior recognition model according to the key frame image. The present invention can improve the accuracy of human behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to a human behavior recognition method and system based on key frame extraction. Background Art

[0002] In recent years, with the increasing demand in the field of intelligent security and video surveillance, human action recognition has attracted more and more attention from researchers, especially in real-time monitoring environments. Human action recognition can be regarded as a behavior detection and recognition task, which aims to determine whether a specific behavior has occurred in a video sequence through computer vision technology. Specifically, given a video to be recognized, the system needs to recognize and classify the behaviors that occur in it in real time, and determine whether a specific behavior occurs based on the monitoring video stream. This has important application significance for intelligent monitoring, public safety, abnormal behavior detection, etc.

[0003] With the development of deep learning technology, deeper and deeper neural networks have been applied to human behavior recognition tasks, achieving remarkable recognition results. However, as the network depth increases, the computer load becomes heavier; at the same time, due to the influence of factors such as camera shooting angle, motion blur and data format, human behavior recognition becomes more complicated, increasing the model's misrecognition rate and missed recognition rate. Summary of the invention

[0004] The purpose of the present invention is to provide a human behavior recognition method and system based on key frame extraction, aiming to solve the problems of high false recognition rate and missed recognition rate in traditional technologies.

[0005] In a first aspect, the present invention provides a method for human behavior recognition based on key frame extraction, the method comprising:

[0006] Perform data enhancement on human monitoring video based on speed coefficient and body size coefficient to obtain target monitoring video;

[0007] Constructing a human skeleton model, the human skeleton model including a plurality of joints, obtaining three-dimensional position information and three-dimensional angle information of each joint, and constructing a position information set of each frame of image according to the three-dimensional position information, and constructing an angle information set of each frame of image according to the three-dimensional angle information;

[0008] The position information set and the angle information set under the same frame image are merged and connected to obtain a multi-dimensional data set for each frame image;

[0009] The images in the target monitoring video are divided into K initial clusters based on the frame sequence, and a left boundary set and a right boundary set are defined according to a number of frame images contained in each initial cluster, and the left boundary set and the right boundary set are iteratively traversed based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results;

[0010] The feature mean set of each final cluster is obtained, and the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs is calculated. The minimum feature distance is screened out from multiple feature distances under the same final cluster, and the key frame image under each final cluster is screened out according to the minimum feature distance, and a human behavior recognition model is constructed based on the key frame image.

[0011] In summary, according to the above-mentioned human behavior recognition method based on key frame extraction, by optimizing the input data form, reducing the impact of external interference factors (such as viewing angle, angle change, etc.) on the recognition accuracy, and combining an efficient key frame extraction algorithm for model training, the overall recognition accuracy can be improved. Specifically, by extracting key frames from the video stream, redundant data is reduced, thereby improving processing efficiency and accuracy. The selection of key frames can not only speed up the processing speed of the video, but also effectively capture the important features of human behavior and improve recognition accuracy. This method is particularly suitable for real-time monitoring scenarios. It can improve the accuracy of human behavior recognition while ensuring high efficiency, especially in monitoring applications in complex environments.

[0012] Furthermore, the step of constructing a human skeleton model, wherein the human skeleton model includes a plurality of joints, obtaining three-dimensional position information and three-dimensional angle information of each joint, and constructing a position information set of each frame of image according to the three-dimensional position information, and constructing an angle information set of each frame of image according to the three-dimensional angle information comprises:

[0013] The location information set is constructed according to the following formula:

[0014]

[0015] in, represents the position information set of the i-th frame image, They represent the x-coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image, respectively. They represent the y coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively. They represent the z coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively;

[0016] The angle information set is constructed according to the following formula:

[0017]

[0018] in, Represents the angle information set of the i-th frame image, They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the X plane of the i-th frame image respectively. They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Y plane of the i-th frame image respectively. They respectively represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Z plane of the i-th frame image.

[0019] Furthermore, the step of merging and connecting the position information set and the angle information set under the same frame image to obtain a multidimensional data set corresponding to each frame image includes:

[0020] Get the cube according to the following formula:

[0021]

[0022] in, A multidimensional dataset representing the i-th frame image, represents the maximum value in the corresponding information set, represents the minimum value in the corresponding information set, represents the normalization operation function, , , , , , Represents a multidimensional dataset The 1st, 2nd, 3rd, 45th, 46th, and 90th dimension data in .

[0023] Furthermore, the step of dividing the images in the target monitoring video into K initial clusters based on the frame sequence, defining a left boundary set and a right boundary set according to a number of frame images included in each initial cluster, and iteratively traversing the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results includes:

[0024] Divide all frame images evenly into K initialization intervals to form initial clusters , select the first frame image of each initial cluster to form the left boundary set , select the last frame image of each initial cluster to form the right boundary set ,in, Respectively represent the first, second, and Kth initial clusters, , Respectively represent the first frame image in the 1st, 2nd, and Kth initial clusters, , Respectively represent the last frame image in the 1st, 2nd, and Kth initial clusters;

[0025] In each iterative traversal process, the right boundary set is traversed first, and then the left boundary set is traversed, and this cycle is repeated until the feature mean sets of the newly formed K clusters do not change, and K final clusters are obtained.

[0026] Furthermore, the steps of traversing the right boundary set include:

[0027] Step 1.1, calculate the feature mean set of each initial cluster, let ;

[0028] Step 2.1: Calculate the right boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the next cluster The characteristic distance and ;

[0029] Step 3.1: Determination Is it greater than ;

[0030] Step 4.1: If Greater than , then the image Move to next cluster middle;

[0031] Step 5.1: Update the current cluster and the next cluster The characteristic mean set and , and again calculate the right boundary concentration and the initial cluster The corresponding new image With the current cluster and the next cluster The new feature distance and , until Less than or equal to ;

[0032] Step 6.1: If Less than or equal to , then let , repeat steps 2.1 to 6.1 until ;

[0033] The steps to traverse the left boundary set include:

[0034] Step 1.2, order ,

[0035] Step 2.2: Calculate the left boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the previous cluster The characteristic distance and ;

[0036] Step 3.2, judgment Is it less than ;

[0037] Step 4.2: If Less than , then the image Move to the previous cluster middle;

[0038] Step 5.2: Update the current cluster and the previous cluster The characteristic mean set and , and again calculate the left boundary concentration and the initial cluster The corresponding new image With the current cluster and the previous cluster The new feature distance and , until Greater than or equal to ;

[0039] Step 6.2: If Greater than or equal to , then let Repeat steps 2.2 to 6.2 until .

[0040] Furthermore, the feature mean set is calculated according to the following formula:

[0041]

[0042] in, Representation and Cluster The corresponding feature mean set, Representation Cluster The last frame of the image, Representation Cluster The first frame image in represents the n-th dimension data in the multidimensional dataset corresponding to image t, , , Respectively represent clusters The mean of the 1st, 2nd, 90th, and nth dimension features in .

[0043] Furthermore, the feature distance is calculated according to the following formula:

[0044]

[0045] in, Represents the i-th frame image and cluster The characteristic distance, Represents the n-th dimension of the multidimensional data set of the i-th frame image.

[0046] In a second aspect, the present invention provides a human behavior recognition system based on key frame extraction, the system comprising:

[0047] A data enhancement module is used to perform data enhancement on the human monitoring video based on the speed coefficient and the body size coefficient to obtain the target monitoring video;

[0048] An information set construction module is used to construct a human skeleton model, the human skeleton model includes multiple joints, obtain three-dimensional position information and three-dimensional angle information of each joint, and construct a position information set of each frame of image according to the three-dimensional position information, and construct an angle information set of each frame of image according to the three-dimensional angle information;

[0049] A data connection module is used to merge and connect the position information set and the angle information set under the same frame image to obtain a multi-dimensional data set for each frame image;

[0050] A clustering cluster division module is used to divide the images in the target monitoring video into K initial clusters based on the frame sequence, and define a left boundary set and a right boundary set according to a number of frame images contained in each initial cluster, and iteratively traverse the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results;

[0051] The key frame extraction module is used to obtain the feature mean set of each final cluster, and calculate the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs, screen out the minimum feature distance from multiple feature distances under the same final cluster, and screen out the key frame image under each final cluster according to the minimum feature distance, and build a human behavior recognition model based on the key frame image.

[0052] In a third aspect, the present invention provides a storage medium storing one or more programs, which, when executed by a processor, implement the above-mentioned human behavior recognition method based on key frame extraction.

[0053] In a fourth aspect, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein:

[0054] The memory is used to store computer programs;

[0055] When the processor is used to execute the computer program stored in the memory, the above-mentioned human behavior recognition method based on key frame extraction is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of a human behavior recognition method based on key frame extraction proposed in one embodiment of the present invention;

[0057] Figure 2 A data enhancement flow chart of an example of an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of a human skeleton model according to an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a key frame extraction result of an example of an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of the structure of a human behavior recognition system based on key frame extraction proposed in one embodiment of the present invention.

[0061] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0063] like Figure 1 As shown, an embodiment of the present invention provides a human behavior recognition method based on key frame extraction, the method comprising steps S101 to S105, wherein:

[0064] Step S101: performing data enhancement on a human monitoring video based on a speed coefficient and a body size coefficient to obtain a target monitoring video;

[0065] It should be noted that if Figure 2 As shown in the figure, in this step, the data enhancement method of the video needs to be reasonable, especially in removing the interference of factors such as height, body shape, viewing angle and movement speed on model training. For people of different heights doing the same action, it is necessary to remove the noise of height difference so that the model training is not affected by factors such as height and body shape. For example, when a person with a height of 1.8 meters and 1.6 meters performs the same action, it is expected that the model can understand their same behavioral characteristics without being disturbed by the height difference.

[0066] To achieve this goal, we choose to construct features based on relative coordinates, and perform data enhancement by introducing the body size coefficient (BS) and the speed coefficient (VS). By multiplying all coordinates by the body size coefficient (BS), we can control the body size differences; and by adjusting the speed coefficient (VS), we can control the differences between different individuals at the same movement speed. This processing method can not only effectively remove the influence of height and body shape on recognition, but also enhance the diversity of data at different speeds, thereby improving the generalization ability of the model.

[0067] Specifically, the body size coefficient (BS) varies within a certain range, such as (0.8, 0.85, 0.9, 0.95, 1.0, 1.05, 1.10, 1.15, 1.2). These coefficients are used to enlarge or reduce the human skeleton sequence to control the change of body shape. The speed coefficient (VS) such as (0.8, 0.9, 1.0, 1.1, 1.2, 1.3) controls the speed of movement. By adjusting the speed coefficient, it helps to simulate different movement speeds. By combining the BS and VS coefficients, the size of the dataset will be expanded to 54 times the original size, thereby increasing the diversity of the data, which will help the model learn more comprehensive feature representations during training, thereby improving the accuracy of classification or detection.

[0068] It should be noted that the selection of BS and VS coefficients should be kept within a reasonable range. For example, for a person with a body size of 1.7 meters, if BS=1.5, the height will be magnified to 2.55 meters, which is obviously inappropriate. Similarly, too high a speed coefficient may also make actions characterized by speed, such as walking and running, difficult to accurately identify. BS=(0.8, 0.85, 0.9, 0.95,1.0, 1.05, 1.10, 1.15, 1.2),VS=(0.8, 0.9, 1.0, 1.1, 1.2, 1.3)is adopted.

[0069] Taking VS=1.1 as an example, when the input human monitoring video is 50 frames, we calculate the final number of frames as 50X1.1=55 frames and evenly insert one frame after 10, 20, 30, 40, and 50 to obtain the target monitoring video.

[0070] The data of the inserted frame is obtained by averaging the previous and next frames at each joint point. This method ensures that when changing the frame, the original action will not be deformed and cause excessive noise.

[0071] Step S102: constructing a human skeleton model, the human skeleton model including a plurality of joints, obtaining three-dimensional position information and three-dimensional angle information of each joint, and constructing a position information set of each frame of image according to the three-dimensional position information, and constructing an angle information set of each frame of image according to the three-dimensional angle information;

[0072] It should be pointed out that if Figure 3 As shown, this step defines 15 joint points, selects the position information of 15 three-dimensional joint nodes in each frame, and represents it as the following 45-dimensional feature vector to construct the position information set:

[0073]

[0074] in, represents the position information set of the i-th frame image, They represent the x-coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image, respectively. They represent the y coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively. They respectively represent the z coordinates of the 1st joint point, the nth joint point, and the 15th joint point in the i-th frame image.

[0075] Considering that the frame difference indicates the magnitude of the movement between frames, in order to better identify some human behaviors with smaller movement amplitudes, such as clapping, and efficiently extract the most representative key frames, this step also adds angle features to improve the accuracy of the key frames. Specifically, the angle information of each joint point and their mapping on three planes are selected and represented as a 45-dimensional vector to construct the angle information set:

[0076]

[0077] in, Represents the angle information set of the i-th frame image, They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the X plane of the i-th frame image respectively. They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Y plane of the i-th frame image respectively. They respectively represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Z plane of the i-th frame image.

[0078] Step S103: merging and connecting the position information set and the angle information set under the same frame image to obtain a multi-dimensional data set for each frame image;

[0079] It should be pointed out that in order to unify the range of the angle information set and the position information set, the two information sets need to be merged and connected. Specifically, the multidimensional data set is obtained according to the following formula:

[0080]

[0081] in, A multidimensional dataset representing the i-th frame image, represents the maximum value in the corresponding information set, represents the minimum value in the corresponding information set, represents the normalization operation function, , , , , , Represents a multidimensional dataset The 1st, 2nd, 3rd, 45th, 46th, and 90th dimension data in .

[0082] Step S104: dividing the images in the target monitoring video into K initial clusters based on the frame sequence, and defining a left boundary set and a right boundary set according to a number of frame images included in each initial cluster, and iteratively traversing the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results;

[0083] It is important to note that traditional key frame extraction algorithms based on unsupervised automatic clustering have a shortcoming: they rely on the Euclidean distance of the skeleton nodes for clustering, ignoring the temporal order, resulting in frames that are far apart being clustered into one category, which will destroy the temporal relationship of the action and affect the accuracy of human action recognition (HAR). Although some algorithms take the timing issue into account, there are errors in their uniformity in time scale, which may lead to unnecessary frame deletion or addition, resulting in information loss or redundancy. To solve this problem, this embodiment proposes an improved KKF algorithm, which optimizes the selection of key frames and ensures the validity of the frames and the consistency of the timing.

[0084] The improved KKF algorithm aims to select K key frames that can best represent the motion trend of the entire action. The ultimate goal of the improved KKF algorithm is to cluster frames with the highest spatiotemporal similarity into one category, while frames with significant spatiotemporal differences are divided into different categories. In order to meet the requirements of temporal stability, all frame images are first evenly divided into K initialization intervals to form an initial cluster. , select the first frame image of each initial cluster to form the left boundary set , select the last frame image of each initial cluster to form the right boundary set ,in, Respectively represent the first, second, and Kth initial clusters, , Respectively represent the first frame image in the 1st, 2nd, and Kth initial clusters, , They represent the last frame images in the 1st, 2nd, and Kth initial clusters respectively. This way of dividing the left and right boundaries meets the requirement that the initial clusters have significant differences in space and reduces the time consumption in the iterative process.

[0085] In addition, in each iterative traversal process, the right boundary set is traversed first, and then the left boundary set is traversed, and this cycle is repeated until the feature mean sets of the newly formed K clusters do not change, and K final clusters are obtained.

[0086] Specifically, in an iterative loop, first traverse the right boundary set, specifically:

[0087] Step 1.1, calculate the feature mean set of each initial cluster, let ;

[0088] Step 2.1: Calculate the right boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the next cluster The characteristic distance and ;

[0089] Step 3.1: Determination Is it greater than ;

[0090] Step 4.1: If Greater than , then the image Move to next cluster middle;

[0091] Step 5.1: Update the current cluster and the next cluster The characteristic mean set and , and again calculate the right boundary concentration and the initial cluster The corresponding new image With the current cluster and the next cluster The new feature distance and , until Less than or equal to ;

[0092] Step 6.1: If Less than or equal to , then let , repeat steps 2.1 to 6.1 until ;

[0093] Then traverse the left boundary set, specifically:

[0094] Step 1.2, order ,

[0095] Step 2.2: Calculate the left boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the previous cluster The characteristic distance and ;

[0096] Step 3.2, judgment Is it less than ;

[0097] Step 4.2: If Less than , then the image Move to the previous cluster middle;

[0098] Step 5.2: Update the current cluster and the previous cluster The characteristic mean set and , and again calculate the left boundary concentration and the initial cluster The corresponding new image With the current cluster and the previous cluster The new feature distance and , until Greater than or equal to ;

[0099] Step 6.2: If Greater than or equal to , then let Repeat steps 2.2 to 6.2 until .

[0100] Step S105: Obtain the feature mean set of each final cluster, and calculate the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs, filter out the minimum feature distance from multiple feature distances under the same final cluster, and filter out the key frame image under each final cluster based on the minimum feature distance, and build a human behavior recognition model based on the key frame image.

[0101] It should be noted that the feature mean set It is composed of clusters The vector derived from the mean of each dimension of all frames in can best comprehensively represent the spatial information in the clustered frames. It will be used to calculate the difference between clusters and frames, so that the obtained frames can best represent the spatial information in the cluster, and the spatial information difference between clusters is the largest, thus ensuring that the complete human behavior information is obtained to the greatest extent. Based on this idea, by calculating the , continuously update the attribution of adjacent cluster boundary frames, so as to achieve spatial information balance of each cluster and avoid cross-time compression. Finally, continuously iterate and optimize the cluster boundary frames, select each cluster The closest frame is used as the keyframe.

[0102] It should also be noted that in human motion analysis, we identify the moments with the most significant and largest changes in motion as key frames, which contain important dynamic information.

[0103] In addition, taking the action waving as an example, 10 key frames extracted by angle and displacement, such as Figure 4 As shown, Figure 4 within Indicates the degree of change between each frame and the first frame in the frame sequence. Indicates the degree of inter-frame change between each frame image and the previous frame image. For the frame numbered 19 and the frame numbered 55, the classic algorithm based on Euclidean distance will classify them into one cluster, and only one frame is used to represent these frames with a larger time sequence. This is because the two frames are consistent in spatial position information, but in practice they should not be clustered in one cluster. Although the two frames are very similar in spatial information, they are very different in time sequence. Such a method is not easy to identify multiple waving behaviors, but if the swing amplitude (angle change) of the bone node is taken into account, it can be well distinguished, and the angle change is indeed very large, such as waving to say hello. Therefore, adding these indicators is helpful for the extraction of key frames. At the same time, in order to prevent the traditional clustering from causing time sequence confusion by performing frame clustering across a larger time sequence, the key frame extraction method of this embodiment no longer iterates for all frames, but iterates for the boundary frames of orderly clusters to extract key frames. This enables the method of this embodiment to extract key frames with stable time sequence. Such precise key frames enable this embodiment to still recognize human behavior and reduce computational consumption while reducing frames. For example, frames 51 and 55 will be grouped into one cluster if only the position and timing factors are considered. However, in fact, the gestures of frames 51 and 55 are very different. If they are placed in the same cluster, the waving behavior cannot be recognized.

[0104] In addition, in some embodiments, the feature mean set is calculated according to the following formula:

[0105]

[0106] in, Representation and Cluster The corresponding feature mean set, Representation Cluster The last frame of the image, Representation Cluster The first frame image in represents the n-th dimension data in the multidimensional dataset corresponding to image t, , , Respectively represent clusters The mean of the 1st, 2nd, 90th, and nth dimension features in ;

[0107] The feature distance is calculated according to the following formula:

[0108]

[0109] in, Represents the i-th frame image and cluster The characteristic distance of Represents the n-th dimension of the multidimensional data set of the i-th frame image.

[0110] In addition, in some embodiments, after extracting several key frames, in order to further reduce parameters while maintaining spatiotemporal motion information, it is also necessary to use the extracted key frames to construct a reasonable network input and convert the action sequence into a matrix that meets the CNN standard. Specifically, it is necessary to calculate the displacement, angle, velocity, and angular velocity of the skeletal joints. All features are based on relative coordinates to enhance viewpoint robustness. Finally, the features are arranged into a dense joint three-dimensional floating-point feature matrix of uniform scale to capture subtle motion changes. Each frame consists of 15 three-dimensional skeletal joint points, extracts 19-dimensional information, and superimposes it with the key frame timing to form a dense joint three-dimensional floating-point feature matrix with stable timing, as follows:

[0111] The relative coordinate characteristics are calculated according to the following formula:

[0112]

[0113] in, Respectively represent the x, y, and z relative coordinates of the nth joint point of the i-th frame image, Respectively represent the x, y, and z coordinates of the first joint point of the i-th frame image;

[0114] The displacement characteristics are calculated according to the following formula:

[0115]

[0116]

[0117] They represent the displacement of the nth joint point in the i-th frame image relative to the nth joint point in the 1st frame image on the X plane, Y plane, and Z plane, respectively. Respectively represent the x, y, and z relative coordinates of the first joint point of the i-th frame image, Represents the displacement feature of the nth joint point of the i-th frame image relative to the nth joint point of the 1st frame image;

[0118] The speed characteristics are calculated according to the following formula:

[0119]

[0120]

[0121] in, Respectively represent the mapping values ​​of the velocity of the nth joint point in the i-th frame image on the X plane, Y plane, and Z plane, They represent the x, y, and z relative coordinates of the nth joint point in the i-1th frame image. It is defined that when i is zero, the three mapping values ​​of the velocity feature are all zero. Represents the velocity feature of the nth joint point in the i-th frame image.

[0122] Finally, the angle feature. The physical meaning of the angle is that it provides the geometric features of the human body structure and represents the geometric relationship between adjacent joints. An angle is defined as consisting of three joints. The corresponding angles of two adjacent joints are defined as the middle joints. Finally, the angle combination is determined as (J 2 , J 1 , J 3 ),(J 1 ,J 2 , J 3 ),(J 2 , J 3 ,J 1 ),(J 2 , J 4 , J 5 ),( J 4 , J 5 , J 6 ),(J 5 , J 6 , J 3 ),(J 2 , J 7 , J 8 ),(J 7 , J 8 , J 9 ),(J 8 , J 9 ,J 3 ),(J 1 , J 10 , J 11 ),(J 10 , J 11 , J 12 ),(J 11 , J 12 , J 3 ),(J 1 , J 13 , J 14 ),(J 13 , J 14 , J 15 ),(J 14 , J 15 ,J 3 ) Each combination represents the angle of the joint point in its middle position. For example, for three adjacent joint points (A, B, C), the angle between the BA vector and the BC vector is used as the angle feature of node B. The calculation method is as follows

[0123]

[0124] is the angle feature of the bone joint point B. By transforming the vector and The dot product of two vectors The modulus product gets the cosine value, and then converts it to radians through the inverse cosine function, thus obtaining our angle feature Then, the angle change per unit time of each joint point and the mapping in three planes are calculated to obtain three mapping values ​​of each joint point. A total of 19 eigenvalues ​​are calculated for each joint point. The main features are concentrated on displacement and angle. 15 joint points of the human skeleton are selected and finally formed. In this embodiment, it is easy to calculate the number of different features according to the needs of the task, such as displacement, speed, direction, acceleration, etc. By selecting different nodes, such as gesture recognition, for example, some detailed bone nodes of the hand can be added to achieve better recognition effect.

[0125] In addition, in some embodiments, a deep convolutional neural network is selected as the basic structure of the human body recognition model. The deep convolutional neural network usually has a strong learning ability, but also requires a larger input scale, and the deep architecture requires more computing resources. To this end, the present embodiment designs a lightweight shallow neural network, retains only three convolutional layers, and reduces the kernel size, step size and pooling window to adapt to the size of the dense joint three-dimensional floating-point feature matrix. In order to accelerate convergence, a batch normalization layer is added after each convolutional layer, and we replace the last maximum pooling layer and flattening layer with an SPP layer to achieve multi-scale training. Specifically, after constructing a multi-scale dense joint three-dimensional floating-point feature matrix based on key frames, it is input into a deep convolutional neural network for iterative multi-scale training, thereby obtaining a human behavior recognition model.

[0126] In summary, according to the above-mentioned human behavior recognition method based on key frame extraction, by optimizing the input data form, reducing the impact of external interference factors (such as viewing angle, angle change, etc.) on the recognition accuracy, and combining an efficient key frame extraction algorithm for model training, the overall recognition accuracy can be improved. Specifically, by extracting key frames from the video stream, redundant data is reduced, thereby improving processing efficiency and accuracy. The selection of key frames can not only speed up the processing speed of the video, but also effectively capture the important features of human behavior and improve recognition accuracy. This method is particularly suitable for real-time monitoring scenarios. It can improve the accuracy of human behavior recognition while ensuring high efficiency, especially in monitoring applications in complex environments.

[0127] like Figure 5As shown, an embodiment of the present invention further proposes a human behavior recognition system based on key frame extraction, the system comprising:

[0128] A data enhancement module 10 is used to perform data enhancement on the human monitoring video based on the speed coefficient and the body size coefficient to obtain a target monitoring video;

[0129] An information set construction module 20 is used to construct a human skeleton model, the human skeleton model includes a plurality of joints, obtain three-dimensional position information and three-dimensional angle information of each joint, and construct a position information set of each frame of image according to the three-dimensional position information, and construct an angle information set of each frame of image according to the three-dimensional angle information;

[0130] A data connection module 30 is used to merge and connect the position information set and the angle information set under the same frame image to obtain a multi-dimensional data set for each frame image;

[0131] A clustering cluster division module 40 is used to divide the images in the target monitoring video into K initial clusters based on the frame sequence, and define a left boundary set and a right boundary set according to a number of frame images included in each initial cluster, and iteratively traverse the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results;

[0132] The key frame extraction module 50 is used to obtain the feature mean set of each final cluster, and calculate the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs, filter out the minimum feature distance from multiple feature distances under the same final cluster, and filter out the key frame image under each final cluster according to the minimum feature distance, and build a human behavior recognition model based on the key frame image.

[0133] Another aspect of the present invention further provides a storage medium on which one or more programs are stored, and when the programs are executed by a processor, the above-mentioned human behavior recognition method based on key frame extraction is implemented.

[0134] On the other hand, the present invention further proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned human behavior recognition method based on key frame extraction.

[0135] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0136] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0137] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0138] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A human behavior recognition method based on key frame extraction, characterized in that: The method comprises: Perform data enhancement on human monitoring video based on speed coefficient and body size coefficient to obtain target monitoring video; Constructing a human skeleton model, the human skeleton model including a plurality of joints, obtaining three-dimensional position information and three-dimensional angle information of each joint, and constructing a position information set of each frame of image according to the three-dimensional position information, and constructing an angle information set of each frame of image according to the three-dimensional angle information; The location information set is constructed according to the following formula: ； in, represents the position information set of the i-th frame image, They represent the x-coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image, respectively. They represent the y coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively. They represent the z coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively; The angle information set is constructed according to the following formula: ； in, Represents the angle information set of the i-th frame image, They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the X plane of the i-th frame image respectively. They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Y plane of the i-th frame image respectively. Respectively represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Z plane of the i-th frame image; The position information set and the angle information set under the same frame image are merged and connected to obtain a multi-dimensional data set for each frame image; The images in the target monitoring video are divided into K initial clusters based on the frame sequence, and a left boundary set and a right boundary set are defined according to a number of frame images contained in each initial cluster, and the left boundary set and the right boundary set are iteratively traversed based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results; The feature mean set of each final cluster is obtained, and the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs is calculated. The minimum feature distance is screened out from multiple feature distances under the same final cluster, and the key frame image under each final cluster is screened out according to the minimum feature distance, and a human behavior recognition model is constructed based on the key frame image.

2. The human behavior recognition method based on key frame extraction according to claim 1 is characterized in that: The step of merging and connecting the position information set and the angle information set under the same frame image to obtain a multidimensional data set corresponding to each frame image includes: Get the cube according to the following formula: ； in, A multidimensional dataset representing the i-th frame image, represents the maximum value in the corresponding information set, represents the minimum value in the corresponding information set, represents the normalization operation function, , , , , , Represents a multidimensional dataset The 1st, 2nd, 3rd, 45th, 46th, and 90th dimension data in .

3. The human behavior recognition method based on key frame extraction according to claim 2 is characterized in that: The steps of dividing the images in the target monitoring video into K initial clusters based on the frame sequence, defining a left boundary set and a right boundary set according to a number of frame images included in each initial cluster, and iteratively traversing the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results include: Divide all frame images evenly into K initialization intervals to form initial clusters , select the first frame image of each initial cluster to form the left boundary set , select the last frame image of each initial cluster to form the right boundary set ,in, Respectively represent the first, second, and Kth initial clusters, , Respectively represent the first frame image in the 1st, 2nd, and Kth initial clusters, , Respectively represent the last frame image in the 1st, 2nd, and Kth initial clusters; In each iterative traversal process, the right boundary set is traversed first, and then the left boundary set is traversed, and this cycle is repeated until the feature mean sets of the newly formed K clusters do not change, and K final clusters are obtained.

4. The human behavior recognition method based on key frame extraction according to claim 3 is characterized in that: The steps to traverse the right boundary set include: Step 1.1, calculate the feature mean set of each initial cluster, let ; Step 2.1: Calculate the right boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the next cluster The characteristic distance and ; Step 3.1: Determination Is it greater than ; Step 4.1: If Greater than , then the image Move to next cluster middle; Step 5.1: Update the current cluster and the next cluster The characteristic mean set and , and again calculate the right boundary concentration and the initial cluster The corresponding new image With the current cluster and the next cluster The new feature distance and , until Less than or equal to ; Step 6.1: If Less than or equal to , then let , repeat steps 2.1 to 6.1 until ; The steps to traverse the left boundary set include: Step 1.2, order , Step 2.2: Calculate the left boundary concentration and the initial cluster Corresponding image Respectively with the current cluster and the previous cluster The characteristic distance and ; Step 3.2, judgment Is it less than ; Step 4.2: If Less than , then the image Move to the previous cluster middle; Step 5.2: Update the current cluster and the previous cluster The characteristic mean set and , and again calculate the left boundary concentration and the initial cluster The corresponding new image With the current cluster and the previous cluster The new feature distance and , until Greater than or equal to ; Step 6.2: If Greater than or equal to , then let Repeat steps 2.2 to 6.2 until .

5. The human behavior recognition method based on key frame extraction according to claim 4 is characterized in that: The feature mean set is calculated according to the following formula: ； in, Representation and Cluster The corresponding feature mean set, Representation Cluster The last frame of the image, Representation Cluster The first frame image in represents the n-th dimension data in the multidimensional dataset corresponding to image t, , , Respectively represent clusters The mean of the 1st, 2nd, 90th, and nth dimension features in .

6. The human behavior recognition method based on key frame extraction according to claim 5 is characterized in that: The feature distance is calculated according to the following formula: ； in, Represents the i-th frame image and cluster The characteristic distance, Represents the n-th dimension of the multidimensional data set of the i-th frame image.

7. A human behavior recognition system based on key frame extraction, characterized in that: The system comprises: A data enhancement module is used to perform data enhancement on the human monitoring video based on the speed coefficient and the body size coefficient to obtain the target monitoring video; An information set construction module is used to construct a human skeleton model, the human skeleton model includes multiple joints, obtain three-dimensional position information and three-dimensional angle information of each joint, and construct a position information set of each frame of image according to the three-dimensional position information, and construct an angle information set of each frame of image according to the three-dimensional angle information; The location information set is constructed according to the following formula: ； in, represents the position information set of the i-th frame image, They represent the x-coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image, respectively. They represent the y coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively. They represent the z coordinates of the 1st joint point, the nth joint point, and the 15th joint point of the i-th frame image respectively; The angle information set is constructed according to the following formula: ； in, Represents the angle information set of the i-th frame image, They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the X plane of the i-th frame image respectively. They represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Y plane of the i-th frame image respectively. Respectively represent the angle information of the 1st joint point, the nth joint point, and the 15th joint point in the Z plane of the i-th frame image; A data connection module is used to merge and connect the position information set and the angle information set under the same frame image to obtain a multi-dimensional data set for each frame image; A clustering cluster division module is used to divide the images in the target monitoring video into K initial clusters based on the frame sequence, and define a left boundary set and a right boundary set according to a number of frame images contained in each initial cluster, and iteratively traverse the left boundary set and the right boundary set based on the improved KKF algorithm to obtain K reclassified final clusters according to the traversal results; The key frame extraction module is used to obtain the feature mean set of each final cluster, and calculate the feature distance between the multidimensional data set of each frame image in each final cluster and the feature mean set of the final cluster to which it belongs, screen out the minimum feature distance from multiple feature distances under the same final cluster, and screen out the key frame image under each final cluster according to the minimum feature distance, and build a human behavior recognition model based on the key frame image.

8. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the human behavior recognition method based on key frame extraction as described in any one of claims 1-6.

9. An electronic device, comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the human behavior recognition method based on key frame extraction as described in any one of claims 1-6.

Images