Human behavior recognition method and system based on three-dimensional floating point joint matrix

Abstract

The present invention provides a method and system for human behavior recognition based on a three-dimensional floating-point joint matrix, the method comprising: obtaining a skeleton sequence model of a profiling human body, selecting multiple target joint points from the joint points in the skeleton sequence model, and constructing a tree skeleton model corresponding to each target joint point according to the connection relationship between any adjacent joint points; taking the target joint point as the starting point, performing a depth-first traversal on the tree skeleton model to obtain a traversal sequence; obtaining a monitoring video of the target human body, and extracting information frame by frame from the monitoring video to obtain physical motion information, and according to the number of the joint point, associating the physical operation information to each joint point in the traversal sequence, and performing frame-by-frame concatenation according to the frame sequence to obtain a three-dimensional floating-point joint matrix; inputting the three-dimensional floating-point joint matrix into a double-branch long-short cascade network for training. The present invention can improve the accuracy and efficiency of human behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to a human behavior recognition method and system based on a three-dimensional floating point joint matrix. Background Art

[0002] In recent years, human action recognition (HAR) based on deep learning technology has attracted widespread attention in the field of computer vision, and related research has made rapid progress. It has been widely used in human-computer interaction, video surveillance, and intelligent medical care.

[0003] Human skeleton joints have received extensive attention in the field of human behavior recognition. In order to store skeleton sequence data and input, many researchers convert skeleton data into images. However, due to the storage range of images and the fact that they can only store integers, the data needs to be normalized, which will lead to more obvious errors and loss of spatial motion information, thus affecting the recognition accuracy of human behavior. Summary of the invention

[0004] The purpose of the present invention is to provide a human behavior recognition method and system based on a three-dimensional floating-point joint matrix, aiming to solve the problem of low recognition accuracy caused by data errors and loss in traditional human behavior recognition technology.

[0005] In a first aspect, the present invention provides a method for human behavior recognition based on a three-dimensional floating-point joint matrix, the method comprising:

[0006] Acquire a skeleton sequence model of a human body, the skeleton sequence model comprising a plurality of joints and a connection relationship between any adjacent joints, select a plurality of target joints from the joints in the skeleton sequence model, and construct a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, the tree skeleton model comprising a plurality of joints and a connection relationship between any two adjacent joints, and the first-level joint is the target joint;

[0007] Taking the target joint point as the starting point, and performing a depth-first traversal on the tree skeleton model according to the connection relationship between any two adjacent joint points, a traversal sequence corresponding to each target joint point is obtained, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship;

[0008] Acquire a monitoring video of the target human body, and extract information frame by frame from the monitoring video to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation in accordance with the frame order to obtain a three-dimensional floating-point joint matrix;

[0009] The three-dimensional floating-point joint matrix is ​​input into a dual-branch long-short cascade network for training to obtain a human behavior recognition model.

[0010] Furthermore, the step of selecting a plurality of target joint points from the joint points in the skeleton sequence model comprises:

[0011] The left shoulder, right shoulder, left hip, right hip, upper spine, lower spine and head of the human body are selected as target joints;

[0012] The physical motion information includes relative position, instantaneous displacement, cumulative total displacement, speed, instantaneous acceleration, cumulative acceleration, joint angle, and instantaneous motion direction between frames.

[0013] Furthermore, the relative position is obtained according to the following formula:

[0014] ;

[0015] in, represents the relative position of the mth joint point of the jth frame image and the mth joint point of the first frame image, The x-axis coordinate, y-axis coordinate, and z-axis coordinate represent the relative position of the m-th joint point in the j-th frame image and the m-th joint point in the first frame image, respectively. They represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-th frame image, respectively. Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the first frame image;

[0016] The instantaneous displacement is calculated according to the following formula:

[0017] ;

[0018] in, represents the instantaneous displacement between the mth joint point of the jth frame image and the mth joint point of the j-1th frame image, Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-1-th frame image;

[0019] The cumulative total displacement is calculated according to the following formula:

[0020] ;

[0021] in, represents the cumulative total displacement of the mth joint point in the jth frame image;

[0022] The speed is calculated according to the following formula:

[0023] ;

[0024] in, represents the velocity of the mth joint point in the jth frame image, Indicates the time interval between adjacent frame images;

[0025] The instantaneous acceleration is calculated according to the following formula:

[0026] ;

[0027] The cumulative acceleration is calculated according to the following formula:

[0028] ;

[0029] in, represents the instantaneous acceleration of the mth joint point in the jth frame image, represents the velocity of the mth joint point in the j-1th frame image, Indicates the velocity of the mth joint point in the first frame image, Represents the cumulative acceleration of the mth joint point in the jth frame image;

[0030] The joint angle is formed by the space vector formed by three adjacent joint points, and the joint angle is calculated according to the following formula:

[0031] ;

[0032] in, represents the cosine value of the joint angle with the mth joint point in the jth frame image as the vertex, represents the spatial vector formed by the mth joint point in the jth frame image and the kth joint point in the jth frame image, Represents the spatial vector formed by the mth joint point in the jth frame image and the nth joint point in the jth frame image;

[0033] When calculating the joint angle, the mth joint point is connected to the jth joint point and the nth joint point respectively;

[0034] The instantaneous motion direction between frames is calculated according to the following formula:

[0035] ;

[0036] in, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xy plane, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane, It represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane. The physical motion information of all joint points in the first frame image is 0.

[0037] Furthermore, the step of associating the physical operation information with each joint point in the traversal sequence according to the joint point number, and performing frame-by-frame concatenation according to the frame sequence to obtain a three-dimensional floating-point joint matrix includes:

[0038] The number of features of physical motion information is the number of columns, the total number of joint points in the traversal sequence is the number of rows, the feature name of the physical motion information is the column name, and the number of the joint point in the traversal sequence is the row name;

[0039] According to the column name, all physical motion information of the cth joint point in the traversal sequence corresponding to the ath target joint point under the bth frame image is filled in the bth row, and a two-dimensional unit joint matrix corresponding to the ath target joint point under the bth frame image is obtained;

[0040] Define the order of each target joint point, and generate a two-dimensional joint matrix from all two-dimensional unit joint matrices in the same frame according to the order of each target joint point;

[0041] The two-dimensional joint matrix is ​​concatenated frame by frame in the frame sequence to obtain a three-dimensional floating-point joint matrix.

[0042] Furthermore, a two-dimensional unit joint matrix is ​​constructed according to the following formula:

[0043] ;

[0044] in, represents the two-dimensional unit joint matrix corresponding to the a-th target joint point in the b-th frame image, Represents the Kth physical motion information of the Cth joint point in the traversal sequence;

[0045] Construct a two-dimensional joint matrix according to the following formula:

[0046] ;

[0047] in, represents the two-dimensional joint matrix corresponding to the b-th frame image, , They respectively represent the two-dimensional unit joint matrices corresponding to the first, second, third, fourth, fifth, sixth, seventh, and eighth target joint points in the b-th frame image.

[0048] Furthermore, the step of inputting the three-dimensional floating-point joint matrix into a dual-branch long-short cascade network for training to obtain a human behavior recognition model includes:

[0049] Taking the two-dimensional joint matrix corresponding to the first frame image as a sampling starting point, the three-dimensional floating-point joint matrix is ​​sliced ​​at intervals of a first preset number of frames to obtain a first target two-dimensional joint matrix;

[0050] Inputting the first target two-dimensional joint matrix into the first channel network, obtaining the probability values ​​of the first motion feature and each human behavior, and screening out the maximum probability value from all the probability values, and determining whether the maximum probability value is greater than a preset confidence threshold;

[0051] If the maximum probability value is less than or equal to a preset confidence threshold, re-slicing the matrix with a second preset number of frames to obtain a second target two-dimensional joint matrix;

[0052] Inputting the second target two-dimensional joint matrix into a second channel network to obtain a second motion feature;

[0053] The first motion feature and the second motion feature are connected, and the connection result is used as the input of the fully connected classifier.

[0054] Furthermore, after the step of determining whether the maximum probability value is greater than a preset confidence threshold, the step further includes:

[0055] If the maximum probability value is greater than a preset confidence threshold, directly using the first motion feature as an input of the fully connected classifier;

[0056] The first preset frame number is greater than the second preset frame number.

[0057] In a second aspect, the present invention provides a human behavior recognition system based on a three-dimensional floating point joint matrix, the system comprising:

[0058] A model building module, used to obtain a skeleton sequence model of the prototyping human body, the skeleton sequence model includes multiple joints and a connection relationship between any adjacent joints, and select multiple target joints from the joints in the skeleton sequence model, and build a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, the tree skeleton model includes multiple levels of joints and a connection relationship between any two adjacent levels of joints, and the first level of joints is the target joint;

[0059] A traversal sequence generation module, used to take the target joint point as a starting point and perform a depth-first traversal on the tree skeleton model according to the connection relationship between any two adjacent joint points, to obtain a traversal sequence corresponding to each target joint point, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship;

[0060] A three-dimensional matrix construction module is used to obtain monitoring videos of the target human body, and extract information frame by frame from the monitoring videos to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation according to the frame order to obtain a three-dimensional floating-point joint matrix;

[0061] The training module is used to input the three-dimensional floating-point joint matrix into a double-branch long-short cascade network for training to obtain a human behavior recognition model.

[0062] In a third aspect, the present invention provides a storage medium storing one or more programs, which, when executed by a processor, implement the above-mentioned human behavior recognition method based on three-dimensional floating-point joint matrix.

[0063] In a fourth aspect, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein:

[0064] The memory is used to store computer programs;

[0065] When the processor is used to execute the computer program stored in the memory, the above-mentioned human behavior recognition method based on the three-dimensional floating-point joint matrix is ​​implemented.

[0066] Compared with the prior art, the present invention has the following advantages:

[0067] 1. In order to simulate the motion dependency between physically connected joints, a tree-like skeleton model is constructed to replace the traditional skeleton sequence model with a fixed order of serial joints to represent the spatial information of the skeleton. The connection relationship between any two joints in the tree-like skeleton model represents the spatial dependency between the joints. The fewer edges, the more relevant the joint connections are, so that more spatial relationships can be retained. Then, a traversal sequence is constructed through the tree-like skeleton model to find a good sequence chain that can help the model obtain more spatial semantic information during feature extraction. In addition, a large-scale three-dimensional floating-point matrix is ​​constructed to store three-dimensional skeleton data. According to the basic physical motion information, the motion features of each joint are extracted frame by frame. The physical motion information is encoded using a special arrangement strategy to provide suitable large-scale input data for the CNN model, thereby providing rich spatiotemporal information for the human behavior recognition model.

[0068] 2. By setting the first preset frame number and the second preset frame number, the three-dimensional floating-point matrix is ​​sampled at different time frequencies to fully utilize the feature information of different time scales. At the same time, the first preset frame number is greater than the second preset frame number, that is, two fast and slow channels are constructed, and the dual-branch method of fast and slow motion is used to capture time and space information. In addition, by setting up this cascaded dual-branch network, the computational complexity can be effectively reduced, which improves the computational efficiency without affecting the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a human behavior recognition method based on a three-dimensional floating-point joint matrix proposed in one embodiment of the present invention;

[0070] Figure 2 The following is a flow chart of generating a traversal sequence according to an example of an embodiment of the present invention, wherein: Figure 2 (a) is a schematic diagram of the structure of a skeleton sequence model according to an embodiment of the present invention. Figure 2 (b) is a schematic diagram of the structure of a tree skeleton model according to an embodiment of the present invention. Figure 2 (c) is a schematic diagram of a traversal sequence according to an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a process for generating a three-dimensional floating-point joint matrix according to an example of an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the structure of a human behavior recognition system based on a three-dimensional floating-point joint matrix proposed in one embodiment of the present invention.

[0073] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0075] like Figure 1As shown, an embodiment of the present invention provides a method for human behavior recognition based on a three-dimensional floating-point joint matrix, the method comprising steps S101 to S104, wherein:

[0076] Step S101: obtaining a skeleton sequence model of a human body, wherein the skeleton sequence model includes a plurality of joints and a connection relationship between any adjacent joints, and selecting a plurality of target joints from the joints in the skeleton sequence model, and constructing a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, wherein the tree skeleton model includes a plurality of joints and a connection relationship between any two adjacent joints, and the first-level joint is the target joint;

[0077] For example, Figure 2 (a) and Figure 2 As shown in (b), the skeleton sequence model has 21 joints. In action recognition based on the skeleton sequence model, the order in which the skeleton joints form a joint chain in each frame is predefined and fixed. Usually, the traditional skeleton sequence model is simply arranged in the order of joint connection (1-2-3-4-5-...-19-20-21). This arrangement ignores the dependencies between joints and adds false connections between body joints that are not tightly connected. For example, compared with joint 8 (right hand part), joint 21 (upper spine point) has a stronger spatial correlation with the joints on the left arm (9, 10, 11, 12). In the CNN model, adjacent joints in the joint sequence are first learned at a lower level, which means that adjacent joints share more spatial relationships. Based on this, in order to find a good sequence chain that can help the model obtain more spatial semantic information during feature extraction and simulate the motion dependency between physically connected joints, this step converts the traditional skeleton sequence model into a tree skeleton model. Whether it is a skeleton sequence model or a tree skeleton model, the connection relationship it contains refers to the edges between adjacent joints, such as Figure 2 As shown in (b), one of the first-level joint points is joint point No. 2, joint point No. 21 and joint point No. 1 are second-level joint points, and so on.

[0078] In addition, the edges (connection relationships) between the joints in the tree skeleton model represent the spatial dependencies between the joints. The fewer the edges, the more relevant the joint connections. Then, the tree structure skeleton is traversed in depth first to ensure that each node can obtain contextual information from its child nodes and parent nodes.

[0079] Step S102: Taking the target joint point as the starting point, and performing a depth-first traversal on the tree skeleton model according to the connection relationship between any two adjacent joint points, to obtain a traversal sequence corresponding to each target joint point, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship;

[0080] It should be noted that, in the process of generating the traversal sequence, the target joint point is first used as the starting point, and then each branch of the tree skeleton model is traversed from left to right. At the same time, any two nodes in the adjacent order are connected to ensure the continuity of the joint points in the traversal sequence. For example, Figure 2 , 17, 18, 19, 20, 19, 18, 17, 1, 2. As shown in (c) in Figure 2, taking target joint point 2 as an example, the traversal sequence with 41 joint points is obtained in the following order: 2, 21, 3, 4, 3, 21, 5, 6, 7, 8,7, 6, 5, 21, 9, 10, 11, 12, 11, 10, 9, 21, 2, 1, 13, 14, 15, 16, 15, 14, 13,1, 17, 18, 19, 20, 19, 18, 17, 1, 2. In this way, the sequence of the original 21 joints is expanded to the sequence of N=41 joints, and the amount of data is also enlarged.

[0081] In addition, in some embodiments, the left shoulder, right shoulder, left hip, right hip, upper spine, lower spine and head of the human body are selected as target joints. These eight joints are selected as target joints because they are relatively stable in most actions relative to other joints and cover five major parts of the human body's physical structure: arms, legs and trunk. Then, the eight tree-like skeleton graphs are traversed in depth first, so that eight groups of skeleton joint chains (traversal sequences) can be obtained in each frame, and each group of joint chains contains N = 41 joints.

[0082] Step S103: Acquire a monitoring video of the target human body, and extract information frame by frame from the monitoring video to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation according to the frame sequence to obtain a three-dimensional floating-point joint matrix;

[0083] It should be noted that each frame of the monitoring video contains a row of joint chains, and a channel is attached to each joint point. Usually each channel represents a joint coordinate axis, that is, each column represents the coordinate value of a joint. In the present invention, relative position will be used instead of directly using the three-dimensional coordinates of the joint, because the coordinates of the original data are affected by the position distance between the camera and the monitored person. In order to better explore the motion information, three-dimensional motion features are extracted at each joint point. In order to avoid the influence of noise during the feature extraction process, combined with the definition of classical physics, some basic physical motion information is selected, and the physical motion information includes relative position, instantaneous displacement, cumulative total displacement, velocity, instantaneous acceleration, cumulative acceleration, joint angle, and instantaneous motion direction between frames, as follows:

[0084] Get the relative position according to the following formula:

[0085] ;

[0086] in, represents the relative position of the mth joint point of the jth frame image and the mth joint point of the first frame image, The x-axis coordinate, y-axis coordinate, and z-axis coordinate represent the relative position of the m-th joint point in the j-th frame image and the m-th joint point in the first frame image, respectively. They represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-th frame image, respectively. Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the first frame image;

[0087] The instantaneous displacement is calculated according to the following formula:

[0088] ;

[0089] in, represents the instantaneous displacement between the mth joint point of the jth frame image and the mth joint point of the j-1th frame image, Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-1-th frame image;

[0090] The cumulative total displacement is calculated according to the following formula:

[0091] ;

[0092] in, represents the cumulative total displacement of the mth joint point in the jth frame image;

[0093] The speed is calculated according to the following formula:

[0094] ;

[0095] in, represents the velocity of the mth joint point in the jth frame image, Indicates the time interval between adjacent frame images;

[0096] The instantaneous acceleration is calculated according to the following formula:

[0097] ;

[0098] The cumulative acceleration is calculated according to the following formula:

[0099] ;

[0100] in, represents the instantaneous acceleration of the mth joint point in the jth frame image, represents the velocity of the mth joint point in the j-1th frame image, Indicates the velocity of the mth joint point in the first frame image, Represents the cumulative acceleration of the mth joint point in the jth frame image;

[0101] The joint angle is formed by the space vector formed by three adjacent joint points, and the joint angle is calculated according to the following formula:

[0102] ;

[0103] in, represents the cosine value of the joint angle with the mth joint point in the jth frame image as the vertex, represents the spatial vector formed by the mth joint point in the jth frame image and the kth joint point in the jth frame image, Represents the spatial vector formed by the mth joint point in the jth frame image and the nth joint point in the jth frame image;

[0104] When calculating the joint angle, the mth joint point is connected to the jth joint point and the nth joint point respectively;

[0105] The instantaneous motion direction between frames is calculated according to the following formula:

[0106] ;

[0107] in, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xy plane, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane, It represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane. The physical motion information of all joint points in the first frame image is 0.

[0108] In addition, when selecting the above physical motion information, both the temporal and spatial information of the motion and the geometric features are considered. In addition, it is also considered that the motion information should include the cumulative motion information compared with the first frame and the inter-frame difference information. This is because the factors for identifying and reflecting motion include the process and final result of the motion.

[0109] In addition, since the original skeleton data only has the coordinates of each joint point in each frame, that is, there is only one absolute position information, resulting in insufficient spatial information. Therefore, motion information such as relative position, instantaneous displacement between frames and cumulative total displacement, velocity and acceleration, joint angle, instantaneous motion direction between frames, etc. are added, thereby expanding the characteristics of physical motion information from the original three dimensions to 17 dimensions, which is also a way to enlarge the scale.

[0110] Furthermore, in some embodiments, Figure 3 As shown, in the process of constructing a three-dimensional floating-point joint matrix, first, the number of features of physical motion information is used as the number of columns, the total number of joint points in the traversal sequence is used as the number of rows, the feature names of physical motion information are used as column names, and the numbers of joint points in the traversal sequence are used as row names; then, according to the column names, all physical motion information of the cth joint point in the traversal sequence corresponding to the ath target joint point under the bth frame image is filled in the bth row respectively, and a two-dimensional unit joint matrix corresponding to the ath target joint point under the bth frame image is obtained; then, the order of each target joint point is defined, and all the two-dimensional unit joint matrices under the same frame are generated into a two-dimensional joint matrix according to the order of each target joint point; finally, the two-dimensional joint matrices are connected frame by frame according to the frame sequence to obtain a three-dimensional floating-point joint matrix.

[0111] Specifically, the two-dimensional unit joint matrix is ​​constructed according to the following formula:

[0112] ;

[0113] in, represents the two-dimensional unit joint matrix corresponding to the a-th target joint point in the b-th frame image, Represents the Kth physical motion information of the Cth joint point in the traversal sequence;

[0114] Construct a two-dimensional joint matrix according to the following formula:

[0115] ;

[0116] in, represents the two-dimensional joint matrix corresponding to the b-th frame image, , They respectively represent the two-dimensional unit joint matrices corresponding to the first, second, third, fourth, fifth, sixth, seventh, and eighth target joint points in the b-th frame image.

[0117] In addition, in some embodiments, after all the physical motion information is obtained, all the physical motion information is split into: relative position x, relative position y, relative position z, instantaneous displacement between frames, instantaneous displacement between frames in the xy plane, instantaneous displacement between frames in the xz plane, instantaneous displacement between frames in the yz plane, cumulative displacement, cumulative displacement in the xy plane, cumulative displacement in the xz plane, cumulative displacement in the yz plane, instantaneous direction in the xy plane, instantaneous direction in the xz plane, instantaneous direction in the yz plane, instantaneous speed, cumulative speed, and joint angle, thereby realizing the expansion from three dimensions to 17. The xy plane, the xz plane, and the xz plane are conventional projection planes.

[0118] In summary, by storing the obtained motion features in a three-dimensional floating-point matrix and arranging the two-dimensional joint matrix through a special arrangement strategy to generate a three-dimensional floating-point matrix, which contains rich spatial and temporal motion information, it can provide suitable large-scale input data for the existing large-dimensional CNN model.

[0119] Step S104: inputting the three-dimensional floating-point joint matrix into a double-branch long-short cascade network for training to obtain a human behavior recognition model.

[0120] It should be pointed out that the constructed three-dimensional floating-point matrix only utilizes basic physical motion information and geometric features, lacking global motion information. Affected by different ages, genders, and body shapes, different people have different motion scales for the same action. Some are faster, some are slower. Therefore, two scales of motion features are introduced, namely fast motion and slow motion. Matrix slices are obtained by sampling at different frame rates to obtain dual-scale motion features. Traditional two-stream networks usually have similar network structures and numbers of parameters. The two-stream network for fast and slow motion here uses two different convolutional models to capture information with different temporal changes. The difference between the two pathways lies mainly in the number of channels, as follows:

[0121] Taking the two-dimensional joint matrix corresponding to the first frame image as the sampling starting point, the three-dimensional floating-point joint matrix is ​​sliced ​​at intervals of a first preset number of frames to obtain a first target two-dimensional joint matrix; the first target two-dimensional joint matrix is ​​input into the first channel network to obtain the probability values ​​of the first motion feature and each human behavior, and the maximum probability value is screened out from all the probability values, and it is determined whether the maximum probability value is greater than a preset confidence threshold; if the maximum probability value is less than or equal to the preset confidence threshold, the matrix is ​​re-sliced ​​at a second preset number of frames to obtain a second target two-dimensional joint matrix; the second target two-dimensional joint matrix is ​​input into the second channel network to obtain a second motion feature; the first motion feature and the second motion feature are connected, and the connection result is used as the input of the fully connected classifier.

[0122] In addition, in some embodiments, if the maximum probability value is greater than a preset confidence threshold, the first motion feature is directly used as an input of the fully connected classifier.

[0123] It should be pointed out that the first preset frame number is greater than the second preset frame number. The first channel network is the fast channel, and the second channel network is the slow channel. The slow channel is used to process slow motion features and pays more attention to spatial modeling capabilities. The convolution model of the slow network can use any pre-trained model as the backbone network; the fast channel is used to process fast motion features and pays more attention to the changes in timing information. Unlike the slow channel, it samples at a higher frame rate, which can make the fast channel lighter and more efficient. For the fast channel, time downsampling is avoided during feature extraction, and global pooling is only performed before the final classification. In this way, high-resolution features are pursued in the time dimension (enhanced time modeling capabilities), while the spatial capacity is moderately reduced in the spatial dimension.

[0124] In addition, since the traditional fast and slow network fuses two paths through lateral connections, the computational cost of the dual-stream network that simultaneously processes fast and slow visual rhythms is very high. Based on this, a cascaded fast and slow network is proposed to simulate the visual information transmitted step by step when the human brain perceives action. First, the input features are sampled at a high frequency to obtain fast motion matrix slices, which are input into the fast channel network to obtain the corresponding fast motion feature matrix. A Softmax layer is connected after the fully connected layer of the recognition network, where the activation function of Softmax is:

[0125]

[0126] It represents the predicted probability of category i after applying the Softmax function to the neural network output Zi. The sum of the probability values ​​of all categories is 1. The category refers to human behavior, such as walking, running, drinking water, etc. Represents the i-th element in Z, where Z represents the array of outputs of multiple neurons in the previous layer.

[0127] Through the Softmax layer, we can get the probability value of each category, take the maximum confidence value, and then compare the obtained confidence with the pre-defined threshold. If it is greater than the threshold, the final recognition result is obtained directly. Otherwise, the input matrix is ​​sampled at a high frequency, and the slow motion feature is obtained through the slow channel network. The previous fast motion feature is fused into the slow channel, and the two features are connected and used as the input of the fully connected classifier to obtain the final recognition result.

[0128] In summary, compared with the prior art, the embodiments of the present invention have the following advantages:

[0129] 1. In order to simulate the motion dependency between physically connected joints, a tree-like skeleton model is constructed to replace the traditional skeleton sequence model with a fixed order of serial joints to represent the spatial information of the skeleton. The connection relationship between any two joints in the tree-like skeleton model represents the spatial dependency between the joints. The fewer edges, the more relevant the joint connections are, so that more spatial relationships can be retained. Then, a traversal sequence is constructed through the tree-like skeleton model to find a good sequence chain that can help the model obtain more spatial semantic information during feature extraction. In addition, a large-scale three-dimensional floating-point matrix is ​​constructed to store three-dimensional skeleton data. According to the basic physical motion information, the motion features of each joint are extracted frame by frame. The physical motion information is encoded using a special arrangement strategy to provide suitable large-scale input data for the CNN model, thereby providing rich spatiotemporal information for the human behavior recognition model.

[0130] 2. By setting the first preset frame number and the second preset frame number, the three-dimensional floating-point matrix is ​​sampled at different time frequencies to fully utilize the feature information of different time scales. At the same time, the first preset frame number is greater than the second preset frame number, that is, two fast and slow channels are constructed, and the dual-branch method of fast and slow motion is used to capture time and space information. In addition, by setting up this cascaded dual-branch network, the computational complexity can be effectively reduced, which improves the computational efficiency without affecting the recognition accuracy.

[0131] like Figure 4 As shown, an embodiment of the present invention further provides a human behavior recognition system based on a three-dimensional floating point joint matrix, the system comprising:

[0132] The model construction module 10 is used to obtain a skeleton sequence model of the prototyping human body, wherein the skeleton sequence model includes a plurality of joints and a connection relationship between any adjacent joints, and select a plurality of target joints from the joints in the skeleton sequence model, and construct a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, wherein the tree skeleton model includes a plurality of joints and a connection relationship between any two adjacent joints, and the first-level joint is the target joint;

[0133] A traversal sequence generation module 20 is used to perform a depth-first traversal of the tree skeleton model with the target joint point as the starting point and according to the connection relationship between any two adjacent joint points to obtain a traversal sequence corresponding to each target joint point, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship;

[0134] A three-dimensional matrix construction module 30 is used to obtain monitoring videos of the target human body, and extract information frame by frame from the monitoring videos to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation according to the frame order to obtain a three-dimensional floating-point joint matrix;

[0135] The training module 40 is used to input the three-dimensional floating-point joint matrix into a double-branch long-short cascade network for training to obtain a human behavior recognition model.

[0136] On the other hand, the present invention further proposes a storage medium on which one or more programs are stored, and when the programs are executed by a processor, the above-mentioned human behavior recognition method based on the three-dimensional floating-point joint matrix is ​​implemented.

[0137] On the other hand, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the above-mentioned human behavior recognition method based on three-dimensional floating-point joint matrix.

[0138] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0139] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0140] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0141] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A human behavior recognition method based on a three-dimensional floating point joint matrix, characterized in that: The method comprises: Acquire a skeleton sequence model of a human body, the skeleton sequence model comprising a plurality of joints and a connection relationship between any adjacent joints, select a plurality of target joints from the joints in the skeleton sequence model, and construct a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, the tree skeleton model comprising a plurality of joints and a connection relationship between any two adjacent joints, and the first-level joint is the target joint; Taking the target joint point as the starting point, and performing a depth-first traversal on the tree skeleton model according to the connection relationship between any two adjacent joint points, a traversal sequence corresponding to each target joint point is obtained, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship; Acquire a monitoring video of the target human body, and extract information frame by frame from the monitoring video to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation in accordance with the frame order to obtain a three-dimensional floating-point joint matrix; The three-dimensional floating-point joint matrix is ​​input into a dual-branch long-short cascade network for training to obtain a human behavior recognition model.

2. The human behavior recognition method based on a three-dimensional floating point joint matrix according to claim 1, characterized in that: The step of selecting a plurality of target joint points from the joint points in the skeleton sequence model comprises: The left shoulder, right shoulder, left hip, right hip, upper spine, lower spine and head of the human body are selected as target joints; The physical motion information includes relative position, instantaneous displacement, cumulative total displacement, speed, instantaneous acceleration, cumulative acceleration, joint angle, and instantaneous motion direction between frames.

3. The human behavior recognition method based on three-dimensional floating point joint matrix according to claim 2, characterized in that: Get the relative position according to the following formula: ； in, represents the relative position of the mth joint point of the jth frame image and the mth joint point of the first frame image, The x-axis coordinate, y-axis coordinate, and z-axis coordinate represent the relative position of the m-th joint point in the j-th frame image and the m-th joint point in the first frame image, respectively. They represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-th frame image, respectively. Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the first frame image; The instantaneous displacement is calculated according to the following formula: ； in, represents the instantaneous displacement between the mth joint point of the jth frame image and the mth joint point of the j-1th frame image, Respectively represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the m-th joint point in the j-1-th frame image; The cumulative total displacement is calculated according to the following formula: ； in, represents the cumulative total displacement of the mth joint point in the jth frame image; The speed is calculated according to the following formula: ； in, represents the velocity of the mth joint point in the jth frame image, Indicates the time interval between adjacent frame images; The instantaneous acceleration is calculated according to the following formula: ； The cumulative acceleration is calculated according to the following formula: ； in, represents the instantaneous acceleration of the mth joint point in the jth frame image, represents the velocity of the mth joint point in the j-1th frame image, Represents the velocity of the mth joint point in the first frame image, Represents the cumulative acceleration of the mth joint point in the jth frame image; The joint angle is formed by the space vector formed by three adjacent joint points, and the joint angle is calculated according to the following formula: ； in, represents the cosine value of the joint angle with the mth joint point in the jth frame image as the vertex, represents the spatial vector formed by the mth joint point in the jth frame image and the kth joint point in the jth frame image, Represents the spatial vector formed by the mth joint point in the jth frame image and the nth joint point in the jth frame image; When calculating the joint angle, the mth joint point is connected to the jth joint point and the nth joint point respectively; The instantaneous motion direction between frames is calculated according to the following formula: ； in, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xy plane, represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane, It represents the projection of the instantaneous motion direction of the mth joint point in the jth frame image on the xz plane. The physical motion information of all joint points in the first frame image is 0.

4. The human behavior recognition method based on a three-dimensional floating point joint matrix according to claim 3, characterized in that: The step of associating the physical operation information with each joint point in the traversal sequence according to the joint point number, and performing frame-by-frame concatenation according to the frame sequence to obtain a three-dimensional floating-point joint matrix comprises: The number of features of physical motion information is the number of columns, the total number of joint points in the traversal sequence is the number of rows, the feature name of the physical motion information is the column name, and the number of the joint point in the traversal sequence is the row name; According to the column name, all physical motion information of the cth joint point in the traversal sequence corresponding to the ath target joint point under the bth frame image is filled in the bth row, and a two-dimensional unit joint matrix corresponding to the ath target joint point under the bth frame image is obtained; Define the order of each target joint point, and generate a two-dimensional joint matrix from all two-dimensional unit joint matrices in the same frame according to the order of each target joint point; The two-dimensional joint matrix is ​​concatenated frame by frame in the frame sequence to obtain a three-dimensional floating-point joint matrix.

5. The human behavior recognition method based on three-dimensional floating point joint matrix according to claim 4, characterized in that: Construct a two-dimensional unit joint matrix according to the following formula: ； in, represents the two-dimensional unit joint matrix corresponding to the a-th target joint point in the b-th frame image, Represents the Kth physical motion information of the Cth joint point in the traversal sequence; Construct a two-dimensional joint matrix according to the following formula: ； in, represents the two-dimensional joint matrix corresponding to the b-th frame image, , They respectively represent the two-dimensional unit joint matrices corresponding to the first, second, third, fourth, fifth, sixth, seventh, and eighth target joint points in the b-th frame image.

6. The human behavior recognition method based on three-dimensional floating point joint matrix according to claim 4 is characterized in that: The step of inputting the three-dimensional floating-point joint matrix into a dual-branch long-short cascade network for training to obtain a human behavior recognition model comprises: Taking the two-dimensional joint matrix corresponding to the first frame image as a sampling starting point, the three-dimensional floating-point joint matrix is ​​sliced ​​at intervals of a first preset number of frames to obtain a first target two-dimensional joint matrix; Inputting the first target two-dimensional joint matrix into the first channel network, obtaining the probability values ​​of the first motion feature and each human behavior, and screening out the maximum probability value from all the probability values, and determining whether the maximum probability value is greater than a preset confidence threshold; If the maximum probability value is less than or equal to a preset confidence threshold, re-slicing the matrix with a second preset number of frames to obtain a second target two-dimensional joint matrix; Inputting the second target two-dimensional joint matrix into a second channel network to obtain a second motion feature; The first motion feature and the second motion feature are connected, and the connection result is used as the input of the fully connected classifier.

7. The human behavior recognition method based on three-dimensional floating point joint matrix according to claim 6 is characterized in that: After the step of determining whether the maximum probability value is greater than a preset confidence threshold, the following step is further performed: If the maximum probability value is greater than a preset confidence threshold, directly using the first motion feature as an input of the fully connected classifier; The first preset frame number is greater than the second preset frame number.

8. A human behavior recognition system based on a three-dimensional floating point joint matrix, characterized in that: The system comprises: A model building module, used to obtain a skeleton sequence model of the prototyping human body, the skeleton sequence model includes multiple joints and a connection relationship between any adjacent joints, and select multiple target joints from the joints in the skeleton sequence model, and build a tree skeleton model corresponding to each target joint according to the connection relationship between any adjacent joints, the tree skeleton model includes multiple levels of joints and a connection relationship between any two adjacent levels of joints, and the first level of joints is the target joint; A traversal sequence generation module, used to take the target joint point as a starting point and perform a depth-first traversal on the tree skeleton model according to the connection relationship between any two adjacent joint points, to obtain a traversal sequence corresponding to each target joint point, wherein the traversal sequence is a sequence including the traversal order of the joint points, and any two nodes in the adjacent order in the traversal sequence have a connection relationship; A three-dimensional matrix construction module is used to obtain monitoring videos of the target human body, and extract information frame by frame from the monitoring videos to obtain physical motion information about each joint point in each frame image, and associate the physical motion information with each joint point in the traversal sequence according to the joint point number, and perform frame-by-frame concatenation according to the frame order to obtain a three-dimensional floating-point joint matrix; The training module is used to input the three-dimensional floating-point joint matrix into a double-branch long-short cascade network for training to obtain a human behavior recognition model.

9. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the human behavior recognition method based on a three-dimensional floating-point joint matrix as described in any one of claims 1-7.

10. An electronic device, comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the human behavior recognition method based on the three-dimensional floating-point joint matrix as described in any one of claims 1-7.

Images