Human behavior recognition method and electronic device based on deep spatiotemporal reasoning network

Abstract

The present invention discloses a method and electronic device for human behavior recognition based on a deep spatiotemporal reasoning network, the method comprising: inputting target video data into a pre-trained deep spatiotemporal reasoning network model, determining the first activation times corresponding to the clustering centers of each node in the deep spatiotemporal reasoning network model; constructing a first behavior feature tree according to the first activation times; obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm; and recognizing human behavior in the target video data according to the first frequent subtree set. The present invention recognizes behavior by using the spatiotemporal features of human dynamic behavior extracted by the deep spatiotemporal reasoning network and the feature relationships of different levels reflected by the frequent subtrees, focusing not only on the inherent features of the human body parts in the behavior, but also on the association and coordination between the parts, thereby improving the feature utilization rate in the network, increasing the recognition rate of the network, and realizing fast and accurate recognition of human behavior in video data.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition technology, and in particular to a human behavior recognition method and electronic equipment based on a deep spatiotemporal reasoning network. Background Art

[0002] With the development of computer video surveillance technology, intelligent monitoring systems are widely used in public places such as traffic intersections, airports, shopping malls, parking lots, and commercial office buildings. Among them, human behavior recognition has become one of the most challenging problems in the field of intelligent monitoring. Its goal is to enable computers to recognize human behavior based on human information, especially motion information.

[0003] According to the different recognition methods, the current methods for behavior recognition can be divided into two categories: traditional methods and deep learning methods. Traditional methods mainly include methods based on template matching, state space, and semantic description. Traditional methods need to build recognition models based on algorithms and feature engineering for specific fields and specific applications, and require complex feature engineering to extract human behavior characteristics. In recent years, human behavior recognition methods based on deep learning have attracted more and more attention from researchers and the industry, but existing human behavior recognition methods based on deep learning can only recognize still images, and cannot recognize human behavior in video data.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a human behavior recognition method and electronic device based on a deep spatiotemporal reasoning network, aiming to solve the problem that the existing human behavior recognition method based on deep learning can only recognize still pictures but cannot recognize human behavior in video data.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for human behavior recognition based on a deep spatiotemporal reasoning network, which includes:

[0008] Inputting the target video data into a pre-trained deep spatiotemporal reasoning network model, and determining a first activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model; wherein the target video data includes a plurality of human behaviors;

[0009] Constructing a first behavior feature tree according to the first activation number;

[0010] Obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm;

[0011] Human behaviors in the target video data are identified according to the first frequent subtree set.

[0012] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of inputting the target video data into a pre-trained deep spatiotemporal reasoning network model and determining the first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model comprises:

[0013] Input the target video data into a pre-trained deep spatiotemporal reasoning network model, and obtain a first observation value corresponding to each node in the deep spatiotemporal reasoning network model;

[0014] According to the first observation value, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model is determined.

[0015] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of determining, based on the first observation value, a first activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model comprises:

[0016] Determine the cluster to which the first observation value belongs according to the first observation value corresponding to each node and the cluster center of each node;

[0017] According to the cluster to which the first observation value belongs, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model is determined.

[0018] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of constructing a first behavior feature tree according to the first activation number includes:

[0019] Determining a target cluster center of each node according to the first activation number;

[0020] A first behavior feature tree is constructed according to the target cluster center.

[0021] In one implementation, in the human behavior recognition method based on a deep spatiotemporal reasoning network, the step of obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm comprises:

[0022] Generating a plurality of first candidate subtrees from the first behavior feature tree based on the rightmost expansion strategy, and determining the support numbers of the plurality of first candidate subtrees;

[0023] A first frequent subtree set is determined according to the support counts of the first candidate subtrees and a preset support count threshold.

[0024] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of identifying human behavior in the target video data according to the first frequent subtree set includes:

[0025] Determine a target first frequent subtree according to the first frequent subtree set; wherein the target first frequent subtree is a first frequent subtree with the highest frequency in the first frequent subtree set;

[0026] According to the correspondence between the target first frequent subtree and the predetermined frequent subtree and the human behavior, the human behavior in the target video data is identified.

[0027] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of inputting the target video data into a pre-trained deep spatiotemporal reasoning network model, comprises:

[0028] Using a training video data set to train a pre-built deep spatiotemporal reasoning network model to obtain a pre-trained deep spatiotemporal reasoning network model; wherein the training video data set includes a plurality of training video data, and the plurality of training video data includes a plurality of human behaviors;

[0029] Inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network to determine frequent subtrees corresponding to a number of human behaviors in the training video data set;

[0030] According to the frequent subtrees corresponding to the plurality of human behaviors in the training video data set, the corresponding relationship between the frequent subtrees and the human behaviors is determined.

[0031] In one implementation, the method for human behavior recognition based on a deep spatiotemporal reasoning network, wherein the step of inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model, and determining frequent subtrees corresponding to a number of human behaviors in the training video data set comprises:

[0032] Inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model, and determining a second activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model;

[0033] constructing a second behavior feature tree according to the second activation number;

[0034] Obtaining a second frequent subtree set from the second behavior feature tree using a frequent subtree mining algorithm;

[0035] According to the second frequent subtree set, frequent subtrees corresponding to a plurality of human behaviors in the training video data set are determined.

[0036] In the second aspect, an embodiment of the present invention also provides an intelligent terminal, which includes: a processor, a storage medium communicatively connected to the processor, the storage medium being suitable for storing multiple instructions; the processor being suitable for calling the instructions in the storage medium to execute the steps of implementing the above-mentioned human behavior recognition method based on deep spatiotemporal reasoning network.

[0037] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to perform the steps of implementing the above-mentioned method for human behavior recognition based on a deep spatiotemporal reasoning network.

[0038] Beneficial effect: When identifying human behaviors in target video data, the present invention utilizes the spatiotemporal features of human dynamic behaviors extracted by the deep spatiotemporal reasoning network model and the feature relationships of different levels reflected by the frequent subtrees to perform behavior recognition, focusing not only on the inherent characteristics of human body parts in the behavior, but also on the association and coordination between the parts, thereby improving the feature utilization rate in the network, improving the recognition rate of the network, and realizing fast and accurate recognition of human behaviors in video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of an embodiment of a method for human behavior recognition based on a deep spatiotemporal reasoning network provided by an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the structure of a deep spatiotemporal reasoning network provided by an embodiment of the present invention;

[0041] Figure 3 is a frequent subtree graph corresponding to the case where the human behavior is waving in the embodiment of the present invention;

[0042] Figure 4 is a frequent subtree graph corresponding to when the human behavior is jumping in the embodiment of the present invention;

[0043] Figure 5 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The human behavior recognition method based on deep spatiotemporal reasoning network provided by the present invention can be applied to smart terminals. Among them, smart terminals can be but are not limited to various personal computers, laptops, mobile phones, tablet computers, car computers and portable wearable devices. The terminal of the present invention adopts a multi-core processor. Among them, the processor of the smart terminal can be at least one of a central processing unit (CPU), a graphics processing unit (GPU), a video processing unit (VPU), etc.

[0047] Exemplary Methods

[0048] In order to solve the problem that the existing human behavior recognition method based on deep learning can only recognize still pictures but cannot recognize human behavior in video data, the present invention provides a human behavior recognition method based on a deep spatiotemporal reasoning network.

[0049] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of a human behavior recognition method based on a deep spatiotemporal reasoning network provided by the present invention.

[0050] In one embodiment of the present invention, the human behavior recognition method based on deep spatiotemporal reasoning network has four steps:

[0051] S100. Input the target video data into a pre-trained deep spatiotemporal reasoning network model, and determine the first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model; wherein the target video data includes a number of human behaviors.

[0052] Specifically, the target video data in this embodiment is video data that needs to be recognized for human behavior, and the target video data includes several human behaviors, such as waving, jumping, etc. The target video data can be collected by an existing device capable of capturing depth data of human behavior, such as a Kinect.

[0053] In this embodiment, in order to recognize human behavior in the target video data, a deep spatiotemporal inference network (DeSTIN network) model is pre-constructed. The structure of the deep spatiotemporal inference network model is as follows: Figure 2 As shown in the figure, the deep spatiotemporal reasoning network model is a multi-layer pyramid-shaped deep network model. Each layer is divided into several 2*2 areas, and a 2*2 area in the N-1th layer is connected to a node in the Nth layer, and the four nodes in the Nth layer are connected to a node in the N+1th layer. In the hidden layer (middle layer) of the deep spatiotemporal reasoning network model, the nodes in each layer contain a certain number of cluster centers.

[0054] Before identifying human behavior in target video data, the deep spatiotemporal reasoning network model needs to be trained in advance. The nodes in the deep spatiotemporal reasoning network model can generate and learn the features of the observation values ​​of this layer through unsupervised learning, and initialize the update step size of the clustering center to obtain a pre-trained deep spatiotemporal reasoning network model.

[0055] Before inputting the target video data into the pre-trained deep spatiotemporal reasoning network model, the target video data needs to be pre-processed first, and the image size in the target video data needs to be normalized, for example, the image size needs to be normalized to 320*320. Then the pre-processed target video data is input into the pre-trained deep spatiotemporal reasoning network model, and the first activation times corresponding to the clustering centers in each node of the deep spatiotemporal reasoning network model are determined. The activation times of the clustering centers in each node of the deep spatiotemporal reasoning network model reflect the spatiotemporal characteristics of human dynamic behavior. In this embodiment, by obtaining the first activation times, human behavior in the target video data can be identified.

[0056] In a specific implementation, step S100 includes the following steps:

[0057] S110, inputting the target video data into a pre-trained deep spatiotemporal reasoning network model, and obtaining a first observation value corresponding to each node in the deep spatiotemporal reasoning network model;

[0058] S120. Determine, based on the first observation value, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model.

[0059] For each layer of the deep spatiotemporal reasoning network model (except the bottom layer), a set of belief values ​​passed from the sublayers are obtained as observation values. After the target video data is input into the pre-trained deep spatiotemporal reasoning network model, at the bottom layer of the deep spatiotemporal reasoning network model, the target video data is directly input as the observation value. Each layer of the deep spatiotemporal reasoning network model can be regarded as an abstraction of the target video data at a certain level. For each layer of the deep spatiotemporal reasoning network model (except the bottom layer), a set of belief values ​​passed from the sublayers are obtained as observation values, that is, the first observation value corresponding to each node of each layer of the deep spatiotemporal reasoning network model is obtained.

[0060] Taking into account that the activation times of cluster centers corresponding to observation values ​​of different clusters are different, in this embodiment, after obtaining the first observation value corresponding to each node of each layer of the deep spatiotemporal reasoning network model, the Euclidean distance from the first observation value corresponding to each node of each layer of the deep spatiotemporal reasoning network model to all cluster centers in the node is calculated, and the cluster to which the first observation value belongs (the activated cluster) is determined according to the calculated Euclidean distance, and then the first activation times of cluster centers in each node of the deep spatiotemporal reasoning network model are determined according to the cluster to which the first observation value belongs, that is, the number of times N that the kth cluster of the jth node in the i-th layer is activated is updated according to the cluster to which the first observation value belongs. ijk ←N ijk +1.

[0061] S200: Construct a first behavior feature tree according to the first activation times.

[0062] Considering that frequent subtrees can reflect the characteristic relationships at different levels of human behavior, in this embodiment, after obtaining the first activation times corresponding to the cluster centers in each node of the deep spatiotemporal reasoning network model, a first behavior feature tree is constructed according to the first activation times. Among them, the height of the first behavior feature tree is the number of layers of the deep spatiotemporal reasoning network model, the nodes of the first behavior feature tree are the nodes of the deep spatiotemporal reasoning network model, and the values ​​of each node in the first behavior feature tree are determined according to the first activation times corresponding to the cluster centers in each node.

[0063] In a specific implementation, step S200 includes the following steps:

[0064] S210, determining a target cluster center of each node according to the first activation number;

[0065] S220: Construct a first behavior feature tree according to the target cluster center.

[0066] Considering that each node of the deep spatiotemporal reasoning network model contains multiple cluster centers, and each cluster center has its corresponding first activation number. In this embodiment, after obtaining the first activation number corresponding to the cluster center in each node of the deep spatiotemporal reasoning network model, first determine the cluster center that is most frequently activated in each node, that is, the cluster center with the largest first activation number, as the target cluster center of each node based on the first activation number. Then calculate the first activation frequency of the target cluster center of each node. Among them, the first activation frequency = max k N ijk / ∑ k N ijk , where max k N ijk is the first activation number corresponding to the target cluster center of the k-th cluster of the j-th node in the i-th layer, ∑ k N ijk It is the sum of the first activation times of all cluster centers of the kth cluster of the jth node in the i-th layer. After calculating the first activation frequency of the target cluster center of each node, the first activation frequency is used as the node value of the tree corresponding to each node to construct the first behavior feature tree.

[0067] S300: Obtain a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm.

[0068] Since the data structure of the frequent subtree mining algorithm (TreeMiner algorithm) is simple, easy to implement, and can uniquely represent a tree, it can improve the design effect of the algorithm. After constructing the first behavior feature tree in this embodiment, the frequent subtree mining algorithm is used to obtain the first frequent subtree set from the first behavior feature tree, so that the human behavior in the target video data can be identified according to the first frequent subtree set in the subsequent steps. In this embodiment, when identifying the human behavior of the target video data, the spatiotemporal features of the human dynamic behavior extracted by the deep spatiotemporal reasoning network and the different levels of feature relationships reflected by the frequent subtrees are used to perform behavior recognition, focusing on both the characteristics of the human body parts in the behavior and the association and coordination between the parts, thereby improving the feature utilization rate in the network model, improving the recognition rate of the network model, and realizing the rapid and accurate recognition of human behavior in the video data.

[0069] In a specific implementation, step S300 specifically includes:

[0070] S310, generating a plurality of first candidate subtrees from the first behavior feature tree based on the rightmost expansion strategy, and determining the support numbers of the plurality of first candidate subtrees;

[0071] S320: Determine a first frequent subtree set according to the support counts of the first candidate subtrees and a preset support count threshold.

[0072] Specifically, in this embodiment, when the frequent subtree mining algorithm is used to obtain the first frequent subtree set, a plurality of first candidate subtrees are first generated from the first behavior feature tree based on the rightmost expansion strategy, and the support number of the plurality of first candidate subtrees is determined. Then, the support number of the plurality of first candidate subtrees determined is compared with a preset support number threshold, and a plurality of first candidate subtrees whose support number is greater than the preset support number threshold are screened out from the plurality of first candidate subtrees as the first frequent subtree set, so that the human behavior in the target video data can be determined according to the first frequent subtree set in the subsequent steps.

[0073] S400: Identify human behaviors in the target video data according to the first frequent subtree set.

[0074] Specifically, in this embodiment, the correspondence between frequent subtrees and human behaviors is predetermined. After the first frequent subtree set corresponding to the target video data is obtained, the human behaviors in the human behavior video data are determined according to the first frequent subtree set and the predetermined correspondence between frequent subtrees and human behaviors.

[0075] In a specific implementation, the step S400 includes the following steps:

[0076] S410: Determine a target first frequent subtree according to the first frequent subtree set; wherein the target first frequent subtree is a first frequent subtree with the highest frequency in the first frequent subtree set;

[0077] S420 : Identify human behaviors in the target video data according to the correspondence between the target first frequent subtree and the predetermined frequent subtree and human behaviors.

[0078] As can be seen from the above steps, the first frequent subtree set includes several first frequent subtrees whose support number is greater than the preset support number threshold. In this embodiment, when human behavior is identified according to the first frequent subtree set, each frequent subtree in the first frequent subtree set is first counted, and the first frequent subtree with the highest frequency, i.e., the largest count value, in the first frequent subtree set is determined, and the first frequent subtree is used as the target first frequent subtree. Then, the target first frequent subtree is compared with the predetermined correspondence between the frequent subtree and the human behavior, and the human behavior corresponding to the target first frequent subtree in the correspondence between the frequent subtree and the human behavior is determined to be the human behavior in the target video data, thereby realizing the identification of the human behavior in the target video data. For example, in the predetermined correspondence between the frequent subtree and the human behavior, the first frequent subtree corresponds to waving, and the second frequent subtree corresponds to jumping. After the first frequent subtree is obtained, it can be determined that the target human behavior corresponding to the first frequent subtree is waving, that is, the human behavior in the human behavior video data is identified as waving.

[0079] In a specific implementation, the following steps are included before step S100:

[0080] S01. Using a training video data set to train a pre-built deep spatiotemporal reasoning network model to obtain a pre-trained deep spatiotemporal reasoning network model; wherein the training video data set includes a plurality of training video data, and the plurality of training video data includes a plurality of human behaviors;

[0081] S02, inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network, and determining frequent subtrees corresponding to a number of human behaviors in the training video data set;

[0082] S03. Determine a corresponding relationship between frequent subtrees and human behaviors according to the frequent subtrees corresponding to the human behaviors in the training video data set.

[0083] Specifically, before the present embodiment recognizes the human behavior in the target video data, it is necessary to pre-train the deep spatiotemporal reasoning network model. When training the deep spatiotemporal reasoning network model, first use a device such as Kinect to capture the depth data of the human behavior and collect the video data of the human behavior, and pre-process the collected video data, such as darkening the collected video data, adding Gaussian noise and salt and pepper noise, etc. to enhance the sample data set, so that the trained deep spatiotemporal reasoning network model has generalization ability and strong robustness. In a specific embodiment, the parameter is 0.9 for darkening, adding 30% Gaussian noise, adding 30% salt and pepper noise, etc. to enhance the sample data. Then the image size of the collected video data is normalized, for example, the image size of the collected video data is normalized to 320*320, and the collected video data is divided into a training video data set and a test video data set according to the proportion, so that the deep spatiotemporal reasoning network model can be trained and tested in the subsequent steps.

[0084] When training the deep spatiotemporal reasoning network model, the training video data in the processed training video data set is input frame by frame into the pre-built deep spatiotemporal reasoning network model. At the bottom layer of the deep spatiotemporal reasoning network model, the training video data is directly input as the observation value. For each layer of the deep spatiotemporal reasoning network model (except the bottom layer), a set of belief values ​​transmitted from the sublayer is obtained as the observation value, and the cluster center is updated by an unsupervised learning method, and then online clustering is performed based on the updated cluster center. While performing online clustering, the deep spatiotemporal reasoning network model feeds back the winning node information (PSSA) of the parent layer clustering to the current layer, updates the cluster center of the current layer and recalculates the belief value, and uses this belief value as the output and transmits it to the parent layer as the observation value of the parent layer, until the preset number of iterations is reached or the change of the cluster center meets the preset threshold, and the pre-trained deep spatiotemporal reasoning network model is obtained.

[0085] In this embodiment, when training the deep spatiotemporal reasoning network model, an unsupervised learning method is used to cluster the observations of each layer. The unsupervised learning method is an online unsupervised clustering method - competitive learning (winner take all, WTA). WTA only adjusts the cluster center that is most similar (closest distance) to the new sample, and other class properties (the location of the cluster center) that are not related to the sample are preserved, thereby improving the training efficiency and robustness of the deep spatiotemporal reasoning network model. The update step size of the cluster center can be set according to user needs. In a specific embodiment, the update step size of the cluster center is set to 0.1.

[0086] After the deep spatiotemporal reasoning network model training is completed, the update of the cluster center of each layer of the network and the information feedback of the parent layer to the current layer are closed, and the number of activations of the k-th cluster center of the j-th node in the i-th layer is initialized N ijk Then, the training video data in the training video dataset is re-input into the trained deep spatiotemporal reasoning network model to determine the frequent subtrees corresponding to several human behaviors in the training video dataset. Finally, according to the frequent subtrees corresponding to several human behaviors in the training video dataset, the corresponding relationship between the frequent subtrees and the human behaviors is determined, so that the human behaviors in the target video data can be identified according to the corresponding relationship between the frequent subtrees and the human behaviors in the subsequent steps.

[0087] In a specific embodiment, step S02 includes the following steps:

[0088] S021, inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model, and determining a second activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model;

[0089] S022. Constructing a second behavior feature tree according to the second activation number;

[0090] S023. Obtain a second frequent subtree set from the second behavior feature tree using a frequent subtree mining algorithm;

[0091] S024. Determine, according to the second frequent subtree set, frequent subtrees corresponding to a plurality of human behaviors in the training video data set.

[0092] Specifically, in this embodiment, when determining the frequent subtrees corresponding to several human behaviors in the training video data set, the training video data in the training video data set is first input into a pre-trained deep spatiotemporal reasoning network model to determine the second activation times corresponding to the cluster centers in each node of the deep spatiotemporal reasoning network model. Then, the second activation frequency corresponding to the most frequently activated cluster center in each node is determined based on the second activation times. The second behavior feature tree is constructed with the number of layers of the deep spatiotemporal reasoning network model as the tree height, the nodes of the deep spatiotemporal reasoning network model as the tree nodes, and the second activation frequency as the node value.

[0093] Similar to the mining step of the aforementioned first behavior feature tree, after constructing the second behavior feature tree, a frequent subtree mining algorithm is used to obtain a second frequent subtree set from the second behavior feature tree, and several second frequent subtrees with the highest frequency are obtained from the second frequent subtree set as frequent subtrees corresponding to several human behaviors in the training video data set, so as to determine the frequent subtrees corresponding to several human behaviors in the training video data set in subsequent steps.

[0094] In a specific embodiment, the inventor uses Kinect to collect a human behavior video dataset, which includes two common human behaviors, waving and jumping. Each behavior has 20 videos, each video is about 10 seconds long, and a total of 80 videos. After data enhancement and normalization, the collected video dataset is divided into a training video dataset and a test video dataset in a ratio of 80%:20%. Then, the following is constructed: Figure 2 The 8-layer deep spatiotemporal reasoning network model shown in the figure has nodes of each layer from bottom to top of 128×128, 64×64, 32×32, 16×16, 8×8, 4×4, 2×2, 1, and the corresponding number of cluster centers in each layer is 2, 2, 4, 8, 32, 16, 8, 3. Using the above-mentioned human behavior recognition method based on deep spatiotemporal reasoning network, the training video data in the training video data set is input into the deep spatiotemporal reasoning network model to train the deep spatiotemporal reasoning network model, and the test video data in the test video data set is input into the trained deep spatiotemporal reasoning network model, and the trained deep spatiotemporal reasoning network model is used to recognize the human behavior in the test video data set, and the following is obtained. Figure 3 and Figure 4 The frequent subtrees corresponding to the two human behaviors of waving and jumping are shown, and the recognition accuracy of the human behavior recognition method based on the deep spatiotemporal reasoning network of the present invention for the human behavior in the test data set reaches more than 85%.

[0095] Exemplary Devices

[0096] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen and a temperature sensor connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human behavior recognition method based on a deep spatiotemporal reasoning network is implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the terminal is pre-set inside the device to detect the current operating temperature of the internal device.

[0097] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In one embodiment, a smart terminal is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, at least the following steps can be implemented:

[0099] Inputting the target video data into a pre-trained deep spatiotemporal reasoning network model, and determining a first activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model; wherein the target video data includes a plurality of human behaviors;

[0100] Constructing a first behavior feature tree according to the first activation number;

[0101] Obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm;

[0102] Human behaviors in the target video data are identified according to the first frequent subtree set.

[0103] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the target video data into a pre-trained deep spatiotemporal reasoning network model to obtain a first observation value corresponding to each node in the deep spatiotemporal reasoning network model; and determining a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model based on the first observation value.

[0104] In one embodiment, when the processor executes the computer program, it can also achieve: determining the cluster to which the first observation value belongs based on the first observation value corresponding to each node and the cluster center of each node; determining the first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model based on the cluster to which the first observation value belongs.

[0105] In one embodiment, when the processor executes the computer program, it can also achieve: determining the target cluster center of each node according to the first activation times; and constructing a first behavior feature tree according to the target cluster center.

[0106] In one embodiment, when the processor executes the computer program, it can also achieve: generating several first candidate subtrees from the first behavior feature tree based on the rightmost expansion strategy, and determining the support numbers of the several first candidate subtrees; determining the first frequent subtree set according to the support numbers of the several first candidate subtrees and a preset support number threshold.

[0107] In one embodiment, when the processor executes the computer program, it can also achieve: determining a target first frequent subtree according to the first frequent subtree set; wherein the target first frequent subtree is the first frequent subtree with the highest frequency in the first frequent subtree set; and identifying human behavior in the target video data according to the correspondence between the target first frequent subtree and the predetermined frequent subtree and human behavior.

[0108] In one embodiment, when the processor executes the computer program, it can also achieve: using a training video data set to train a pre-built deep spatiotemporal reasoning network model to obtain a pre-trained deep spatiotemporal reasoning network model; wherein the training video data set includes a number of training video data, and the number of training video data includes a number of human behaviors; inputting the training video data in the training video data set into the pre-trained deep spatiotemporal reasoning network to determine the frequent subtrees corresponding to the number of human behaviors in the training video data set; and determining the correspondence between the frequent subtrees and the human behaviors based on the frequent subtrees corresponding to the number of human behaviors in the training video data set.

[0109] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model, determining the second activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model; constructing a second behavior feature tree based on the second activation number; obtaining a second frequent subtree set from the second behavior feature tree using a frequent subtree mining algorithm; and determining frequent subtrees corresponding to several human behaviors in the training video data set based on the second frequent subtree set.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] In summary, the present invention discloses a method and electronic device for human behavior recognition based on a deep spatiotemporal reasoning network, the method comprising: inputting target video data into a pre-trained deep spatiotemporal reasoning network model, determining a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model; constructing a first behavior feature tree according to the first activation number; obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm; and recognizing human behavior in the target video data according to the first frequent subtree set. The present invention recognizes behavior through the spatiotemporal features of human dynamic behavior extracted by a deep spatiotemporal reasoning network and the feature relationships of different levels reflected by frequent subtrees, focusing not only on the inherent characteristics of human body parts in the behavior, but also on the association and coordination between parts, thereby improving the feature utilization rate in the network, increasing the recognition rate of the network, and realizing rapid and accurate recognition of human behavior in video data.

[0112] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A human behavior recognition method based on deep spatiotemporal reasoning network, characterized in that: include: Inputting the target video data into a pre-trained deep spatiotemporal reasoning network model, and determining a first activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model; wherein the target video data includes a plurality of human behaviors; Constructing a first behavior feature tree according to the first activation number; Obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm; According to the first frequent subtree set, identifying human behavior in the target video data; The constructing a first behavior feature tree according to the first activation number includes: Determine the target cluster center of each node according to the first activation number; Calculate the first activation frequency of the target cluster center of each node; Using the first activation frequency as a node value of a tree corresponding to each node, to construct a first behavior feature tree; Among them, the first activation frequency = max k N ijk / ∑ k N ijk , Among them, max k N ijk is the first activation number corresponding to the target cluster center of the k-th cluster of the j-th node in the i-th layer, ∑ k N ijk is the sum of the first activation times of all cluster centers of the k-th cluster of the j-th node in the i-th layer; The step of identifying human behavior in the target video data according to the first frequent subtree set comprises: Determine a target first frequent subtree according to the first frequent subtree set; wherein the target first frequent subtree is the first frequent subtree with the highest frequency in the first frequent subtree set; According to the correspondence between the target first frequent subtree and the predetermined frequent subtree and the human behavior, the human behavior in the target video data is identified.

2. The method for human behavior recognition based on deep spatiotemporal reasoning network according to claim 1, characterized in that: The step of inputting the target video data into a pre-trained deep spatiotemporal reasoning network model and determining the first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model comprises: Input the target video data into a pre-trained deep spatiotemporal reasoning network model, and obtain a first observation value corresponding to each node in the deep spatiotemporal reasoning network model; According to the first observation value, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model is determined.

3. The human behavior recognition method based on deep spatiotemporal reasoning network according to claim 2 is characterized in that: The step of determining, according to the first observation value, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model comprises: Determine the cluster to which the first observation value belongs according to the first observation value corresponding to each node and the cluster center of each node; According to the cluster to which the first observation value belongs, a first activation number corresponding to the cluster center of each node in the deep spatiotemporal reasoning network model is determined.

4. The human behavior recognition method based on deep spatiotemporal reasoning network according to claim 1 is characterized in that: The step of obtaining a first frequent subtree set from the first behavior feature tree using a frequent subtree mining algorithm comprises: Generating a plurality of first candidate subtrees from the first behavior feature tree based on the rightmost expansion strategy, and determining the support numbers of the plurality of first candidate subtrees; A first frequent subtree set is determined according to the support counts of the first candidate subtrees and a preset support count threshold.

5. The human behavior recognition method based on deep spatiotemporal reasoning network according to claim 1 is characterized in that: Before inputting the target video data into the pre-trained deep spatiotemporal reasoning network model, the following steps are also included: Using a training video data set to train a pre-built deep spatiotemporal reasoning network model to obtain a pre-trained deep spatiotemporal reasoning network model; wherein the training video data set includes a plurality of training video data, and the plurality of training video data includes a plurality of human behaviors; Inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network to determine frequent subtrees corresponding to a number of human behaviors in the training video data set; According to the frequent subtrees corresponding to the plurality of human behaviors in the training video data set, the corresponding relationship between the frequent subtrees and the human behaviors is determined.

6. The method for human behavior recognition based on deep spatiotemporal reasoning network according to claim 5, characterized in that: The step of inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model to determine frequent subtrees corresponding to a plurality of human behaviors in the training video data set comprises: Inputting the training video data in the training video data set into a pre-trained deep spatiotemporal reasoning network model, and determining a second activation number corresponding to a cluster center of each node in the deep spatiotemporal reasoning network model; constructing a second behavior feature tree according to the second activation number; Obtaining a second frequent subtree set from the second behavior feature tree using a frequent subtree mining algorithm; According to the second frequent subtree set, frequent subtrees corresponding to a plurality of human behaviors in the training video data set are determined.

7. An intelligent terminal, characterized in that: include: A processor and a storage medium communicatively connected to the processor, wherein the storage medium is suitable for storing multiple instructions; the processor is suitable for calling the instructions in the storage medium to execute the steps of the human behavior recognition method based on the deep spatiotemporal reasoning network as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to perform the steps of implementing the human behavior recognition method based on a deep spatiotemporal reasoning network as described in any one of claims 1 to 6 above.

Images