A behavior recognition method, device, and storage medium

Abstract

The present invention discloses a behavior recognition method, apparatus and storage medium. The method includes: obtaining data to be detected; the data to be detected is three-dimensional point cloud data; using a preset behavior recognition model to recognize the data to be detected to obtain a behavior result; the behavior recognition model is trained based on a neural network; the neural network adopts a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing context in different directions.

Description

Technical Field

[0001] The present invention relates to the field of networks, and in particular, to a behavior recognition method, apparatus, and storage medium. Background Art

[0002] Currently, for behavior recognition in a home environment, video surveillance is still mainly used. The video surveillance solution requires the use of behavior recognition algorithms, and the behavior recognition algorithms need to be analyzed based on a time series. The computing power required by the time series itself is relatively higher than that of a single timestamp. In order to optimize the model, the method is often to increase the depth, width, and cardinality of the network, all of which require more resources. And an overly large model will be difficult to run on the edge side and can only rely on cloud analysis. However, cloud analysis is not conducive to privacy protection. In the event of a data leakage risk, the user's identity information will be completely exposed. How to maintain the lightweight of the model while optimizing the model prediction effect is one of the existing problems. Summary of the Invention

[0003] In view of this, the main purpose of the present invention is to provide a behavior recognition method, apparatus, and storage medium.

[0004] To achieve the above object, the technical solution of the present invention is implemented as follows:

[0005] An embodiment of the present invention provides a behavior recognition method, the method including:

[0006] Obtain data to be detected; the data to be detected is three-dimensional point cloud data;

[0007] Use a preset behavior recognition model to recognize the data to be detected, and obtain a behavior result;

[0008] The behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing context in different directions.

[0009] In the above solution, the using a preset behavior recognition model to recognize the data to be detected and obtaining a behavior result includes:

[0010] According to the data to be detected, obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence; the first three-dimensional spatio-temporal sequence includes a horizontal direction heat map and a time dimension corresponding to the horizontal direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical direction heat map and a time dimension corresponding to the vertical direction heat map;

[0011] Extract a first direction feature according to the first three-dimensional spatio-temporal sequence, and extract a second direction feature according to the second three-dimensional spatio-temporal sequence;

[0012] Fuse the first direction feature and the second direction feature to obtain a target feature;

[0013] Decode and predict the target sample feature to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1;

[0014] Based on the three-dimensional positions of the N skeleton nodes, obtain the behavior result.

[0015] In the above solution, the method further includes: generating a preset behavior recognition model; the generating of the preset behavior recognition model includes:

[0016] Obtain a training sample set; the training sample set includes: at least one training sample and the behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data.

[0017] Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain the behavior recognition model.

[0018] In the above solution, the training of the neural network according to the at least one training sample and the behavior label corresponding to each training sample includes:

[0019] Obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and the time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and the time dimension corresponding to the sample vertical direction heat map;

[0020] Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

[0021] In the above solution, the neural network includes: a first neural network part, a second neural network part, and a fully connected layer;

[0022] The training of the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data includes:

[0023] The first neural network part extracts a first sample direction feature according to the first sample three-dimensional spatio-temporal sequence;

[0024] The second neural network part extracts a second sample direction feature according to the second sample three-dimensional spatio-temporal sequence;

[0025] Fuse the first sample direction feature and the second sample direction feature to obtain a target sample feature;

[0026] Use the fully connected layer to decode and predict the target sample feature to obtain the sample three-dimensional positions of N skeleton nodes; obtain a sample prediction result based on the sample three-dimensional positions of the N skeleton nodes;

[0027] Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

[0028] An embodiment of the present invention provides a behavior recognition device, and the device includes:

[0029] An acquisition module, configured to acquire data to be detected; the data to be detected is three-dimensional point cloud data;

[0030] An identification module, configured to identify the data to be detected by using a preset behavior recognition model to obtain a behavior result;

[0031] The behavior recognition model is trained based on a neural network; the neural network adopts a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing contexts in different directions.

[0032] In the above solution, the identification module is configured to obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence according to the data to be detected; the first three-dimensional spatio-temporal sequence includes a horizontal direction heat map and a time dimension corresponding to the horizontal direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical direction heat map and a time dimension corresponding to the vertical direction heat map;

[0033] Extract a first direction feature according to the first three-dimensional spatio-temporal sequence, and extract a second direction feature according to the second three-dimensional spatio-temporal sequence;

[0034] Fuse the first direction feature and the second direction feature to obtain a target feature;

[0035] Decode and predict the target sample feature to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1;

[0036] Obtain the behavior result based on the three-dimensional positions of the N skeleton nodes.

[0037] In the above solution, the device further includes: a preprocessing module, configured to obtain a training sample set; the training sample set includes: at least one training sample and a behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data.

[0038] Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain the behavior recognition model.

[0039] In the above solution, the preprocessing module is used to obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and a time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and a time dimension corresponding to the sample vertical direction heat map.

[0040] Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

[0041] In the above solution, the neural network includes: a first neural network part, a second neural network part, and a fully connected layer.

[0042] The preprocessing module is used to extract first sample direction features from the first sample three-dimensional spatio-temporal sequence through the first neural network part.

[0043] Extract second sample direction features from the second sample three-dimensional spatio-temporal sequence through the second neural network part.

[0044] Fuse the first sample direction features and the second sample direction features to obtain target sample features.

[0045] Use the fully connected layer to decode and predict the target sample features to obtain the sample three-dimensional positions of N skeleton nodes; obtain a sample prediction result based on the sample three-dimensional positions of the N skeleton nodes.

[0046] Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

[0047] An embodiment of the present invention provides a behavior recognition device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the behavior recognition method described in any one of the above are implemented.

[0048] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the behavior recognition method described in any one of the above are implemented.

[0049] A behavior recognition method, device, and storage medium provided by an embodiment of the present invention. The method includes: obtaining data to be detected; the data to be detected is three-dimensional point cloud data; using a preset behavior recognition model to recognize the data to be detected to obtain a behavior result; the behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangle pooling layer for capturing context in different directions; thus, behavior recognition is performed based on three-dimensional point cloud data, reducing recognition costs and power consumption, and since the resolution of three-dimensional point cloud data is low, personal information cannot be directly seen from the three-dimensional point cloud data, and it has a still object elimination function to protect privacy, and is particularly suitable for use in scenarios with strong privacy such as at home. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic flowchart of a behavior recognition method provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic flowchart of another behavior recognition method provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the network structure of the Resnet block provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the structure of the spatial attention layer provided by an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the structure of the channel attention layer provided by an embodiment of the present invention;

[0055] Figure 6 It is a schematic diagram of rectangle pooling provided by an embodiment of the present invention;

[0056] Figure 7 It is a schematic diagram of a behavior recognition result provided by an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram of the structure of a behavior recognition device provided by an embodiment of the present invention;

[0058] Figure 9 It is a schematic diagram of the structure of another behavior recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The method provided by an embodiment of the present invention obtains an image to be detected, uses a behavior recognition model to identify the image to be detected, and obtains a first feature; matches the first feature with at least one feature in a feature retrieval library, and determines a first distance between the first feature and each of the at least one feature; adjusts the first distance corresponding to each of the features based on a weight; and determines the type of the image to be detected based on the adjusted first distance corresponding to each of the features.

[0060] The present invention will be further described in detail below in conjunction with the embodiments.

[0061] Figure 1 A flow chart of a behavior recognition method provided by an embodiment of the present invention; Figure 1 As shown, the method is applied to a smart device; the smart device may be a server, a computer, etc. The method includes:

[0062] Step 101, obtaining data to be detected; the data to be detected is three-dimensional (3D) point cloud data;

[0063] Step 102: Use a preset behavior recognition model to recognize the data to be detected and obtain a behavior result;

[0064] The behavior recognition model is obtained based on neural network training; the neural network adopts blocks based on spatial attention layers and channel attention layers, as well as a rectangular pooling layer for capturing contexts in different directions.

[0065] Here, any commercial millimeter wave radar can be used to collect 3D point cloud data. The transmitting and receiving antennas of the millimeter wave radar are required to be 3 transmitters and 4 receivers or more, and the millimeter wave radar's built-in static object elimination algorithm can be used to retain only the information of active human bodies.

[0066] The behavior result may include: a behavior and a probability corresponding to the behavior; for example, the behavior is squatting, and the probability is 90%; the probability represents the credibility of the corresponding behavior.

[0067] Here, the neural network may adopt a Resnet network, and accordingly, the block based on the spatial attention layer and the channel attention layer is a residual block (Resnet block) based on the spatial attention layer and the channel attention layer;

[0068] In addition, the original maximum pooling layer and average pooling layer are replaced by a rectangular pooling layer for capturing contexts in different directions.

[0069] The neural network may also adopt other neural networks by applying the spatial attention layer and the channel attention layer thereto and using the rectangular pooling layer instead of other pooling layers.

[0070] In one embodiment, using the preset behavior recognition model to recognize the data to be detected and obtain a behavior result includes:

[0071] According to the data to be detected, obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence; the first three-dimensional spatio-temporal sequence includes a horizontal direction heat map and a time dimension corresponding to the horizontal direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical direction heat map and a time dimension corresponding to the vertical direction heat map;

[0072] Extract a first direction feature according to the first three-dimensional spatio-temporal sequence, and extract a second direction feature according to the second three-dimensional spatio-temporal sequence;

[0073] Fuse the first direction feature and the second direction feature to obtain a target feature;

[0074] Decode and predict the target sample feature to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1;

[0075] Based on the three-dimensional positions of the N skeleton nodes, obtain the behavior result.

[0076] The description of obtaining the first three-dimensional spatio-temporal sequence and the second three-dimensional spatio-temporal sequence is as follows. The three-dimensional point cloud data presents energy values and corresponds to three-dimensional coordinates (x, y, z); the energy values of the three-dimensional point cloud data can be respectively mapped into the horizontal plane (x, y) and the vertical plane (x, z) in an interpolation manner. Processing the three-dimensional point cloud data along the time axis in the above way, 2 columns of data with a dimension of 3 are obtained, that is, a horizontal direction heat map with a time dimension (i.e., the first three-dimensional spatio-temporal sequence) and a horizontal direction heat map with a time dimension (i.e., the second three-dimensional spatio-temporal sequence) are obtained.

[0077] In one embodiment, the method further includes: generating a preset behavior recognition model.

[0078] Specifically, generating the preset behavior recognition model includes:

[0079] Obtain a training sample set; the training sample set includes: at least one training sample and a behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data.

[0080] Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain a trained neural network as the behavior recognition model.

[0081] Wherein, training the neural network according to the at least one training sample and the behavior label corresponding to each training sample includes:

[0082] Obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and the time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and the time dimension corresponding to the sample vertical direction heat map;

[0083] Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

[0084] The behavior label may include at least one of the following: standing up, sitting down, falling, coughing, back pain, chest pain, etc. Accordingly, three-dimensional point cloud sample data corresponding to different behaviors can be collected, and the length of each action segment during collection can be 5 seconds. The behaviors included are not limited to the above behaviors.

[0085] Specifically, regarding obtaining three-dimensional sample point cloud data, a millimeter-wave radar can be placed on a horizontal table about 1 meter in height, and the subject makes corresponding actions at a distance of 2-5 meters from the radar.

[0086] Specifically, the neural network includes: a first neural network part, a second neural network part, and a fully connected layer;

[0087] The training of the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data includes:

[0088] The first neural network part extracts first sample direction features according to the first sample three-dimensional spatio-temporal sequence;

[0089] The second neural network part extracts second sample direction features according to the second sample three-dimensional spatio-temporal sequence;

[0090] Fuse the first sample direction features and the second sample direction features to obtain target sample features;

[0091] Use the fully connected layer to decode and predict the target sample features to obtain the sample three-dimensional positions of N bone nodes; obtain a sample prediction result based on the sample three-dimensional positions of the N bone nodes;

[0092] Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

[0093] The above-mentioned feature fusion can be a simple connection. For example, if the first-direction feature is a 1*n-dimensional vector and the second-direction feature is another 1*n-dimensional vector, the fused result is a 1*2n vector as the target feature.

[0094] The fusion of the sample direction features is the same and will not be elaborated here.

[0095] The method provided by the embodiments of the present invention is applicable to any suitable behavior recognition. For example, for the recognition of the falling behavior in a home environment, it can identify whether a user in the environment has fallen and issue an alarm if so, thereby ensuring the safety of users such as the elderly and children.

[0096] Figure 2 It is a schematic diagram of a neural network provided by the embodiments of the present invention; as Figure 2 shown, the neural network includes: a first neural network part, a second neural network part, a fully connected layer, and a softmax layer.

[0097] The first neural network part can adopt a Resnet part, and the second neural network part can adopt a Resnet part.

[0098] The Resnet part adopts the structure as shown in Table 1 below:

[0099]

[0100] Table 1

[0101] Here, the neural network uses a Resnet structure. Among them, the block based on the spatial attention layer and the channel attention layer is each residual block (Resnet block) in the Resnet structure.

[0102] That is, for each block in the Resnet structure, an attention layer is added; the attention layer is used to learn the attention weights, perform a weighted operation on the weights and the input, and obtain the attention features. The whole structure is divided into two parts, first through the spatial attention layer, and then through the channel attention layer.

[0103] The attention layer includes: a spatial attention layer and a channel attention layer. The network structure diagram of each specific Resnet block is as Figure 3 shown. A spatial attention layer and a channel attention layer are added to each original Resnet block, and first through the spatial attention layer, and then through the channel attention layer. In the figure, represents element multiplication.

[0104] Figure 4 It is a schematic diagram of the structure of the spatial attention layer provided by the embodiments of the present invention; combined with Figure 4 the spatial attention layer is described as follows;

[0105] The spatial attention layer is used to compress the channels. Max-pooling operation (Maxpool) and average-pooling operation (Average pool) are respectively performed on the channel dimension. The number of extractions during the pooling operation is the height multiplied by the width. Then, the results after Max pool and Average pool are connected in parallel, passed through a convolution operation, reduced to one channel, and finally passed through the sigmoid function to generate the spatial attention weight. The input is weighted with this weight to obtain the final feature.

[0106] The algorithm of the spatial attention layer can be expressed as: M s (F) = δ(f([AvgPool(F); MaxPool(F)]))

[0107] Among them, δ represents the sigmoid function, f() represents the convolution operation, AvgPool(F) represents the average-pooling operation; MaxPool(F) represents the max-pooling operation.

[0108] Figure 5 FIG. is the structural schematic diagram of the channel attention layer provided by the embodiment of the present invention; in combination with Figure 5 The channel attention layer is described as follows;

[0109] The channel attention layer compresses the feature map in the spatial dimension to obtain a one-dimensional vector. When performing spatial compression, max pool and average pool are respectively used to aggregate spatial information. Then, the results after max pool and average pool are sent into a shared fully-connected network, and finally the spatial dimension is compressed and merged by element-wise summation to generate a channel attention map.

[0110] For a single image, channel attention focuses on which contents on this image play an important role. Average pooling provides feedback to each pixel point on the feature map, while during the calculation of gradient backpropagation, only the place with the maximum response in the feature map has gradient feedback in max pooling.

[0111] The algorithm expression of the channel attention layer is: Mc(F) = δ(MLP([AvgPool(F); MaxPool(F)]))

[0112] Among them, δ represents the sigmoid function, MLP represents the multi-layer perceptron; AvgPool(F) represents the average-pooling operation; MaxPool(F) represents the max-pooling operation.

[0113] The rectangular pooling layer is described as follows.

[0114] In the Resnet network structure, rectangular pooling is used instead of global pooling in the Resnet network to more effectively capture long-range dependencies. Rectangular pooling has two advantages:

[0115] First, it deploys a long strip-shaped pooling kernel shape along one spatial dimension, so it can capture long-range relationships in isolated regions. Second, it maintains a narrow kernel shape in other spatial dimensions, which is convenient for capturing local context and preventing irrelevant regions from interfering with label prediction. Integrating such long and narrow pooling kernels enables the semantic segmentation network to aggregate global and local context simultaneously. This is essentially different from traditional pooling that collects context from a fixed square region.

[0116] Figure 6 Schematic diagram of the rectangular pooling provided by the embodiment of the present invention; rectangular pooling consists of two paths, which respectively focus on capturing remote context in the horizontal and vertical directions. As shown in Figure 6, a feature map is input, and the actual input is C*H*W (C represents the number of output channels, H represents the height, and W represents the width). For the convenience of representation, only one channel is drawn in the figure, and the processing method for each channel is the same. The following also takes one channel as an example for introduction. The input feature map becomes H*1 and 1*W after horizontal rectangular pooling and vertical rectangular pooling. The pooling method is to calculate the mean value of the elements in the pooling kernel and use this value as the pooling output value. Subsequently, one-dimensional convolution is used to amplify the two output feature vectors along the vertical and horizontal directions respectively. After amplification, the two feature maps have the same size, and the amplified feature maps are fused. Finally, the final output result is obtained by multiplying the convolution and sigmoid processing with the original input map.

[0117] In this process, each position in the output is associated with each position in the input. Therefore, through multiple aggregation processes, long-term dependencies can be constructed throughout the scene. In addition, the final dot product operation can also be regarded as a kind of attention learning.

[0118] Compared with global pooling, rectangular pooling considers a longer but narrower range, which can avoid establishing unnecessary links between positions that are far apart. Compared with the nolocal structure (non-local structure) that focuses on global modules but needs to calculate the relationship between each pair of positions, the calculation of rectangular pooling is lightweight, and it greatly improves the ability of the algorithm to capture remote spatial dependencies without consuming a large amount of computing power cost and time cost.

[0119] The embodiment of the present invention also provides a method for training the above neural network to obtain a behavior recognition model. The specific method includes: obtaining 3D point cloud data output by ordinary commercial millimeter wave radar in the behavior as sample 3D point cloud data, and generating a horizontal heat map and a vertical heat map (the corresponding heat map has a time dimension) based on the sample 3D point cloud data processing as the input value of the student network.

[0120] Among them, ordinary commercial millimeter-wave radar manufacturers such as TI, NXP, Continent, etc. require 3 transmit and 4 receive antennas or more. The millimeter-wave radar system's built-in static object elimination algorithm can be used to retain only the information of active human bodies; based on the sample 3D point cloud data and the behavior recognition information corresponding to the sample 3D point cloud data, the neural network is trained and optimized to obtain the best behavior recognition model for predicting behavior.

[0121] Regarding obtaining 3D point cloud data, the millimeter wave radar can be placed on a horizontal table at a height of about 1 meter, and the subject can make corresponding actions at a distance of 2-5 meters from the radar, including: standing up, sitting down, falling, coughing, back pain, chest pain, etc. The length of each action clip is 5 seconds. The behaviors include but are not limited to the above behaviors.

[0122] Regarding data processing, the millimeter wave point cloud data can be mapped into a three-dimensional space by interpolation, with an input dimension of 4, that is, a three-dimensional stereogram on the time axis.

[0123] Regarding the output of the model, the above behavior recognition model calculates the probability that the current behavior belongs to a certain behavior in the Softmax layer and outputs the behavior with the highest probability.

[0124] Based on the differences in the trained actions, the method provided in the embodiment of the present invention can be used to identify dangerous behaviors such as falls and fights.

[0125] Correspondingly, the behavior recognition method further includes: issuing a dangerous behavior alarm based on the behavior recognition result.

[0126] Specifically, when dangerous behavior is detected, a warning message will be sent to relevant agencies (such as community service centers, police stations, etc.), and the relevant agencies will take corresponding actions. For example, they can send people to help elderly people living alone, or conduct security checks in relevant areas to prevent fights, thereby helping to improve the level of social governance.

[0127] Figure 7 A schematic diagram of a behavior recognition result provided by an embodiment of the present invention; Figure 7 As shown, the behavior recognition result is standing up.

[0128] Considering that low-resolution millimeter-wave radars are relatively sparse and it is difficult to learn subtle human posture features from them, optimization is required based on existing models to achieve better results. At present, the main means of network optimization is to increase, deepen, and widen the network, which will lead to a higher time cost for prediction and a greater requirement for computing power. The timeliness of the fall prediction task itself and the portability requirements for edge-side operation require that the optimization of the algorithm must be as lightweight and efficient as possible. Based on this, through the method provided in the embodiments of the present invention, a spatial attention-channel attention module is proposed for optimization. The spatial attention-channel attention module is a lightweight general module, so the overhead of this module can be ignored and it can be seamlessly integrated into any convolutional neural network (CNN) architecture, and can be trained end-to-end with the basic CNN. The entire optimization process is lightweight and efficient, without excessive additional computational costs.

[0129] Moreover, the neural network of the present invention adopts a rectangular pooling layer to improve the long-distance dependence relationship of the model. The point cloud, as a mapping of the human body structure, is mostly long-strip point clouds. The rectangular pooling kernel is better adapted to the current task than the square pooling kernel, reducing the contaminated information from irrelevant regions. At the same time, rectangular pooling can capture the long-distance dependence relationship. Compared with existing methods, rectangular pooling consumes less memory and computing power and has obvious advantages.

[0130] Figure 8 It is a schematic structural diagram of a behavior recognition device provided by an embodiment of the present invention; as Figure 8 shown, the device includes:

[0131] An acquisition module, configured to acquire data to be detected; the data to be detected is three-dimensional point cloud data;

[0132] An identification module, configured to identify the data to be detected by using a preset behavior recognition model to obtain a behavior result;

[0133] The behavior recognition model is trained based on a neural network; the neural network adopts a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing contexts in different directions.

[0134] In one embodiment, the identification module is configured to obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence according to the data to be detected; the first three-dimensional spatio-temporal sequence includes a horizontal-direction heat map and the time dimension corresponding to the horizontal-direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical-direction heat map and the time dimension corresponding to the vertical-direction heat map;

[0135] Extract the first direction feature according to the first three-dimensional spatio-temporal sequence, and extract the second direction feature according to the second three-dimensional spatio-temporal sequence;

[0136] Fuse the first direction feature and the second direction feature to obtain a target feature;

[0137] Decode and predict the target sample feature to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1;

[0138] Based on the three-dimensional positions of the N skeleton nodes, obtain the behavior result.

[0139] In one embodiment, the device further includes: a preprocessing module, configured to obtain a training sample set; the training sample set includes: at least one training sample and a behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data.

[0140] Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain the behavior recognition model.

[0141] In one embodiment, the preprocessing module is configured to obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and a time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and a time dimension corresponding to the sample vertical direction heat map;

[0142] Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

[0143] In one embodiment, the neural network includes: a first neural network part, a second neural network part, and a fully connected layer;

[0144] The preprocessing module is configured to extract a first sample direction feature from the first sample three-dimensional spatio-temporal sequence through the first neural network part;

[0145] Extract a second sample direction feature from the second sample three-dimensional spatio-temporal sequence through the second neural network part;

[0146] Fuse the first sample direction feature and the second sample direction feature to obtain a target sample feature;

[0147] Use the fully connected layer to decode and predict the target sample features to obtain the sample three-dimensional positions of N skeleton nodes; obtain the sample prediction result based on the sample three-dimensional positions of the N skeleton nodes;

[0148] Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

[0149] It should be noted that when the behavior recognition device provided in the above embodiment implements the corresponding behavior recognition method, only the above division of each program module is used for illustration. In actual applications, the above processing can be allocated to different program modules according to needs, that is, the internal structure of the server is divided into different program modules to complete all or part of the above-described processing. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be repeated here.

[0150] Figure 9 FIG. is a schematic structural diagram of another behavior recognition device provided by an embodiment of the present invention. As Figure 9 shown, the device 90 includes: a processor 901 and a memory 902 for storing a computer program that can run on the processor; when the processor 901 runs the computer program, it executes: obtaining data to be detected; the data to be detected is three-dimensional point cloud data; using a preset behavior recognition model to recognize the data to be detected to obtain a behavior result; the behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing context in different directions.

[0151] The behavior recognition device can execute the method as Figure 1 shown, which belongs to the same concept as the method embodiment as Figure 1 shown, and the specific implementation process can be seen in the method embodiment, which will not be repeated here.

[0152] In actual application, the device 90 may further include: at least one network interface 903. Each component in the device 90 is coupled together through a bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 9 all kinds of buses are labeled as the bus system 904. Among them, the number of the processors 901 can be at least one. The network interface 903 is used for wired or wireless communication between the device 90 and other devices.

[0153] The memory 902 in the embodiments of the present invention is used to store various types of data to support the operation of the device 90.

[0154] The method disclosed in the embodiments of the present invention above can be applied to the processor 901 or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 901 or instructions in the form of software. The above-mentioned processor 901 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiments of the present invention, it can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, and this storage medium is located in the memory 902. The processor 901 reads the information in the memory 902 and combines its hardware to complete the steps of the foregoing method.

[0155] In an exemplary embodiment, the device 90 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontroller units (MCUs), microprocessors, or other electronic components, and is used to execute the foregoing method.

[0156] The embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored; when the computer program is run by a processor, it executes: obtaining data to be detected; the data to be detected is three-dimensional point cloud data; using a preset behavior recognition model to recognize the data to be detected to obtain a behavior result; the behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangle pooling layer for capturing contexts in different directions.

[0157] When the computer program is run by a processor, it can execute as Figure 1 shown in the method, which belongs to the same concept as the Figure 1 method embodiment shown. For the specific implementation process, please refer to the method embodiment and will not be elaborated here.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the displayed or discussed components can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0159] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, in each embodiment of the present invention, each functional unit can be all integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in a unit; the above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0161] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: various media that can store program codes such as removable storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical discs.

[0162] Alternatively, if the above integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as removable storage devices, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0163] It should be noted that: "first", "second", etc. are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0164] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0165] As described above, the above are only the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A behavior recognition method, characterized in that, The method includes: Obtain the data to be detected; the data to be detected is three-dimensional point cloud data; Use a preset behavior recognition model to recognize the data to be detected, and obtain a behavior result; The behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangle pooling layer for capturing context in different directions; The step of using a preset behavior recognition model to recognize the data to be detected and obtain a behavior result includes: According to the data to be detected, obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence; the first three-dimensional spatio-temporal sequence includes a horizontal direction heat map and the time dimension corresponding to the horizontal direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical direction heat map and the time dimension corresponding to the vertical direction heat map; Extract first direction features according to the first three-dimensional spatio-temporal sequence, and extract second direction features according to the second three-dimensional spatio-temporal sequence; Fuse the first direction features and the second direction features to obtain target features; Decode and predict the target features to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1; Based on the three-dimensional positions of the N skeleton nodes, obtain the behavior result.

2. The method according to claim 1, wherein The method further includes: generating a preset behavior recognition model; the step of generating a preset behavior recognition model includes: Obtain a training sample set; the training sample set includes: at least one training sample and the behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data; Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain the behavior recognition model.

3. The method according to claim 2, wherein The step of training the neural network according to the at least one training sample and the behavior label corresponding to each training sample includes: Obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and the time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and the time dimension corresponding to the sample vertical direction heat map; Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

4. The method according to claim 3, wherein The neural network includes: a first neural network part, a second neural network part, and a fully connected layer; The step of training the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data includes: The first neural network part extracts first sample direction features according to the first sample three-dimensional spatio-temporal sequence; The second neural network part extracts second sample direction features according to the second sample three-dimensional spatio-temporal sequence; Fuse the first sample direction features and the second sample direction features to obtain target sample features; Decode and predict the target sample features using the fully connected layer to obtain the sample three-dimensional positions of N skeleton nodes; obtain the sample prediction result based on the sample three-dimensional positions of the N skeleton nodes; Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

5. A behavior recognition device, characterized in that, The device includes: An acquisition module, configured to acquire data to be detected; the data to be detected is three-dimensional point cloud data; An identification module, configured to identify the data to be detected by using a preset behavior recognition model to obtain a behavior result; The behavior recognition model is trained based on a neural network; the neural network uses a block based on a spatial attention layer and a channel attention layer, and a rectangular pooling layer for capturing contexts in different directions; The identification module is specifically configured to obtain a first three-dimensional spatio-temporal sequence and a second three-dimensional spatio-temporal sequence according to the data to be detected; the first three-dimensional spatio-temporal sequence includes a horizontal direction heat map and a time dimension corresponding to the horizontal direction heat map; the second three-dimensional spatio-temporal sequence includes a vertical direction heat map and a time dimension corresponding to the vertical direction heat map; Extract first direction features according to the first three-dimensional spatio-temporal sequence, and extract second direction features according to the second three-dimensional spatio-temporal sequence; Fuse the first direction features and the second direction features to obtain target features; Decode and predict the target features to obtain the three-dimensional positions of N skeleton nodes; N is greater than or equal to 1; Obtain the behavior result based on the three-dimensional positions of the N skeleton nodes.

6. The device according to claim 5, characterized in that, The device further includes: a preprocessing module, configured to obtain a training sample set; the training sample set includes: at least one training sample and a behavior label corresponding to each training sample; the training sample is three-dimensional point cloud sample data; Train the neural network according to the at least one training sample and the behavior label corresponding to each training sample to obtain the behavior recognition model.

7. The device according to claim 6, characterized in that The preprocessing module is configured to obtain three-dimensional point cloud sample data; the three-dimensional point cloud sample data includes: a first sample three-dimensional spatio-temporal sequence and a second sample three-dimensional spatio-temporal sequence; the first sample three-dimensional spatio-temporal sequence includes a sample horizontal direction heat map and a time dimension corresponding to the sample horizontal direction heat map; the second sample three-dimensional spatio-temporal sequence includes a sample vertical direction heat map and a time dimension corresponding to the sample vertical direction heat map; Train the neural network based on the first sample three-dimensional spatio-temporal sequence, the second sample three-dimensional spatio-temporal sequence, and the behavior label corresponding to the three-dimensional point cloud sample data.

8. The device according to claim 7, characterized in that The neural network includes: a first neural network part, a second neural network part, and a fully connected layer; The preprocessing module is configured to extract first sample direction features according to the first sample three-dimensional spatio-temporal sequence through the first neural network part; Extract second sample direction features according to the second sample three-dimensional spatio-temporal sequence through the second neural network part; Fuse the first sample direction features and the second sample direction features to obtain target sample features; Use the fully connected layer to decode and predict the target sample features to obtain the sample three-dimensional positions of N bone nodes; obtain a sample prediction result based on the sample three-dimensional positions of the N bone nodes; Compare the sample prediction result with the behavior label corresponding to the three-dimensional point cloud sample data, and optimize the neural network based on the comparison result.

9. An action recognition device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images