A dual-mode human behavior recognition method based on RGB data and skeleton data

Abstract

The present application relates to a kind of dual-mode human behavior recognition method based on RGB data and skeleton data, by using different network structure, can accurately capture the action information in skeleton and spatial information in RGB mode, solve the difficult problem that skeleton information and RGB information in the Transformer framework are difficult to fuse and play the maximum effect.First, generate pseudo-heat map for skeleton data, this can avoid the stability caused by the skeleton expression as graphics, the problem that multiple people scene cannot be handled.Then a double-flow Transformer architecture with different attention layers and different window sizes is designed, and the pseudo-heat map and RGB frame are input into the double-flow structure with different time and spatial resolution.Finally, through experimental verification, the proposed behavior recognition method has higher accuracy and can solve the behavior recognition in multiple people scene.The double-flow structure and skeleton heat map generation method based on this method are suitable for behavior recognition in various public monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human behavior recognition, in particular to the field of human behavior recognition based on video, and specifically to a human behavior recognition method based on RGB data and skeleton heat map data. BACKGROUND

[0002] Human behavior recognition refers to obtaining human behavior data through monitoring video, motion capture, etc., and searching and recognizing human behaviors therein. Human behavior recognition has broad application prospects in public security, intelligent transportation, medical monitoring, and production safety. The human behavior recognition technology that is more widely applied can be divided into two categories according to the different input data: human behavior recognition based on RGB video, and human behavior recognition based on skeleton data. Compared with skeleton data, RGB video is very convenient to collect and can provide detailed and rich appearance information such as shape, color, and texture, but it is usually very sensitive to lighting conditions and shooting angles, and when the light is weak, the accuracy of human behavior recognition algorithm based on RGB video will be greatly reduced. On the contrary, although skeleton data cannot provide specific appearance information, it is very compatible with lighting conditions and shooting angles, and can also obtain good recognition accuracy through methods such as graph neural network (GCN). However, skeleton data is usually difficult to collect. Through a motion capture system, accurate skeleton key point information can be obtained, but in many application scenarios, the motion capture system is difficult to popularize and use. The development of pose estimation algorithm provides a more convenient and fast way for the extraction of skeleton data. By inputting RGB video into the pose estimation algorithm, the skeleton data corresponding to the video can be obtained. Combining RGB data and skeleton data as the input of human behavior recognition algorithm will obtain higher accuracy than algorithms relying only on RGB data or skeleton data.

[0003] Now there are works that combine RGB data and skeleton data to improve the accuracy of human behavior recognition. Luvizon et al. link the problems of pose estimation and behavior recognition, and realize efficient pose extraction and behavior recognition through the sharing mechanism of the two tasks. Das et al. proposed a pose-driven spatiotemporal attention mechanism and applied it to 3DCNN to perform human behavior recognition. In its subsequent work, it further increases the attention mechanism to the topology of the skeleton to calculate the spatiotemporal features. Li et al. proposed a dual-stream network structure, which has three main modules, an ST-GCN module to extract skeleton features, an R(2+1)D network to extract RGB features, and a module using the two features to enhance the action-related information in the RGB video. Finally, the score fusion method is used to obtain the classification result. Cai et al. also adopted a dual-stream network structure, but different from the above, in addition to the skeleton data stream, the other stream input is the aligned key point stream data extracted from the RGB video. The above works all process the skeleton data as a topological graph, but the topological graph is not a robust expression compared to the RGB data, and the loss of some skeleton key point information will have a great impact on the entire skeleton data. Jing and Wang proposed a ViT-based dual-path network, in which the input of the dual-path network is RGB frames with different frame rates and different resolutions, and the skeleton data is fused with the RGB data through three different fusion methods to obtain the final classification result. The fusion method of the skeleton data is to encode the skeleton as a token as an embedding and fuse it with the RGB data. Such a fusion method still cannot avoid the instability of the skeleton data, and also increases the feature dimension and the algorithm complexity. SUMMARY

[0004] The purpose of the present application is to propose a skeleton and RGB dual-modal behavior recognition method based on Transformer. A dual-stream Transformer architecture is proposed, taking RGB frames and skeleton heat maps as input. The present application inputs RGB frames and skeleton heat maps into the Transformer at different temporal and spatial resolutions, and uses different attention layer numbers and window sizes for the two inputs inside the network. In addition, to avoid the stability problems brought by representing the skeleton as a graph, a heat map is generated for the skeleton, improving the stability and solving the problem of behavior recognition in multi-person scenes. A double-stream transverse connection fusion method is proposed, which reduces the influence of noisy background in behavior recognition algorithms and solves the problem of missing key shape information in character interaction behaviors.

[0005] The purpose of the present application is achieved as follows:

[0006] A dual-stream Transformer architecture taking RGB and skeletal modalities as input is proposed. By using pose estimation algorithm and generating skeletal heat map, the problem of reduced stability in expressing skeleton in graphics can be avoided, and the fusion with RGB information can be more convenient. Based on the different temporal and spatial resolutions of the input and the different structures in the network, the RGB stream can capture spatial information and the skeletal stream can accurately capture action information. Finally, the fusion method proposed in the application is used in the network to fully fuse the appearance information in the RGB modality and the action information in the skeletal modality, solve the problem of insufficient single modality information, and reduce the influence of noisy background.

[0007] The specific manner is as follows:

[0008] A dual-stream RGB and skeletal dual-modality behavior recognition method based on Transformer includes the following steps:

[0009] Step 1: using a pose estimation algorithm to obtain skeletal information in the RGB video.

[0010] Step 2: generating a skeletal heat map.

[0011] Step 3: sampling the RGB video and the skeletal heat map.

[0012] Step 4: inputting a dual-stream Transformer structure.

[0013] Step 5: obtaining classification token information for horizontal fusion.

[0014] Step 6: obtaining network output and mapping it to classification results in a linear classifier.

[0015] Further, in step 1, the data owner uses a pre-trained model of HRNet on COCO-keypoint key points to perform pose estimation on the RGB video.

[0016] Further, in step 2, after obtaining the result of step 1, the 2D pose, a skeletal heat map is generated using the following method. For each skeletal joint:

[0017]

[0018] where (x k ,y k ) represents the coordinates of the kth point, and c k represents the confidence of the kth point.

[0019] For the limbs:

[0020]

[0021] where D((i,j),seg[a k ,b k ]) denotes the distance from a point to a line, and seg[a k ,b k ] denotes the line segment between two points. Then all the heatmaps are organized along the time dimension in the form of a heatmap video.

[0022] Further, step three uniformly samples the heatmap video and RGB video obtained in step two. For the heatmap video, 32 frames of heatmaps should be uniformly sampled along the time dimension. For the RGB video, 8 frames of RGB frames should be uniformly sampled along the time dimension.

[0023] Further, before inputting the Transformer, for the RGB video, the video will be resized to 320 pixels on the short side. In the training stage, it is cropped to 224x224 using a random cropping method. In the validation stage, it is cropped to 224x224 using a center cropping method. In the test stage, it is cropped to 224x224 at three spatial positions: the upper left, the center, and the lower right, and the average of the softmax values is taken as the final result. For the heatmap video, when generating the heatmaps, the smallest detection box covering all the target persons is obtained, zero padding is performed within the detection box, and the background unrelated to human behavior recognition outside the detection box is cut off. The final generated heatmap has a size of 224x224. No cropping or resizing operation is required.

[0024] For the RGB video and the heatmap video, the same decomposition operation is adopted. Taking the heatmap video as an example, the heatmap video is decomposed into N non-overlapping spatio-temporal "tubes" x1,x2,...,x N ∈R t×h×w×3 , where N=K*T. Next, each tube x i is linearly mapped into an encoding z i , z i =Ex i . Finally, all the encodings z i are concatenated into a vector z 0 . A special learnable vector z cls ∈R d is added to the first position, representing the embedding of the classification label. The position embedding p pos ∈R (N+1)×d is also added to this sequence.

[0025] RGB stream and skeleton stream use the same attention mechanism. The mechanism first computes frame-level spatial attention under the same time pointer. The number of spatial attention layers of skeleton stream is L' = 10, and the number of spatial attention layers of RGB stream is L = 12. For the spatial attention module of the l-th layer, first compute query / key / value:

[0026]

[0027] where a = 1,..., A represents attention head, p = 1,..., N represents spatial position, represents time position. represents the output of the previous layer. Then compute spatial attention:

[0028]

[0029] where

[0030] Then the output of the l-th layer can be obtained by the following formula:

[0031]

[0032] where s represents the output vector of all attention heads.

[0033] After computing the L' layer, the output result is transmitted to the MLP layer, which contains a GELU function and two linear layers divided by the GELU function:

[0034]

[0035] At this time, the data owner obtains the frame-level spatial attention expression which can be regarded as classification features Therefore we can express it as frame-level expression h i ∈R d , and all frame-level expressions are combined as:

[0036]

[0037] After the spatial attention block output is obtained, the dual-stream fusion can be performed.

[0038] Further, step five averages the frame-level expression obtained by the skeleton stream in step four, that is, every four frames are averaged, so that the frame-level expression obtained by the skeleton stream can be combined with the frame-level expression obtained by the RGB stream at the same time position.

[0039] Further, the module of spatial attention will be sent to the time attention module, and the attention is as follows:

[0040]

[0041] After that, the frame-level representation is obtained consistent with the spatial attention.

[0042] After passing through L t =4 layers of temporal attention, the obtained classification tokens will be input into the MLP head.

[0043] Further, in step 6, the data owner inputs the output of step 5 into a linear classification layer, and we obtain the class scores. By averaging the scores of the two streams, we obtain the final classification result.

[0044] The positive effects of the present application are:

[0045] (1) A dual-stream Transformer framework for RGB and skeleton dual modalities is proposed, which integrates the advantages of skeleton data and RGB data and improves the accuracy of behavior recognition.

[0046] (2) The framework generates a skeleton heat map from the skeleton data, avoiding the problem of different pose extractors affecting the accuracy of behavior recognition caused by expressing the skeleton as a graph, and the problem of being unable to solve the rapidly growing computational load in a multi-person scene. Secondly, the skeleton heat map and the RGB frame are input into the dual-stream network with different temporal and spatial resolutions. Inside the network, the skeleton heat map uses fewer attention layers. This makes the skeleton stream more accurately extract motion information.

[0047] (3) At the same time, a new fusion method is proposed to perform horizontal connection within the dual-stream, which can fuse information from the skeleton stream with information from the RGB stream. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a dual-stream framework structure diagram.

[0049] Figure 2 is a skeleton stream attention mechanism diagram.

[0050] Figure 3 is a dual-stream horizontal connection detail diagram. DETAILED DESCRIPTION

[0051] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] Step 1, as shown, we obtain skeleton data from the original RGB video using a pose extractor and generate a skeleton heat map in step 2. Figure 1

[0053] Step 3, as shown, we obtain the RGB frame and the skeleton heat map from the previous step and input them into the dual-stream framework. Figure 1 ​As shown, the skeleton heat map and the RGB frame have different temporal resolutions when inputting the dual-stream Transformer. The frame rate of the heat map is 4 times that of the frame map. In step 4, both use the same attention mechanism inside the Transformer. First, the spatial attention is calculated to obtain the frame-level expression. After the lateral connection in step 5, the temporal attention is calculated, and finally the MLP head is input. After obtaining the class scores of the dual stream, the final classification result is obtained by averaging.

[0054] The detailed structure of step 4 is as shown in Figure 2 As shown, the input data is first decomposed to obtain complementary overlapping tubes. After mapping to tokens and adding classification tokens, the input attention mechanism is input. After L' layers of spatial attention, the output is sent to the temporal attention, and finally sent to the MLP head.

[0055] The detailed structure of step 5 is as shown in Figure 3 As shown, after the spatial attention layer, both streams obtain frame-level expressions. After averaging the frame-level expressions of the skeleton stream, the frame-level expressions of the dual stream are merged according to the same time position, and then input into the temporal attention layer.

Claims

1. A dual-flow RGB and skeleton dual-modal behavior recognition method based on Transformer, comprising the following steps: Step 1: using a pose estimation algorithm to obtain skeleton information in the RGB video; Step 2: generating a skeleton heat map; Step 3: sampling the RGB video and the skeleton heat map; Step 4: inputting a dual-flow Transformer structure; Step 5: obtaining classification token information for horizontal fusion; before inputting the Transformer, the data owner adjusts the size of the RGB video to 320 pixels on the short side, in the training stage, uses a random cutting method to crop to 224x224 size, in the verification stage, uses a center cutting method to crop to 224x224 size, in the test stage, crops 224x224 size at the upper left, center, and lower right three spatial positions respectively and inputs the network to take the average of the softmax values as the final result; for the heat map video, when generating the heat map, obtain the smallest detection box that can cover all target persons, perform zero padding in the detection box, and cut the background unrelated to human behavior recognition outside the detection box; the final generated heat map size is 224x224; no cropping or size adjustment operation is required; For RGB videos and heat map videos, the same decomposition operation is adopted. Take heat map videos as an example, a heat map video is decomposed into N non-overlapping spatio-temporal "tubes" x1, x2,..., xN, respectively, where xk= {xk,1, xk,2,..., xk, T}, k = 1, 2,..., N, and xk, t = {xk, t, xk, t+1,..., xk, t+T-1}, t = 1, 2,..., T, where T is the number of frames in a video. N ∈R t×h×w×3 , where Next, each tube xkis linearly mapped into an encoding z i , where i z i = Ex i ; finally, all the encodings z i are concatenated into a vector z 0 ; In the first position a special learnable vector z is added cls ∈ R d , denotes the embedding of the classification label; and a position embedding p pos ∈ R (N+1)×d Also added to this sequence is a position embedding p The RGB flow and the skeleton flow use the same attention mechanism; The mechanism first calculates the frame-level spatial attention under the same time pointer; the number of layers of the spatial attention of the skeleton flow is L' = 10, and the number of layers of the spatial attention of the RGB flow is L = 12; for the spatial attention module of the l-th layer, first calculate query / key / value: where a = 1,..., A denotes attention heads, p = 1,..., N denotes spatial positions, denotes temporal positions; denotes the output of the previous layer; then the spatial attention is computed: wherein D = (N + 1) x d. Then the output of the l-th layer can be obtained by the following formula: Where s represents the output vector of all attention heads; After calculating the L' layer, the output result is transmitted to the MLP layer, which includes a GELU function and two linear layers divided by the GELU function: At this time, the data owner obtains the frame-level spatial attention expression It can be regarded as a classification feature Therefore we can express it as a frame-level expression h i ∈R d , and all frame-level expressions are combined into: After obtaining the spatial attention block output, dual-flow fusion can be performed; Step 6: obtaining network output, mapping to classification results in a linear classifier.

2. The recognition method according to claim 1, wherein the data owner uses a pre-trained model of HRNet on COCO-keypoint key points to perform pose estimation on the RGB video.

3. The recognition method according to claim 1, wherein the data owner uses the following method to generate a skeleton heat map after obtaining the result of step 1 and the 2D pose in step 2; for each skeletal joint: wherein (x k ,y k ) denotes the coordinates of the kth point, c k denotes the confidence of the kth point; For the limbs: where D((i,j), seg[a k ,b k ]) denotes the distance from a point to a line, and seg[a k ,b k ] denotes the line segment between two points; and then all the heat maps are organized along the time dimension in the form of a heat map video.

4. The recognition method according to claim 1, wherein the data owner uniformly samples the heat map video and the RGB video obtained in step two in step three; for the heat map video, 32 heat maps should be uniformly sampled along the time dimension; for the RGB video, 8 RGB frames should be uniformly sampled along the time dimension.

5. The recognition method according to claim 1, wherein the data owner averages the frame-level expressions obtained by the skeleton flow in step five, that is, average every four frames, so that the frame-level expressions obtained by the skeleton flow can be combined with the frame-level expressions obtained by the RGB flow at the same time position. Further, the spatial attention module is sent to the temporal attention module, which pays attention as follows: After that, the frame-level representation is obtained consistent with spatial attention; after passing through L t = 4 layers of temporal attention layers, the classification tokens obtained will be input into the MLP head.

6. The identification method of claim 1, wherein in step 6, the data owner inputs the output of step 5 into a linear classification layer to obtain a class score; and the final classification result is obtained by averaging the scores of the two streams.

Images