Behavior recognition method based on three-flow network and Transform

Abstract

The invention discloses a behavior recognition method based on a three-stream network and a Transform, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an inter-frame difference view, an edge view and a depth view through employing an obtained video; performing feature extraction on the inter-frame difference view, the edge view and the depth view by using a backbone network to respectively obtain inter-frame difference view features, edge view features and depth view features; respectively inputting the fusion features of the inter-frame difference view and the edge view and the fusion features of the edge view and the depth view into a cross Transform for training, and then performing classification by an FC layer; a multi-modal fusion mechanism enriches data information, reduces a lot of background information which does not need to be learned, and accelerates the learning speed; and in addition, the three-flow network is fused with the Transform, so that the calculation complexity is reduced, and the processing of the long sequence is more efficient.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a behavior recognition method based on a three-stream network and a Transformer. Background Art

[0002] Human Action Recognition (HAR) is a technology that collects raw data containing people through hardware devices such as cameras and surveillance, and processes and analyzes these data to identify human actions in this video containing people. Its research content includes many academic fields such as machine learning, pattern recognition, computer graphics, image processing, computer vision, artificial intelligence, etc. It is widely used in the fields of elderly fall detection, smart cars, smart homes, smart communities, smart security and human-computer interaction, and has great research prospects and considerable economic benefits. In the fields of empty-nest elderly fall detection and smart homes, research on human action recognition has important social value and research significance.

[0003] Extensive and in-depth research has been carried out worldwide on the development of visible light video, as reflected in the establishment and research of other datasets such as the HMDB-51 dataset, the UCF-101 dataset, and the Kinetics-400\600\700 dataset. However, visible light video surveillance has high requirements for the environment and can only work when the vision is clear and the light is sufficient. In bad weather, it is impossible to monitor the target area in real time. Intelligent video surveillance based on visible light is difficult to play a role at night when criminal activities are prone to occur, so research based on infrared video is very necessary. Since any object will emit infrared rays, which are mainly thermal radiation phenomena, the temperature of the object determines the intensity of its radiation, that is, the higher the surface temperature of the object, the greater the infrared thermal radiation energy emitted outward, which is ultimately reflected in the higher grayscale value during imaging, and infrared images are thus generated. Infrared video can not only realize full-sky video surveillance, but also eliminate interference from harsh environments such as smoke, rain, snow and dust. With the rapid development of artificial intelligence technology and the formation of application scale, the manufacturing cost of infrared equipment has been reduced, attracting more and more scholars to conduct research in this direction, providing possibilities for the development of infrared video. Therefore, the research on HAR based on infrared thermal imaging technology has great development potential.

[0004] Modern science began to study HAR in 1973. It was first proposed by a Swedish psychologist named Johanson. The idea was to model the joints of the human skeleton to obtain the motion information of different actions and then identify the actions. Although people only widely studied this research field in the 1990s, many scholars have proposed various methods and theories related to HAR. The data representation forms are also diverse: infrared images, depth images, RGB images, point cloud images, radar images, etc. There are three main types of HAR methods: (1) Research on methods based on wearable devices: let the subjects wear accelerometers, signal receivers, smart watches and other devices to obtain relevant signal features; (2) Research on methods based on images and videos: For single-modal images or videos, shallow methods (SA) and deep learning methods (DLM) are used to work; (3) Research on methods based on multi-modality: after training and feature extraction of multiple modalities, these features are fused and then input into the classifier to carry out classification tasks.

[0005] The existing research on infrared video behavior recognition currently encounters the following problems:

[0006] (1) There are problems with feature extraction of infrared information. Although infrared video can capture human behavior in special scenes, it lacks a lot of information because it is composed of surface thermal radiation information of people or objects. Although infrared video has many unique advantages, it lacks texture, color and other information compared to visible light data, which brings great challenges to the model's extraction of spatiotemporal features. Therefore, a deeper understanding of the principles and characteristics of infrared imaging is needed to better establish models and recognition and classification.

[0007] (2) There are few application scenarios and the recognition speed is slow. Vision-based behavior recognition and detection technologies focus on abnormal behavior detection. At the same time, the data sets currently studied are mostly the behavior actions of a single person, and there is little recognition of the behavior of multiple people or group interactions, which cannot meet the multi-person interaction, diversity, and complexity of daily life. Based on the above problems, there are few application scenarios for HAR research. Most of the current HAR-related research is based on training on large-scale computing platforms, which requires not only a large amount of computing resources, but also a lot of time to train the model. Intelligent video surveillance requires real-time processing of the captured information, and lacks high computing power. Reducing network model parameters while ensuring recognition rate and improving the recognition efficiency of the model are also one of the current problems. Summary of the invention

[0008] In order to make up for the deficiencies of the existing technical problems, the purpose of the present invention is to provide a behavior recognition method based on a three-stream network and Transformer, which fuses the RGB view, infrared view and depth view, and compensates for the lack of texture, color and other information in the infrared view. The depth view uses the distance information between the subject and the sensor as the pixel value of the image, which directly reflects the geometric shape of the subject's surface, reduces a lot of background information that does not need to be learned, and speeds up the learning process. In addition, the three-stream network is integrated with the Transformer, and the self-attention mechanism introduces the attention matrix to parallelize the calculation of the attention weights. Under the premise of greatly reducing the complexity of the calculation, the processing of long sequences is also more efficient. To achieve the above purpose, the present invention provides the following technical solutions:

[0009] A behavior recognition method based on a three-stream network and a Transformer includes the following steps:

[0010] (1) Obtain video information through the camera, process the acquired video, and obtain RGB view, infrared view and depth view data sets respectively;

[0011] (2) Process the RGB view in step (1) by using the inter-frame difference method to obtain an inter-frame difference view; process the infrared view by using the Sobel operator to obtain an edge view; and resize the depth map to obtain a depth view;

[0012] (3) Using the backbone network that integrates the Res3D-18 and CBAM modules, the inter-frame difference view, edge view, and depth view are extracted to obtain inter-frame difference view features, edge view features, and depth view features, respectively;

[0013] (4) The fused features of the inter-frame difference view and the edge view, as well as the fused features of the edge view and the depth view are respectively input into the cross Transformer for training, and then classified by the FC layer.

[0014] The backbone network of the present invention using the fusion of Res3D-18 and CBAM modules includes 5 convolutional layers, a maximum pooling layer and an average pooling layer of the Res3D-18 network and a CBAM module.

[0015] The specific process of extracting features using the backbone network fused with Res3D-18 and CBAM modules in the present invention is as follows:

[0016] First, the edge view is processed by five convolutional layers and one average pooling layer of the Res3D-18 network in sequence to obtain the infrared edge feature, and then the feature is input into the subsequent CBAM layer for further feature processing, and finally a feature with a dimension of 512 is obtained, and the weight is saved;

[0017] The weight is applied to the network for training inter-frame difference view data and depth view data, and finally the features under three views are obtained, namely, inter-frame difference view features, infrared edge view features and depth view features.

[0018] The process of inputting the features of the inter-frame difference view and the edge view and the features of the edge view and the depth view into the cross Transformer for training in step (4) of the present invention is as follows:

[0019] The features of the RGB view And the characteristics of infrared view The superposition of the characteristics

[0020] Y is passed into a transformer unit, where the query, key, and value of the feature are constructed using equations (7), (8), and (9), respectively.

[0021]

[0022] The attention is scaled as defined in equation (10),

[0023]

[0024] Among them, Attention(K Y ,Q Y ,V Y ) represents the attention operation of the subspace, d k represents the dimension of K; the core idea of ​​the attention mechanism is simplified to formula (11),

[0025]

[0026] Then the features of RGB view and infrared view are updated as equations (12) and (13),

[0027]

[0028] Input equations (12) and (13) into the feedforward sublayer to obtain the output of the conversion unit in the cross-attention module, and write its expression as:

[0029]

[0030] In order to obtain the final representation results of the RGB view and the infrared view, Y c Split into and The features of the RGB view are averaged and pooled again, and the features of the infrared view are max-pooled to obtain the final output features F of the two. RGB With F IR ; Then add the two features to obtain the final fusion feature representation of RGB view and infrared view, that is,

[0031] F=F RGB +F IR (15)

[0032] The same processing is performed on the features of the edge view and the depth view to obtain the fusion feature representation of the RGB view and the depth view, that is,

[0033] F′=F RGB ′+F DP ' (twenty three).

[0034] The classification in the present invention adopts the scoring decision fusion method to make decisions, that is, the features F and F′ are mapped to output vectors P and P′ through the fully connected layer. The formula is as follows:

[0035] P=FC(F) (24)

[0036] P′=FC(F′) (25)

[0037] Finally, the output vectors are added together, and the index of the maximum value in the vector is obtained as the final prediction result, that is,

[0038] Label = Max(P + P′) (26).

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention proposes a multimodal fusion mechanism to fuse RGB views, infrared views and depth views. The RGB view is mostly used in actual scenes. It is not only easy to obtain, but also can provide more texture detail information. The infrared view can not only well show the thermal target characteristics of the image, but also is not affected by light, which makes up for the shortcoming of RGB view that loses information in the lighting environment. The depth view uses the distance information between the subject and the sensor as the pixel value of the image, which directly reflects the geometric shape of the subject's surface, reduces a lot of background information that does not need to be learned, and speeds up the learning process.

[0041] (2) The present invention integrates the three-stream network with the Transformer, and introduces the self-attention mechanism into the attention matrix to parallelize the calculation of the attention weights. Under the premise of greatly reducing the complexity of the calculation, the processing of long sequences is also more efficient. In addition, the self-attention mechanism can capture the dependency between any two positions in the sequence to provide richer context information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the method of the present invention.

[0043] Figure 2 It is a flow chart of the two-frame difference method in the present invention.

[0044] Figure 3 It is a schematic diagram of the three-frame difference method in the present invention.

[0045] Figure 4 For the infrared view of the present invention, the Sobel operator is used to obtain the edge view.

[0046] Figure 5 Schematic diagram of the cross-attention mechanism principle in the present invention.

[0047] Figure 6 Schematic diagram of the structure of the cross-attention model in the present invention.

[0048] Figure 7 This is a partial category map of the NTU RGB+D dataset in the present invention.

[0049] Figure 8 It is a table diagram of the confusion matrix of different modal fusion in the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0051] like Figure 1-8 As shown in the figure, a behavior recognition method based on a three-stream network and Transformer is as follows:

[0052] 1. Identification Method Framework

[0053] Compared with the traditional attention mechanism, the self-attention mechanism introduces the attention matrix to parallelize the calculation of attention weights. Under the premise of greatly reducing the complexity of calculation, the processing of long sequences is also more efficient. In addition, the self-attention mechanism can capture the dependency between any two positions in the sequence to provide richer context information. Based on the above advantages, the present invention proposes a behavior recognition method based on a three-stream network and Transformer, which is mainly divided into a data preprocessing module, a feature extraction module and a feature fusion classification module. The flowchart of the present invention is as follows Figure 1 The main process is as follows: (1) Since the data obtained by the camera is video information (including RGB view, infrared view and depth view information), and video information cannot be input into the network for feature learning and extraction. First, the infrared video is cut at 24 frames per second to obtain the infrared view, and the video of each view is cut at 24 frames per second to obtain different view data. (2) Secondly, the obtained RGB view data is subjected to the inter-frame difference method, the Sobel operator in edge detection is used for the infrared view, and the depth map is resized to obtain the corresponding inter-frame difference view, edge view, and depth view respectively; (3) Then, the pre-processed infrared edge view is input into the Res3D-18 feature extraction network with the CBAM module added, and the weights are obtained by training. The weights are applied to the network for training the inter-frame difference view data and the depth view data; features with a dimension of 512 are extracted before the fully connected layer of the Res3D-18 network; (4) Finally, the features of the inter-frame difference view and edge view, as well as the features of the edge view and depth view are respectively input into the cross Transformer for training, and then classified by the FC layer.

[0054] (II) Data preprocessing

[0055] (1) RGB view. Temporal Difference is an algorithm for detecting target motion. By performing differential operations on consecutive frame images, it is possible to determine whether the target is moving. If the target in the scene is not moving, the change between consecutive frames will be small. Otherwise, the change will be more obvious. The algorithm calculates the difference in grayscale values ​​of corresponding pixels in two or three frames of images, and determines whether the difference exceeds a preset threshold. When the difference in grayscale values ​​exceeds the threshold, it is determined that the target has moved. The present invention uses the temporal difference method to process the RGB view.

[0056] 1. Two-frame difference method: The flowchart of the two-frame difference method is as follows Figure 2 As shown. Since the processed image is an RGB view, the RGB view must be grayscaled to obtain a grayscale image. The grayscale processing formula is as follows:

[0057] F(x,y)=0.299R(x,y)+0.587G(x,y)+0.114B(x,y) (1)

[0058] Among them, R(x,y), G(x,y) and B(x,y) are the three channels of the original RGB view, and 0.299, 0.587 and 0.114 are the weights of the three channels. The RGB time series after grayscale conversion of the nth frame and the n+1th frame are respectively denoted as F n With F n+1 , the gray value of the pixel corresponding to these two frames is F n (x,y) and F n+1 (x, y). According to formula (2), the absolute value of the difference between the grayscale values ​​of the pixels corresponding to the two frames of view is taken, and the obtained absolute value is used as the grayscale value D of the image. n (x,y).

[0059] D n (x,y)=|F n+1 (x,y)-F n (x,y)|(2)

[0060] With T as the threshold, each pixel is binarized according to formula (3) to obtain the binarized image. When the gray value is greater than the threshold, the point is marked as a moving point; when the gray value is less than or equal to the threshold, the point is marked as a background point.

[0061]

[0062] 2. Three-frame difference method: In the scenario where the target moves slowly, the two-frame difference method can work well. However, when the target moves too fast, the position of the target will be quite different between two adjacent frames. This phenomenon makes it difficult to obtain complete motion information after subtracting the two frames. To solve the above problem, some scholars have developed the three-frame difference method. The flowchart of the three-frame difference method is as follows: Figure 3 As shown. The RGB time series of the n-1th frame, the nth frame, and the n+1th frame are respectively recorded as F n-1 (x,y),F n (x,y) and F n+1 (x, y), and the difference image D is obtained according to formula (4-2) n (x,y) and the difference image D n+1 (x, y), and then according to formula (4) the difference image D n (x,y) and the difference image D n+1 (x,y) and operation, get the image

[0063]

[0064] In order to obtain the image R with complete motion information n (x, y), threshold processing and connectivity analysis are required first. By selecting a suitable threshold T, the noise in the differential image can be effectively suppressed or the masking of some information of the moving target can be reduced. When the threshold T is set to a fixed value, this method is difficult to adapt to the impact of light changes in the scene and other situations. In response to the above problems, some scholars have proposed adding an additional item that is sensitive to the overall illumination to the judgment condition, and modifying the judgment condition to,

[0065]

[0066] Among them, N A is the total number of pixels in the area to be tested, λ is the light suppression coefficient, and A is set to the entire image. It represents the change of illumination in the whole image. When the value of this item approaches 0, it indicates that the illumination change in the scene is small; when the value of this item increases significantly, it indicates that the illumination change in the scene is obvious, so that the judgment condition on the right side of equation (5) becomes larger adaptively, and the final judgment result is that there is no moving target. This method effectively suppresses the influence of light changes on the target motion detection result.

[0067] (2) Infrared view. Edge detection uses operators to extract the edge area of ​​the image. The edge is usually represented as a contour that constitutes the important attributes of the image, which facilitates the understanding of the image structure and shape, eliminates a large amount of background and other information that is not related to the action, greatly reduces the amount of data, and saves a lot of network training time. The Sobel operator is an edge detection operator and is often used under coarse precision. Its advantages are that it has directionality, can efficiently detect edges, and can suppress noise well, but its disadvantage is that it is not very sensitive to changes in grayscale. The traditional Sobel operator has convolution templates in the 0° direction and the 90° direction. Assuming s i is the gradient in the 0° direction, s j is the gradient in the 90° direction, the edge extraction process of the Sobel operator is as follows:

[0068] (1) The convolution template of the Sobel operator at the 0° and 90° directions consists of two 3×3 convolution kernels, I and J. First, the grayscale of the neighboring points at the 0° and 90° directions is weighted averaged. I and J are, respectively,

[0069]

[0070] Select View F IRA point in (x, y) is taken as the center point and the eight nearest points around it form a 3×3 area. IR After taking the product of each point in (x,y) and I(m,n) and adding them together, we get s i , F IR After taking the product of each point in (x,y) and J(m,n) and adding them together, we get s j . Taking point (i,j) as an example,

[0071]

[0072] Among them, F IR (i, j) is the view to be processed by the Sobel operator; s i is the angle of view in the 0° direction, s j is the angle of the view in the 90° direction.

[0073] (2) Calculate the angle of view at 0° and 90° to get the gradient amplitude, which is: Finally, an appropriate threshold is selected for judgment, and the extracted edge point set is the edge image F S (x,y). Figure 4 It is the process of obtaining edge view after edge detection of infrared view through Sobel operator. It takes about 320 minutes to process part of the data used by the NTU RGB+D dataset.

[0074] (3) Depth view. After the depth map is obtained, its size needs to be modified so that it can be input into the network for training. The process is as follows: D (x,y)←resize(F d (x,y)), where F d (x, y) is the original depth view, F D (x,y) is the processed depth view, and its size is 224×224.

[0075] (III) Res3D-18 feature extraction network with CBAM module added

[0076] The network structure used in the present invention is a Res3D-18 feature extraction network with a CBAM module added, and its network parameter settings are shown in Table 4-1.

[0077]

[0078] 4. Transformer

[0079] Cross-attention module. The cross-attention mechanism combines two independent embedding sequences of the same dimension in an asymmetric way, one of which takes Q as input and the other takes K and V as input, such as Figure 5 Compared with cross attention, the input of self-attention is a sequence of single embeddings.

[0080] Although the self-attention module makes good use of the relationship within the modality, it does not explore the relationship between the modalities well, such as the relationship between RGB views, infrared views, and depth views. Figure 6 As shown, the features of the RGB view And the characteristics of infrared view The superposition characteristics of As the input of the criss-cross attention module.

[0081] Y is passed into a transformer unit, where the query, key, and value of the feature are constructed using equations (7), (8), and (9), respectively.

[0082]

[0083] The attention is scaled as defined in equation (10),

[0084]

[0085] Among them, Attention(K Y ,Q Y ,V Y ) represents the attention operation of the subspace, d k represents the dimension of K. The core idea of ​​the attention mechanism can be simplified as formula (11),

[0086]

[0087] Then the features of RGB view and infrared view can be updated as equations (12) and (13),

[0088]

[0089] Figure 6 The network model structure of multimodal data cross-attention is presented. MHA stands for multi-head attention mechanism, η represents the mean value, and σ represents the standard deviation. In the network model of multimodal data cross-attention, MHA is used to discover the interaction between different modalities. The multi-head attention sublayer takes into account both the inter-modal and intra-modal relationships, and then inputs equations (12) and (13) into the feedforward sublayer to obtain the output of the conversion unit in the cross-attention module, which is expressed as,

[0090]

[0091] In order to obtain the final representation results of the RGB view and the infrared view, Y c Split into and The features of the RGB view are averaged and pooled again, and the features of the infrared view are max-pooled to obtain the final output features F of the two. RGB With F IR Then the two features are added together to obtain the final RGB view and infrared view feature representation, that is,

[0092] F=F RGB +F IR (15)

[0093] For the cross fusion of RGB view and depth view, the features of RGB view are And the characteristics of the depth view The superposition characteristics of As the input of the criss-cross attention module.

[0094] Y is passed into a transformer unit, where the query, key, and value of the feature are constructed using equations (16), (17), and (18), respectively.

[0095]

[0096] The attention is scaled as defined in equation (19),

[0097]

[0098] Among them, Attention(K Y ,Q Y ,V Y ) represents the attention operation of the subspace, d k Represents the dimension of K. Similar to the calculation process of formula 11, the features of the RGB view and the depth view can be updated as formulas (20) and (21),

[0099]

[0100] In the network model of cross-attention of multimodal data, MHA is used to discover the interaction between different modalities. The multi-head attention sublayer takes into account both the inter-modal and intra-modal relationships, and then inputs equations (20) and (21) into the feedforward sublayer to obtain the output of the conversion unit in the cross-attention module, which is expressed as:

[0101]

[0102] In order to obtain the final representation results of the RGB view and the depth view, Y c Split into and The features of the RGB view are averaged and pooled again, and the features of the depth view are max-pooled to obtain the final output features F of the two. RGB ′ and F DP ′. Then the two features are added together to obtain the final RGB view and depth view feature representation, that is,

[0103] F′=F RGB ′+F DP ' (twenty three)

[0104] After that, for the classification results, the score decision fusion method is adopted to make a decision, that is, the features F and F′ are mapped to output vectors P and P′ through the fully connected layer. The formula is as follows:

[0105] P=FC(F) (24)

[0106] P′=FC(F′) (25)

[0107] Finally, the output vectors are added together, and the index of the maximum value in the vector is obtained as the final prediction result, that is,

[0108] Label=Max(P+P′) (26)

[0109] (V) Data Verification

[0110] (5.1) Dataset Introduction

[0111] This dataset was collected and organized by Nanyang Technological University, Singapore, and contains data in various forms, such as depth maps, 3D skeleton data, RGB videos, and infrared videos. The dataset was released in 2016 and covers 60 action categories including daily actions, health-related, and two-person interactions. 40 volunteers aged 10 to 35 were invited to record 56,880 samples. There are two ways to divide this dataset. The first is the cross-target evaluation method (CS) and the second is the cross-view evaluation method (CV). The cross-target evaluation method divides the volunteers into two groups according to their numbers, each with 20 volunteers, one group as a training set and the other as a test set. The cross-view evaluation method is divided according to the camera number. The data taken by the camera numbered 1 is the test set, and the data taken by the other two are the training sets. Due to the large scale of this dataset and high hardware requirements, infrared videos with category numbers [A001-A010] are selected to verify the method of the present invention. The dataset example is as follows Figure 7 shown.

[0112] (II) Experimental environment configuration and parameter setting

[0113] Table 4-2 shows the main configuration and environment for conducting experiments on the computer, including software and hardware related information.

[0114] Table 4-2 Specific experimental environment configuration

[0115]

[0116] (III) Experimental results and analysis

[0117] Table 3 shows the accuracy of the depth map under different network models and different batch sizes. The results show that there are some differences in the results obtained by conducting experiments under the conditions of different depth networks and different batch sizes. Among them, when the network model is Res3D-18+CBAM and the batch size is 8, the effect is the best, with an accuracy of 81.41%.

[0118] Table 3 Comparison of experimental results of different models and different batch sizes (%)

[0119]

[0120] As shown in Table 4, the RGB view is trained to obtain edge information after Sobel operator extraction, and the effect is better than that of the RGB view. The present invention performs two-frame and three-frame differential processing on the RGB view to obtain the corresponding differential image, and inputs the differential sequence into the network for training. The result obtained after three-frame differential processing is better than that of two-frame differential, and is also better than that of edge view and original RGB view. At this time, the time series of the data after three-frame differential processing is 16, and the training speed is one third faster.

[0121] Table 4 Comparison of experimental results of RGB view under different processing methods (%)

[0122]

[0123] It can be seen from Table 5 that the type and number of modes have an impact on the experimental results.

[0124] Table 5 Comparison of experimental results of different modalities under different times of parallel calculation of attention modules (%)

[0125]

[0126] The results with the same number of parallel calculations of the attention module are better than those of the two modalities. Since the embedding dimension is strictly divisible by the number of parallel calculations of the attention module, the number of parallel calculations of the attention module N is taken as 1, 2, 4 and 8. The experimental results are better as the value of N increases.

[0127] As can be seen from Table 6, the number of Transformer layers affects the experimental results, which is manifested in that the experimental effect improves with the increase of the number of layers.

[0128] Table 6 Comparison of experimental results of different modes at different numbers of layers (%)

[0129]

[0130] The confusion matrix of the present invention fused in different modalities is as follows: Figure 8 As shown, the results show that the effect of three-modal fusion is better than that of two-modal fusion, and there is a more obvious improvement in multiple actions. At the same time, the final recognition result is also higher than that in Chapter 3, which verifies the effectiveness of the method of the present invention.

[0131] Table 7 compares the research results of the relevant literature on this data. The method of the present invention has certain advantages in recognition effect.

[0132] Table 7 Comparison of the method of the present invention with other methods (%)

[0133]

[0134] 1. Wang Chao, et al., Infrared human action recognition method based on multimodal attention network. Computer Science: p.1-12.

[0135] 2. Ren, Z., et al., Segment spatial-temporal representation and cooperative learning of convolution neural networks for multimodal-based action recognition. Neurocomputing, 2021.433: p.142-153.

[0136] 3.Lee,S.,et al.Modality mixer for multi-modal action recognition.inProceedings of the IEEE / CVF Winter Conference on Applications ofComputerVision.2023.

[0137] 4. Xiao,

[0138] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. The behavior recognition method based on three-stream network and Transformer is characterized by: The following steps are involved: (1) Obtain video information through the camera, process the acquired video, and obtain RGB view, infrared view and depth view data sets respectively; (2) Processing the RGB view in step (1) by using an inter-frame difference method to obtain an inter-frame difference view; Perform Sobel operator processing on the infrared view to obtain the edge view; Resizing the depth map to obtain a depth view; (3) Using the backbone network that integrates the Res3D-18 and CBAM modules, the inter-frame difference view, edge view, and depth view are extracted to obtain inter-frame difference view features, edge view features, and depth view features, respectively; (4) The fused features of the inter-frame difference view and the edge view, as well as the fused features of the edge view and the depth view are respectively input into the cross Transformer for training, and then classified by the FC layer.

2. The behavior recognition method based on three-stream network and Transformer according to claim 1 is characterized in that: The backbone network using the fusion of Res3D-18 and CBAM modules includes 5 convolutional layers, a maximum pooling layer and an average pooling layer of the Res3D-18 network and a CBAM module.

3. The behavior recognition method based on three-stream network and Transformer according to claim 2 is characterized in that: The specific process of extracting features using the backbone network fused with the Res3D-18 and CBAM modules in step (3) is as follows: First, the edge view is processed by five convolutional layers and one average pooling layer of the Res3D-18 network in sequence to obtain the infrared edge feature, and then the feature is input into the subsequent CBAM layer for further feature processing, and finally a feature with a dimension of 512 is obtained, and the weight is saved; The weight is applied to the network for training inter-frame difference view data and depth view data, and finally the features under three views are obtained, namely, inter-frame difference view features, infrared edge view features and depth view features.

4. The behavior recognition method based on three-stream network and Transformer according to claim 1 is characterized in that: In step (4), the process of inputting the features of the inter-frame difference view and the edge view, as well as the features of the edge view and the depth view into the cross Transformer for training is as follows: The features of the RGB view And the characteristics of infrared view The superposition of the characteristics Y is passed into a transformer unit, where the query, key, and value of the feature are constructed using equations (7), (8), and (9), respectively. The attention is scaled as defined in equation (10), Among them, Attention(K Y ,Q Y ,V Y ) represents the attention operation of the subspace, d k represents the dimension of K; the core idea of ​​the attention mechanism is simplified to formula (11), Then the features of RGB view and infrared view are updated as equations (12) and (13), Input equations (12) and (13) into the feedforward sublayer to obtain the output of the conversion unit in the cross-attention module, and write its expression as: In order to obtain the final representation results of the RGB view and the infrared view, Y c Split into and The features of the RGB view are averaged and pooled again, and the features of the infrared view are max-pooled to obtain the final output features F of the two. RGB With F IR ; Then add the two features to obtain the final fusion feature representation of RGB view and infrared view, that is, F=F RGB +F IR (15) The same processing is performed on the features of the edge view and the depth view to obtain the fusion feature representation of the RGB view and the depth view, that is, F′=F RGB ′+F DP ′ (23)。 5. The behavior recognition method based on three-stream network and Transformer according to claim 4 is characterized in that: The classification in step (4) is decided by score decision fusion, that is, the features F and F′ are mapped to output vectors P and P′ through the fully connected layer. The formula is as follows: P=FC(F) (24) P′=FC(F′) (25) Finally, the output vectors are added together, and the index of the maximum value in the vector is obtained as the final prediction result, that is, Label = Max(P + P′) (26).

Images