Behavior recognition method based on transformer semantic information enhancement

Abstract

The application relates to a behavior recognition method based on semantic information enhancement of a Transformer, wherein a light flow method and a clustering method are used to extract video key frames as input, and a convolutional neural network is used to extract low-level information; then, a hollow convolution is used to capture local information between adjacent key vectors, and self-attention for excavating global information is further triggered by using the local information; finally, local and global context information is fused; compared with a traditional self-attention mechanism, the method can simultaneously learn local features in the position aspect and content-based global interaction, thereby enhancing visual representation capability; in addition, under the premise of ensuring the calculation cost, the application effectively improves the accuracy of behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a behavior recognition method based on semantic information enhancement of Transformer. BACKGROUND

[0002] Video understanding and recognition is one of the basic tasks of computer vision. In the field of video understanding research, human behavior understanding is a challenging task with high practical application value; human behavior understanding includes recognizing, locating and predicting human behavior. The task of recognizing human behavior in a video is called behavior recognition, and recognizing human behavior has a long history and attracts various disciplines including psychology, biology and computer science. From video monitoring to human-computer interaction, retail analysis, user interface design, robot learning, network video search and retrieval, medical diagnosis, quality of life improvement for the elderly, sports analysis and other social applications, their core is a computing algorithm that can understand human behavior.

[0003] The behavior recognition task can be achieved by traditional methods and deep learning methods. Traditional methods use hand-designed features to represent behavior and use statistical learning classification methods to classify and recognize behavior, which requires a lot of time, for example, manually identifying abnormal activities from video recordings, which is both expensive and difficult. In addition, 24-hour monitoring of the area of interest is tedious and can introduce human errors, and a long list of emerging technologies and applications shows that manual analysis of behavior and behavior data is not ideal. In order to solve these problems, automatic modeling of human behavior can be used, and automatic modeling of behavior is a process of mapping a specific behavior to a description of the behavior instantiation label using deep learning methods. Deep learning is based on representation learning of data, using efficient algorithms for feature learning and hierarchical feature extraction to automatically extract features instead of manually obtaining features, and deep learning has become popular due to its strong learning ability, high adaptability, portability and other advantages.

[0004] However, deep learning methods are not a panacea and even bring new problems, such as non-single action label, dimension disaster, increased algorithm complexity, increased parameters, increased computing load, and unstable recognition accuracy. The most important problem is the lack of semantic information extraction, which includes the expression of shallow features (such as color, geometry, texture, etc.) and the expression of high-level features (such as global information, local information), and recognizing human behavior requires understanding both short-term action-specific motion information and long-term temporal information. However, many models in behavior recognition lack semantic information, so new models need to be constructed to obtain sufficient information. SUMMARY

[0005] To overcome at least one of the deficiencies in the prior art, the present application provides a behavior recognition method based on semantic information enhancement of a Transformer.

[0006] In a first aspect, a behavior recognition method based on semantic information enhancement of a Transformer is provided, comprising:

[0007] obtaining a key frame sequence of a video to be recognized, the key frame sequence including a plurality of key frames;

[0008] obtaining spatio-temporal information of the key frame sequence;

[0009] inputting the spatio-temporal information of the key frame sequence into a convolutional neural network to extract an information matrix of the key frame sequence containing low-level information;

[0010] inputting the information matrix of the key frame sequence containing low-level information into a self-attention mechanism to extract local information and global information, and fusing the local information and the global information to obtain fused information;

[0011] inputting the fused information into a first feedforward neural network module;

[0012] inputting an output of the first feedforward neural network module into a second feedforward neural network module to determine a category of the video to be recognized.

[0013] In one embodiment, obtaining the key frame sequence of the video to be recognized comprises:

[0014] determining total optical flow information of each video frame in the video to be recognized based on an optical flow method;

[0015] dividing all video frames into a plurality of categories using a clustering method according to the total optical flow information of each video frame, and determining key frames in each category;

[0016] all key frames constitute the key frame sequence.

[0017] In one embodiment, obtaining the spatio-temporal information of the key frame sequence comprises:

[0018] inputting the key frame sequence into a linear layer to extract spatial information of the key frame sequence;

[0019] extracting time sequence information of the key frame sequence according to an index of each key frame in the key frame sequence;

[0020] fusing the spatial information and the time sequence to obtain the spatio-temporal information of the key frame sequence.

[0021] In one embodiment, inputting the information matrix of the key frame sequence containing low-level information into the self-attention mechanism to extract local information and global information comprises:

[0022] mapping the information matrix of the key frame sequence containing low-level information to obtain mapped information;

[0023] determining a query vector Q, a key vector K and a value vector V according to the mapped information;

[0024] processing the key vector K by using a hollow convolution to obtain local information;

[0025] splicing the local information and the query vector Q, and performing two continuous convolution processing to obtain an attention matrix;

[0026] convolving the attention matrix and the value vector V to obtain global information.

[0027] In a second aspect, a behavior recognition device based on semantic information enhancement of a Transformer is provided, and the device comprises:

[0028] a key frame sequence acquisition module configured to acquire a key frame sequence of a video to be recognized, the key frame sequence comprising a plurality of key frames;

[0029] a spatiotemporal information acquisition module configured to acquire spatiotemporal information of the key frame sequence;

[0030] a convolutional neural network configured to input the spatiotemporal information of the key frame sequence into the convolutional neural network to extract an information matrix of the key frame sequence containing low-level information;

[0031] a self-attention mechanism configured to input the information matrix of the key frame sequence containing low-level information into the self-attention mechanism to extract local information and global information, and to fuse the local information and the global information to obtain fused information;

[0032] a first feedforward neural network module configured to input the fused information into the first feedforward neural network module;

[0033] a second feedforward neural network module configured to input an output of the first feedforward neural network module into the second feedforward neural network module to determine a category of the video to be recognized.

[0034] In an embodiment, the key frame sequence acquisition module is further configured to:

[0035] determine total optical flow information of each video frame in the video to be recognized based on an optical flow method;

[0036] divide all the video frames into a plurality of categories by using a clustering method according to the total optical flow information of each video frame, and determine key frames in each category;

[0037] all the key frames constitute the key frame sequence.

[0038] In one embodiment, the spatio-temporal information acquisition module is further configured to:

[0039] input the key frame sequence into a linear layer to extract spatial information of the key frame sequence;

[0040] extract time sequence information of the key frame sequence according to an index of each key frame in the key frame sequence;

[0041] fuse the spatial information and the time sequence to obtain spatio-temporal information of the key frame sequence.

[0042] In one embodiment, the self-attention mechanism is further configured to:

[0043] map an information matrix containing low-level information of the key frame sequence to obtain mapped information;

[0044] determine a query vector Q, a key vector K and a value vector V according to the mapped information;

[0045] process the key vector K using a cavity convolution to obtain local information;

[0046] concatenate the local information and the query vector Q, and perform two consecutive convolution processes to obtain an attention matrix;

[0047] convolve the attention matrix and the value vector V to obtain global information.

[0048] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned behavior recognition method based on semantic information enhancement of a Transformer.

[0049] In a fourth aspect, a computer program product is provided, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the computer programs / instructions implement the above-mentioned behavior recognition method based on semantic information enhancement of a Transformer.

[0050] Compared with the prior art, the application has the following beneficial effects: the application extracts video key frames as input by using a sampling optical flow method and a clustering algorithm, and extracts low-level information by using a convolutional neural network; then, local information between adjacent key vectors is captured by using a cavity convolution, and further, self-attention for mining global information is triggered by using the local information, and finally, local and global context information is fused; compared with a traditional self-attention mechanism, the method can simultaneously learn local features in terms of position and content-based global interaction, thereby enhancing visual representation capability; in addition, the application effectively improves the accuracy of behavior recognition under the premise of ensuring the calculation cost. BRIEF DESCRIPTION OF DRAWINGS

[0051] The present application can be better understood with reference to the following description taken in conjunction with the accompanying drawings, which are included as a part of this specification and which illustrate by way of example a preferred embodiment. In the drawings:

[0052] Figure 1 A flow chart of a behavior recognition method based on semantic information enhancement of a Transformer according to an embodiment of the present application is shown;

[0053] Figure 2 A structural block diagram of a behavior recognition device based on semantic information enhancement of a Transformer according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In the following, exemplary embodiments of the present application will be described with reference to the accompanying drawings. In the specification, not all the features of the actual embodiments have been described for the sake of clarity and conciseness. It should be appreciated, however, that many embodiment-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific objectives of the developers, and these decisions can vary from embodiment to embodiment.

[0055] It should also be noted here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, and other details not closely related to the present application are omitted.

[0056] It should be understood that the present application is not limited to the described embodiments by virtue of the following description with reference to the drawings. In this context, the embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in an embodiment.

[0057] The present application aims to solve the problem of insufficient semantic information extraction in the process of existing human behavior recognition. By using related knowledge in the field of deep learning, the key frames of a video are extracted as input, and the Transformer is organically combined with the convolutional neural network to perform self-attention learning under the guidance of local context, so as to obtain context information that combines local and global information to promote visual representation learning, thereby achieving the human behavior recognition task.

[0058] An embodiment of the present application provides a behavior recognition method based on semantic information enhancement of a Transformer, Figure 1 A flow chart of a behavior recognition method based on semantic information enhancement of a Transformer according to an embodiment of the present application is shown, and the method comprises:

[0059] In step S1, a key frame sequence of a video to be recognized is obtained, and the key frame sequence includes a plurality of key frames.

[0060] In step S2, the spatio-temporal information of the key frame sequence is obtained.

[0061] In step S3, the spatio-temporal information of the key frame sequence is input into a convolutional neural network to extract an information matrix containing low-level information of the key frame sequence.

[0062] In this step, a 4-layer 3x3 convolutional neural network can be used to extract the information matrix containing low-level information, so as to reduce the inductive bias.

[0063] In step S4, the information matrix containing low-level information of the key frame sequence is input into a self-attention mechanism to extract local information and global information, and the local information and the global information are fused to obtain fused information.

[0064] In step S5, the fused information is input into a first feedforward neural network module.

[0065] In step S6, the output of the first feedforward neural network module is input into a second feedforward neural network module to determine the category of the video to be recognized.

[0066] In this step, the first feedforward neural network module and the second feedforward neural network module are both existing structures of Transformer, and will not be described in detail. The second feedforward neural network module has a softmax normalization layer for outputting probabilities of the video to be recognized belonging to different categories. The category with the largest probability value is selected as the category of the video to be recognized, or the categories corresponding to the top 5 largest probabilities are selected as possible category results of the video to be recognized. For example, the video category can be determined as kicking a football, drawing, or running, etc.

[0067] In one embodiment, obtaining the key frame sequence of the video to be recognized in step S1 can include:

[0068] First, based on the optical flow method, the total optical flow information of each video frame in the video to be recognized is determined. Here, determining the total optical flow information of each video frame in the video to be recognized is to calculate the motion vector of each pixel point in the video frame.

[0069] Then, according to the total optical flow information of each video frame, a clustering method is used to divide all video frames into multiple categories, and determine the key frame in each category. Here, the k-means clustering algorithm can be used to divide the total optical flow information as a clustering target, divide the video frames into multiple categories, and determine the key frame in each category.

[0070] Then, all the key frames constitute a key frame sequence.

[0071] In this embodiment, the method of extracting key frames can remove inter-frame redundancy to reduce the interference of redundant information.

[0072] In one embodiment, in step S2, acquiring the spatio-temporal information of the key frame sequence can include:

[0073] In step S21, the spatial information of the key frame sequence is extracted by inputting the key frame sequence into a linear layer.

[0074] In step S22, the temporal information of the key frame sequence is extracted according to the index of each key frame in the key frame sequence.

[0075] In step S23, the spatial information and the temporal information are fused to obtain the spatio-temporal information of the key frame sequence.

[0076] In one embodiment, in step S4, inputting the information matrix containing low-level information of the key frame sequence into the self-attention mechanism to extract local information and global information can include:

[0077] In step S41, the information matrix containing low-level information of the key frame sequence is mapped to obtain the mapped information X.

[0078] In step S42, the query vector Q, the key vector K and the value vector V are determined according to the mapped information X; here, Q=X, K=X, and V=XW v , W v is an embedding matrix.

[0079] In step S43, the key vector K is processed by using a dilated convolution to obtain the local information k 局部 ; here, the key vector K includes multiple key vectors corresponding to multiple key frames, and for each key frame, multiple key vectors are processed by using a dilated convolution within a spatially defined grid of a certain size, and the dilated rate is 2 to obtain the local information representing the context information of the key vector K.

[0080] In step S44, the local information and the query vector Q are spliced, and after two consecutive convolution processing, the attention matrix A is obtained.

[0081] A=[K 局部 ,Q]W θ W δ

[0082] Where [.] represents splicing, W θ is a convolution operation with a ReLU activation function, and W δ is a convolution operation without an activation function.

[0083] In step S45, the attention matrix and the value vector V are convolved to obtain the global information K 全局 .

[0084]

[0085] wherein, denotes convolution.

[0086] In this embodiment, in view of the problem that the existing visual Transformer mainly relies on the interaction between the independent query vector Q and the key vector K in the self-attention mechanism to measure the attention matrix, ignoring the rich context relationship between adjacent key vectors K and being difficult to extract fine-grained local features, the context mining between key vectors K and the self-attention learning are unified in a single architecture, thereby avoiding introducing additional computational cost for context mining.

[0087] Technically, first, the representation of the key vector is contextualized by performing a dilated convolution on all adjacent keys. The contextualized key features can be regarded as a local representation of the input, which reflects the local context between local neighbors. Then, the contextualized key features and the input query are connected into two consecutive 1x1 convolutions, the purpose of which is to produce an attention matrix. This process naturally utilizes the mutual relationship between each query vector Q and all key vectors K, and performs self-attention learning under the guidance of local context. The learned attention matrix is further used to aggregate all input values, so as to realize the global context representation of the input to describe the global information. In this embodiment, both of the above two spatial contexts in the input are captured, i.e., the local context through the dilated convolution and the global context based on the self-attention, to facilitate visual representation learning.

[0088] Based on the same inventive concept as the Transformer-based semantic information enhanced behavior recognition method, the embodiment also provides a Transformer-based semantic information enhanced behavior recognition device, Figure 2 A structural block diagram of the Transformer-based semantic information enhanced behavior recognition device according to the embodiment of the present application is shown, and the device comprises:

[0089] The key frame sequence acquisition module 21 is configured to acquire a key frame sequence of a video to be recognized, and the key frame sequence comprises a plurality of key frames.

[0090] The spatiotemporal information acquisition module 22 is configured to acquire spatiotemporal information of the key frame sequence.

[0091] The convolutional neural network 23 is configured to input the spatiotemporal information of the key frame sequence into the convolutional neural network, and extract an information matrix of the key frame sequence containing low-level information.

[0092] The self-attention mechanism 24 is configured to input the information matrix containing low-level information of the key frame sequence into the self-attention mechanism, extract local information and global information, and fuse the local information and the global information to obtain fused information.

[0093] The first feedforward neural network module 25 is configured to input the fused information into the first feedforward neural network module.

[0094] The second feedforward neural network module 26 is configured to input the output of the first feedforward neural network module into the second feedforward neural network module to determine the category of the video to be recognized.

[0095] In an embodiment, the key frame sequence acquisition module 21 is further configured to:

[0096] determine total optical flow information of each video frame in the video to be recognized based on an optical flow method;

[0097] divide all the video frames into multiple categories by using a clustering method according to the total optical flow information of each video frame, and determine key frames in each category;

[0098] all the key frames constitute the key frame sequence.

[0099] In an embodiment, the spatio-temporal information acquisition module 22 is further configured to:

[0100] input the key frame sequence into a linear layer to extract spatial information of the key frame sequence;

[0101] extract time sequence information of the key frame sequence according to an index of each key frame in the key frame sequence;

[0102] fuse the spatial information and the time sequence to obtain spatio-temporal information of the key frame sequence.

[0103] In an embodiment, the self-attention mechanism 24 is further configured to:

[0104] map the information matrix containing low-level information of the key frame sequence to obtain mapped information;

[0105] determine a query vector Q, a key vector K and a value vector V according to the mapped information;

[0106] process the key vector K by using a cavity convolution to obtain local information;

[0107] splice the local information and the query vector Q, and perform two continuous convolution processes to obtain an attention matrix;

[0108] convolve the attention matrix and the value vector V to obtain global information.

[0109] The embodiment of the application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the behavior recognition method based on semantic information enhancement of the Transformer.

[0110] The embodiment of the application provides a computer program product, including computer programs / instructions, which are executed by a processor to implement the behavior recognition method based on semantic information enhancement of the Transformer.

[0111] In order to verify the identification performance of the identification method of the application, on the Kinetics-400 dataset, by comparing with common behavior recognition algorithms in recent years such as ViViT, ARTNet, TEINet, SmallBigNet, TDN, LGD, all models are pre-trained on the Imagenet-1k dataset, all evaluation results are rounded to 1 after the decimal point, and the experimental results are as shown in Table 1.

[0112] Table 1 Comparison of the method of the application with other video recognition algorithms

[0113] Method Main model Frame number x sampling number x clipping number Top-1 Top-5 TFLOPs TimeSformer ViT 8×10×3 76.9 93.6 0.59 ViViT - Not counted x 1 x 1 77.8 93.7 4.218 ARTNet ResNet18 16×25×10 70.7 89.3 5.87 TEINet ResNet50 16×10×3 76.3 92.7 2.38 LGD ResNet50 128 x not counted 77.6 91.6 - TDN ResNet50 8×10×3 75.3 92.8 1.08 TDN ResNet50 16×10×3 76.8 92.8 2.16 TDN ResNet50 (8+16)×10×3 78.1 94.4 4.94 SmallBigNet ResNet50 (8+32)×4×3 78.4 93.9 5.7 Method of the present application ViT 8×4×3 80.5 94.3 0.97

[0114] The evaluation indexes adopted are top-1 accuracy and top-5 accuracy, hereinafter referred to as top-1 and top-5. In the prediction result, top-1 indicates that the label with the largest value is selected as the classification result, and if the result is consistent with the true label, the classification is correct, otherwise it is incorrect. Top-5 indicates that the labels with the top five largest values are selected as the result, and if the true label is contained in the five results, the classification is correct, otherwise it is incorrect. In order to measure the complexity of the model, the total number of calculation operations that the model must perform can be calculated. Here, TFLOP (Tera Floating Point Operation) is used, that is, floating point operation. These operation operations include addition, subtraction, multiplication, division and any other operations related to floating point numbers. The TFLOPs of a model is the sum of all FLOPs of the model. From the experimental results, it can be seen that the behavior recognition method based on semantic information enhancement of the Transformer can effectively improve the accuracy of behavior recognition.

[0115] In summary, the behavior recognition method based on the semantic information enhancement of the Transformer in the application can improve the accuracy of behavior recognition under the condition of controlling the computing cost. By analyzing the existing problems of behavior recognition technology and studying the relationship between convolutional neural network and Transformer, first, the key frames are extracted from the video data set as input by using the optical flow method and clustering algorithm, then the low-level information is extracted by using 4 layers of 3*3 convolution, and the adjacent context between adjacent keys is captured by using the hollow convolution, and further it is used to trigger the self-attention to mine the global context, finally the local and global context information are fused, the method can learn the local features in the position and the global interaction based on the content at the same time, thereby enhancing the visual representation ability.

[0116] The above merely describes various embodiments of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A behavior recognition method based on Transformer semantic information enhancement, characterized in that, The method comprises the following steps: obtaining a key frame sequence of a video to be identified, the key frame sequence comprising a plurality of key frames; obtaining spatio-temporal information of the key frame sequence; inputting the spatio-temporal information of the key frame sequence into a convolutional neural network to extract an information matrix of the key frame sequence containing low-level information; inputting the information matrix of the key frame sequence containing low-level information into a self-attention mechanism to extract local information and global information, and fusing the local information and the global information to obtain fused information; inputting the fused information into a first feedforward neural network module; inputting an output of the first feedforward neural network module into a second feedforward neural network module to determine a category of the video to be identified; wherein the spatio-temporal information of the key frame sequence is obtained by: inputting the key frame sequence into a linear layer to extract spatial information of the key frame sequence; extracting time sequence information of the key frame sequence according to an index of each key frame in the key frame sequence; fusing the spatial information and the time sequence to obtain the spatio-temporal information of the key frame sequence; wherein the information matrix of the key frame sequence containing low-level information is inputted into the self-attention mechanism to extract the local information and the global information by: mapping the information matrix of the key frame sequence containing low-level information to obtain mapped information; determining a query vector Q, a key vector K and a value vector V according to the mapped information; processing the key vector K by using a hollow convolution to obtain local information; splicing the local information and the query vector Q, and processing by two consecutive convolutions to obtain an attention matrix; convolving the attention matrix and the value vector V to obtain global information.

2. The method of claim 1, wherein, wherein, the key frame sequence of the video to be identified is obtained by: determining total optical flow information of each video frame in the video to be identified based on an optical flow method; dividing all video frames into a plurality of categories by using a clustering method according to the total optical flow information of each video frame, and determining key frames in each category; all key frames constitute the key frame sequence.

3. A behavior recognition device based on semantic information enhancement of a Transformer, characterized by, The method comprises the following steps: a key frame sequence obtaining module is configured to obtain a key frame sequence of a video to be identified, the key frame sequence comprising a plurality of key frames; a spatio-temporal information obtaining module is configured to obtain spatio-temporal information of the key frame sequence; a convolutional neural network is configured to input the spatio-temporal information of the key frame sequence into the convolutional neural network to extract an information matrix of the key frame sequence containing low-level information; a self-attention mechanism is configured to input the information matrix of the key frame sequence containing low-level information into the self-attention mechanism to extract local information and global information, and fuse the local information and the global information to obtain fused information; a first feedforward neural network module is configured to input the fused information into the first feedforward neural network module; a second feedforward neural network module is configured to input an output of the first feedforward neural network module into the second feedforward neural network module to determine a category of the video to be identified; the spatio-temporal information obtaining module is further configured to: input the key frame sequence to a linear layer, and extract spatial information of the key frame sequence; extract time sequence information of the key frame sequence according to an index of each key frame in the key frame sequence; fuse the spatial information and the time sequence to obtain space-time information of the key frame sequence; the self-attention mechanism is further configured to: map an information matrix containing low-level information of the key frame sequence to obtain mapped information; determine a query vector Q, a key vector K and a value vector V according to the mapped information; process the key vector K by using a cavity convolution to obtain local information; concatenate the local information and the query vector Q, and perform two continuous convolution processes to obtain an attention matrix; perform convolution on the attention matrix and the value vector V to obtain global information.

4. The apparatus of claim 3, wherein, wherein, the key frame sequence acquisition module is further configured to: determine total optical flow information of each video frame in the to-be-identified video based on an optical flow method; determine key frames in each category by using a clustering method according to the total optical flow information of each video frame, and divide all video frames into multiple categories; all the key frames constitute the key frame sequence.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the behavior recognition method based on semantic information enhancement of the Transformer in claim 1 or 2.

6. A computer program product, characterised in that, The computer program / instruction is executed by the processor to implement the behavior recognition method based on semantic information enhancement of the Transformer in claim 1 or 2.

Images