A human behavior recognition method based on a dual-flow network architecture ViT

Abstract

The application discloses a human behavior recognition method based on a ViT of a double-flow network architecture, first pre-processes an input video to obtain a scene image and an image after sparse sampling, then calculates a flow image corresponding to the image through the processed image, then respectively establishes a spatial feature extraction network and a time sequence feature extraction network model, respectively inputs the RGB image and the flow image into the network model, trains to obtain a human behavior classification result, and weights the results of the two networks to obtain a final result. Through the above mode, the human behavior recognition of the video is completed, the application can fully utilize the time sequence feature of the action to assist the image recognition of the occurrence of the behavior action, through the advantage of the time sequence feature processing of the ViT, the time sequence feature is integrated, and the accuracy of the behavior recognition is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly designs a human behavior recognition method based on a ViT under a dual-flow network architecture. BACKGROUND

[0002] Human action recognition (HAR) is an important topic in the field of computer vision, as it has a wide range of applications in areas such as assisted living, intelligent surveillance systems, human-computer interaction, computer games, and affective computing. Depending on the target application, action recognition technology can be used to recognize full-body behavior, as well as partial body gesture recognition and facial recognition.

[0003] The research on video-based behavior recognition technology has caused a research boom in recent times due to the introduction of the VisionTranformer architecture in the field of vision. This is because the importance and wide range of video behavior recognition technology should be used in the scene, so people's research on it has been high. However, unlike the target recognition field, behavior recognition not only needs to analyze the spatial dependence of the target, but also needs to analyze the historical change information of the target in the past time. For example, when monitoring the elderly and children, full-body behavior recognition is essential, while in human-computer interaction, a gesture recognition system is more suitable. Due to the background clutter, partial occlusion, scale, viewpoint, light and appearance changes, it is often a challenging task to recognize human motion from video or image sequences. In addition, there are large intra-class and inter-class differences, and the same action may have great differences in different people, and the speed of the person performing the action changes greatly over time, making it difficult to determine the starting point of the action, thereby affecting the feature representation of the video when extracting features.

[0004] Therefore, how to effectively obtain the action information features of the human body from the video is an important technical basis for improving the performance of the behavior recognition algorithm. The traditional behavior recognition technology is to extract hand-crafted features (HOG, HOF, DenseTrajectories, etc.) first, and then use a classifier for classification. However, the limitations of hand-crafted features are relatively large, and the effect is not very ideal. The action feature extraction of the deep learning method is to extract deep action features through a feature extraction network, and then perform recognition processing. The temporal features of the action are not utilized. SUMMARY

[0005] The application provides a ViT human recognition method based on a dual-flow network architecture to solve the problem of integrating action temporal features into the human behavior recognition method in the prior art and improve the recognition accuracy.

[0006] In order to achieve the above purpose, the application adopts the following technical scheme:

[0007] A human behavior recognition method based on a dual-stream network architecture ViT, comprising the following steps:

[0008] Step 1: Preprocess the input video to convert the video stream data into image RGB frames and TVL1 optical flow images;

[0009] Step 2: Establish feature extraction models for image RGB frames and TVL1 optical flow images, respectively;

[0010] Step 3: Perform Softmax classification on the features of the RGB frames and TVL1 optical flow images, and perform weighted sum on the results between multiple frames to obtain the result;

[0011] Step 4: Input the corresponding video data, obtain the RGB frames and corresponding optical flow images of the video after preprocessing, and input them into the trained dual-stream ViT model to obtain the classification result.

[0012] Further, the specific steps of step 1 are:

[0013] Step 11: Extract the video into frame images using tools;

[0014] Step 12: Use the sparse sampling method proposed by TSN to sample the input video, divide the frame set of the video into N segments on average, randomly extract 1 frame from each video segment, and use opencv to calculate the TVL1 optical flow of the image to obtain the corresponding x, y optical flow map, and also perform segmented frame extraction;

[0015] Step 13: Crop the obtained images to 224x224 and perform data enhancement processing at the same time.

[0016] Further, the specific steps of step 2 are:

[0017] Step 21: Select a composite network recognition model using Resnet50+ViT-B / 16, which has been pre-trained on ImageNet21K;

[0018] Step 22: Train the optical flow network and the RGB image network separately, the RGB image network mainly obtains the spatial features of the video action, and the optical flow network mainly obtains the time sequence features of the video action. At the same time, since the optical flow picture is single-channel data and the RGB image is three-channel data, the pre-convolution of the optical flow network part will be changed to double-channel to adapt to the input of the optical flow image.

[0019] Further, the specific steps of step 3 are:

[0020] Step 31: Given a video V, combined with the processing of video data in step 1, it is divided into K segments, K video segments are recorded as {S1, S2, S3, …, S K}, the recognition result R(T1, T2, T3, …, T K ) of each video is represented as:

[0021] R(T1, T2, T3, …, T K ) = F(H(H(T1, W), G(T2, W), …, G(T K , W))

[0022] Where: (T1, T2, T3, …, T K ) is a segment sequence, which is randomly sampled from each frame T K in the corresponding S K video segment; G(T K , W) is the recognition result of T K frame in all categories by the recognition model with parameter W; the function H is used to combine the output results of multiple frames, the function H selects the avg function to calculate the average value in each category, and the function F is a prediction function for predicting the result of the whole video. This patent trains the model by combining the Softmax function and the classification cross-entropy loss function, and the loss function L(m, R) is represented as:

[0023]

[0024] Where: C is the number of action categories, i and j are specific categories, m i is the classification label i corresponding to each video, R i and R j are the recognition results of each video in the corresponding category.

[0025] Step 32: Given a video V, combined with the processing of video data in step 1, it is divided into K segments, K video segments are recorded as {S1, S2, S3, …, S K}, the recognition result R(T x1,y1 , T x2,y2 , T x3,y3 , …, T xK,yK ) of each video is represented as:

[0026] R(T x1,y1 , T x2,y2 , T x3,y3 , …, T xK,yK ) = F(H(G(T x1,y1 , W), G(T x2,y2 , W), …, G(T xK,yKW)

[0027] wherein: (T x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK ) is a fragment sequence, the fragment sequence is randomly sampled from each frame T K in the corresponding S K video segment; G(T xK,yK ,W) is the identification result of the x,y direction optical flow of the frame T K in all categories by the identification model with the parameter W; the function H is used to combine the output results of multiple frames, the function H selects the avg function to obtain the average value in each category, the F function is a prediction function for predicting the result of the whole video, the model is trained by combining the Softmax function and the classification cross-entropy loss function, and the loss loss function L(m,R) is represented as:

[0028]

[0029] wherein: C is the number of action categories, i and j are specific categories, m i is the classification label i corresponding to each video, R i and R j are the identification results of each video in the corresponding category,

[0030] Step 33: after the video is input into the network, the respective classification result weights of the RGB network and the optical flow network are obtained, and finally the results of the two networks are combined by weighting to obtain a prediction result.

[0031] Compared with the prior art, the application has the following beneficial effects:

[0032] The human body recognition method of the ViT based on the dual-stream network architecture of the application first pre-processes the input video to obtain a scene image and a sparsely sampled image, then calculates the optical flow image corresponding to the processed image, then establishes a spatial feature extraction network and a time sequence feature extraction network model respectively, inputs the RGB image and the optical flow image into the network model respectively, trains to obtain a human behavior classification result, and weights the results of the two networks to obtain a final result. Through the above mode, the human behavior recognition of the video is completed, and the application can fully utilize the time sequence features of the action to assist the image recognition of the action, and through the advantages of the ViT in processing the time sequence features, the time sequence features are integrated, and the accuracy of the behavior recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1A flowchart of the present application;

[0034] Figure 2 A human behavior recognition network model diagram of the ViT of the dual-stream network architecture;

[0035] Figure 3 A ViT network structure diagram of the dual-stream network architecture. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with examples.

[0037] Example 1

[0038] As shown in the figure, a human recognition method based on the ViT of the dual-stream network architecture comprises the following steps: Figure 1

[0039] Step 1: Preprocess the input video to convert the video stream data into image RGB frames and TVL1 optical flow images;

[0040] Step 2: Establish feature extraction models for image RGB frames and TVL1 optical flow images, respectively;

[0041] Step 3: Perform Softmax classification on the features of the RGB frames and TVL1 optical flow images, and obtain the results by weighting the results between multiple frames;

[0042] Step 4: Input the corresponding video data, obtain the RGB frames and corresponding optical flow images of the video after the previous preprocessing, and input them into the trained dual-stream ViT model to obtain the classification results.

[0043] The step 1 in the present example 1 is specifically:

[0044] Divide all images in the video into single image frames, average them into N segments according to the number of frames, randomly extract a frame from each segment, complete sparse random sampling, and use OpenCV to calculate the TVL1 optical flow image on the divided image frames, and also use the random sampling method to randomly extract the optical flow images in x and y directions from each segment;

[0045] The preprocessed images are all center-cropped and resized to 224x224.

[0046] The step 2 in the present example 1 is specifically:

[0047] 21. Select a composite network recognition model using Resnet50+ViT-B / 16, which has been pre-trained on ImageNet21K

[0048] ​22. The optical flow network and the RGB image network are trained separately. The RGB image network mainly acquires the spatial features of video actions, while the optical flow network mainly acquires the temporal features of video actions.

[0049] In this embodiment, step 3 specifically refers to:

[0050] 31. Given a video V, using the video data processing steps in step one, divide it into K equal segments {S1, S2, S3, ..., S...}. K The recognition result for each video can be represented as:

[0051] R(T1,T2,T3,…,T K )=F(H(G(T1,W),G(T2,W),…,G(T K ,W)))

[0052] Here (T1, T2, T3, ..., T k ) is a sequence of segments, derived from the corresponding S k Each frame T randomly sampled from the video segment k G(T) k ,W) represents the recognition result of the recognition model with parameter W for frame H in all categories. Function T is used to combine the output results of multiple frames; this patent chooses the avg function to calculate its average value for each category. Function F is the prediction function for predicting the result of the entire video; this patent uses it to train the model by combining the Softmax function with the classification cross-entropy loss function. The loss function is expressed as: Where C is the number of action categories, y i Tags corresponding to each video.

[0053] 32. The above describes the RGB process of an image. The recognition process of an optical flow network is similar. Finally, when combining the results of the two networks, the results of the two networks are weighted and combined to obtain the prediction result.

[0054] Specifically: based on the human recognition model of the ViT under the dual-flow network architecture, the composite model of Resnet50 and ViT-B / 16 is adopted, the recognition model of Vision Transformer Hybrid used in the model is different from the classic Vision Transformer model, which directly maps the image into token for input and feature extraction, and then recognizes, in the Hybrid model, the image will first pass through the ResNet network for feature extraction, and the extracted features will be flattened and mapped into the feature vector of the Token size required by the Transformer architecture through the Linear layer, that is, the input of the Vision Transformer in the Hybrid model is the feature map extracted by the Resnet network.

[0055] The convolution in the network uses the weight standardization convolution, and the three Block blocks in Stage4 in ResNet50 are moved into Stage3. The multi-frame input of the video will make the BatchSize of the input smaller, and the normalization of the BN layer is not as good as the effect of the Group Norm layer in the case of smaller BatchSize, so the GN layer is used in the Resnet network in the Hybrid model to normalize the single-frame multi-channel features and improve the recognition ability. In order to improve the generalization ability of the model, a data increasing module is also added to the model to improve the recognition effect of the model by flipping the image up and down and left and right.

[0056] The training process and test of the model are as shown in Figure 2 The spatial feature extraction network and the temporal feature extraction network are trained respectively, the image frames extracted from each video are input into the network, the classification results of each frame in all behavior classifications are obtained, and finally the multi-frame results are averaged to obtain the final classification result. Back propagation, random gradient update, and finally the converged model is obtained.

[0057] During the model test, the image frames and the optical flow frames corresponding to the video are input into the spatial feature extraction network and the temporal feature extraction network, respectively, the results of the two networks are weighted and summed, and the spatial network result and the temporal network result are added in a 1:1 ratio, and then input into the Softmax selector, and finally the prediction result is obtained.

[0058] The step 4 in the embodiment 1 is specifically:

[0059] The corresponding video data is input, the RGB frame corresponding to the video and the corresponding optical flow image after pre-processing are obtained, and the trained dual-flow ViT model is input to obtain the classification result.

[0060] The application is a human behavior recognition method based on a ViT of a double-flow network architecture, which can complete video multi-task content recognition, fully utilizes the time sequence characteristics of actions, and uses the advantages of the ViT model in processing time sequence data to assist the RGB space features to complete the behavior recognition task, thereby improving the behavior recognition accuracy.

[0061] Embodiment 2

[0062] A human behavior recognition method based on a ViT of a double-flow network architecture, characterized in that it comprises the following steps:

[0063] Step 1: pre-processing the input video to convert the video stream data into image RGB frames and TVL1 optical flow images;

[0064] The specific steps of step 1 are:

[0065] Step 11: extracting the video into frame images through a tool;

[0066] Step 12: using the sparse sampling method proposed by TSN to sample the input video, dividing the frame set of the video into N segments on average, randomly extracting 1 frame from each video segment, and using opencv to calculate the TVL1 optical flow of the image to obtain the corresponding x, y optical flow map, and also performing segmented frame extraction;

[0067] Step 13: cropping the obtained image into 224x224 and simultaneously performing data enhancement processing.

[0068] Step 2: respectively establishing feature extraction models for image RGB frames and TVL1 optical flow images;

[0069] The specific steps of step 2 are:

[0070] Step 21: selecting a composite network recognition model using Resnet50+ViT-B / 16, which has been pre-trained on ImageNet21K;

[0071] Step 22: training the optical flow network and the RGB image network separately, the RGB image network mainly obtains the spatial features of the video action, and the optical flow network mainly obtains the time sequence features of the video action. At the same time, since the optical flow picture is single-channel data and the RGB image is three-channel data, the pre-convolution of the optical flow network part will be changed to double-channel to adapt to the input of the optical flow image.

[0072] Step 3: performing Softmax classification on the features of the RGB frames and the TVL1 optical flow images, and weighting the results between multiple frames to obtain the result;

[0073] The specific steps of step 3 are:

[0074] Step 31: Given a video V, and combining the video data processing from Step 1, divide it into K equal segments, denoted as {S1, S2, S3, ..., S...}. K The recognition result for each video is R(T1,T2,T3,…,T). K ) is represented as:

[0075] R(T1,T2,T3,…,T K )=F(H(G(T1,W),G(T2,W),…,G(T K ,W)))

[0076] Where: (T1,T2,T3,…,T) K ) is a sequence of segments, which is derived from the corresponding S K Each frame T randomly sampled from the video segment K G(T) K (W) represents the recognition model with parameter W for T. K The recognition results of the frame in all categories; function H is used to combine the output results of multiple frames. The function H selects the avg function and calculates its average value in each category. The F function is the prediction function for predicting the result of the entire video. In this embodiment 2, the model is trained by combining the Softmax function with the classification cross-entropy loss function. The loss function L(m,R) is expressed as:

[0077]

[0078] Where: C represents the number of action categories, i and j are specific categories, and m i For each video, the corresponding category label i, R i and R j It is the recognition result of each video in the corresponding category;

[0079] Step 32: Given a video V, and combining the video data processing from Step 1, divide it into K equal segments, denoted as {S1, S2, S3, ..., S...}. K The recognition result R(T) for each video x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK ) is represented as:

[0080] R(T x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK )=F(H(G(T x1,y1 ,W),G(T x2,y2T xK,yK T

[0081] T x1,y1 T x2,y2 T x3,y3 T xK,yK is a segment sequence, which is randomly sampled from the corresponding S K video segment; T K corresponds to the optical flow image in x, y direction; G(T xK,yK , W) is the recognition result of the optical flow in x, y direction of T K frame in all categories by the recognition model with parameter W; the function H is used to combine the output results of multiple frames, the function H selects the avg function to calculate the average value in each category, and the function F is a prediction function for predicting the result of the whole video; the embodiment 2 is to train the model by combining the Softmax function and the classification cross-entropy loss function, and the loss function L(m, R) is expressed as:

[0082]

[0083] wherein C is the number of categories of actions, i and j are specific categories, m i is the classification label i corresponding to each video, R i and R j are the recognition results of each video in the corresponding category,

[0084] Step 33: After the video is input into the network, the respective classification result weights of the RGB network and the optical flow network are obtained, and the results of the two networks are combined by weighting when the two network results are combined, to obtain the prediction result.

[0085] Step 4: The corresponding video data is input, the RGB frame and the corresponding optical flow image corresponding to the video after the pre-processing are obtained, and the double-flow ViT model trained is input to obtain the classification result.

[0086] The above only describes the preferred embodiments of the present application, and it should be noted that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A human behavior recognition method based on a dual-flow network architecture ViT, characterized in that, The method comprises the following steps: Step 1: preprocessing the input video, converting the video stream data into image RGB frames and TVL1 optical flow images; Step 2: respectively establishing feature extraction models of the image RGB frames and the TVL1 optical flow images; The step 2 comprises the following steps: Step 21: selecting a composite network recognition model using Resnet50+ViT-B / 16, which is pre-trained on ImageNet21K; Step 22: training the optical flow network and the RGB image network separately, the RGB image network mainly obtains the spatial features of the video action, and the optical flow network mainly obtains the time sequence features of the video action, and since the optical flow picture is single-channel data and the RGB image is three-channel data, the pre-convolution of the optical flow network part is changed to double channels to adapt to the input of the optical flow image; Step 3: performing Softmax classification on the features of the RGB frames and the TVL1 optical flow images, and weighting the results between multiple frames to obtain the result; The step 3 comprises the following steps: Step 31: Given a video V, and combining the video data processing from Step 1, divide it into K equal segments, denoted as {S1, S2, S3, ..., S...}. K The recognition result for each video is R(T1,T2,T3,…,T). K ) is represented as: R(T1, T2, T3,..., T K ) = F(H(G(T1, W), G(T2, W),..., G(T K , W))) Where: (T1,T2,T3,…,T) K ) is a sequence of segments, which is derived from the corresponding S K Each frame T randomly sampled from the video segment K G(T) K (W) represents the recognition model with parameter W for T. K The recognition results of the frame across all categories; function H is used to combine the output results of multiple frames. The avg function is chosen for function H to calculate its average value across each category. Function F is the prediction function for the entire video. The model is trained using a combination of the Softmax function and the classification cross-entropy loss function. The loss function L(m,R) is expressed as: wherein: C is the number of categories of actions, i and j are one specific category, m i is the classification label i corresponding to each video, R i and R j is the recognition result of each video in the corresponding category; Step 4: inputting the corresponding video data, obtaining the corresponding RGB frames and the corresponding optical flow images of the video after the pre-processing, and inputting the trained double-flow ViT model to obtain the classification result.

2. The human behavior recognition method based on the dual-flow network architecture ViT according to claim 1, characterized in that, The step 1 comprises the following steps: Step 11: extracting the video into frame images through a tool; Step 12: sampling the input video by using the sparse sampling method proposed by TSN, dividing the frame set of the video into N segments, randomly extracting 1 frame from each video segment, calculating the corresponding x, y optical flow map by using opencv for TVL1 optical flow calculation, and also performing segment frame extraction; Step 13: cropping the obtained images into 224x224 and simultaneously performing data enhancement processing.

3. The human behavior recognition method of the ViT based on the dual-flow network architecture according to claim 1, characterized in that, The step 3 further comprises the following steps: Step 32: Given a video V, combine the processing of the video data in step 1 to divide it evenly into K segments, the K video segments are denoted as {S1, S2, S3, …, SK}, the recognition result of each video is denoted as R(T1, T2, T3, …, TK). K} and the recognition result of each video is denoted as R(T x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK ). R(T x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK ) = F(H(G(T x1,y1 ,W), G(T x2,y2 ,W), …, G(T xK,yK ,W))) where: (T x1,y1 ,T x2,y2 ,T x3,y3 ,…,T xK,yK ) is a sequence of segments, which is randomly sampled from the corresponding S K video segment; each frame T K corresponds to the optical flow image in x, y direction; G(T xK,yK ,W) is the recognition result of the optical flow in x, y direction of the frame T K in all categories by the recognition model with parameters W; the function H is used to combine the output results of multiple frames, and the function H selects the avg function to obtain the average value in each category; the F function is a prediction function for predicting the result of the whole video, and the model is trained by combining the Softmax function and the classification cross-entropy loss function; and the loss function L(m,R) is represented as: where: C is the number of categories of actions, i and j are one specific category, m i R is the recognition result of each video in the corresponding category, and i R is the recognition result of each video in the corresponding category, and j is the recognition result of each video in the corresponding category, Step 33: after inputting the video into the network, obtaining the respective classification result weights of the RGB network and the optical flow network, and when combining the results of the two networks, the results of the two networks are weighted and combined to obtain the prediction result.

Images