A video behavior recognition system and method based on spatiotemporal sequence model

Abstract

The present invention discloses a video behavior recognition system and method based on a spatiotemporal sequence model, including a brightness stream sampling module, a time modeling module, a space modeling module and a classification output module. The brightness stream sampling module is used to sample brightness data of a video file or a video stream and use it as the input of a neural network; the time modeling module is used to perform time modeling on spatiotemporal feature data; the space modeling module is used to perform space modeling on spatiotemporal feature data; the classification output module classifies and fuses the spatiotemporal feature data to obtain a classification result for the entire video. The present invention solves the problem of low recognition rate of RGB input mode through the brightness stream sampling module; and solves the problems of high complexity, slow running speed and low recognition accuracy of 3D spatiotemporal modeling methods through spatiotemporal sequence modeling.

Description

Technical Field

[0001] The present invention belongs to the technical field of video behavior recognition, and in particular relates to a video behavior recognition method based on spatiotemporal sequence modeling. Background Art

[0002] In the information age, a large amount of video data is generated all the time. The development of edge computing and communication technology has made video tasks increasingly important. Action recognition is at the core of video tasks. The purpose of action recognition is to determine the behavior category of a person in a video. With the development of the industry, the requirements for action recognition are becoming higher and higher, such as autonomous driving. Compared with images, videos have one more dimension of time series information. How to make good use of video time series information is the key to action recognition.

[0003] In the field of video behavior recognition, the most common method is to use 3D networks for spatiotemporal modeling, such as I3D and X3D. 3D networks have achieved good results, but they have many parameters, large computational workload, and high memory consumption, which limits their practicality. Other methods use two-stream networks to model space and time respectively, training RGB input (spatial stream) and optical flow input (temporal stream) respectively, and finally taking the average of the two data streams to obtain the final prediction result. Two-stream networks have been proven to be effective, but optical flow calculations require huge computational workloads and storage space, and it is difficult to meet the real-time requirements of videos, which limits the application of optical flow. Other researchers have proposed (2+1)D networks, which decompose 3D networks into 2D spatial networks and 1D temporal networks. By decomposing the 3D network, the 3D convolution kernel size is reduced from m*n*t to m*n+t (m and n are the spatial convolution kernel size, and t is the temporal convolution kernel size), and achieved recognition results comparable to those of 3D networks. For the (2+1)D network, the spatial network can be pre-trained on the image dataset to obtain the initial parameters of the spatial network. However, the (2+1)D network does not explore the initialization method of the temporal network parameters, which results in the entire network model not necessarily being optimal. Summary of the invention

[0004] The present invention proposes a video behavior recognition method based on a spatiotemporal sequence model, which performs time modeling, space modeling, classification and fusion on spatiotemporal feature data to obtain a classification result for the entire video.

[0005] The present invention is implemented by the following technical solutions:

[0006] A video behavior recognition method based on a spatiotemporal sequence model, the method comprising the following steps:

[0007] Step 1, YCbCr data segmentation sampling: sampling the brightness data of the video file or video stream, specifically: obtaining YCbCr color space data from the video file or video stream; segmenting and sampling the YCbCr color space data to extract the brightness component data; the relevant model is described as follows:

[0008] The YCbCr data {f1, f2, ..., f N}Evenly divide it into T segments, and randomly sample 3 consecutive frames of YCbCr data from each segment. The expression is as follows:

[0009] {f1, f2, ..., f N}={{f1,f2,...,f n}, {f n+1 , f n+2 , ..., f 2n}, ..., {f (T-1)n+1 , f (T-1)n+2 , ..., f N}}

[0010]

[0011] Among them, S represents a random function, k represents a segment, represents three consecutive frames of YCbCr data randomly sampled from the kth segment, and n represents the number of frames contained in each segment;

[0012] For compressed video files and video streams, you need to first decode the video file or video stream to obtain the video color space YCbCr data, the expression is as follows:

[0013] {f1, f2, ..., f N}=Decode(input)

[0014] Where Decode represents a video decoder, input represents an input video file or video stream, N represents the number of frames of a video file or video stream, {f1, f2, ..., f N} indicates that YCbCr data is obtained after video decoding. For video devices using YCbCr color space, this step can be skipped to directly obtain YCbCr data;

[0015] Step 2: Extract brightness data and overlay: For three consecutive frames of randomly sampled YCbCr data The brightness component (Y component) data is extracted separately and superimposed to form the brightness stream data. The expression is as follows:

[0016]

[0017] Among them, H represents Extract the brightness component data, Represents the brightness component data, Y k Indicates that The brightness stream data is composed of superposition, h and w represent the width and height of the video, Represents Y k is a 3×h×w matrix belonging to the real number field;

[0018] Get the sampling data X of the entire video file, the expression is as follows:

[0019] in:

[0020] Where T represents the number of video segments.

[0021] Step 3: Extract spatiotemporal feature data: Input the brightness stream data into a 7×7 two-dimensional convolutional layer to obtain the spatiotemporal feature data F, which is expressed as follows:

[0022] F=K0*X

[0023] Among them, K0 represents a two-dimensional convolutional layer with a kernel size of 7×7, and * represents a convolution operation;

[0024] Step 4: Reduce the dimension of feature channels: Input the spatiotemporal feature data F into a two-dimensional convolutional layer with a kernel size of 1×1 to obtain the spatiotemporal feature data F d , the expression is as follows:

[0025] F d =K1*F

[0026] Among them, K1 represents a two-dimensional convolutional layer with a kernel size of 1×1, and F represents the input spatiotemporal feature data;

[0027] Step 5: Time modeling: transform the spatiotemporal feature data F d Input a one-dimensional convolutional layer with a kernel size of 3 to obtain the temporal model F t , the expression is as follows:

[0028] F t =K t *F d

[0029] InitK t F t =F d

[0030] Among them, K t represents a one-dimensional convolutional layer with a kernel size of 3, F t InitK represents the spatiotemporal feature data after time modeling. t Ft =F d K t The parameters of F are specially initialized so that t =F d ;

[0031] Step 6: Spatial modeling: transform the spatiotemporal feature data F t Input a 2D convolutional layer with a kernel size of 3×3 to obtain the spatial modeling F s , the expression is as follows:

[0032] F s =K s *F t

[0033] Among them, K s represents a 2D convolutional layer with a kernel size of 3×3, F s Represents the spatiotemporal feature data after spatial modeling;

[0034] Step 7: Restore the feature channel dimension: Restore the spatiotemporal feature data F s Input a two-dimensional convolutional layer with a kernel size of 1×1 to obtain the spatiotemporal feature data F u , the expression is as follows:

[0035] F u =K2*F s

[0036] Among them, K2 represents a two-dimensional convolutional layer with a kernel size of 1×1;

[0037] Step 8: loop the above steps 4 to 7 N times, where N is set according to the actual situation, including the GPU memory size, parameter size, calculation size, real-time requirements, and accuracy requirements;

[0038] Step 9: Get the segmentation result through the fully connected layer: transform the spatiotemporal feature data F u Input the fully connected layer to get the classification result z for each segment t , the expression is as follows:

[0039] z t =Z(F k ), 1≤k≤T

[0040] Among them, Z represents the fully connected layer, F k represents the spatiotemporal feature data of the kth time period, T represents the number of video segments, z t Represents the classification result of the tth time period;

[0041] Step 10: Classification results of all segments z1, z2, ..., z T Perform average fusion;

[0042] Step 11: Get the classification result L of the entire video, expressed as follows:

[0043]

[0044] Here, Avg represents the average function.

[0045] Compared with the prior art, the beneficial effects and advantages of the present invention are:

[0046] 1. The present invention discloses a video behavior recognition method based on a spatiotemporal sequence model. The brightness stream sampling module included in the method has the following advantages: replacing the RGB image of the traditional method with the brightness stream can improve the spatiotemporal modeling ability of the model; the brightness stream is based on the video color space YCbCr, which is more conducive to deployment on video devices; the input size of the brightness stream is the same as that of the RGB image, and there is no need to modify the known network model.

[0047] 2. The present invention discloses a video behavior recognition method based on a spatiotemporal sequence model. The method includes a temporal modeling module and a spatial modeling module, which decomposes the spatiotemporal modeling into temporal modeling and spatial modeling executed in sequence, thereby reducing the complexity of the spatiotemporal modeling, reducing the amount of calculation, and improving the recognition accuracy.

[0048] 3. Compared with the prior art, the beneficial effects brought about by the technical solution of the present invention are: the present invention improves the video behavior recognition system in terms of performance and efficiency, and achieves system acceleration and video memory saving. We verified the method of the present invention on the video classification task. Among them, on the Something-SomethingV1 dataset, compared with the TEA method, the training time of the method of the present invention was reduced by 36.3%, the accuracy was improved by 0.8%, the throughput was improved by 11.7%, and the latency was reduced by 43.6%. Experiments show that the method of the present invention has great advantages in terms of accuracy and efficiency compared with the baseline system (TEA, I3D). BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A module diagram of a video behavior recognition system based on a spatiotemporal sequence model of the present invention;

[0050] Figure 2 A flow chart of a video behavior recognition method based on a spatiotemporal sequence model of the present invention;

[0051] Figure 3 The figure is a schematic diagram of an embodiment of a video behavior recognition method based on a spatiotemporal sequence model of the present invention. DETAILED DESCRIPTION

[0052] The specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, it is a module diagram of a video behavior recognition system based on a spatiotemporal sequence model of the present invention. The system includes a brightness flow sampling module 100, a time modeling module 200, a space modeling module 300, and a classification output module 400. Among them:

[0054] The brightness stream sampling module 100 is used to sample brightness data of a video file or a video stream and use it as the input of a neural network, wherein: the video file or the video stream is decoded to obtain YCbCr color space data; the decoded YCbCr data is segmented and sampled, and the brightness component (Y component) data is extracted; and three consecutive frames of brightness component data are superimposed to form a brightness stream as the input of the neural network.

[0055] The temporal modeling module 200 is used to perform temporal modeling on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 1×1 to reduce the feature channel dimension; and the reduced spatiotemporal feature data is input into a one-dimensional convolutional layer with a kernel size of 3 to perform temporal modeling.

[0056] The spatial modeling module 300 is used to perform spatial modeling on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 3×3 to perform spatial modeling.

[0057] The classification output module 400 classifies and fuses the spatiotemporal feature data to obtain the classification result of the entire video, wherein: the spatiotemporal feature data of each segment is input into the fully connected layer to obtain the classification result of each segment; and the classification results of all segments are averaged and fused to obtain the classification result of the entire video.

[0058] like Figure 2 As shown in FIG. 1 , a flow chart of a video behavior recognition method based on a spatiotemporal sequence model of the present invention is shown. The specific steps included in the flow chart are as follows:

[0059] Step 1, YCbCr data segmentation sampling: sampling the brightness data of the video file or video stream, specifically: obtaining YCbCr color space data from the video file or video stream; segmenting and sampling the YCbCr color space data to extract the brightness component data; the relevant model is described as follows:

[0060] The YCbCr data {f1, f2, ..., f N}Evenly divide it into T segments, and randomly sample 3 consecutive frames of YCbCr data from each segment. The expression is as follows:

[0061] {f1, f2, ..., f N}={{f1,f2,...,f n}, {fn+1 , f n+2 , ..., f 2n}, ..., {f (T-1)n+1 , f (T-1)n+2 , ..., f N}}

[0062]

[0063] Among them, S represents a random function, k represents a segment, represents three consecutive frames of YCbCr data randomly sampled from the kth segment, and n represents the number of frames contained in each segment;

[0064] For compressed video files and video streams, you need to first decode the video file or video stream to obtain the video color space YCbCr data, the expression is as follows:

[0065] {f1, f2, ..., f N}=Decode(input)

[0066] Where Decode represents a video decoder, input represents an input video file or video stream, N represents the number of frames of a video file or video stream, {f1, f2, ..., f N} indicates that YCbCr data is obtained after video decoding. For video devices using YCbCr color space, this step can be skipped to directly obtain YCbCr data;

[0067] Step 2: Extract brightness data and overlay: For three consecutive frames of randomly sampled YCbCr data The brightness component (Y component) data is extracted separately and superimposed to form the brightness stream data. The expression is as follows:

[0068]

[0069] Among them, H represents Extract the brightness component data, Represents the brightness component data, Y k Indicates that The brightness stream data is composed of superposition, h and w represent the width and height of the video, Represents Y k is a 3×h×w matrix belonging to the real number field;

[0070] Get the sampling data X of the entire video file, the expression is as follows:

[0071] in:

[0072] Where T represents the number of video segments.

[0073] Step 3: Extract spatiotemporal feature data: Input the brightness stream data into a 7×7 two-dimensional convolutional layer to obtain the spatiotemporal feature data F, which is expressed as follows:

[0074] F=K0*X

[0075] Among them, K0 represents a two-dimensional convolutional layer with a kernel size of 7×7, and * represents a convolution operation;

[0076] Step 4: Reduce the dimension of feature channels: Input the spatiotemporal feature data F into a two-dimensional convolutional layer with a kernel size of 1×1 to obtain the spatiotemporal feature data F d , the expression is as follows:

[0077] F d =K1*F

[0078] Among them, K1 represents a two-dimensional convolutional layer with a kernel size of 1×1, and F represents the input spatiotemporal feature data;

[0079] Step 5: Time modeling: transform the spatiotemporal feature data F d Input a one-dimensional convolutional layer with a kernel size of 3 to obtain the temporal model F t , the expression is as follows:

[0080] F t =K t *F d

[0081] Init K t F t =F d

[0082] Among them, K t represents a one-dimensional convolutional layer with a kernel size of 3, F t Init K represents the spatiotemporal feature data after time modeling. t F t =F d K t The parameters of F are specially initialized so that t =F d ;

[0083] Step 6: Spatial modeling: transform the spatiotemporal feature data F t Input a 2D convolutional layer with a kernel size of 3×3 to obtain the spatial modeling F s , the expression is as follows:

[0084] F s =K s *F t

[0085] Among them, Ks represents a 2D convolutional layer with a kernel size of 3×3, F s Represents the spatiotemporal feature data after spatial modeling;

[0086] Step 7: Restore the feature channel dimension: Restore the spatiotemporal feature data F s Input a two-dimensional convolutional layer with a kernel size of 1×1 to obtain the spatiotemporal feature data F u , the expression is as follows:

[0087] F u =K2*F s

[0088] Among them, K2 represents a two-dimensional convolutional layer with a kernel size of 1×1;

[0089] Step 8: loop the above steps 4 to 7 N times, where N is set according to the actual situation, including the GPU memory size, parameter size, calculation size, real-time requirements, and accuracy requirements;

[0090] Step 9: Get the segmentation result through the fully connected layer: transform the spatiotemporal feature data F u Input the fully connected layer to get the classification result z for each segment t , the expression is as follows:

[0091] z t =Z(F k ), 1≤k≤T

[0092] Among them, Z represents the fully connected layer, F k represents the spatiotemporal feature data of the kth time period, T represents the number of video segments, z t Represents the classification result of the tth time period;

[0093] Step 10: Classification results of all segments z1, z2, ..., z T Perform average fusion;

[0094] Step 11: Get the classification result L of the entire video, expressed as follows:

[0095]

[0096] Here, Avg represents the average function.

[0097] In summary, a video behavior recognition method based on a spatiotemporal sequence model of the present invention first samples brightness data of a video file or a video stream and uses it as the input of a neural network, wherein: the video file or the video stream is decoded to obtain YCbCr color space data; the decoded YCbCr data is segmented and sampled, and the brightness component (Y component) data is extracted; three consecutive frames of brightness component data are superimposed to form a brightness stream as the input of the neural network; then, temporal modeling is performed on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 1×1 to reduce the dimension of the feature channel; the reduced spatiotemporal feature data is input into a one-dimensional convolutional layer with a kernel size of 3 to perform temporal modeling; and spatial modeling is performed on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 3×3 to perform spatial modeling; the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 1×1 to restore the dimension of the feature channel. Temporal modeling and spatial modeling are repeatedly superimposed, and the number of superpositions is set according to the actual situation, including the GPU memory size, parameter size, calculation size, real-time requirements, and accuracy requirements. Finally, the spatiotemporal feature data is classified and fused to obtain the classification result of the entire video, wherein the spatiotemporal feature data of each segment is input into the fully connected layer to obtain the classification result of each segment; the classification results of all segments are averaged and fused to obtain the classification result of the entire video.

[0098] The present invention has conducted experiments on multiple datasets, including Kinetics-400, Something-SomethingV1&V2, UCF-101 and HMDB-51 datasets. Kinetics-400 is the first large-scale, high-quality video behavior recognition dataset. The Something-Something dataset records the interaction between people and some objects in daily life. It has two versions, V1 and V2. UCF-101 and HMDB-51 are two early video datasets. Table 1 lists the statistical information of each dataset.

[0099] Table 1 Statistics of the dataset

[0100] Dataset category Number of training videos Verification video number Kinetics-400 400 238,796 19,877 Something-Something V1 174 86,017 11,522 Something-Something V2 174 168,913 24,777 UCF-101 101 9,537 3,783 HMDB-51 51 3,570 1,530

[0101] First, the effectiveness of the method of the present invention is verified on 5 data sets, and the accuracy is used as the evaluation index of effectiveness. The specific experimental results are shown in Table 2.

[0102] Table 2 Experimental results of each method on 5 datasets

[0103] Model Kinetics-400 STH-V1 STH-V2 UCF-101 HMDB-51 TSN 69.1％ 19.7％ 27.8％ 86.4％ 53.7％ TSM 74.1％ 45.6％ 59.1％ 96.0％ 73.2％ STM 73.7％ - - 96.2％ 72.2％ TEA 75.0％ 48.9％ - 96.9％ 73.3％ I3D 74.2％ 41.6％ - 95.6％ 74.8％ Method of the present invention 75.3％ 49.7％ 62.2％ 96.7％ 76.6％

[0104] As shown in the table above, the method of the present invention has achieved the best accuracy on the four datasets of Kinetics-400, STH-V1, STH-V2, and HMDB-51. Compared with TSM, the accuracy of the method of the present invention on the Kinetics-400, STH-V1, STH-V2, UCF-101 and HMDB-51 datasets has increased by 1.2%, 4.1%, 3.1%, 0.7% and 3.4%, respectively. From the experimental results, it can be seen that compared with other behavior recognition methods, the method of the present invention has a significant accuracy advantage.

[0105] Next, the method of the present invention is tested on the Something-Something V1 dataset. We use accuracy as its effectiveness evaluation index, and latency and throughput as its efficiency evaluation index. The specific experimental results are shown in Table 3.

[0106] Table 3 Experimental results of various methods on STH-V1 dataset

[0107] Model Accuracy(%) Latency(ms) Throughput(vid / s) I3D 41.6 165.3 6.1 TSM 45.6 20.2 85.0 TEINet 47.4 36.5 46.9 TEA 48.9 36.7 74.6 Method of the present invention 49.7 20.7 83.3

[0108] As shown in Table 3, the method of the present invention achieves the best accuracy in the Something-Something V1 dataset. Compared with I3D, TSM, TEINet, and TEA, the accuracy of the method of the present invention is improved by 8.1%, 4.1%, 2.3%, and 0.8%, respectively. In addition, compared with the TEA method, the delay time of the method of the present invention is reduced by 10ms and the throughput is increased by 8.7vid / s. From the experimental results, it can be seen that compared with other methods, the method of the present invention has achieved obvious advantages in recognition accuracy, delay time, and throughput.

[0109] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Those skilled in the art may make various modifications, equivalent substitutions or variations without departing from the spirit and scope of the present invention, and all modifications, equivalent substitutions or variations made within the spirit and principles of the present application fall within the protection scope of the present invention as defined by the appended claims.

Claims

1. A video behavior recognition system based on spatiotemporal sequence model, characterized in that: The system includes a brightness flow sampling module, a time modeling module, a space modeling module and a classification output module, wherein: The brightness stream sampling module is used to sample brightness data of a video file or a video stream and use it as the input of a neural network, wherein: the video file or the video stream is decoded to obtain YCbCr color space data; the decoded YCbCr data is segmented and sampled, and brightness component data is extracted; three consecutive frames of brightness component data are superimposed to form a brightness stream as the input of the neural network; The temporal modeling module is used to perform temporal modeling on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 1×1 to reduce the feature channel dimension; the reduced spatiotemporal feature data is input into a one-dimensional convolutional layer with a kernel size of 3 to perform temporal modeling; The spatial modeling module is used to perform spatial modeling on the spatiotemporal feature data, wherein: the spatiotemporal feature data is input into a two-dimensional convolutional layer with a kernel size of 3×3 to perform spatial modeling; The classification output module classifies and fuses the spatiotemporal feature data to obtain the classification result of the entire video, wherein: the spatiotemporal feature data of each segment is input into the fully connected layer to obtain the classification result of each segment; the classification results of all segments are averaged and fused to obtain the classification result of the entire video.

2. A video behavior recognition method based on a spatiotemporal sequence model, characterized in that: The method comprises the following steps: Step 1: Obtain YCbCr color space data from a video file or video stream, segment and sample the YCbCr color space data, and extract brightness component data; the relevant model is as follows: The YCbCr data {f1,f2,…,f N }Evenly divide it into T segments, each segment contains n = N / T frames, and then randomly sample 3 consecutive frames of YCbCr data from each segment. The expression is as follows: {f1,f2,...f N }={f1,f2,…f n },{f n+1 ,f n+2 ,...,f 2n },...,{f (T-1)n+1 ,f (T-1)n+2 ,…,f N }} Among them, S represents a random function, k represents a segment, represents three consecutive frames of YCbCr data randomly sampled from the kth segment, and N represents the number of frames in the video file or video stream; Step 2: Extract brightness data and overlay: For three consecutive frames of randomly sampled YCbCr data The brightness component data is extracted separately and superimposed to form the brightness stream data. The expression is as follows: Among them, H represents Extract the brightness component data, Represents the brightness component data, Y k Indicates that The brightness stream data is composed of superposition, h and w represent the width and height of the video, Represents Y k is a 3×h×w matrix belonging to the real number field; Get the sampling data X of the entire video file, the expression is as follows: in: Where T represents the number of video segments; Step 3: Extract spatiotemporal feature data: Input the brightness stream data into a 7×7 two-dimensional convolutional layer to obtain the spatiotemporal feature data F, which is expressed as follows: F=K0*X Among them, K0 represents a two-dimensional convolutional layer with a kernel size of 7×7, and * represents a convolution operation; Step 4: Reduce the dimension of feature channels: Input the spatiotemporal feature data F into a two-dimensional convolutional layer with a kernel size of 1×1 to reduce the dimension of feature channels and obtain the spatiotemporal feature data F d , the expression is as follows: F d =K1*F Among them, K1 represents a two-dimensional convolutional layer with a kernel size of 1×1, and F represents the input spatiotemporal feature data; Step 5: Time modeling: transform the spatiotemporal feature data F d Input a one-dimensional convolution layer with a kernel size of 3 to perform time modeling and obtain the spatiotemporal feature data F after time modeling. t , the expression is as follows: F t =K t *F d Init K t ,s.t.F t =F d Among them, K t represents a one-dimensional convolutional layer with a kernel size of 3, F t InitK represents the spatiotemporal feature data after time modeling. t F t =F d K t The parameters of F are specially initialized so that t =F d ; Step 6: Spatial modeling: transform the spatiotemporal feature data F t Input a 2D convolutional layer with a kernel size of 3×3 to perform spatial modeling and obtain the spatiotemporal feature data F after spatial modeling. s , the expression is as follows: F S =K S *F t Among them, K s represents a 2D convolutional layer with a kernel size of 3×3, F s Represents the spatiotemporal feature data after spatial modeling; Step 7: Restore the feature channel dimension: Restore the spatiotemporal feature data F s Input a two-dimensional convolutional layer with a kernel size of 1×1 to obtain the spatiotemporal feature data F u , the expression is as follows: F u =K2*F s Among them, K2 represents a two-dimensional convolutional layer with a kernel size of 1×1; Step 8: loop the above steps 4 to 7 N times, where N is set according to the actual situation, including the GPU memory size, parameter size, calculation size, real-time requirements, and accuracy requirements; Step 9: Get the segmentation result through the fully connected layer: transform the spatiotemporal feature data F u Input the fully connected layer to get the classification result z for each segment t , the expression is as follows: With t =Z(F k ),1≤k≤T Among them, Z represents the fully connected layer, F k represents the spatiotemporal feature data of the kth time period, T represents the number of video segments, z t Represents the classification result of the tth time period; Step 10: Classification results of all segments z1, z2, …, z T Perform average fusion; Step 11: Get the classification result L of the entire video, expressed as follows: Here, Avg represents the average function.

3. The video behavior recognition method based on the spatiotemporal sequence model as claimed in claim 2, characterized in that: The step 1 specifically includes the following processing: For compressed video files and video streams, you need to first decode the video file or video stream to obtain the video color space YCbCr data, the expression is as follows: {f1,f2,...,f N }=Decode(input) Where Decode represents a video decoder, input represents an input video file or video stream, N represents the number of frames of a video file or video stream, {f1,f2,…,f N } indicates that YCbCr data is obtained after video decoding. For video devices that use the YCbCr color space, this step can be skipped to directly obtain YCbCr data.

Images