Video Behavior Recognition Method and System Based on Spatiotemporal Modeling

Abstract

The present invention discloses a video behavior recognition method and system based on spatiotemporal modeling, wherein the method comprises: S1: inputting a complete video into a video image sampling module, adopting a segmented random video frame sampling method to sample the video image, and obtaining a plurality of sampling frame images; S2: constructing a spatial attention module based on a triplet loss function, and performing feature extraction on the action parts of the plurality of sampling frame images to obtain spatial feature information; S3: constructing a temporal pyramid module, and sampling the obtained spatial feature information at different frame rates to obtain multiple groups of feature information; S4: adopting an early fusion method to fuse the obtained multiple groups of feature information, specifically adopting an add operation to splice the feature vectors of the multiple groups of features; S5: inputting the fused feature information into a Softmax classifier to perform video behavior classification, generating a probability label for each video behavior, and outputting the recognition result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a video behavior recognition method and system based on spatiotemporal modeling, which can be used for analysis and research of video behavior recognition. Background Art

[0002] Visual information accounts for a significant portion of the vast amount of information generated by human interaction with the outside world. In today's rapidly evolving world, with the rapid development of information transmission technology and the rise of short videos based on mobile devices and the internet, a vast amount of visual information is generated every moment. This makes it easier to acquire large amounts of video information, but at the same time, processing, identifying, and classifying this vast amount of video information presents a significant challenge. The cost of manually extracting information from numerous and complex videos is increasing, and the demand for using artificial intelligence to rapidly filter information is growing. Therefore, research on video behavior recognition is particularly important. Video behavior recognition is an important research direction in computer vision and a key task in video understanding. With the increasing adoption of informatization and the booming development of computer vision, video behavior recognition has found widespread applications in areas such as intelligent surveillance, smart classrooms, and intelligent early warning, and has become a research hotspot. Taking the protection field as an example, from accident warnings in cities to border protection warnings, they are all within the scope of safety protection. By applying video behavior recognition to this field, and the fact that computers can monitor for a long time, the specific circumstances of the accident can be quickly analyzed and early warnings can be issued, thereby greatly improving the efficiency of safety protection and reducing the consumption of manpower and material resources. It can bring great convenience to people's lives and has broad application prospects.

[0003] Currently, video action recognition methods are broadly divided into two categories: those based on traditional feature extraction and those based on deep learning, which utilizes neural networks to learn features. Traditional feature extraction-based methods achieve action recognition by manually extracting features that represent actions, but are also limited by the features extracted. While traditional manually extracted features have demonstrated promising performance through continuous improvement, they still have limitations, such as noise that significantly impacts performance. Continuous attempts are needed to improve manually extracted features, but the cost is prohibitive, leaving limited room for improvement. More powerful and robust spatiotemporal features are needed for better action recognition. To further improve the recognition rate of algorithms, with the advent of deep learning methods, methods based on automatic feature extraction using deep neural networks have been applied to human action recognition. However, existing deep learning-based human action recognition methods, such as two-stream convolutional neural networks and their variants, and 3D convolutional neural networks, while leveraging temporal information, still face challenges, such as long computational time, high computational cost, and poor real-time performance. In addition, existing deep learning methods do not fully utilize temporal information when extracting temporal features. When extracting spatial features, they are easily interfered by background information, extracting useless features and affecting the accuracy of video behavior recognition. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a video behavior recognition method and system based on spatiotemporal modeling. In view of the problem that existing deep learning methods are easily interfered by background information and easily extract invalid information when extracting spatial features, a spatial attention module based on a triplet loss function is constructed. By adopting the triplet loss function, after continuous learning, the extracted features are concentrated on the action part of the video image, and the extraction of invalid features in the background part is reduced, thereby improving the effectiveness of feature extraction and improving the video behavior recognition performance; in view of the problem that time information is not fully utilized in the existing technology, a time module is constructed. In the time module, the features of the single-layer output of the neural network are sampled according to different sampling frames, thereby constructing a time pyramid, performing multi-level extraction of time features in the video while reducing the difficulty and time requirements of fusion; then, through early feature fusion, features of different frame rates are fused, thereby making overall use of the short-term features and global time features of the video; finally, a Softmax classifier is used for classification, so that accurate judgment of the behavior in the input video can be obtained.

[0005] The present invention provides a method and system for video action recognition based on spatiotemporal modeling. The core of the system is a deep neural network that models spatiotemporal information. The network uses the ResNet (residual network) model as the backbone network, introduces a triplet loss function, and creates an attention module based on triplet loss. This module focuses only on the action part and ignores the background part to improve the network's extraction of action features, thereby achieving spatial information modeling. A temporal pyramid structure is introduced to construct a temporal module, which constructs a temporal pyramid structure at the feature level. This allows for multi-level utilization of temporal information, thereby achieving temporal information modeling. By modeling the spatiotemporal feature information in the video, the performance of video action recognition is ultimately improved.

[0006] To achieve the above objectives, the present invention provides a video behavior recognition method based on spatiotemporal modeling, which specifically includes:

[0007] Step S1: input the complete video into the video image sampling module, and use the segmented random video frame sampling method to sample the video image to obtain multiple sampling frame images;

[0008] Step S2: Construct a spatial attention module based on the triplet loss function, and extract features of the action parts of multiple sample frame images to obtain spatial feature information;

[0009] Step S3: constructing a temporal pyramid module and sampling the obtained spatial feature information at different frame rates to obtain multiple sets of feature information;

[0010] Step S4: using early fusion to fuse the obtained multiple sets of feature information, specifically using the add operation to splice the feature vectors of the multiple sets of features;

[0011] Step S5: Input the fused feature information into the Softmax classifier to classify the video behavior, generate a probability label for each video behavior, and output the recognition result.

[0012] In one embodiment of the present invention, the specific process of step S1 includes:

[0013] Step S11: Divide the input complete video sequence V into U segments evenly, namely S1, S2...S U , where U is a constant;

[0014] Step S12: Randomly sample each segment to obtain sample frame images T1, T2...T U .

[0015] In one embodiment of the present invention, the step S2 of constructing a spatial attention module based on a triplet loss function is specifically as follows:

[0016] Step S201: inserting a triplet loss function into a spatial module for extracting spatial features of an image in a backbone network, so as to selectively collect feature information in the spatial domain, wherein the backbone network is a ResNet network;

[0017] Step S202: Multiple feature maps are fed into a spatial module that inserts a triplet loss function for selection, and appropriate anchor examples, positive examples, and negative examples are selected, where the anchor example is obtained by subtracting the average value of each element of the positive example from each element of the positive example;

[0018] Step S203: The training space module continuously learns to make the feature extraction of the anchor example closer to the positive example, that is, the loss function is minimized. The loss function is:

[0019]

[0020] in, represents the Euclidean distance metric between the positive example and the anchor example, represents the Euclidean distance metric between the negative example and the anchor example, and α is the minimum interval between the two Euclidean distance metrics.

[0021] In one embodiment of the present invention, the specific process of extracting features from the action parts of the plurality of sample frame images to obtain spatial feature information in step S2 is as follows:

[0022] Step S211: performing spatial feature extraction on multiple sample frame images to obtain multiple frame feature maps, where the dimensions of each frame feature map are C×H×W, where C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0023] Step S212: reshape the features of any feature map A and convert the feature dimension to C×N, where N is the number of pixels of the feature map A;

[0024] Step S213: Use the Top-K ranking function to extract the top K attention graphs S p and the next K attention maps S n , where S p S is the feature that affects the behavior classification. n Features that are not related to behavior identification;

[0025] Step S214: p After reshaping to K×H×W dimensions, average pooling is performed;

[0026] Step S215: Use the Sigmoid function to perform normalization to obtain the spatial attention weight of the emphasized action part;

[0027] Step S216: Multiply the obtained attention weight by the feature map A element by element to obtain the output A of the spatial attention module * for:

[0028] A * =σ(Avgp(Resh(sP)))*A (2)

[0029] Among them, A * To emphasize the feature map of the action part, Resh is the dimension conversion, Avgp is the average pooling, and σ is the Sigmoid function.

[0030] In one embodiment of the present invention, step S3 is specifically as follows:

[0031] Step S301: The multiple feature maps of the same layer extracted by the spatial attention module are used as the features of the layer and recorded as F base , its feature dimension is C×T×H×W, where C is the number of channels, T is the number of frames, H and W are the height and width of the feature map respectively;

[0032] Step S302: Select m different sampling ratios on the dimension T, which are:

[0033] {r1, r2, ..., r m ; r1<r2<…<r m} (3);

[0034] Step S303: According to m different sampling ratios, the layer feature F is respectively base Sampling is performed to obtain multiple sets of feature maps, which are:

[0035]

[0036] Among them, the dimensions corresponding to the multiple sets of feature maps are:

[0037]

[0038] In one embodiment of the present invention, the specific process of generating a probability label for each video behavior in step s5 is as follows:

[0039] Step S51: Assume that the feature set to be classified {(z (1) , g (1) ),...,(z (n) , g (n) )}There are k categories, g (i) ∈{1, 2, 3, ..., k}, then the probability of each class corresponding to any input z is p(g = j | z), j = (1, 2, ..., k);

[0040] Step S52: By function hθ (z) Output a k-dimensional vector representing the probability value corresponding to k classification estimates:

[0041]

[0042] Among them, θ1, θ2, ..., is the model parameter of Softmax, and T is the transposed symbol.

[0043] To achieve the above object, the present invention further provides a video behavior recognition system based on spatiotemporal modeling, which is used to execute the above method, and comprises:

[0044] a video image sampling module, configured to perform segmented random video frame sampling on the input video;

[0045] a spatial attention module, connected to the video image sampling module, wherein the spatial attention module is a spatial attention module based on a triplet loss function and is used to extract spatial feature information;

[0046] a temporal pyramid module, connected to the spatial attention module, and outputting multiple sets of feature information by sampling spatial feature information at different frame rates in the feature dimension;

[0047] a feature fusion module, connected to the temporal pyramid module, for fusing multiple sets of feature information;

[0048] A Softmax module includes a Softmax classifier and is connected to the feature fusion module, and is used to classify video behaviors based on the fused feature information and output recognition results.

[0049] The video action recognition method and system based on spatiotemporal modeling provided by the present invention have at least the following advantages compared with the prior art:

[0050] 1) The spatial attention module built based on the triplet loss function can fully leverage the advantages of the triplet loss function. Through continuous learning, it focuses on extracting action-related features, thereby improving the effectiveness of feature extraction and enhancing the performance of video action recognition.

[0051] 2) By constructing a temporal pyramid module, single-layer features are sampled at different frame rates at the feature level, achieving multi-level extraction of temporal features while reducing the difficulty of fusion, thereby modeling temporal relationships at different time scales;

[0052] 3) The spatiotemporal information in the video is modeled separately through the constructed spatiotemporal modeling network, so as to fully extract and utilize the spatiotemporal features in the video, thereby improving the video behavior recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of video behavior recognition based on spatiotemporal modeling in one embodiment of the present invention;

[0055] Figure 2 Schematic diagram of segmented random video frame sampling in one embodiment of the present invention;

[0056] Figure 3 Schematic diagram of triplet loss function;

[0057] Figure 4 This is a flow chart of feature extraction performed by a spatial attention module based on a triplet loss function in one embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the effect of feature extraction using a spatial attention module in one embodiment of the present invention;

[0059] Figure 6 Schematic diagram of a time module sampling a feature map at different frame rates in one embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0061] The present invention proposes a neural network structure model based on spatiotemporal modeling. The network uses the currently popular ResNet network model as the backbone network and introduces a triplet loss function. Through continuous iteration, the extraction of action part features is improved, the extraction of invalid background parts is reduced, and the effectiveness of the extracted features is improved. The feature map output by the neural network is then sampled at different frame rates, thereby modeling the temporal information at the feature level. The early fusion method is then used to perform feature fusion on the feature maps trained at different frame rates. Finally, the Softmax classifier is used for classification to obtain the video behavior recognition result. The network of the present invention achieves modeling of the spatiotemporal information in the video by introducing a triplet loss function and constructing a temporal pyramid structure at the feature level, thereby further improving the network's extraction and utilization of spatiotemporal features, and ultimately improving the video behavior recognition performance of the entire network.

[0062] Figure 1 FIG. 1 is a flow chart of video behavior recognition based on spatiotemporal modeling in one embodiment of the present invention. Figure 1 As shown, this embodiment provides a video behavior recognition method based on spatiotemporal modeling, which specifically includes:

[0063] Step S1: input the complete video into the video image sampling module, and use the segmented random video frame sampling method to sample the video image to obtain multiple sampling frame images;

[0064] In the field of video feature recognition, when a complete video is input into a convolutional neural network, considering that there is a large amount of similar information between adjacent frames and the uneven distribution of key information in the video, if feature extraction is performed on each frame of the video, a large amount of information redundancy will occur, resulting in unnecessary computing costs and affecting the recognition speed. In addition, the length of the input video is also difficult to unify, so it is very necessary to perform reasonable sampling processing on the input video. The video image sampling module of this embodiment adopts a segmented random video frame sampling method for video image sampling. Segmented random video frame sampling refers to first averaging multiple video frame segments from a complete video sequence, and then performing random sampling in these segments. The reason why this sampling method is adopted in this embodiment is that the subsequent time module needs to perform secondary sampling at the feature level. Therefore, the segmented random video frame sampling method can reduce redundant information while ensuring the integrity and randomness of the sampled video frames for video behavior as much as possible.

[0065] Figure 2 FIG. 1 is a schematic diagram of segmented random video frame sampling in one embodiment of the present invention. Figure 2 As shown, in this embodiment, the specific process of step S1 includes:

[0066] Step S11: Divide the input complete video sequence V into U segments evenly, namely S1, S2...S U , where U is a constant;

[0067] Step S12: Randomly sample each segment to obtain sample frame images T1, T2...T U .

[0068] Step S2: Construct a spatial attention module based on the triplet loss function, and extract features of the action parts of multiple sample frame images to obtain spatial feature information;

[0069] Figure 3 is a schematic diagram of the triplet loss function, as shown Figure 3 As shown in the figure, the triplet loss function is a loss function proposed by the Google research team in 2015. Its advantage lies in detail differentiation. Specifically, when two inputs are similar, the triplet loss function can better model the details, which is equivalent to adding a measure of the difference between the two inputs, thereby learning a better representation of the inputs. The triplet in the triplet loss function is a triplet, which is a random sample selected from the training dataset. This sample is called an anchor. Then, a sample belonging to the same class as the anchor is randomly selected, called a positive sample, and a sample of a different class from the anchor is randomly selected, called a negative sample. The three samples of anchor, positive, and negative constitute a triplet. The goal of the triplet loss function is to make features with the same label as close as possible in spatial position, while features with different labels are as far apart as possible in spatial position. In other words, the distance between positive samples and anchor samples of the same class is getting closer, while the distance between negative samples and anchor samples of different classes is getting farther and farther.

[0070] For each element in the triplet, a network with shared or non-shared parameters is trained to obtain the feature expressions of the three elements, which are recorded as and The purpose of triplet loss is to make and The distance between them should be as small as possible. and The distance between them should be as large as possible, and at the same time, and The distance between and The distance between them has a minimum interval α, so a basic triplet loss function formula is:

[0071]

[0072] in, Represents the Euclidean distance metric between Positive samples and Anchor samples, Represents the Euclidean distance metric between negative samples and anchor samples.

[0073] Based on the above characteristics, the triplet loss function is often used in face recognition tasks. Considering that in the field of video behavior recognition, when extracting features, it is easy to be interfered by background information, thereby extracting invalid features, the spatial feature extraction of this embodiment requires the construction of a spatial attention module based on the triplet loss function.

[0074] In this embodiment, the spatial attention module based on the triplet loss function is constructed in step S2 as follows:

[0075] Step S201: inserting a triplet loss function into a spatial module that extracts spatial features from an image in a backbone network, to selectively collect feature information in the spatial domain, wherein the backbone network is a ResNet network (residual network);

[0076] Step S202: Multiple feature maps are fed into a spatial module that inserts a triplet loss function for selection, and appropriate anchor examples, positive examples, and negative examples are selected, where the anchor example is obtained by subtracting the average value of each element of the positive example from each element of the positive example;

[0077] Step S203: The training space module continuously learns to make the feature extraction of the anchor example closer to the positive example, that is, the loss function is minimized. The loss function is:

[0078]

[0079] in, represents the Euclidean distance metric between the positive example and the anchor example, represents the Euclidean distance metric between the negative example and the anchor example, and α is the minimum interval between the two Euclidean distance metrics.

[0080] Since the triplet loss function has the characteristic of being able to better model details, this embodiment uses the triplet loss function to selectively collect features in the spatial domain, constructs a feature map by weighting, and realizes the selective collection of spatial domain features through continuous learning, so that the extracted features are more focused on the action and ignore the influence of background information, thereby improving the effectiveness of the extracted feature information.

[0081] Figure 4 FIG. 1 is a flow chart of feature extraction performed by a spatial attention module based on a triplet loss function in one embodiment of the present invention. Figure 4As shown, in this embodiment, the specific process of extracting features from the action parts of multiple sampling frame images in step S2 to obtain spatial feature information is as follows:

[0082] Step S211: performing spatial feature extraction on multiple sample frame images to obtain multiple frame feature maps, where the dimensions of each frame feature map are C×H×W, where C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0083] Step S212: reshape the features of any feature map A, converting the feature dimension to C×N, where N is the number of pixels of the feature map A;

[0084] Step S213: Use the Top-K ranking function to extract the top K attention graphs S p and the next K attention maps S n , where S p S is the feature that affects the behavior classification. n The Top-K ranking function is a commonly used ranking function, which is used to obtain the top K largest values or the top K smallest values and their labels of a certain dimension. When the spatial attention module is trained, the features required for each classification can be activated through the training samples. Therefore, when sorting the features, the value of the correlation with the required features can be obtained, thereby extracting S p and S n .

[0085] Step S214: p Reshape it into K×H×W dimensions and then perform average pooling (AvgPool);

[0086] Step S215: Use the Sigmoid function for normalization to obtain the spatial attention weight that emphasizes the action part; the Sigmoid function is called the S-shaped growth curve. In information science, due to its monotonic increasing and inverse monotonic increasing properties, the Sigmoid function is often used as an activation function of a neural network to map variables between 0 and 1.

[0087] Step S216: Multiply the obtained attention weight by the feature map A element by element to obtain the output A of the spatial attention module * for:

[0088] A * =σ(Avgp(Resh(S P )))*A (2)

[0089] Among them, A * To emphasize the feature map of the action part, Resh is the dimension conversion, Avgp is the average pooling, and σ is the Sigmoid function.

[0090] Figure 5 FIG. 1 is a schematic diagram showing the effect of feature extraction by the spatial attention module in one embodiment of the present invention. Figure 5 As shown, the significance of applying the triplet loss function in the spatial attention module of this embodiment is that the anchor example is closer to the positive example and farther away from the negative example after continuous learning, so as to achieve S p and S n The clear distinction between the body and the action area ultimately makes the entire network pay more attention to the body action area and reduces the interference of background information.

[0091] Step S3: constructing a temporal pyramid module and sampling the obtained spatial feature information at different frame rates to obtain multiple sets of feature information;

[0092] The goal of this module is to utilize temporal information at multiple levels and scales. Therefore, in order to fully utilize the implicit advantage of neural networks in containing multi-scale information, this embodiment proposes to construct a temporal pyramid module to mine different temporal relationships, thereby modeling temporal relationships at different time scales. Unlike the traditional feature pyramid structure, since the spatial attention module of this embodiment is a spatial attention module based on a triplet loss function, when constructing the pyramid structure, this embodiment chooses to sample only a single layer of features in the temporal dimension, thereby constructing a temporal pyramid. This reduces the difficulty and time requirements of feature fusion while achieving multi-level utilization of temporal information.

[0093] Figure 6 FIG. 1 is a schematic diagram of a time module sampling a feature map at different frame rates according to an embodiment of the present invention. Figure 6 As shown, in this embodiment, step S3 is specifically as follows:

[0094] Step S301: The multiple feature maps of the same layer extracted by the spatial attention module are used as the features of the layer and recorded as F base , its feature dimension is C×T×H×W, where C is the number of channels, T is the number of frames, H and W are the height and width of the feature map respectively;

[0095] Step S302: Select m different sampling ratios on the dimension T, which are:

[0096] {r1, r2, ..., r m ; r1<r2<…<r m} (3);

[0097] Step S303: According to m different sampling ratios, the layer feature F is respectively base Sampling is performed to obtain multiple sets of feature maps, which are:

[0098]

[0099] Among them, the dimensions corresponding to the multiple sets of feature maps are:

[0100]

[0101] The above-mentioned features are multiple sets of feature information obtained after being processed by the temporal pyramid module. This embodiment can better utilize temporal information by sampling the feature map at different frame rates in the feature dimension.

[0102] Step S4: using early fusion to fuse the obtained multiple sets of feature information, specifically using the add operation to splice the feature vectors of the multiple sets of features;

[0103] Feature fusion methods are generally categorized as early fusion and late fusion. Early fusion involves first fusing features and then training on the fused features. This type of method, known as skip connection, involves connecting features using the concat (concatenation) and add (addition) operations before obtaining the recognition result. The concat operation directly concatenates two features. For example, if the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p + q. The add operation is a parallel strategy, combining two feature vectors into a complex vector. For example, for input features x and y, the output feature z = x + iy, where i is an imaginary unit. Late fusion involves first classifying the features and then fusing the classification results. The advantage of late fusion is that errors in the fusion model are independent of each other, preventing further accumulation of errors. Because this embodiment samples feature maps at different frame rates at the feature level in the aforementioned steps, thereby modeling temporal relationships at different scales, resulting in different sets of feature maps, this embodiment chooses an early fusion approach to fuse features in order to fully utilize temporal relationships at different scales.

[0104] Step S5: Input the fused feature information into the Softmax classifier for video behavior classification, generate a probabilistic label for each video behavior, and output the recognition result. The Softmax regression model is a generalization of the logistic regression model for multi-classification problems and is widely used in multi-classification problems.

[0105] In this embodiment, the specific process of generating a probability label for each video behavior in step S5 is as follows:

[0106] Step S51: Assume that the feature set to be classified {(z (1) , g (1) ),...,(z (n) , g (n))}There are k categories, g (i) ∈{1, 2, 3, ..., k}, then the probability of each class corresponding to any input z is p(g = j | z), j = (1, 2, ..., k);

[0107] Step S52: By function h θ (z) Output a k-dimensional vector representing the probability value corresponding to k classification estimates:

[0108]

[0109] in, is the model parameter of Softmax, and T is the transposed symbol.

[0110] Please refer to Figure 1 Another embodiment of the present invention provides a video behavior recognition system based on spatiotemporal modeling, which is used to perform the above method, and includes:

[0111] a video image sampling module, configured to perform segmented random video frame sampling on the input video;

[0112] a spatial attention module, connected to the video image sampling module, wherein the spatial attention module is a spatial attention module based on a triplet loss function and is used to extract spatial feature information;

[0113] a temporal pyramid module, connected to the spatial attention module, and outputting multiple sets of feature information by sampling spatial feature information at different frame rates in the feature dimension;

[0114] a feature fusion module, connected to the temporal pyramid module, for fusing multiple sets of feature information;

[0115] A Softmax module includes a Softmax classifier and is connected to the feature fusion module, and is used to classify video behaviors based on the fused feature information and output recognition results.

[0116] Existing deep learning methods, such as two-stream networks and 3D convolutional neural networks, have made some use of temporal information. However, they suffer from issues such as the time-consuming optical flow map extraction of two-stream networks and the high computational complexity of 3D convolutional neural networks. Furthermore, existing networks do not fully utilize temporal information, and are susceptible to background interference when extracting features, leading to the extraction of useless features. The video action recognition method and system based on spatiotemporal modeling provided by the present invention have at least the following advantages over existing technologies:

[0117] 1) The spatial attention module built based on the triplet loss function can fully leverage the advantages of the triplet loss function. Through continuous learning, it focuses on extracting action-related features, thereby improving the effectiveness of feature extraction and enhancing the performance of video action recognition.

[0118] 2) By constructing a temporal pyramid module, single-layer features are sampled at different frame rates at the feature level, achieving multi-level extraction of temporal features while reducing the difficulty of fusion, thereby modeling temporal relationships at different time scales;

[0119] 3) The spatiotemporal information in the video is modeled separately through the constructed spatiotemporal modeling network, so as to fully extract and utilize the spatiotemporal features in the video, thereby improving the video behavior recognition performance.

[0120] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0121] Those skilled in the art will appreciate that the modules in the apparatuses of the embodiments may be distributed in the apparatuses of the embodiments as described in the embodiments, or may be located in one or more apparatuses different from the embodiments with corresponding changes. The modules in the above embodiments may be combined into one module or further divided into multiple sub-modules.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video behavior recognition method based on spatiotemporal modeling, characterized in that: include: Step S1: input the complete video into the video image sampling module, and use the segmented random video frame sampling method to sample the video image to obtain multiple sampling frame images; Step S2: Construct a spatial attention module based on the triplet loss function, and extract features from the action parts of multiple sample frame images to obtain spatial feature information. The construction of the spatial attention module based on the triplet loss function is specifically as follows: Step S201: inserting a triplet loss function into a spatial module for extracting spatial features of an image in a backbone network, so as to selectively collect feature information in the spatial domain, wherein the backbone network is a ResNet network; Step S202: Multiple feature maps are fed into a spatial module that inserts a triplet loss function for selection, and appropriate anchor examples, positive examples, and negative examples are selected, where the anchor example is obtained by subtracting the average value of each element of the positive example from each element of the positive example; Step S203: The training space module continuously learns to make the feature extraction of the anchor example closer to the positive example, that is, the loss function is minimized. The loss function is: in, represents the Euclidean distance metric between the positive example and the anchor example, represents the Euclidean distance metric between the negative example and the anchor example, and α is the minimum interval between the two Euclidean distance metrics; Step S3: Construct a temporal pyramid module and sample the obtained spatial feature information at different frame rates to obtain multiple sets of feature information, specifically: Step S301: The multiple feature maps of the same layer extracted by the spatial attention module are used as the features of the layer and recorded as F base , its feature dimension is C×T×H×W, where C is the number of channels, T is the number of frames, H and W are the height and width of the feature map respectively; Step S302: Select m different sampling ratios on the dimension T, which are: {r1，r2，…，r m ；r1<r2<…<r m} (3)； Step S303: According to m different sampling ratios, the layer feature F is respectively base Sampling is performed to obtain multiple sets of feature maps, which are: Among them, the dimensions corresponding to the multiple sets of feature maps are: Step S4: using early fusion to fuse the obtained multiple sets of feature information, specifically using the add operation to splice the feature vectors of the multiple sets of features; Step S5: Input the fused feature information into the Softmax classifier to classify the video behavior, generate a probability label for each video behavior, and output the recognition result.

2. The video behavior recognition method based on spatiotemporal modeling according to claim 1 is characterized in that: The specific process of step S1 includes: Step S11: Divide the input complete video sequence V into U segments evenly, namely S1, S2...S U , where U is a constant; Step S12: Randomly sample each segment to obtain sample frame images T1, T2...T U .

3. The video behavior recognition method based on spatiotemporal modeling according to claim 1 is characterized in that: In step S2, the specific process of extracting features from the action parts of multiple sampling frame images to obtain spatial feature information is as follows: Step S211: performing spatial feature extraction on multiple sample frame images to obtain multiple frame feature maps, where the dimensions of each frame feature map are C×H×W, where C is the number of channels, and H and W are the height and width of the feature map, respectively. Step S212: reshape the features of any feature map A and convert the feature dimension to C×N, where N is the number of pixels of the feature map A; Step S213: Use the Top-K ranking function to extract the top K attention graphs S p and the next K attention maps S n , where S p S is the feature that affects the behavior classification. n Features that are not related to behavior identification; Step S214: p After reshaping to K×H×W dimensions, average pooling is performed; Step S215: Use the Sigmoid function to perform normalization to obtain the spatial attention weight of the emphasized action part; Step S216: Multiply the obtained attention weight by the feature map A element by element to obtain the output A of the spatial attention module * for: A * =σ(Avgp(Resh(S P )))*A (2) Among them, A * To emphasize the feature map of the action part, Resh is the dimension conversion, Avgp is the average pooling, and σ is the Sigmoid function.

4. The video behavior recognition method based on spatiotemporal modeling according to claim 1 is characterized in that: The specific process of generating a probability label for each video behavior in step S5 is as follows: Step S51: Assume that the feature set to be classified {(z (1) ,g (1) ),...,(z (n) ,g (n) )}There are k categories, g (i) ∈{1,2,3,...,k}, then the probability of each class corresponding to any input z is p(g=j|z), j=(1,2,…,k); Step S52: By function h θ (z) Output a k-dimensional vector representing the probability value corresponding to k classification estimates: in, is the model parameter of Softmax, and T is the transposed symbol.

5. A video behavior recognition system based on spatiotemporal modeling, used to execute the method of any one of claims 1 to 4, characterized in that: include: a video image sampling module, configured to perform segmented random video frame sampling on the input video; a spatial attention module, connected to the video image sampling module, wherein the spatial attention module is a spatial attention module based on a triplet loss function and is used to extract spatial feature information; a temporal pyramid module, connected to the spatial attention module, and outputting multiple sets of feature information by sampling spatial feature information at different frame rates in the feature dimension; a feature fusion module, connected to the temporal pyramid module, for fusing multiple sets of feature information; A Softmax module includes a Softmax classifier and is connected to the feature fusion module, and is used to classify video behaviors based on the fused feature information and output recognition results.

Images