A spatiotemporal action detection method based on hierarchical relationship modeling

Abstract

This invention relates to the fields of computer data processing and video surveillance technology, and particularly to a method for spatiotemporal action detection of individuals in surveillance videos. A spatiotemporal action detection method based on hierarchical relationship modeling is proposed. Feature extraction is performed on the input video frames, individual features are enhanced, and then a self-attention method is used to model complex spatial relationships. The modeled features are then merged to obtain hierarchical relationship features. Action classification is performed based on these hierarchical relationship features to obtain the desired spatiotemporal action detection results. This invention addresses complex multi-person scenarios by capturing the interaction relationships between people and the environment, and between people themselves, through self-attention, thereby improving the accuracy of action classification and enhancing the spatiotemporal action detection effect. Furthermore, by enhancing individual features, the ability to recognize fine-grained action categories is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer data processing and video surveillance technology, and in particular to a method for detecting the spatiotemporal movements of individuals in surveillance videos. Background Technology

[0002] With the development of computer technology, video has gradually become the dominant form of multimedia data presentation on internet platforms. In the security field, the volume of surveillance video data has surged due to the national push for "smart cities." Furthermore, monitoring and analysis technologies in industrial production environments are increasingly based on video. With the growth in video data volume and the increasing demand for video data analysis and processing across industries, spatiotemporal motion detection technology urgently needs exploration and development. Spatiotemporal motion detection is an advanced task in video behavior classification, requiring the location of the temporal position of each action occurring in the video and the spatial position of the action subject, followed by accurate action classification.

[0003] For spatiotemporal action detection tasks, methods can be categorized into bottom-up frame-by-frame detection methods and top-down tube-level detection methods, depending on the processing approach. Frame-by-frame detection methods typically begin by spatially locating the subject of the action in each frame of the video, using common object detection algorithms such as SSD and Faster R-CNN. The detection results from each frame are then linked to form an action sequence, which is finally classified. The core of this type of method lies in effectively extracting spatiotemporal information. Current methods include optical flow-based two-stream methods, 3D convolutional neural network-based methods, and recurrent neural network-based methods. These methods are effective at capturing short-term spatiotemporal information, but they are not effective at handling long-term dependencies. Tube-level detection methods often improve upon existing object detection methods by expanding the two-dimensional anchor boxes into three-dimensional tubes. Human detection is first performed on keyframes, then the detection boxes are expanded to adjacent frames to form action tubes. Finally, the detection boxes are refined and actions are classified.

[0004] An individual's actions are not only related to their physical movements but also closely linked to the environment and the people or environment they interact with. Existing methods do not fully utilize spatial relationships, fail to model the relationships between people in multi-person scenarios, and do not capture the interaction between people and the environment, thus missing crucial information for action classification. Furthermore, for some fine-grained action categories, such as speaking or answering a phone call, existing methods do not enhance fine-grained features, resulting in poor recognition performance for these actions. In complex multi-person scenarios, spatiotemporal action detection tasks require the detection of action categories including actions interacting with each other, actions interacting with the environment, and independent actions performed by the individual. These complex actions cannot be effectively classified using existing spatiotemporal information capture methods alone. Summary of the Invention

[0005] The technical problem this invention aims to solve is to propose a spatiotemporal action detection method based on hierarchical spatial relationship modeling to improve the accuracy of action classification. Furthermore, it enhances individual features to improve the ability to recognize fine-grained action categories.

[0006] To address the aforementioned technical problems, this invention proposes a spatiotemporal action detection method based on hierarchical relationship modeling, mainly comprising the following steps P1 to P4:

[0007] P1: Extract features from the input video frames, extracting the spatial features of keyframes and the spatiotemporal features of the entire video segment; perform individual spatial localization based on spatial features, obtain spatial candidate boxes, and obtain the spatiotemporal features of the individual.

[0008] P2: Based on the spatiotemporal characteristics of individuals and the spatiotemporal characteristics of the entire video segment, perform hierarchical relationship modeling based on self-attention, including modeling the relationship between individuals and the context, and modeling the relationship between individuals.

[0009] P3: Merge the modeled features to obtain hierarchical relationship features;

[0010] P4: Link the spatial candidate boxes and classify actions based on hierarchical relationship features to obtain the required spatiotemporal action detection results.

[0011] Preferably, the spatial feature extraction of the key frame in step P1 is specifically as follows: for the input video frame, a frame is selected as a key frame at regular intervals according to the uniform sampling method, and spatial features are generated for the key frame using a two-dimensional convolutional neural network.

[0012] Furthermore, the spatiotemporal feature extraction of the video segment mentioned in step P1 specifically involves inputting the scaled-down entire video segment into a three-dimensional convolutional neural network for complete spatiotemporal feature extraction.

[0013] Preferably, obtaining the spatiotemporal characteristics of an individual in step P1 specifically includes:

[0014] Using a set of spatial candidate boxes, the region of interest pooling method is employed to extract a portion of the candidate boxes from the complete spatiotemporal features, i.e., the background environment features, as the spatiotemporal features of the individual, thus obtaining the set of individual spatiotemporal features.

[0015] A further improvement is to perform fine-grained enhancement processing on the spatiotemporal features of the individual after obtaining the spatiotemporal features of the individual in step P1 and before step P2, thereby obtaining fine-grained action-sensitive spatiotemporal features of the individual.

[0016] Preferably, the individual spatiotemporal features undergo fine-grained enhancement processing, including the following steps:

[0017] The spatiotemporal features of individuals are compressed in the time dimension using max pooling, thus transforming the four-dimensional feature tensor into a three-dimensional feature tensor.

[0018] Fine-grained feature enhancement is performed on the 3D feature tensor using point convolution, and attention weights are obtained through the Softmax function.

[0019] By using attention weights to perform a weighted summation of the three-dimensional feature tensor in the spatial dimension, a one-dimensional feature with individual feature enhancement is obtained.

[0020] Furthermore, the spatiotemporal characteristics of the individual are the spatiotemporal characteristics of the individual after fine-grained enhancement processing.

[0021] Preferably, the relationship between the individual and its context is modeled, specifically including the following steps:

[0022] The context features, i.e. the complete spatiotemporal features, are compressed in the time dimension using max pooling, so that the four-dimensional feature tensor is transformed into a three-dimensional context feature tensor.

[0023] By embedding individual spatiotemporal feature vectors, the query vectors required for relation modeling are obtained;

[0024] The three-dimensional environmental feature tensor is embedded to obtain the required key vector and value vector;

[0025] Attention coefficients are calculated using query vectors and key vectors, and attention-weighted summation is performed on value vectors to obtain the spatiotemporal characteristics of an individual after interacting with the environment.

[0026] Furthermore, the relationships between these individuals are modeled, specifically including the following steps:

[0027] The individual spatiotemporal features are used to embed new query vectors, key vectors, and value vectors;

[0028] By using self-attention to perform feature interaction and action relationship reasoning, features modeled after individual interaction are obtained.

[0029] Preferably, step P4 specifically includes the following steps:

[0030] The candidate bounding boxes for spatial detection are linked using the Viterbi algorithm to obtain a complete sequence of motion boxes for each individual appearing in the video frame.

[0031] The spatiotemporal features of individuals obtained before hierarchical relationship modeling are combined with the spatiotemporal features of individuals after hierarchical relationship modeling.

[0032] Action classification is performed using a fully connected network and a sigmoid activation function to obtain the probability distribution of action categories;

[0033] The action category with the higher probability is taken as the individual's final action category.

[0034] The beneficial effects of this invention include:

[0035] (1) For individual fine-grained independent actions, point attention method is used to enhance individual features, so that the algorithm focuses on the body parts of the individual that are closely related to the action;

[0036] (2) In complex scenarios involving multiple people, the interaction between people and the environment, and between people, is captured through self-attention, and the identification of related behaviors is enhanced through relational reasoning.

[0037] (3) To explore the modeling order problem of multiple relationship modeling, a hierarchical modeling method is adopted, proceeding step by step and reasoning gradually. Attached Figure Description

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Figure 1 This is a system framework diagram for a spatiotemporal action detection method based on hierarchical relationship modeling.

[0040] Figure 2 The logic diagram of the individual fine-grained feature enhancement algorithm based on point attention is shown.

[0041] Figure 3 A flowchart illustrating the logic of modeling self-attention-based interaction relationships. Detailed Implementation

[0042] Combination Figure 1 As shown, this invention relates to a spatiotemporal action detection method based on hierarchical relationship modeling, used for accurate action detection and classification in complex multi-person scenes, mainly comprising four steps S1 to S4:

[0043] S1: The input video frames are scaled up and then feature extraction is performed, including spatial feature extraction of keyframes and spatiotemporal feature extraction of video segments. Then, individual spatial localization is performed based on spatial features to obtain spatial candidate boxes. The spatiotemporal features of the individual are obtained through the region of interest (RoI) pooling method.

[0044] S2: Using the point attention method, the spatiotemporal features of individuals obtained in S1 are enhanced in a fine-grained manner, thereby obtaining fine-grained action-sensitive spatiotemporal features.

[0045] S3: Based on the spatiotemporal features of the individuals obtained in S2 and the spatiotemporal features of the entire segment obtained in S1, perform hierarchical relationship modeling based on self-attention. First, model the relationship between the individual and the environment, then model the relationship between the individuals, and finally merge the features.

[0046] S4: Link the candidate boxes obtained in S1, and classify the actions based on the hierarchical relationship features obtained in S3, so as to obtain the final spatiotemporal action detection results.

[0047] Step S1, spatiotemporal feature extraction, includes the following three aspects:

[0048] S11: For the input video clip, the frame size is first scaled down to 256×256. Then, using a uniform sampling method, one frame is selected as a keyframe every second. A 2D convolutional neural network (ResNet-50) is used to extract features from the keyframes. Afterward, a region proposal network is used to generate individual candidate boxes from the extracted features. The generation method adopts a common two-stage object detection algorithm: a 3×3 sliding window traverses the entire feature map extracted by the 2D convolutional neural network, generating 9 candidate boxes at the center of each window according to a predetermined ratio and scale. Then, a fully connected layer is used to perform binary classification (determining whether it is a human or background) and preliminary detection box regression on each candidate box. Finally, a more accurate candidate box is output through the convolutional neural network. The output candidate boxes are set as follows: Where b j The coordinates and dimensions of the candidate bounding box are represented by , and p represents the number of individuals contained in the keyframe of the detection.

[0049] S12: The scaled-down video clip is input into the SlowFast 3D convolutional neural network for complete spatiotemporal feature extraction. The SlowFast network is a classic 3D convolutional network, divided into fast and slow branches. The slow branch captures spatial features, while the fast branch captures temporal features of high-speed motion. This structure enables it to extract spatiotemporal features more effectively. The extracted feature map is set to F. i , where i represents the i-th video segment;

[0050] S13: The set of candidate boxes obtained using S11 The spatiotemporal features F obtained from S12 are obtained using the Region of Interest Pooling (RoI Pooling) method. i A portion of the candidate bounding box is extracted as the spatiotemporal feature of an individual. Let the obtained set of individual spatiotemporal features be... in Taking the complete spatiotemporal features as background environmental features, denoted as...

[0051] The spatiotemporal features of the individuals obtained from S1 are enhanced, such as... Figure 2 As shown, the specific steps are as follows:

[0052] S21: Compress individual spatiotemporal features in the time dimension using max pooling, resulting in a four-dimensional feature tensor. Transform into a three-dimensional feature tensor

[0053] S22: Next, fine-grained feature enhancement is performed on the 3D feature tensor using point convolution (i.e., 1×1 convolution), and attention weights are obtained through the Softmax function, as shown in the following formula, where... The feature value representing the spatial location k of the j-th person is... This represents the attention weight value;

[0054]

[0055] S23: Then, using the attention weights obtained in S22, the three-dimensional feature tensor obtained in S21 is weighted and summed in the spatial dimension to obtain a one-dimensional feature with individual feature enhancement. The formula is expressed as follows.

[0056]

[0057] The hierarchical relationship modeling performed in step S3 specifically includes the following steps:

[0058] S31: First, model the relationship between the individual and its context. Compress the context features (i.e., complete spatiotemporal features) in the time dimension using max-pooling, resulting in a four-dimensional feature tensor. Transformed into a three-dimensional environment feature tensor

[0059] S32: The individual feature vector obtained in S23 Feature embedding is performed to obtain the query vector required for relation modeling. The formula is Where f θ This represents a single-layer feedforward neural network with weight parameters θ.

[0060] S33: Transform the three-dimensional environment feature tensor Perform feature embedding to obtain the required key vector. Sum value vector The formula is expressed as: Where g θ This represents a single-layer feedforward neural network with weight parameters as follows: maxpool represents the maximum value pooling operation.

[0061] like Figure 3 As shown, attention coefficients are calculated using query vectors and key vectors, and then attention-weighted summations are performed on the value vectors to obtain individual characteristics after interaction with the environment. The formula is as follows:

[0062]

[0063]

[0064] S34: Then, model the relationships between individuals. (The above individual characteristics...) A new query vector is obtained through feature embedding. key vector Sum value vector Furthermore, feature interaction and action relationship reasoning are performed through self-attention, with the specific formula as follows:

[0065]

[0066]

[0067] This yields features modeled through individual interactions.

[0068] Step S4 specifically includes the following steps:

[0069] S41: The spatial detection candidate boxes obtained in S1 are linked together using the Viterbi algorithm to obtain a complete action box sequence for each individual appearing in the video frame;

[0070] S42: Individual features enhanced with fine-grained detail obtained from S2 Individual features obtained from S3 after hierarchical interaction modeling The data is concatenated, then passed through a fully connected network and a Sigmoid activation function for action classification to obtain the action category probability distribution. The category with the higher probability is then taken as the final action category for the individual.

[0071] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A spatiotemporal action detection method based on hierarchical relationship modeling, characterized in that, Includes the following steps: P1: Extract features from the input video frames, extracting the spatial features of keyframes and the spatiotemporal features of the entire video segment; perform individual spatial localization based on spatial features, obtain spatial candidate boxes, and obtain the spatiotemporal features of the individual. P2: Based on the spatiotemporal characteristics of individuals and the spatiotemporal characteristics of the entire video segment, perform hierarchical relationship modeling based on self-attention, including modeling the relationship between individuals and the context, and modeling the relationship between individuals. P3: Merge the modeled features to obtain hierarchical relationship features; P4: Link the spatial candidate boxes and classify actions based on hierarchical relationship features to obtain the required spatiotemporal action detection results; Modeling the relationship between the individual and its context involves the following steps: The context features, i.e. the complete spatiotemporal features, are compressed in the time dimension using max pooling, so that the four-dimensional feature tensor is transformed into a three-dimensional context feature tensor. By embedding individual spatiotemporal feature vectors, the query vectors required for relation modeling are obtained; The three-dimensional environmental feature tensor is embedded to obtain the required key vector and value vector; Attention coefficients are calculated using query vectors and key vectors, and attention-weighted summation is performed on value vectors to obtain the spatiotemporal characteristics of an individual after interacting with the environment. Modeling the relationships between these individuals involves the following steps: The individual spatiotemporal features are used to embed new query vectors, key vectors, and value vectors; By using self-attention to perform feature interaction and action relationship reasoning, features modeled after individual interaction are obtained. Step P4 specifically includes the following steps: The candidate bounding boxes for spatial detection are linked using the Viterbi algorithm to obtain a complete sequence of motion boxes for each individual appearing in the video frame. The spatiotemporal features of individuals obtained before hierarchical relationship modeling are combined with the spatiotemporal features of individuals after hierarchical relationship modeling. Action classification is performed using a fully connected network and a Sigmoid activation function to obtain the probability distribution of action categories; The action category with the higher probability is taken as the individual's final action category.

2. The spatiotemporal action detection method based on hierarchical relationship modeling according to claim 1, characterized in that, The spatial feature extraction of keyframes in step P1 is specifically as follows: for the input video frames, a frame is selected as a keyframe at regular intervals according to the uniform sampling method, and spatial features are generated for the keyframes using a two-dimensional convolutional neural network.

3. The spatiotemporal action detection method based on hierarchical relationship modeling according to claim 2, characterized in that, The spatiotemporal feature extraction of the video segment mentioned in step P1 specifically involves inputting the scaled-down entire video segment into a three-dimensional convolutional neural network for complete spatiotemporal feature extraction.

4. The spatiotemporal action detection method based on hierarchical relationship modeling according to claim 3, characterized in that, The acquisition of the spatiotemporal characteristics of an individual as described in step P1 specifically includes: Using a set of spatial candidate boxes, the region of interest pooling method is employed to extract a portion of the candidate boxes from the complete spatiotemporal features, i.e., the background environment features, as the spatiotemporal features of the individual, thus obtaining the set of individual spatiotemporal features.

5. The spatiotemporal action detection method based on hierarchical relationship modeling according to claim 1, characterized in that, After obtaining the spatiotemporal features of an individual in step P1 and before step P2, the spatiotemporal features of the individual are subjected to fine-grained enhancement processing to obtain fine-grained motion-sensitive spatiotemporal features of the individual. The individual spatiotemporal features undergo fine-grained enhancement processing, including the following steps: The spatiotemporal features of individuals are compressed in the time dimension using max pooling, thus transforming the four-dimensional feature tensor into a three-dimensional feature tensor. Fine-grained feature enhancement is performed on the 3D feature tensor using point convolution, and then... The function yields the attention weights; By using attention weights to perform a weighted summation of the three-dimensional feature tensor in the spatial dimension, a one-dimensional feature with individual feature enhancement is obtained.

6. The spatiotemporal action detection method based on hierarchical relationship modeling according to claim 5, characterized in that, The spatiotemporal characteristics of the individual are the spatiotemporal characteristics of the individual after fine-grained enhancement processing.

Images