Behavior detection method and system based on joint regular behavior subject tracking

Abstract

The invention discloses a behavior detection method and system based on joint regular behavior subject tracking, and the method comprises the steps: obtaining a feature Fs with significant spatial semantics through the spatial features of an input video, and carrying out the constraint of the feature Fs through target offset mutual information cross entropy loss; splicing the time sequence feature Fv of the input video with the feature Fs to obtain a feature F, and performing edge feature down-sampling on the feature F to obtain a feature F <->; the feature F <-> is sent to a multi-scale Transform network, different weights are distributed to the feature F <-> through an attention mechanism, a multi-scale feature Z is obtained, the multi-scale feature Z is classified and regressed in a Transform decoder, and a behavior detection task is completed; in the process of completing behavior detection, the accuracy of behavior detection is improved by utilizing masking reconstruction. By utilizing the method, the behavior detection performance can be obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of behavior detection in video understanding, and specifically is a behavior detection method based on joint regular behavior subject tracking. Background Art

[0002] With the popularity of various surveillance devices, people's lives are increasingly exposed to the attention of various cameras, resulting in a large amount of visual content. In addition, with the rise of various short video social platforms, the visual content on the Internet is growing at an exponential rate every day. Therefore, understanding and analyzing massive videos has become crucial. Video understanding is a basic research problem in the field of computer vision, which aims to automatically identify and analyze the content in the video through intelligent technologies such as deep learning.

[0003] A large number of behaviors are inevitable in videos. How to identify the category of behaviors in long videos and how to determine the time when behaviors occur have become research hotspots in the field of video understanding. Therefore, the behavior detection task was proposed, which aims to predict the category of behaviors in videos and the start and end time of behaviors. With the help of this technology, a certain behavior in the video can be quickly located. It is applied in the field of social public security to effectively ensure public safety. It is applied in the field of autonomous driving to realize pedestrian detection and vehicle behavior analysis, and improve the vehicle's perception and prediction of the environment. It is applied in the field of sports events to realize athlete posture analysis, game statistics and data analysis, and provide coaches, athletes and spectators with more valuable information and data support... Compared with the similar task of action recognition in video understanding, the behavior detection task is more difficult than the behavior recognition task due to its characteristics such as the complexity of the scene and the uncertainty of the duration of the behavior.

[0004] In recent years, there have been new research results in the field of behavior detection. There are three main methods: the first is anchor-based methods, which predefine the number and size of anchor points and generate multiple candidate behavior segments at different scales at each anchor point, but this type of method is complex to operate and sensitive to hyperparameters; the second is actionness-guided methods, which first predict the probability of containing behavior at each time point, and then generate behavior candidate video segments based on the probability, but this type of method has high computational cost and cannot be performed end-to-end; the third is anchor-free methods, which can directly classify behaviors and predict the time boundaries of the start and end of behaviors and reduce the complexity of model design. This method has received more and more attention from scholars, but its positioning accuracy is low, which hinders the development of behavior detection tasks. Summary of the invention

[0005] In view of the shortcomings in the prior art, the present invention provides a behavior detection method based on joint regular behavior subject tracking, which significantly improves the performance of behavior detection.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A behavior detection method based on joint regular behavior subject tracking:

[0008] Extract the temporal features F of the input video v and spatial feature K f , the spatial feature K f Through the Transformer network based on quadtree attention, the feature F with significant spatial semantics is obtained s ; The feature F with significant spatial semantics s Use target offset mutual information cross entropy loss for constraints;

[0009] The time series feature F v With the feature F that has significant spatial semantics s Perform feature concatenation to obtain feature F; downsample feature F through the first edge feature to obtain feature F↓;

[0010] The feature F↓ is sent to the first multi-scale Transformer network, and different weights are assigned to the feature F↓ through the attention mechanism to obtain a multi-scale feature Z; the multi-scale feature Z is classified and regressed in the Transformer decoder to complete the behavior detection task;

[0011] In the process of completing behavior detection, masked reconstruction is used to improve the accuracy of behavior detection. The masked reconstruction is as follows: after randomly masking the feature F, it is downsampled by the second edge feature and positionally encoded, and then sent to the second multi-scale Transformer network for encoding to obtain the feature F e , the feature F e It is sent together with the initialization mask to the reconstruction decoder for feature reconstruction.

[0012] In a further technical solution, the feature F with significant spatial semantics s The target offset mutual information cross entropy loss is used for constraint, specifically:

[0013] Construct a target-background feature pool and treat the features in the target pool as positive examples k + , the features in the background pool are considered as negative examples k - ; Use the following formula to shorten the distance between features of the same object in the positive example, and at the same time increase the distance between the positive example and the negative example, so as to track the behavior subject:

[0014]

[0015] Where: q j represents the positive example k + The jth feature in Indicates that the positive example is divided by q j Other feature sets besides represents the feature set of negative examples, τ1 and τ2 represent hyperparameters, dis() calculates the distance between features, and area It consists of P patches with the highest attention scores in the pyramid features.

[0016] A further technical solution is that the feature F is downsampled by edge features to obtain feature F↓, specifically: the feature F is divided into feature blocks without overlap according to a fixed time step, and each feature block is downsampled separately through a shared convolutional layer, and the feature after downsampling is F↓.

[0017] In a further technical solution, the random masking, the second edge feature downsampling, the second multi-scale Transformer network and the reconstruction decoder constitute a masked reconstruction model, and the first edge feature downsampling, the first multi-scale Transformer network and the Transformer decoder constitute a behavior detection model.

[0018] A further technical solution is to jointly train the mask reconstruction model and the behavior detection model. The total loss function during training is:

[0019]

[0020] Among them, L cls Indicates the use of focal loss to constrain behavior classification, L reg Indicates the use of DIoU loss to constrain behavior regression positioning, L rec represents the masked reconstruction loss, ι ct represents an indicator function, marking whether there is a behavior in the time step t, T + represents the total number of behavior samples, λ reg , rec Represent the coefficients of behavior localization loss and masked reconstruction loss respectively.

[0021] In a further technical solution, the masked reconstruction loss is:

[0022] L rec (F,F c )=MSE(F,F c )

[0023] Among them: F c Represents the reconstruction feature.

[0024] According to a further technical solution, the structure of the second edge feature downsampling is the same as the network structure of the first edge feature downsampling, and the parameters are shared.

[0025] According to a further technical solution, the second multi-scale Transformer network has the same structure as the first multi-scale Transformer network and shares parameters.

[0026] A behavior detection system based on joint regular behavior subject tracking, comprising:

[0027] Feature encoding module, used to extract temporal and spatial features of the video;

[0028] The behavior subject extraction module inputs spatial features and outputs features with significant spatial semantics to determine the behavior subject;

[0029] The edge feature downsampling module concatenates the input temporal features with the features with significant spatial semantics to obtain the distinguishing features in the time dimension.

[0030] The behavior detection module inputs the downsampled features and outputs the behavior category and the time range of occurrence;

[0031] The masked reconstruction module assists the behavior detection module in learning video timing information and improving the accuracy of behavior detection.

[0032] The beneficial effects of the present invention are:

[0033] (1) The present invention provides a behavior detection method based on joint regular behavior subject tracking to solve the problems of inaccurate capture of spatial behavior salient information and blurred behavior boundaries in videos.

[0034] (2) The present invention utilizes the quadtree attention mechanism to quickly skip irrelevant areas, capture features with significant spatial semantics, determine the subject of behavior from complex scenes, and reduce the interference of scene information and behavior-irrelevant target information; it uses the target offset mutual information cross entropy loss to make the significant targets focused on in consecutive video frames consistent, thereby achieving tracking of the subject of behavior.

[0035] (3) The present invention introduces edge feature downsampling, and by independently downsampling within the feature block, the feature is made more distinguishable in the time dimension, which is conducive to locating the behavior boundary; at the same time, the operation can be performed in parallel, greatly reducing the calculation time.

[0036] (4) The present invention reconstructs the entire feature sequence after downsampling, encoding, adding position coding, etc. on the feature sequence after random masking. The reconstruction algorithm can learn the temporal correlation and assist in the behavior detection task. At the same time, the joint training of masked reconstruction and behavior detection can improve the robustness and anti-interference ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the behavior detection method based on joint regular behavior subject tracking according to the present invention;

[0038] Figure 2 Schematic diagram of the behavior detection method based on joint regular behavior subject tracking described in the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0040] like Figure 1 , 2 As shown, a behavior detection method based on joint regular behavior subject tracking specifically includes the following steps:

[0041] Step (1): for the input video, use the I3D (Inflated 3D) network to extract the temporal features F of the video v

[0042] Step (2), extract the spatial feature K of the video f ;

[0043] Step (2.1), use OpenCV library function to read the video and convert it into video frames; the sampling frequency of the video frame is consistent with the timing characteristics;

[0044] Step (2.2), feed the video frame into the pre-trained neural network (ResNet50) to extract the spatial feature K of the video f .

[0045] Step (3): The extracted spatial features are fed into the Transformer network based on quadtree attention to obtain the features F with significant spatial semantics. s , the subject of the behavior can be determined;

[0046] Step (3.1), the spatial feature K of the video f Using the learnable parameter matrix W q , W k , W v Perform projection and downsampling to construct L s Layer pyramid feature Q s , K s 、V s , as shown in formula (1); where Q s , K s Use the average pooling layer to construct the pyramid, V s Use convolution activation layers to construct pyramids;

[0047] Q s =W q K f

[0048] K s =W k K f (1)

[0049] V s =W v K f

[0050] Step (3.2) iteratively calculates the attention score of each layer of the pyramid feature. During the iteration process, the calculation of the next layer is only performed within the P patches with the highest attention scores in the previous layer, as shown in formula (2); where Attention() is the attention score calculation formula (as shown in formula (3)), It is a region The matrix formed by all k and v superimposed inside, It consists of P patches with the highest attention scores, k and v are K s 、V s Characteristics of each layer;

[0051]

[0052]

[0053] Where: l represents L s The lth layer in represents the i-th feature of the l-th layer, Representation characteristics The dimension of , T represents transpose;

[0054] Then each layer of information in the pyramid Perform weighted summation to obtain the semantic feature F with significant space s , as shown in formula (4), where are learnable weights;

[0055]

[0056] Step (4) uses the target offset mutual information cross entropy loss to optimize the feature F with significant spatial semantics. s For adjacent sampling frames, two feature pools of “target-background” are constructed, where the target pool contains Γ i L The features in the target pool are considered as positive examples k. + , the features in the background pool are considered as negative examples k- ; Since the positive examples may contain multiple objects, the distance between the features in the positive examples is calculated. If the distance is less than the threshold The distance between the features of the same object in the positive example is shortened by using formula (5), while the distance between the positive example and the negative example is increased, so as to achieve the purpose of tracking the subject of behavior; the distance between features is calculated as shown in formula (6):

[0057]

[0058]

[0059] Among them, q j represents the positive example k + The jth feature in Indicates that the positive example is divided by q j Other feature sets besides represents the feature set of negative examples, τ1 and τ2 represent hyperparameters, dis() calculates the distance between features, and a x 、a y represents the horizontal and vertical coordinates of point a, b x , b y Represents the horizontal and vertical coordinates of point b.

[0060] Step (5), transform the time series feature F v With the feature F that has significant spatial semantics s Feature splicing is performed to obtain feature F, which contains both the video temporal information and the significant spatial semantic information of the behavior subject, thereby improving the accuracy of the classification task in the behavior detection task.

[0061] Step (6), sending feature F to the first edge feature for downsampling to obtain feature F↓;

[0062] Traditional global downsampling, due to the inductive bias characteristics of CNN, causes the features to become smooth and reduces the resolution of global features. In contrast, this embodiment only downsamples within the feature block, which can maintain the distinguishability of the features in time series and is more conducive to determining the boundary information of the behavior. This operation can be performed in parallel, thereby reducing the computational complexity. This embodiment divides the feature F into feature blocks without overlap according to a fixed time step, and downsamples each feature block separately through a shared convolutional layer to maintain the distinguishability of the features in the time dimension. The downsampled feature is F↓:

[0063] F↓=local-downsample(F)(7)

[0064] Among them, local-downsample() represents the first edge feature downsampling layer.

[0065] Step (7), feed the feature F↓ into the encoder of the multi-scale Transformer network to obtain L t Layer-wise multi-scale feature representation

[0066] In step (7.1), the feature F↓ undergoes a shallow convolution and RELU activation function to obtain Z 0 =[E(f↓1),E(f↓2),...,E(f↓ N )] T , where E() is a convolutional network with RELU as activation function, f↓ n represents the nth video feature, N represents the number of input videos, E(f↓ n ) represents the feature after embedding, and T represents transposition;

[0067] Step (7.2), Z 0 Input into the multi-scale Transformer network, where the windowed local self-attention is used to calculate the attention score, that is, to calculate the similarity score between features in the time range [t, t+ω], which can reduce the amount of calculation while retaining relevant timing information; where ω represents the time offset step, and ω<<T′ (T′ represents the time length of the input video); for input Z 0 , using the learnable matrix Projection is performed to obtain feature Q t , K t 、V t , calculate the attention score and compare it with V t After multiplication, we get the feature As shown in formula (8):

[0068]

[0069] in, Represents the characteristic Q t Dimension.

[0070] Step (7.3) uses the downsampling layer to construct a feature pyramid to capture the behavior of features at different time scales; the network used for downsampling here has a total of L t Transformer layers, each layer consists of alternating local multi-head self-attention (MSA) and MLP, and LayerNorm (LN) is applied before each MSA or MLP block; in order to capture the behavior of different scales, a downsampling operation is added, as shown in formula (9):

[0071]

[0072]

[0073]

[0074] in, Indicates the first t -1 layer pyramid features, in particular, the input of the first layer feature pyramid is express After layer normalization, local multi-head attention and residual connection, the lth t Layer pyramid features; express After layer normalization, MLP and residual connection, the lth t Layer feature pyramid feature; is a learnable channel scaling factor, downsample() uses a one-dimensional convolution; l t The value range is [1,2,...,L t ].

[0075] The Transformer network encoder is connected to the downsampling layer to obtain the feature pyramid

[0076] Step (8), randomly mask the feature F in step (5), reconstruct the entire feature sequence, and force the masked and reconstructed model to learn temporal correlation;

[0077] Step (8.1), randomly mask 50% of feature F in step (5) to obtain feature F m ;

[0078] Step (8.2), the unmasked feature F m After the second edge feature is downsampled and embedded in the position code, the feature F is obtained. p ; The masked position encoding is consistent with the absolute position in the original feature sequence F, as shown in formula (10); the downsampling layer network here has the same structure as step (6) and shares parameters;

[0079] F p =Be(e)+local-downsample(F m )(10)

[0080] Among them, Be() represents the position encoder, e represents the initialization position encoding, and local-downsample() represents the second feature edge downsampling layer.

[0081] Step (8.3), feature F p It is sent to the encoder of the first multi-scale Transformer network for encoding to obtain the feature F e, as shown in formula (11); the encoder network here has the same structure as step (7) and shares parameters;

[0082] F e =E(F p )(11)

[0083] In step (8.4), traditional self-supervised learning only reconstructs the masked part and fine-tunes the model for downstream tasks. Traditional methods can make the model pay more attention to the masked local information. In comparison, reconstructing the complete feature sequence can force the model to pay attention to global information and learn temporal correlation, which is more conducive to the positioning task in behavior detection; in order to reconstruct the feature F, a shared, learnable mask is copied and interpolated into the embedding layer at the corresponding position to form a reconstruction decoder; after decoding, the reconstructed feature F is obtained c , F c The calculation process is shown in formula (12).

[0084] F c =D(F e ,B m (m))(12)

[0085] Among them, D, B m , m represent the decoder, mask embedding layer and initialization mask respectively.

[0086] Step (9), calculate feature F and reconstruct feature F c The reconstruction loss between makes the reconstructed features as similar as possible to the original features. The reconstruction loss is shown in formula (13):

[0087] l rec (F,F c )=MSE(F,F c )(13)

[0088] Random masking, second edge feature downsampling, second multi-scale Transformer network and reconstruction decoder constitute the masked reconstruction model.

[0089] Step (10): Send the feature pyramid Z to the lightweight Transformer decoder and decode it into a sequence label Each behavior instance By start time (start time) s i , end time (offset time)e i and behavior label a i Composition, i ∈[1,T′], e i ∈[1,T′], a i ∈{1,...,C} (C is a predefined behavior category), and s i<e i The decoder consists of a classifier and a regressor.

[0090] In step (10.1), the classifier checks at each moment in all L t Layer pyramid features, and predict the probability p(a t ); the classifier contains 3 layers of one-dimensional convolution (convolution kernel size is 3), layer normalization (first 2 layers) and RELU activation. The output dimension of each layer of pyramid features after the classifier is added with a σ function to predict the probability a of C behavior category t , as shown in formula (14);

[0091]

[0092] Step (10.2), the regressor is similar to the classifier, except that the regression head predicts the start time and offset time of the behavior only when the current time step t contains the behavior The regression head is also implemented using a one-dimensional convolutional network, following the same design as the classification head, with an additional RELU at the end for distance estimation, resulting in (s i ,e i ).

[0093] Step (11), during the training process, the masked reconstruction model is used as an auxiliary regularization task and is jointly trained with the behavior detection model (including the first edge feature downsampling, the first multi-scale Transformer network and the Transformer decoder); the total loss function during training is shown in formula (15):

[0094]

[0095] Among them, L cls Indicates the use of focal loss to constrain behavior classification, L reg Indicates the use of DIoU loss (Distance-IoU) to constrain behavior regression positioning, L rec represents the masked reconstruction loss, ι ct represents an indicator function, marking whether there is a behavior in the time step t, T + represents the total number of behavior samples, λ reg , rec Represent the coefficients of behavior localization loss and masked reconstruction loss respectively.

[0096] A behavior detection system based on joint regular behavior subject tracking, comprising:

[0097] Feature encoding module, used to extract temporal and spatial features of the video;

[0098] The behavior subject extraction module inputs spatial features and outputs features with significant spatial semantics to determine the behavior subject;

[0099] The edge feature downsampling module inputs the time series feature and the features concatenated with the above-mentioned features with significant spatial semantics to obtain distinguishing features in the time dimension;

[0100] The masking reconstruction module is used to assist the behavior detection module in learning video timing information and improve the accuracy of behavior detection;

[0101] The behavior detection module inputs the downsampled features and outputs the category of the behavior and the time range of occurrence.

[0102] The technical solution of the present application, or the part that contributes to the prior art, or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0103] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A behavior detection method based on joint regular behavior subject tracking, Features: Extract the temporal features F of the input video v and spatial feature K f , the spatial feature K f Through the Transformer network based on quadtree attention, the feature F with significant spatial semantics is obtained s ; The feature F with significant spatial semantics s Use target offset mutual information cross entropy loss for constraints; The time series feature F v With the feature F that has significant spatial semantics s Perform feature concatenation to obtain feature F; downsample feature F through the first edge feature to obtain feature F↓; The feature F↓ is sent to the first multi-scale Transformer network, and different weights are assigned to the feature F↓ through the attention mechanism to obtain a multi-scale feature Z; the multi-scale feature Z is classified and regressed in the Transformer decoder to complete the behavior detection task; In the process of completing behavior detection, masked reconstruction is used to improve the accuracy of behavior detection. The masked reconstruction is as follows: after randomly masking the feature F, it is downsampled by the second edge feature and positionally encoded, and then sent to the second multi-scale Transformer network for encoding to obtain the feature F e , the feature F e It is sent together with the initialization mask to the reconstruction decoder for feature reconstruction.

2. The behavior detection method based on joint regular behavior subject tracking according to claim 1, It is characterized in that The feature F with significant spatial semantics s The target offset mutual information cross entropy loss is used for constraint, specifically: Construct a target-background feature pool and treat the features in the target pool as positive examples k + , the features in the background pool are considered as negative examples k - ; Use the following formula to shorten the distance between features of the same object in the positive example, and at the same time increase the distance between the positive example and the negative example, so as to track the behavior subject: Where: q j represents the positive example k + The jth feature in Indicates that the positive example is divided by q j Other feature sets besides Represents the feature set of negative examples, τ 1 , τ 2 represents the hyperparameter, dis() calculates the distance between features, and the area Γ i l It consists of P patches with the highest attention scores in the pyramid features.

3. The behavior detection method based on joint regular behavior subject tracking according to claim 1, It is characterized in that The feature F is downsampled by edge features to obtain feature F↓, specifically: feature F is divided into feature blocks without overlap according to a fixed time step, and each feature block is downsampled separately through a shared convolutional layer, and the feature after downsampling is F↓.

4. The behavior detection method based on joint regular behavior subject tracking according to claim 1, It is characterized in that The random masking, the second edge feature downsampling, the second multi-scale Transformer network and the reconstruction decoder constitute a masked reconstruction model, and the first edge feature downsampling, the first multi-scale Transformer network and the Transformer decoder constitute a behavior detection model.

5. The behavior detection method based on joint regular behavior subject tracking according to claim 4, It is characterized in that The masked reconstruction model and the behavior detection model are jointly trained, and the total loss function during training is: Among them, L cls Indicates the use of focal loss to constrain behavior classification, L reg Indicates the use of DIoU loss to constrain behavior regression positioning, L rec represents the masked reconstruction loss, ι ct represents an indicator function, marking whether there is a behavior in the time step t, T + represents the total number of behavior samples, λ reg , rec Represent the coefficients of behavior localization loss and masked reconstruction loss respectively.

6. The behavior detection method based on joint regular behavior subject tracking according to claim 5, It is characterized in that The masked reconstruction loss is: L rec (F,F c )=MSE(F,F c ) Among them: F c Represents the reconstruction feature.

7. The behavior detection method based on joint regular behavior subject tracking according to claim 1, It is characterized in that The structure of the second edge feature downsampling is the same as the network structure of the first edge feature downsampling, and the parameters are shared.

8. The behavior detection method based on joint regular behavior subject tracking according to claim 1, It is characterized in that The second multi-scale Transformer network has the same structure as the first multi-scale Transformer network and shares parameters.

9. A system for implementing the behavior detection method based on joint regular behavior subject tracking as described in any one of claims 1 to 8, It is characterized in that include: Feature encoding module, used to extract temporal and spatial features of the video; The behavior subject extraction module inputs spatial features and outputs features with significant spatial semantics to determine the behavior subject; The edge feature downsampling module concatenates the input temporal features with the features with significant spatial semantics to obtain the distinguishing features in the time dimension. The behavior detection module inputs the downsampled features and outputs the category of the behavior and the time range of occurrence.

10. The system according to claim 9, It is characterized in that It also includes a masked reconstruction module to assist the behavior detection module in learning video timing information to improve the accuracy of behavior detection.

Images