Space-time behavior detection method

Abstract

The invention provides a time-space behavior detection method, which is characterized by comprising the following steps of: S1, slicing a video to obtain a plurality of continuous image frames; s2, all the image frames are input into a self-adaptive cross-scale feature fusion codec network, a space-time behavior detection result is obtained, the self-adaptive cross-scale feature fusion codec network comprises a self-adaptive cross-scale feature fusion space-time encoder and a shared detection head decoder, and the self-adaptive cross-scale feature fusion codec network comprises a self-adaptive cross-scale feature fusion space-time encoder and a shared The adaptive cross-scale feature fusion space-time encoder comprises a 2D backbone module, an AIFI module, an adaptive cross-scale space fusion module, a 3D backbone module and a splicing module, wherein the adaptive cross-scale space fusion module is used for extracting high-quality space information and space-time information and fusing the high-quality space information and the space-time information. In a word, the method can generate an accurate space-time behavior detection result for the video.

Description

Technical Field

[0001] The present invention relates to the field of video understanding, and in particular to a spatiotemporal behavior detection method. Background Art

[0002] Spatiotemporal behavior detection is a branch task in the video understanding task. Its content is to mark the location of the target actor in the video sequence in space and classify the actor's ongoing actions. At present, spatiotemporal behavior detection has been applied in many scenarios and plays an important role, such as abnormal safety behavior detection of factory workers, site security detection, illegal behavior detection such as fighting in public places, and detection of autonomous driving and driver driving behavior. The proposed detection methods are all committed to improving the detection effect of behavioral actions, so improving the learning and understanding ability of the detection model has become a current research hotspot.

[0003] At present, spatiotemporal behavior detection based on deep learning is divided into two categories: clip-level and frame-level detection according to the form of the prediction candidates obtained. The Clip-Level detection method is a detection process that obtains a 3D spatiotemporal Tubelet object that needs to be predicted for a slice in the video and predicts the object. The frame-level method generates a target suggestion for the action actor for each frame in the video and classifies the action suggestion into behaviors.

[0004] However, the existing spatiotemporal behavior detection algorithms are not comprehensive enough for the extraction of spatiotemporal information features. Some of the same action information at different scales may not be detected, and its detection performance needs to be further improved. Summary of the invention

[0005] The present invention is made to solve the above-mentioned problem, and its purpose is to provide a spatiotemporal behavior detection method.

[0006] The present invention provides a spatiotemporal behavior detection method, which has the following characteristics and comprises the following steps: step S1, slicing a video to obtain a plurality of continuous image frames; step S2, inputting all the image frames into an adaptive cross-scale feature fusion codec network to obtain a spatiotemporal behavior detection result, wherein the adaptive cross-scale feature fusion codec network comprises an adaptive cross-scale feature fusion spatiotemporal encoder and a shared detection head decoder, and the adaptive cross-scale feature fusion spatiotemporal encoder comprises: a 2D backbone module, which is used to sample the last image frame to generate a plurality of semantic features with different scales, wherein all the semantic features include a first semantic feature, a second semantic feature and a third semantic feature; an AIFI module, which is used to encode the first semantic feature to obtain an encoded feature; an adaptive cross-scale spatial fusion module, which is used to encode the encoded feature, the second semantic feature and the third semantic feature. The first spatial feature, the second spatial feature and the third spatial feature are obtained by row fusion; the 3D backbone module is used to extract the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature with different scales from all image frames; the splicing module is used to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature to obtain the spatiotemporal feature encoding, the first semantic feature, the second semantic feature and the third semantic feature are respectively different by n times of the spatial sampling rate, and the scale of the first semantic feature is the smallest. The shared detection head decoder includes: a screening module, which is used to perform an Object screening operation on the spatiotemporal feature encoding to generate a content query and a position query; a decoding module is used to decode according to the content query and the position query to obtain the prediction results of the position and category of the action in each image frame as the spatiotemporal behavior detection result.

[0007] The spatiotemporal behavior detection method provided by the present invention may also have the following characteristics: wherein, in the adaptive cross-scale spatial fusion module, the first spatial feature, the second spatial feature and the third spatial feature are fused according to the coding feature, the second semantic feature and the third semantic feature, and the calculation expression of the first spatial feature, the second spatial feature and the third spatial feature is obtained as follows: S′ 5 =σ SiLU (BN(Conv 1×1 (F 5 ))), S′ 5up =UpSample n ×(S′ 5 ), S 4,5up =Fusion(S′ 5up , S 4 ), S′ 4 =σ SiLU (BN(Conv 1×1 (S 4，5up ))), S′ 4up =UpSample(S′ 4),S 3,4up =Fusion(S′ 4up ,S 3 ),S′ 3 =σ SiLU (BN(Conv 1×1 (S 3，4up ))),S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))),i=3,4,5,S 54 =ASFF 1 (DownSample n× (S″ 4 ),S″ 5 ),S 45 =ASFF 2 (UpSample n× (S″ 5 ),S″ 4 ),S′ 54 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 54 ))))),S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))),S 534 =ASFF 3 (S′ 54 ,DownSample n× (S′ 45 ),DownSample 2n× (S″ 3 )),S 435 =ASFF 4 (UpSample n× (S′ 54 ),S′ 45 ,DownSample n× (S″ 3 )),S 345 =ASFF 5 (UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″ 3 ),P 5 =BN(Conv 3×3 (ReLU(BN(Conv 3×3(S 534 ))))),P 4 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))),P 3 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))), ASFF d (a 1 , a 2 , …, a m )=Conv 3×3 (α 1 a 1 +α 2 a 2 +…+α m a m ), m≥2, d=1, 2, 3, 4, 5, α i =Softmax(Conv 1×1 (σ ReLU (BN(conv 1×1 (a i ))))), 1≤i≤m, where F 5 is the encoding feature, Conv 1×1 is processed by a 1×1 convolution layer, BN is processed by a BatchNorm layer, σ SiLU To activate the operation via SiLU, UpSample n× is n times upsampling, S 4 is the second semantic feature, σ ReLU DownSample is activated by ReLU. n× is n times down-sampled, Conv 3×3 To process through a 3×3 convolutional layer, UpSample 2n× 2n times upsampling, DownSample 2n× is 2n times downsampling, P 5 is the first spatial feature, P 4 is the second spatial feature, P 3 is the third space feature, Concat is the connection operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N It is N-layer RepBlock processing, and Softmax is Softmax processing.

[0008] The spatiotemporal behavior detection method provided by the present invention may also have the following feature: wherein, in the splicing module, the calculation expression of the spatiotemporal feature encoding is obtained according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature:

[0009] F i1 =Conv 1×1 (Conv 3×3 (Concat[P i , V i ])), i = 3, 4, 5,

[0010] F i2 =Reshape(F i1 ), i=3,4,5,

[0011]

[0012] F i =Reshape(F i3 ), i=3,4,5,

[0013] F st =Concat(F3, F4, F5),

[0014] Where P 5 , P 4 and P 3 They are the first spatial feature, the second spatial feature and the third spatial feature, V 5 、V 4 and V 3 They are the first space-time feature, the second space-time feature, and the third space-time feature, respectively. st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After 1×1 convolutional layer processing, Conv 3×3 It is processed by a 3×3 convolution layer, Reshape is a dimension deformation operation (compressing the HW dimension into one dimension and restoring one dimension to the HW dimension are both called reshaping), and Softmax is a Softmax process.

[0015] In the spatiotemporal behavior detection method provided by the present invention, it can also have the following characteristics: wherein, the decoding module includes a plurality of decoder layers and a shared submodule connected in sequence, the decoder layer includes: a multi-head self-attention unit, used to perform multi-head self-attention processing on the input first data to obtain the second data; a cross-attention unit, used to perform cross-attention processing on the first data, the spatiotemporal feature encoding and the second data to obtain the third data; an FFN unit, including a feedforward neural network, used to generate output features according to the second data and the third data, the shared submodule includes a shared regression head and a shared classification head, used to generate the corresponding position coordinates of the predicted behavior object and the confidence scores of each category of the predicted behavior object according to the output features of each decoder layer, the first data of the first decoder layer is the content query and the position query, the first data of the second to the last decoder layers are the position coordinates and output features corresponding to the previous decoder layer, and the spatiotemporal behavior detection result is obtained according to the position coordinates of the predicted behavior object corresponding to the last decoder layer and the confidence scores of each category of the predicted behavior object.

[0016] In the spatiotemporal behavior detection method provided by the present invention, the following features may also be provided: wherein, in the training process of the adaptive cross-scale feature fusion codec network, the contrast denoising noise is generated by GroundTruth, and combined with the multi-constraint loss function Perform parameter optimization and multi-constraint loss function The calculation expression is: GT i ={c i ,b i}, Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the ith GroundTruth object, c i For GroundTruth object GT i The categories in b i For GroundTruth object GT i The bounding box in is the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the results The prediction classification in For the results The regression box prediction in ,λ VFL, L1 , GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT i The corresponding one-to-many prediction result set of the l-th decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i is the ith GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

[0017] Functions and Effects of the Invention

[0018] According to the spatiotemporal behavior detection method involved in the present invention, the adaptive cross-scale spatial fusion module uses an adaptive cross-scale method to extract high-quality spatial information from each scale, and the spatiotemporal information extracted by the 3D backbone module is fused through the splicing module, and then provided to the subsequent modules for learning and understanding, thereby obtaining accurate spatiotemporal behavior detection results. Therefore, the spatiotemporal behavior detection method of the present invention can generate accurate spatiotemporal behavior detection results for videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flow chart of a spatiotemporal behavior detection method in an embodiment of the present invention;

[0020] Figure 2is a block diagram of an adaptive cross-scale feature fusion codec network in an embodiment of the present invention;

[0021] Figure 3 is a block diagram of an adaptive cross-scale feature fusion spatiotemporal encoder in an embodiment of the present invention;

[0022] Figure 4 is a block diagram of a decoding module in an embodiment of the present invention;

[0023] Figure 5 is a block diagram of a decoding layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the spatiotemporal behavior detection method of the present invention.

[0025] This embodiment provides a spatiotemporal behavior detection device for obtaining a spatiotemporal behavior detection result of a video.

[0026] Figure 1 It is a flow chart of the spatiotemporal behavior detection method in an embodiment of the present invention.

[0027] like Figure 1 As shown, the spatiotemporal behavior detection method is used to obtain the spatiotemporal behavior detection result of the video, including the following steps:

[0028] Step S1, slicing the video to obtain a plurality of continuous image frames.

[0029] Step S2: input all image frames into an adaptive cross-scale feature fusion codec network to obtain spatiotemporal behavior detection results.

[0030] Figure 2 is a block diagram of an adaptive cross-scale feature fusion codec network in an embodiment of the present invention.

[0031] like Figure 2 As shown, the adaptive cross-scale feature fusion encoder-decoder network 200 includes an adaptive cross-scale feature fusion spatiotemporal encoder 21 and a shared detection head decoder 22.

[0032] Figure 3 is a block diagram of an adaptive cross-scale feature fusion spatiotemporal encoder in an embodiment of the present invention.

[0033] like Figure 3 As shown, the adaptive cross-scale feature fusion spatiotemporal encoder 21 includes a 2D backbone module 211, an AIFI module 212, an adaptive cross-scale spatial fusion module 213, a 3D backbone module 214 and a splicing module 215.

[0034] The 2D backbone module 211 is used to sample the last image frame to generate multiple semantic features with different scales, wherein all semantic features include a first semantic feature, a second semantic feature and a third semantic feature. The first semantic feature, the second semantic feature and the third semantic feature are respectively different by n times of the spatial sampling rate, and the scale of the first semantic feature is the smallest.

[0035] In this embodiment, the 2D backbone module 211 is the existing 2D backbone of ResNet, which performs multi-scale semantic feature extraction on the last image frame. The semantic features of each adjacent level scale differ by n=2 times of the spatial sampling rate, and the semantic feature S is obtained. 1 , S 2 , S 3 , S 4 and S 5 , and the semantic feature S 5 As the first semantic feature, the semantic feature S 4 As the second semantic feature, and the semantic feature S 3 As the third semantic feature. In other embodiments, the 2D backbone module 211 can generate at least three semantic features according to actual needs, and select a semantic feature with richer information as the first semantic feature, and then select the second semantic feature and the third semantic feature according to the first semantic feature. In addition, in other embodiments, the spatial sampling rate can be adjusted according to actual needs, for example, the spatial sampling rate of adjacent semantic features differs by n=4 times.

[0036] The AIFI module 212 is used to encode the first semantic feature to obtain the encoded feature. In this embodiment, the AIFI module 212 is an existing Attention-based Intrascale Feature Interaction module.

[0037] The adaptive cross-scale spatial fusion module 213 is used to fuse the coding feature, the second semantic feature and the third semantic feature to obtain the first spatial feature, the second spatial feature and the third spatial feature.

[0038] The adaptive cross-scale spatial fusion module 213 fuses the coding feature, the second semantic feature and the third semantic feature to obtain the calculation expression of the first spatial feature, the second spatial feature and the third spatial feature as follows:

[0039] S′ 5 =σ SiLU (BN(Conv 1×1 (F 5 ))),

[0040] S′ 5up =UpSample n×(S′ 5 ),

[0041] S 4,5up =Fusion(S′ 5up ,S 4 ),

[0042] S′ 4 =σ SiLU (BN(Conv 1×1 (S 4，5up ))),

[0043] S′ 4up =UpSample(S′ 4 ),

[0044] S 3,4up =Fusion(S′ 4up ,S 3 ),

[0045] S′ 3 =σ SiLU (BN(Conv 1×1 (S 3，4up ))),

[0046] S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))),i=3,4,5,

[0047] S 54 =ASFF 1 (DownSample n× (S″ 4 ),S″ 5 ),

[0048] S 45 =ASFF 2 (UpSample n× (S″ 5 ),S″ 4 ),

[0049] S′ 54 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 54 ))))),

[0050] S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))),

[0051] S 534 =ASFF 3 (S′ 54 ,DownSample n× (S′ 45 ), DownSample 2n× (S″ 3 )),

[0052] S 435 =ASFF 4 (UpSample n× (S′ 54 ),S′ 45 ,DownSample n× (S″ 3 )),

[0053] S 345 =ASFF 5 (UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″ 3 ),

[0054] P 5 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 534 ))))),

[0055] P 4 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))),

[0056] P 3 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))),

[0057]

[0058] AFF d (to 1 ,to 2 ,…,to m )=Conv 3×3 (α 1 A 1 +α 2 A 2 +…+αm a m ), m≥2, d=1, 2,3,4,5,

[0059] α i =softmax(Conv 1×1 (σ ReLU (BN(Conv 1×1 (a i ))))), 1≤i≤m,

[0060] Where F 5 is the encoding feature, Conv 1×1 is processed by a 1×1 convolution layer, BN is processed by a BatchNorm layer, σ SiLU To activate the operation via SiLU, UPSample n× is n times upsampling, S 4 is the second semantic feature, σ ReLU For the ReLU activation operation, DOWNSample n× is n times down-sampled, Conv 3×3 After 3×3 convolutional layer processing, UPSample 2n× 2n times upsampling, DOWNSample 2n× is 2n times downsampling, P 5 is the first spatial feature, P 4 is the second spatial feature, P 3 is the third space feature, Concat is the connection operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N The ASFF of the adaptive cross-scale feature fusion codec network 200 trained in this embodiment is as follows: d Operations are performed separately by computing modules with different weights.

[0061] In this embodiment, the adaptive cross-scale spatial fusion module 213 is a cross-scale fusion network that inserts a fusion block composed of a convolutional layer network and multiple adaptive structural feature fusion blocks, namely ASFF operations, on the fusion path of multiple scale features. Among the multiple scale features input to the adaptive cross-scale spatial fusion module 213, only the highest semantic feature is obtained through the transformer attention operation, and other scale features do not need attention calculation, so the computational burden can be reduced well. In addition, the attention information of the highest semantic feature is fused into the feature information at other scales through the adaptive structural feature fusion block, so that other scales can also understand the information weight distribution on the entire spatial feature, so that high-quality information can be extracted at different scales.

[0062] The adaptive cross-scale spatial fusion module 213 also uses the asymptotic architecture to further transform S′ 5 The weight information of the feature at its own scale is gradually propagated to the feature information of all other scales. At the same time, the model uses an adaptive method to determine the weight assigned to each scale during fusion, so that each scale feature has the weight from S′ while ensuring the features of its own scale. 5 The attention feature information of the neighboring scale and the features containing attention information from the neighboring scale can prevent the loss caused by information degradation.

[0063] The 3D backbone module 214 is used to extract first spatiotemporal features, second spatiotemporal features, and third spatiotemporal features of different scales from all image frames.

[0064] The splicing module 215 is used to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature to obtain the spatiotemporal feature code.

[0065] In the splicing module 215, the calculation expression of the spatiotemporal feature coding is obtained according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature:

[0066] F i1 =Conv 1×1 (Conv 3×3 (Concat[P i , V i ])), i = 3, 4, 5,

[0067] F i2 =Reshape(F i1 ), i=3,4,5,

[0068]

[0069] Fi =Reshape(F i3 ), i=3,4,5,

[0070] F st =Concat(F3, F4, F5),

[0071] Where P 5 , P 4 and P 3 They are the first spatial feature, the second spatial feature and the third spatial feature, V 5 、V 4 and V 3 They are the first space-time feature, the second space-time feature, and the third space-time feature, respectively. st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After 1×1 convolutional layer processing, Conv 3×3 It is processed by a 3×3 convolution layer, Reshape is a dimension deformation operation, which includes compressing the HW dimension into one dimension and restoring a compressed dimension to the HW dimension, and Softmax is a Softmax process.

[0072] The shared detection head decoder 22 includes a screening module 221 and a decoding module 222 .

[0073] The screening module 221 is used to perform an object screening operation on the spatiotemporal feature coding to generate content query and location query. In this embodiment, the screening module 221 performs an object screening operation on the spatiotemporal feature coding to screen the top K feature blocks with the highest confidence, and the object obtained thereby has better content query and location query. This screening process eliminates insignificant query object information and reduces the computational burden of the network.

[0074] The decoding module 222 is used to perform decoding according to the content query and the position query to obtain the prediction results of the position and category of the action in each image frame as the spatiotemporal behavior detection result.

[0075] Figure 4 is a block diagram of a decoding module in an embodiment of the present invention.

[0076] like Figure 4 As shown, the decoding module 222 includes a plurality of decoding layers 2221 and a sharing submodule 2222 connected in sequence. In this embodiment, the number of the decoding layers 2221 is 6.

[0077] Figure 5 is a block diagram of a decoding layer in an embodiment of the present invention.

[0078] like Figure 5As shown, the decoding layer 2221 includes a multi-head self-attention unit 22211, a cross-attention unit 22212 and a FFN unit 22213.

[0079] The multi-head self-attention unit 22211 is used to perform multi-head self-attention processing on the input first data to obtain second data. The function of the multi-head self-attention processing in this embodiment is to calculate the relationship between the content query embedded in the position information and itself, and then select the most likely candidate object based on this attention information, so as to infer the object position information of the encoding position. Furthermore, the multi-head self-attention processing adopts the self-attention method commonly used in Transformer.

[0080] The cross attention unit 22212 is used to perform cross attention processing on the first data, the spatiotemporal feature encoding, and the second data to obtain third data. In this embodiment, the cross attention processing is to further obtain or optimize the candidate object based on the output of the position query and the multi-self-attention, that is, the second data. Furthermore, the cross attention processing adopts a deformable attention mechanism that only needs to interact with the partial pixels obtained based on sampling.

[0081] The FFN unit 22213 includes a feedforward neural network for generating output features based on the second data and the third data.

[0082] The shared submodule 2222 includes a shared regression head and a shared classification head. The shared regression head generates the position coordinates of the predicted behavior object corresponding to the decoder layer 2221 based on the output features of the decoder layer 2221. The shared classification head generates the confidence scores of each category of the predicted behavior object corresponding to the decoder layer 2221 based on the output features of the decoder layer 2221.

[0083] Among them, the first data of the first decoder layer 2221 is the content query and the location query. The first data of the second to the last decoder layer 2221 are the location coordinates and output features corresponding to the previous decoder layer. The spatiotemporal behavior detection result is obtained according to the location coordinates of the predicted behavior object corresponding to the last decoder layer 2221 and the confidence scores of each category of the predicted behavior object.

[0084] Since the prior art only uses the 1-to-1 loss function of the bipartite graph matching method, the learning process of the Encoder-Decoder for spatiotemporal features has problems such as insufficient understanding and insufficient constraints. To this end, this embodiment combines Contrastive Denoising (CDN) method, one-to-one supervision method and one-to-many supervision method to construct a multi-constraint loss function. Thereby optimizing the parameters of the adaptive cross-scale feature fusion codec network 200.

[0085] The CDN method generates contrast denoising noise based on GroundTruth, and the contrast denoising noise includes noise content query and noise position query. During the training process, the noise content query and the content query are concatenated as the first data of the first decoder layer 2221, and the noise position query and the position query are concatenated as the second data of the first decoder layer 2221.

[0086] The multi-constrained loss function The calculation expression is:

[0087]

[0088]

[0089] GT i ={c i , b i},

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the ith GroundTruth object, c i For GroundTruth object GT i The categories in b i For GroundTruth object GT i The bounding box in is the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the results The prediction classification in For the results The regression box prediction in ,λ VFL , L1 , GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT iThe corresponding one-to-many prediction result set of the l-th decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i is the ith GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder 22 and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

[0096] In this embodiment, the adaptive cross-scale feature fusion codec network 200, namely Our Method, is compared with the existing advanced algorithms in the field, namely T-CNN, ACT, MPS, SAMOC, STEP, YOWO and TubeR, using the existing UCF101-24 dataset. The specific results are shown in the following table:

[0097] algorithm Frame-mAP param FLOPS Video-mAP T-CNN 41.4％ - - 51.3％ ACT 69.5％ - - - MPS - - - 82.4％ SAMOC 79.3％ - - 80.5％ STEP 75.0％ - - 83.1％ TubeR 81.3％ 120M 132G - YOWO 80.4％ 121.4M 54.7G 82.5％ Our Method 81.52％ 120.54M 57.51G 76.97％

[0098] The first column in the table above is each algorithm, and the second to fifth columns are the Frame-mAP index, model parameter param, and calculation FLOPS of each algorithm when the IoU threshold is 0.5, and the Video-mAP index when the IoU threshold is 0.1. The higher the Frame-mAP index and Video-mAP index, the lower the model parameter param and calculation FLOPS, indicating a better model. For example, the cell in the third column of the ninth row indicates that the model parameter of Our Method is 120.54M.

[0099] It can be seen that in terms of algorithm accuracy, i.e., Frame-mAP indicator when the IoU threshold is 0.5, OurMethod has improved by 1.1% and 0.2% respectively compared with YOWO and TubeR, which have better performance. This is because the adaptive cross-scale feature fusion codec network 200 gradually propagates the attention information in the high-level information to other scale features through the cross-scale adaptive fusion method, so that each scale learns which information is more important, that is, which should be noticed more. Compared with other scale fusion methods, this method loses less information and has better results.

[0100] In terms of model complexity, i.e., model parameters param and computational FLOPS, Our Method simplifies the computational complexity by about 75G compared to the pure 3D Transformer method TubeR, while the model parameters are 0.9M fewer than YOWO. This is mainly because the adaptive cross-scale feature fusion encoder-decoder network 200 adopts a simplified encoder-decoder structure, i.e., the encoder only performs attention mechanism calculations on deep semantics, and the detection head of the decoder is designed to be shared.

[0101] In this embodiment, an ablation experiment is also performed on the adaptive cross-scale spatial fusion module 213. The adaptive cross-scale feature fusion codec network 200 without the adaptive cross-scale spatial fusion module 213 is recorded as CCFM, and the original adaptive cross-scale feature fusion codec network 200 is recorded as ACCFM. The specific comparative performance is shown in the following table:

[0102]

[0103] The first column in the above table is each method, the second column is the f-mAP index corresponding to each method, and the third to fifth columns are the v-mAP indexes under IoU of 0.1, 0.2 and 0.5 respectively. The higher the above index is, the better the model is. It can be seen that in terms of the f-mAP index, ACCFM effectively improves the detection accuracy by 0.5% compared to CCFM. In addition, in terms of the v-mAP index with IoU of 0.1 and 0.2, ACCFM also improves by 0.6% and 0.8% respectively compared to CCFM. Therefore, the adaptive cross-scale feature fusion codec network 200 can improve the accuracy of spatiotemporal action behavior detection.

[0104] In this embodiment, the multi-constraint loss function Conduct ablation experiments to Loss and The loss was tested and the results are as follows:

[0105]

[0106] The first column in the table above shows the various ablation settings, and the second to fourth columns show the F-mAP index, classification accuracy, and recall rate under each setting. For example, the cell in the second column of the fifth row shows that the F-mAP of OurMethod is 81.52%. This shows that when the multi-constraint loss function cannot be fully used, , the performance of the model showed different degrees of decline, so the model uses a multi-constrained loss function Learning and understanding on the training set can effectively improve the detection effect.

[0107] Functions and Effects of the Embodiments

[0108] According to the spatiotemporal behavior detection method involved in this embodiment, the adaptive cross-scale spatial fusion module uses an adaptive cross-scale method to extract high-quality spatial information from each scale, and the spatiotemporal information extracted by the 3D backbone module is fused through the splicing module, and then provided to the subsequent modules for learning and understanding, thereby obtaining accurate spatiotemporal behavior detection results. In short, this method can generate accurate spatiotemporal behavior detection results for videos.

[0109] Furthermore, by jointly constraining the training of the entire model through a multi-constrained loss function, not only the screening results of the adaptive cross-scale feature fusion spatiotemporal encoder are optimized, but also the understanding and learning of the screening results by the decoding module is improved.

[0110] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A spatiotemporal behavior detection method for obtaining a spatiotemporal behavior detection result of a video, characterized in that: The following steps are involved: Step S1, slicing the video to obtain a plurality of continuous image frames; Step S2, inputting all the image frames into an adaptive cross-scale feature fusion codec network to obtain the spatiotemporal behavior detection result, The adaptive cross-scale feature fusion codec network includes an adaptive cross-scale feature fusion spatiotemporal encoder and a shared detection head decoder. The adaptive cross-scale feature fusion spatiotemporal encoder includes: A 2D backbone module, used for sampling the last image frame to generate a plurality of semantic features with different scales, wherein all the semantic features include a first semantic feature, a second semantic feature and a third semantic feature; An AIFI module is used to encode the first semantic feature to obtain an encoded feature; An adaptive cross-scale spatial fusion module, used for fusing the coding feature, the second semantic feature and the third semantic feature to obtain a first spatial feature, a second spatial feature and a third spatial feature; A 3D backbone module, used for extracting first spatiotemporal features, second spatiotemporal features and third spatiotemporal features with different scales from all the image frames; a splicing module, configured to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature to obtain a spatiotemporal feature code, The first semantic feature, the second semantic feature and the third semantic feature are respectively different in spatial sampling rate by n times, and the scale of the first semantic feature is the smallest. The shared detection head decoder comprises: A screening module, used to perform an Object screening operation on the spatiotemporal feature code to generate a content query and a location query; A decoding module is used to perform decoding according to the content query and the position query to obtain the prediction results of the position and category of the action in each of the image frames as the spatiotemporal behavior detection result.

2. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, In the adaptive cross-scale spatial fusion module, the encoding feature, the second semantic feature and the third semantic feature are fused to obtain the calculation expression of the first spatial feature, the second spatial feature and the third spatial feature: S′5=σ SiLU (BN(Conv 1×1 (F5))), S′ 5up =UpSample n× (S′5), S 4,5up =Fusion(S′ 5up ,S4), S′4=σ SiLU (BN(Conv 1×1 (S 4,5up ))), S′ 4up =UpSample(S′4), S 3,4up =Fusion(S′ 4up ,S3), S′3=σ SiLU (BN(Conv 1×1 (S 3,4up ))), S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))),i=3,4,5, S 54 =ASFF1(DownSample n× (S″4),S″5), S 45 =ASFF2(UpSample n× (S″5),S″4), S′ 54 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 54 ))))), S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))), S 534 =ASFF3(S′ 54 ,DownSample n× (S′ 45 ),DownSample 2n× (S″3)), S 435 =ASFF4(UpSample n× (S′ 54 ),S′ 45 ,DownSamole n× (S″3)), S 345 =ASFF5(UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″3), P5=BN(Conv. 3×3 (ReLU(BN(Conv 3×3 (S 534 ))))), P4=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))), P3=BN(Conv. 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))), ASFF d (a1,a2,...,a m )=Conv 3×3 (α1a1+α2a2+…+α m to m ),m≥2,d=1,2,3,4,5, a i =Softmax(Conv 1×1 (s ReLU (BN(Conv 1×1 (a i ))))),1≤i≤m, Where F5 is the encoding feature, Conv 1×1 is processed by a 1×1 convolutional layer, BN is processed by a BatchNorm layer, σ SiLU To activate the operation via SiLU, UpSample n× is n times upsampling, S4 is the second semantic feature, σ ReLU For the ReLU activation operation, DownSamplen n× is n times down-sampled, Conv 3×3 After 3×3 convolutional layer processing, UPSample 2n× 2n times upsampling, DownSample 2n× is 2n times downsampling, P5 is the first spatial feature, P4 is the second spatial feature, P3 is the third spatial feature, Concat is a concatenation operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N It is N-layer RepBlock processing, and Softmax is Softmax processing.

3. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, In the splicing module, the calculation expression for obtaining the spatiotemporal feature coding according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature is: F i1 =Conv 1×1 (Conv 3×3 (Concat[P i ,V i ])),i=3,4,5, F i2 =Reshape(F i1 ),i=3,4,5, F i =Reshape(F i3 ),i=3,4,5, F st =Concat(F3,F4,F5), Wherein P5, P4 and P3 are the first spatial feature, the second spatial feature and the third spatial feature respectively, V5, V4 and V3 are the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature respectively, and F st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After 1×1 convolutional layer processing, Conv 3×3 It is processed by a 3×3 convolution layer, Reshape is a dimension deformation operation, and Softmax is a Softmax process.

4. The spatiotemporal behavior detection method according to claim 1, Features: The decoding module includes a plurality of decoder layers and shared submodules connected in sequence. The decoder layer comprises: A multi-head self-attention unit, used for performing multi-head self-attention processing on the input first data to obtain second data; a cross attention unit, configured to perform cross attention processing on the first data, the spatiotemporal feature code and the second data to obtain third data; an FFN unit, comprising a feedforward neural network, for generating output features according to the second data and the third data, The shared submodule includes a shared regression head and a shared classification head, which are used to generate the position coordinates of the corresponding predicted behavior object and the confidence scores of each category of the predicted behavior object according to the output features of each decoder layer, The first data of the first decoder layer is the content query and the location query, The first data of the second to the last decoder layer are the position coordinates and the output features corresponding to the previous decoder layer, The spatiotemporal behavior detection result is obtained according to the position coordinates of the predicted behavior object corresponding to the last decoder layer and the confidence scores of each category of the predicted behavior object.

5. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, During the training process of the adaptive cross-scale feature fusion codec network, contrast denoising noise is generated by GroundTruth and combined with a multi-constrained loss function Optimize the parameters. The multi-constraint loss function The calculation expression is: GT i ={c i ,b i }, Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the ith GroundTruth object, c i For GroundTruth object GT i The categories in b i For GroundTruth object GT i The bounding box in is the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the result The prediction classification in For the result The regression box prediction in ,λ VFL , L1 , GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT i The corresponding one-to-many prediction result set of the l-th decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i is the ith GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

Images