Spatiotemporal behavior detection method

Abstract

The application provides a spatio-temporal behavior detection method, which has the characteristics that comprises the following steps: S1, slicing a video to obtain a plurality of continuous image frames; S2, inputting all the image frames into an adaptive cross-scale feature fusion codec network to obtain a spatio-temporal behavior detection result, wherein the adaptive cross-scale feature fusion codec network comprises an adaptive cross-scale feature fusion spatio-temporal encoder and a shared detection head decoder, the adaptive cross-scale feature fusion spatio-temporal encoder comprises a 2D backbone module, an AIFI module, an adaptive cross-scale spatial fusion module, a 3D backbone module and a splicing module, and the adaptive cross-scale spatial fusion module is used for extracting high-quality spatial information and spatio-temporal information and fusing the same. In summary, the method can generate an accurate spatio-temporal behavior detection result for a video.

Description

Technical Field

[0001] The present invention relates to the field of video understanding, and in particular to a spatiotemporal behavior detection method. Background Art

[0002] Spatiotemporal behavior detection is a sub-task within video understanding. It involves spatially annotating the target actor's location in a video sequence and classifying the actor's actions. Currently, spatiotemporal behavior detection is widely used and plays a vital role in many scenarios, such as detecting abnormal factory worker safety behaviors, ensuring workplace security, detecting illegal behaviors such as fights in public places, and detecting driver behavior in autonomous driving. Existing detection methods are focused on improving the effectiveness of action detection, and therefore, enhancing the learning and understanding capabilities of detection models has become a current research hotspot.

[0003] Currently, deep learning-based spatiotemporal action detection is categorized into two main types: clip-level and frame-level detection, based on the form of the predicted candidates obtained. Clip-level detection involves obtaining a 3D spatiotemporal Tubelet object to be predicted for each slice in a video and performing a prediction on it. Frame-level detection generates target suggestions for the action actor for each frame in the video and classifies the action suggestions into behaviors.

[0004] However, the existing spatiotemporal behavior detection algorithms are not comprehensive enough for the extraction of spatiotemporal information features. Some of the same action information at different scales may not be detected, and its detection performance needs to be further improved. Summary of the Invention

[0005] The present invention is made to solve the above problems, and its purpose is to provide a spatiotemporal behavior detection method.

[0006] The present invention provides a spatiotemporal behavior detection method, which has the following characteristics: step S1, slicing the video to obtain multiple continuous image frames; step S2, inputting all image frames into an adaptive cross-scale feature fusion codec network to obtain a spatiotemporal behavior detection result, wherein the adaptive cross-scale feature fusion codec network includes an adaptive cross-scale feature fusion spatiotemporal encoder and a shared detection head decoder, and the adaptive cross-scale feature fusion spatiotemporal encoder includes: a 2D backbone module for sampling the last image frame to generate multiple semantic features with different scales, all semantic features including a first semantic feature, a second semantic feature and a third semantic feature; an AIFI module for encoding the first semantic feature to obtain a coded feature; an adaptive cross-scale spatial fusion module for encoding the coded feature, the second semantic feature and the third semantic feature. The first spatial feature, the second spatial feature and the third spatial feature are obtained by row fusion; the 3D backbone module is used to extract the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature with different scales from all image frames; the splicing module is used to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature to obtain the spatiotemporal feature encoding, the first semantic feature, the second semantic feature and the third semantic feature are respectively different by n times the spatial sampling rate, and the scale of the first semantic feature is the smallest. The shared detection head decoder includes: a screening module for performing an Object screening operation on the spatiotemporal feature encoding to generate a content query and a location query; a decoding module for decoding according to the content query and the location query to obtain the prediction results of the position and category of the action in each image frame as the spatiotemporal behavior detection result.

[0007] The spatiotemporal behavior detection method provided by the present invention may also have the following characteristics: wherein, in the adaptive cross-scale spatial fusion module, the first spatial feature, the second spatial feature and the third spatial feature are fused according to the coding feature, the second semantic feature and the third semantic feature, and the calculation expression of the first spatial feature, the second spatial feature and the third spatial feature is obtained as follows: S′5=σ SiLU (BN(Conv 1×1 (F5))), S′ 5up =UpSample n ×(S′5), S 4,5up =Fusion(S′ 5up , S4), S′4=σ SiLU (BN(Conv 1×1 (S 4，5up ))), S′ 4up =UpSample(S′4), S 3,4up =Fusion(S′ 4up , S3), S′3=σ SiLU (BN(Conv1×1 (S 3，4up ))),S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))),i=3,4,5,S 54 =ASFF1(DownSample n× (S″4),S″5),S 45 =ASFF2(UpSample n× (S″5),S″4),S′ 54 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 54 ))))),S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))),S 534 =ASFF3(S′ 54 ,DownSample n× (S′ 45 ),DownSample 2n× (S″3)),S 435 =ASFF4(UpSample n× (S′ 54 ),S′ 45 ,DownSample n× (S″3)),S 345 =ASFF5(UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″3),P5=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 534 ))))),P4=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))),P3=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))), ASFF d (a1,a2,…,a m )=Conv 3×3(α1a1+α2a2+…+α m a m ), m≥2, d=1, 2, 3, 4, 5, α i =Softmax(Conv 1×1 (σ ReLU (BN(conv 1×1 (a i ))))), 1≤i≤m, where F5 is the encoding feature, Conv 1×1 is processed by 1×1 convolution layer, BN is processed by BatchNorm layer, σ SiLU To activate the operation via SiLU, UpSample n× is n times upsampling, S4 is the second semantic feature, σ ReLU For the ReLU activation operation, DownSample n× For n-fold downsampling, Conv 3×3 To process through 3×3 convolution layer, UpSample 2n× 2n times upsampling, DownSample 2n× is 2n times downsampling, P5 is the first spatial feature, P4 is the second spatial feature, P3 is the third spatial feature, Concat is the connection operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N It is N-layer RepBlock processing, and Softmax is Softmax processing.

[0008] The spatiotemporal behavior detection method provided by the present invention may further have the following feature: wherein, in the splicing module, the calculation expression for the spatiotemporal feature coding is obtained according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature, and the third spatial feature:

[0009] F i1 =Conv 1×1 (Conv 3×3 (Concat[P i , V i ])), i=3,4,5,

[0010] F i2 =Reshape(F i1 ), i=3,4,5,

[0011]

[0012] F i =Reshape(F i3 ), i=3,4,5,

[0013] F st =Concat(F3, F4, F5),

[0014] Where P5, P4 and P3 are the first spatial feature, the second spatial feature and the third spatial feature respectively, V5, V4 and V3 are the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature respectively, and F st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After processing by 1×1 convolution layer, Conv 3×3 The network is processed by a 3×3 convolution layer, Reshape is a dimension deformation operation (compressing the HW dimension into one dimension and restoring one dimension to the HW dimension are both called reshaping), and Softmax is a Softmax process.

[0015] In the spatiotemporal behavior detection method provided by the present invention, it can also have the following characteristics: wherein, the decoding module includes multiple decoder layers and shared sub-modules connected in sequence, and the decoder layer includes: a multi-head self-attention unit, which is used to perform multi-head self-attention processing on the input first data to obtain second data; a cross-attention unit, which is used to perform cross-attention processing on the first data, the spatiotemporal feature encoding, and the second data to obtain third data; an FFN unit, including a feedforward neural network, which is used to generate output features based on the second data and the third data, and the shared sub-module includes a shared regression head and a shared classification head, which are used to generate the corresponding position coordinates of the predicted behavior object and the confidence scores of each category of the predicted behavior object according to the output features of each decoder layer. The first data of the first decoder layer is content query and position query, and the first data of the second to last decoder layers are the position coordinates and output features corresponding to the previous decoder layer. The spatiotemporal behavior detection result is obtained based on the position coordinates of the predicted behavior object corresponding to the last decoder layer and the confidence scores of each category of the predicted behavior object.

[0016] The spatiotemporal behavior detection method provided by the present invention may also have the following features: wherein, during the training process of the adaptive cross-scale feature fusion codec network, contrast denoising noise is generated by GroundTruth, and combined with a multi-constraint loss function Perform parameter optimization and multi-constraint loss function The calculation expression is: GT i ={c i ,b i}, Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the i-th GroundTruth object, c i GroundTruth object GT i Category in b i GroundTruth object GT i The bounding box in For the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the results The predicted classification in For the results Regression box prediction in ,λ VFL ,λ L1 ,λ GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT i The corresponding one-to-many prediction result set of the lth decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i For the i-th GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

[0017] Functions and effects of the invention

[0018] The spatiotemporal behavior detection method of the present invention uses an adaptive cross-scale spatial fusion module to extract high-quality spatial information from each scale. The spatiotemporal information extracted by the 3D backbone module is then fused by the splicing module and provided to subsequent modules for learning and understanding, thereby obtaining accurate spatiotemporal behavior detection results. Therefore, the spatiotemporal behavior detection method of the present invention can generate accurate spatiotemporal behavior detection results for videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a flow chart of a spatiotemporal behavior detection method according to an embodiment of the present invention;

[0020] Figure 2 is a block diagram of an adaptive cross-scale feature fusion codec network in an embodiment of the present invention;

[0021] Figure 3 is a block diagram of an adaptive cross-scale feature fusion spatiotemporal encoder according to an embodiment of the present invention;

[0022] Figure 4 is a block diagram of a decoding module in an embodiment of the present invention;

[0023] Figure 5 is a block diagram of a decoding layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, objectives and effects of the present invention easier to understand, the following embodiments and accompanying drawings specifically illustrate the spatiotemporal behavior detection method of the present invention.

[0025] This embodiment provides a spatiotemporal behavior detection device for obtaining spatiotemporal behavior detection results of a video.

[0026] Figure 1 4 is a flow chart of a spatiotemporal behavior detection method in an embodiment of the present invention.

[0027] like Figure 1 As shown, the spatiotemporal behavior detection method is used to obtain the spatiotemporal behavior detection results of the video, including the following steps:

[0028] Step S1: Slice the video to obtain multiple continuous image frames.

[0029] In step S2, all image frames are input into the adaptive cross-scale feature fusion codec network to obtain the spatiotemporal behavior detection results.

[0030] Figure 2 is a block diagram of an adaptive cross-scale feature fusion codec network in an embodiment of the present invention.

[0031] like Figure 2 As shown, the adaptive cross-scale feature fusion codec network 200 includes an adaptive cross-scale feature fusion spatiotemporal encoder 21 and a shared detection head decoder 22.

[0032] Figure 3 4 is a block diagram of an adaptive cross-scale feature fusion spatiotemporal encoder in an embodiment of the present invention.

[0033] like Figure 3 As shown, the adaptive cross-scale feature fusion spatiotemporal encoder 21 includes a 2D backbone module 211, an AIFI module 212, an adaptive cross-scale spatial fusion module 213, a 3D backbone module 214 and a splicing module 215.

[0034] The 2D backbone module 211 is configured to sample the last image frame to generate a plurality of semantic features having different scales, wherein the plurality of semantic features include a first semantic feature, a second semantic feature, and a third semantic feature. The first semantic feature, the second semantic feature, and the third semantic feature are sequentially spaced at n times the spatial sampling rate, and the first semantic feature has the smallest scale.

[0035] In this embodiment, the 2D backbone module 211 is the 2D backbone of the existing ResNet, which performs multi-scale semantic feature extraction on the last image frame, and the semantic features of each adjacent level scale differ by n=2 times the spatial sampling rate, obtaining semantic features S1, S2, S3, S4 and S5, and using semantic feature S5 as the first semantic feature, semantic feature S4 as the second semantic feature, and semantic feature S3 as the third semantic feature. In other embodiments, the 2D backbone module 211 can generate at least three semantic features according to actual needs, and select the semantic feature with richer information as the first semantic feature, and then select the second semantic feature and the third semantic feature based on the first semantic feature. In addition, in other embodiments, the spatial sampling rate can be adjusted according to actual needs, for example, the spatial sampling rate of adjacent semantic features can be made to differ by n=4 times.

[0036] The AIFI module 212 is used to encode the first semantic feature to obtain an encoded feature. In this embodiment, the AIFI module 212 is an existing Attention-based Intrascale Feature Interaction module.

[0037] The adaptive cross-scale spatial fusion module 213 is used to fuse the coding features, the second semantic features and the third semantic features to obtain the first spatial features, the second spatial features and the third spatial features.

[0038] The adaptive cross-scale spatial fusion module 213 fuses the coding feature, the second semantic feature, and the third semantic feature to obtain the calculation expressions of the first spatial feature, the second spatial feature, and the third spatial feature:

[0039] S′5=σ SiLU (BN(Conv 1×1 (F5))),

[0040] S′ 5up =UpSample n× (S′5),

[0041] S 4,5up =Fusion(S′ 5up , S4),

[0042] S′4=σ SiLU (BN(Conv 1×1 (S 4，5up ))),

[0043] S′ 4up =UpSample(S′4),

[0044] S 3,4up =Fusion(S′ 4up , S3),

[0045] S′3=σ SiLU (BN(Conv 1×1 (S 3，4up ))),

[0046] S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))), i=3,4,5,

[0047] S 54 =ASFF1(DownSample n× (S″4), S″5),

[0048] S 45 =ASFF2(UpSample n× (S″5), S″4),

[0049] S′ 54 =BN(Conv 3×3(ReLU(BN(Conv 3×3 (S 54 ))))),

[0050] S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))),

[0051] S 534 =ASFF3(S′ 54 ,DownSample n× (S′ 45 ),DownSample 2n× (S″3)),

[0052] S 435 =ASFF4(UpSample n× (S′ 54 ),S′ 45 ,DownSample n× (S″3)),

[0053] S 345 =ASFF5(UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″3),

[0054] P5=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 534 ))))),

[0055] P4=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))),

[0056] P3=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))),

[0057]

[0058] ASFF d (a1,a2,…,a m )=Conv 3×3 (α1a1+α2a2+…+α m tom ), m≥2, d=1, 2,3,4,5,

[0059] α i =softmax(Conv 1×1 (σ ReLU (BN(Conv 1×1 (a i ))))),1≤i≤m,

[0060] Where F5 is the encoding feature, Conv 1×1 is processed by 1×1 convolution layer, BN is processed by BatchNorm layer, σ SiLU To activate the operation via SiLU, UPSample n× is n times upsampling, S4 is the second semantic feature, σ ReLU For ReLU activation operation, DOWNSample n× For n-fold downsampling, Conv 3×3 To process through 3×3 convolution layer, UPSample 2n× 2n times upsampling, DOWNSample 2n× is 2n times downsampling, P5 is the first spatial feature, P4 is the second spatial feature, P3 is the third spatial feature, Concat is the connection operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N The ASFFs of the trained adaptive cross-scale feature fusion codec network 200 in this embodiment are d Operations are performed separately by calculation modules with different weights.

[0061] In this embodiment, the adaptive cross-scale spatial fusion module 213 is a cross-scale fusion network that inserts a fusion block composed of a convolutional layer network and multiple adaptive structural feature fusion blocks, namely ASFF operations, into the fusion path of multiple scale features. Among the multiple scale features input to the adaptive cross-scale spatial fusion module 213, only the highest semantic feature is obtained through the transformer attention operation, and the other scale features do not require attention calculation, so the computational burden can be greatly reduced. In addition, the attention information of the highest semantic feature is fused into the feature information at other scales through the adaptive structural feature fusion block, so that other scales can also understand the information weight distribution on the entire spatial feature, thereby extracting high-quality information at different scales.

[0062] The adaptive cross-scale spatial fusion module 213 also utilizes an asymptotic architecture to gradually propagate the weight information of the S′5 feature at its own scale to the features at all other scales. Simultaneously, the model adaptively determines the weight assigned to each scale during fusion, ensuring that each scale retains its own characteristics while also incorporating attention features from S′5 and features from neighboring scales containing attention information, thus preventing information loss due to degradation.

[0063] The 3D backbone module 214 is used to extract the first spatiotemporal features, the second spatiotemporal features and the third spatiotemporal features of different scales from all image frames.

[0064] The splicing module 215 is used to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature to obtain a spatiotemporal feature code.

[0065] In the splicing module 215, the calculation expression of the spatiotemporal feature code is obtained according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature and the third spatial feature:

[0066] F i1 =Conv 1×1 (Conv 3×3 (Concat[P i , V i ])), i=3,4,5,

[0067] F i2 =Reshape(F i1 ), i=3,4,5,

[0068]

[0069] F i =Reshape(F i3 ), i=3,4,5,

[0070] F st =Concat(F3, F4, F5),

[0071] Where P5, P4 and P3 are the first spatial feature, the second spatial feature and the third spatial feature respectively, V5, V4 and V3 are the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature respectively, and F st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After processing by 1×1 convolution layer, Conv 3×3The network is processed through a 3×3 convolution layer. Reshape is a dimension deformation operation, which includes compressing the HW dimension into one dimension and restoring the compressed dimension to the HW dimension. Softmax is a Softmax process.

[0072] The shared detection head decoder 22 includes a screening module 221 and a decoding module 222 .

[0073] Filtering module 221 is used to perform object filtering on the spatiotemporal feature encodings to generate content and location queries. In this embodiment, filtering module 221 performs object filtering on the spatiotemporal feature encodings, selecting the top K feature blocks with the highest confidence. The resulting objects are more suitable for content and location queries. This filtering process eliminates insignificant query object information, reducing the computational burden on the network.

[0074] The decoding module 222 is used to perform decoding according to the content query and the position query, and obtain the prediction results of the position and category of the action in each image frame as the spatiotemporal behavior detection result.

[0075] Figure 4 is a block diagram of a decoding module in an embodiment of the present invention.

[0076] like Figure 4 As shown, the decoding module 222 includes a plurality of decoding layers 2221 and a sharing submodule 2222 connected in sequence. In this embodiment, the number of the decoding layers 2221 is 6.

[0077] Figure 5 is a block diagram of a decoding layer in an embodiment of the present invention.

[0078] like Figure 5 As shown, the decoding layer 2221 includes a multi-head self-attention unit 22211, a cross-attention unit 22212 and a FFN unit 22213.

[0079] The multi-head self-attention unit 22211 is used to perform multi-head self-attention processing on the input first data to obtain second data. In this embodiment, the multi-head self-attention processing calculates the relationship between the content query embedded in the location information and itself. Then, based on this attention information, the most likely candidate objects are selected, thereby inferring the object location information of the encoded location. Furthermore, the multi-head self-attention processing uses the self-attention method commonly used in Transformers.

[0080] Cross-attention unit 22212 is configured to perform cross-attention processing on the first data, the spatiotemporal feature encoding, and the second data to obtain third data. In this embodiment, cross-attention processing further obtains or optimizes candidate objects based on the output of the location query and multi-self-attention, i.e., the second data. Furthermore, cross-attention processing utilizes a deformable attention mechanism that only requires interactive calculations with a subset of pixels obtained based on sampling.

[0081] The FFN unit 22213 includes a feedforward neural network for generating output features based on the second data and the third data.

[0082] The shared submodule 2222 includes a shared regression head and a shared classification head. The shared regression head generates the position coordinates of the predicted behavior object corresponding to the decoder layer 2221 based on the output features of the decoder layer 2221. The shared classification head generates the confidence scores of each category of the predicted behavior object corresponding to the decoder layer 2221 based on the output features of the decoder layer 2221.

[0083] The first data of the first decoder layer 2221 is the content query and location query. The first data of the second to last decoder layers 2221 are the location coordinates and output features corresponding to the previous decoder layer. The spatiotemporal behavior detection result is obtained based on the location coordinates of the predicted behavior object corresponding to the last decoder layer 2221 and the confidence scores of each category of the predicted behavior object.

[0084] Since the existing technology only uses the 1-to-1 loss function of the bipartite graph matching method, the Encoder-Decoder learning process of spatiotemporal features has problems such as insufficient understanding and insufficient constraints. To this end, this embodiment combines Contrastive Denoising (CDN) method, one-to-one supervision method and one-to-many supervision method to construct a multi-constraint loss function. Thereby, the parameters of the adaptive cross-scale feature fusion codec network 200 are optimized.

[0085] The CDN method generates contrastive denoising noise based on GroundTruth. This contrastive denoising noise includes a noisy content query and a noisy location query. During training, the noisy content query and the content query are concatenated as the first data of the first decoder layer 2221, and the noisy location query and the location query are concatenated as the second data of the first decoder layer 2221.

[0086] The multi-constraint loss function The calculation expression is:

[0087]

[0088]

[0089] GT i ={c i , b i},

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the i-th GroundTruth object, c i GroundTruth object GT i Category in b i GroundTruth object GT i The bounding box in For the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the results The predicted classification in For the results Regression box prediction in ,λ VFL ,λ L1 ,λ GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT i The corresponding one-to-many prediction result set of the lth decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i For the i-th GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder 22 and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

[0096] In this example, the performance of the adaptive cross-scale feature fusion codec network 200 (Our Method) is compared with existing state-of-the-art algorithms in the field, namely T-CNN, ACT, MPS, SAMOC, STEP, YOWO, and TubeR, using the existing UCF101-24 dataset. The specific results are shown in the following table:

[0097] algorithm Frame-mAP param FLOPS Video-mAP T-CNN 41.4％ - - 51.3％ ACT 69.5％ - - - MPS - - - 82.4％ SAMOC 79.3％ - - 80.5％ STEP 75.0％ - - 83.1％ TubeR 81.3％ 120M 132G - YOWO 80.4％ 121.4M 54.7G 82.5％ Our Method 81.52％ 120.54M 57.51G 76.97％

[0098] The first column in the table above lists each algorithm. The second through fifth columns show the Frame-mAP metric, model parameters, and computational FLOPS for each algorithm at an IoU threshold of 0.5, as well as the Video-mAP metric at an IoU threshold of 0.1. Higher Frame-mAP and Video-mAP metrics, and lower model parameters and computational FLOPS, indicate better models. For example, the cell in the third column of the ninth row indicates that the model parameters for Our Method are 120.54M.

[0099] As can be seen, in terms of algorithm accuracy, or the Frame-mAP metric at an IoU threshold of 0.5, Our Method achieves improvements of 1.1% and 0.2%, respectively, compared to the higher-performing YOWO and TubeR. This is because the adaptive cross-scale feature fusion encoder-decoder network 200 uses a cross-scale adaptive fusion method to progressively propagate attention information from high-level information to features at other scales, allowing each scale to learn which information is more important and should be paid more attention. Compared to other scale fusion methods, this method loses less information and achieves better results.

[0100] In terms of model complexity (param and FLOPS), our method reduces computational overhead by approximately 75 GB compared to the pure 3D Transformer method TubeR, while having 0.9 million fewer model parameters than YOWO. This is primarily due to the adaptive cross-scale feature fusion encoder-decoder network 200's simplified encoder-decoder architecture, where the encoder only performs attention calculations on depth semantics, while the decoder's detection head is designed to be shared.

[0101] In this embodiment, an ablation experiment is also conducted on the adaptive cross-scale spatial fusion module 213. The adaptive cross-scale feature fusion codec network 200 without the adaptive cross-scale spatial fusion module 213 is recorded as CCFM, while the original adaptive cross-scale feature fusion codec network 200 is recorded as ACCFM. The specific comparative performance is shown in the following table:

[0102]

[0103] The first column in the table above shows the various methods, the second column shows the f-mAP index corresponding to each method, and the third to fifth columns show the v-mAP index under IoU of 0.1, 0.2, and 0.5, respectively. The higher the above index, the better the model. It can be seen that in terms of the f-mAP index, ACCFM effectively improves the detection accuracy by 0.5% compared to CCFM. In addition, in terms of the v-mAP index with IoU of 0.1 and 0.2, ACCFM also improves by 0.6% and 0.8% respectively compared to CCFM. Therefore, the adaptive cross-scale feature fusion codec network 200 can improve the accuracy of spatiotemporal action behavior detection.

[0104] In this embodiment, the multi-constraint loss function Conduct ablation experiments to Loss and The loss is tested and the results are as follows:

[0105]

[0106] The first column in the table above shows the various ablation settings, and the second to fourth columns show the F-mAP index, classification accuracy, and recall rate under each setting. For example, the cell in the second column of the fifth row shows that the F-mAP of OurMethod is 81.52%. This shows that when the multi-constraint loss function cannot be fully used, When , the performance of the model showed different degrees of decline, so the model adopted a multi-constraint loss function Learning and understanding on the training set can effectively improve the detection effect.

[0107] Functions and Effects of the Embodiments

[0108] The spatiotemporal behavior detection method of this embodiment uses an adaptive cross-scale spatial fusion module to extract high-quality spatial information from each scale. The spatiotemporal information extracted by the 3D backbone module is then fused by the stitching module and provided to subsequent modules for learning and understanding, resulting in accurate spatiotemporal behavior detection results. In short, this method can generate accurate spatiotemporal behavior detection results for videos.

[0109] Furthermore, by jointly constraining the training of the entire model through the multi-constraint loss function, not only the screening results of the adaptive cross-scale feature fusion spatiotemporal encoder are optimized, but also the understanding and learning of the screening results by the decoding module are improved.

[0110] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A spatiotemporal behavior detection method for obtaining a spatiotemporal behavior detection result of a video, characterized in that: The following steps are involved: Step S1, slicing the video to obtain a plurality of continuous image frames; Step S2: input all the image frames into an adaptive cross-scale feature fusion codec network to obtain the spatiotemporal behavior detection result. The adaptive cross-scale feature fusion codec network includes an adaptive cross-scale feature fusion spatiotemporal encoder and a shared detection head decoder. The adaptive cross-scale feature fusion spatiotemporal encoder includes: a 2D backbone module, configured to sample the last image frame to generate a plurality of semantic features having different scales, wherein all the semantic features include a first semantic feature, a second semantic feature, and a third semantic feature; An AIFI module is used to encode the first semantic feature to obtain an encoded feature; an adaptive cross-scale spatial fusion module, configured to fuse the coding feature, the second semantic feature, and the third semantic feature to obtain a first spatial feature, a second spatial feature, and a third spatial feature; A 3D backbone module, configured to extract first spatiotemporal features, second spatiotemporal features, and third spatiotemporal features having different scales from all the image frames; a splicing module, configured to splice and fuse the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature, and the third spatial feature to obtain a spatiotemporal feature code, The first semantic feature, the second semantic feature, and the third semantic feature are sequentially different in spatial sampling rate by n times, and the scale of the first semantic feature is the smallest. The shared detection head decoder comprises: A screening module, configured to perform an object screening operation on the spatiotemporal feature code to generate a content query and a location query; A decoding module is configured to perform decoding according to the content query and the position query to obtain prediction results of the position and category of the action in each of the image frames as the spatiotemporal behavior detection results.

2. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, In the adaptive cross-scale spatial fusion module, the encoding feature, the second semantic feature, and the third semantic feature are fused to obtain the calculation expression of the first spatial feature, the second spatial feature, and the third spatial feature: S′5=σ SiLU (BN(Conv 1×1 (F5))), S′ 5up =UpSample n× (S′5), S 4,5up =Fusion(S′ 5up ,S4), S′4=σ SiLU (BN(Conv 1×1 (S 4,5up ))), S′ 4up =UpSample(S′4), S 3,4up =Fusion(S′ 4up ,S3), S′3=σ SiLU (BN(Conv 1×1 (S 3,4up ))), S″ i =σ ReLU (BN(Conv 1×1 (S′ i ))),i=3,4,5, S 54 =ASFF1(DownSample n× (S″4),S″5), S 45 =ASFF2(UpSample n× (S″5),S″4), S′ 54 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 54 ))))), S′ 45 =BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 45 ))))), S 534 =ASFF3(S′ 54 ,DownSample n× (S′ 45 ),DownSample 2n× (S″3)), S 435 =ASFF4(UpSample n× (S′ 54 ),S′ 45 ,DownSamole n× (S″3)), S 345 =ASFF5(UpSample 2n× (S′ 54 ),UpSample 2n× (S′ 45 ),S″3), P5=BN(Conv. 3×3 (ReLU(BN(Conv 3×3 (S 534 ))))), P4=BN(Conv 3×3 (ReLU(BN(Conv 3×3 (S 435 ))))), P3=BN(Conv. 3×3 (ReLU(BN(Conv 3×3 (S 345 ))))), ASFF d (a1,a2,...,a m )=Conv 3×3 (α1a1+α2a2+…+α m to m ),m≥2,d=1,2,3,4,5, a i =Softmax(Conv 1×1 (s ReLU (BN(Conv 1×1 (a i ))))),1≤i≤m, Where F5 is the encoding feature, Conv 1×1 is processed by 1×1 convolution layer, BN is processed by BatchNorm layer, σ SiLU To activate the operation via SiLU, UpSample n× is n times upsampling, S4 is the second semantic feature, σ ReLU For the ReLU activation operation, DownSamplen n× For n-fold downsampling, Conv 3×3 To process through 3×3 convolution layer, UPSample 2n× 2n times upsampling, DownSample 2n× is 2n times downsampling, P5 is the first spatial feature, P4 is the second spatial feature, P3 is the third spatial feature, Concat is a concatenation operation, To add the eigenvalues ​​of the corresponding positions of two eigenvectors, RepBlock ×N It is N-layer RepBlock processing, and Softmax is Softmax processing.

3. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, In the splicing module, a calculation expression for obtaining a spatiotemporal feature code according to the first spatiotemporal feature, the second spatiotemporal feature, the third spatiotemporal feature, the first spatial feature, the second spatial feature, and the third spatial feature is: F i1 =Conv 1×1 (Conv 3×3 (Concat[P i ,V i ])),i=3,4,5, F i2 =Reshape(F i1 ),i=3,4,5, F i =Reshape(F i3 ),i=3,4,5, F st =Concat(F3,F4,F5), Where P5, P4 and P3 are the first spatial feature, the second spatial feature and the third spatial feature respectively, V5, V4 and V3 are the first spatiotemporal feature, the second spatiotemporal feature and the third spatiotemporal feature respectively, and F st is the spatiotemporal feature encoding, Concat is the connection operation, Conv 1×1 After processing by 1×1 convolution layer, Conv 3×3 It is processed by a 3×3 convolution layer, Reshape is a dimension deformation operation, and Softmax is a Softmax processing.

4. The spatiotemporal behavior detection method according to claim 1, Its characteristics are: The decoding module includes a plurality of decoder layers and shared submodules connected in sequence. The decoder layer includes: A multi-head self-attention unit, configured to perform multi-head self-attention processing on the input first data to obtain second data; a cross attention unit, configured to perform cross attention processing on the first data, the spatiotemporal feature code, and the second data to obtain third data; An FFN unit, comprising a feedforward neural network, configured to generate output features according to the second data and the third data, The shared submodule includes a shared regression head and a shared classification head, which are used to generate the corresponding position coordinates of the predicted behavior object and the confidence scores of each category of the predicted behavior object according to the output features of each decoder layer. The first data of the first decoder layer is the content query and the location query, The first data of the second to last decoder layers are the position coordinates and the output features corresponding to the previous decoder layer, The spatiotemporal behavior detection result is obtained according to the position coordinates of the predicted behavior object corresponding to the last decoder layer and the confidence scores of each category of the predicted behavior object.

5. The spatiotemporal behavior detection method according to claim 1, characterized in that: in, During the training process of the adaptive cross-scale feature fusion codec network, contrast denoising noise is generated by GroundTruth and combined with a multi-constraint loss function Optimize parameters, The multi-constraint loss function The calculation expression is: GT i ={c i ,b i }, Where k is the total number of decoder layers, n is the number of GroundTruth objects, and GT i is the i-th GroundTruth object, c i GroundTruth object GT i Category in b i GroundTruth object GT i The bounding box in For the lth decoder layer and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, For the results The predicted classification in For the results Regression box prediction in ,λ VFL ,λ L1 ,λ GIoU are the weights of the corresponding loss terms, is the VFL loss function, is the GIoU loss function, For GroundTruth objects GT i The corresponding one-to-many prediction result set of the lth decoder layer, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object category results, For the prediction result set The j-th prediction and the i-th GroundTruth object GT i The corresponding predicted object regression result, K i For the i-th GroundTruth object GT i The number of predictions in the corresponding prediction result set, is the classification loss, is the regression box loss, is the auxiliary loss function, and its calculation formula is the same as same, The prediction result output by the shared detection head decoder and the i-th GroundTruth object GT i The corresponding results after one-to-one matching, is the i-th GroundTruth object GT i The corresponding set of prediction results of j denosie, is the jth predicted category, is the jth regression box prediction, and d is the preset number of denosings.

Images