Video target detection method based on motion modulation attention spatiotemporal feature pyramid

Abstract

The application discloses a video target detection method based on motion modulation attention space-time feature pyramid, comprising the following steps: 1) CNN feature extraction; 2) motion feature extraction; 3) position coding; 4) feature vector conversion; 5) motion information enhancement; 6) hierarchical space attention calculation; 7) time attention calculation; 8) prediction target detection result. The video target detection method can effectively utilize the space-time features of the video sequence, enhance the dynamic scene adaptability, improve the multi-scale target detection performance, realize end-to-end detection, and meet the demand of high-performance target detection in the field of automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video target detection, and particularly relates to a video target detection method based on motion modulation attention spatiotemporal feature pyramid. BACKGROUND

[0002] With the rapid development of artificial intelligence and computer vision technology, target detection technology plays an increasingly important role in many fields. Especially in the field of autonomous driving, accurate and robust target detection is a core component of the environment perception system, directly related to the safety and reliability of autonomous vehicles. However, existing target detection technologies still face many challenges in processing video sequences, especially in dynamic scenes:

[0003] Insufficient use of spatiotemporal features: Traditional image target detection methods mainly focus on the spatial features of single-frame images, ignoring the temporal dimension information contained in video sequences, and cannot effectively capture the motion patterns and temporal dependencies of targets, resulting in decreased detection performance in cases of motion blur, occlusion, etc.

[0004] Poor adaptability to dynamic scenes: In the autonomous driving scenario, both the target and the scene itself can be in motion, and traditional static image target detection methods are difficult to effectively adapt to such dynamic changes, and are prone to missed detection, false detection, etc.

[0005] Multi-scale target detection challenge: In the autonomous driving scenario, the scale of the target varies widely, with near and far targets coexisting, and traditional single-scale or shallow multi-scale feature fusion methods are difficult to effectively handle such scale changes, resulting in poor small target detection performance.

[0006] Therefore, there is an urgent need for a video target detection method that can effectively utilize spatiotemporal features of video sequences, enhance adaptability to dynamic scenes, improve multi-scale target detection performance, and realize end-to-end detection, to meet the demand for high-performance target detection in the field of autonomous driving and other fields. SUMMARY

[0007] In view of the above-mentioned defects or shortcomings in the prior art, it is desirable to provide a video target detection method based on motion modulation attention spatiotemporal feature pyramid, which can effectively utilize spatiotemporal features of video sequences, enhance adaptability to dynamic scenes, improve multi-scale target detection performance, and realize end-to-end detection, to meet the demand for high-performance target detection in the field of autonomous driving and other fields.

[0008] The present application provides a video target detection method based on motion modulation attention spatiotemporal feature pyramid, comprising the following steps:

[0009] 1) CNN feature extraction to obtain three scale feature maps;

[0010] 2) motion feature extraction, generate three scale motion residual maps;

[0011] 3) position encoding, add position encoding to the three scale feature maps and motion residual maps to explicitly encode spatial position information;

[0012] 4) feature vector conversion, the three scale feature maps after adding position encoding and the three scale motion residual maps after adding position encoding are respectively linearly transformed by linear layers to convert them into vector form;

[0013] 5) motion information enhancement, the three scale linearly transformed feature vectors of each frame of video sequence image are respectively element-wise multiplied with the corresponding scale linearly transformed motion residual vectors to enhance the motion information of the feature vectors;

[0014] 6) perform hierarchical spatial attention calculation to obtain N sequence hierarchical spatial attention feature vectors;

[0015] The hierarchical spatial attention calculation includes the following steps:

[0016] 61) scale correspondence;

[0017] Q1: 1 / 16 scale modulated feature vector;

[0018] K1, V1: 1 / 8 scale modulated feature vector;

[0019] Q2: 1 / 32 scale modulated feature vector;

[0020] 62) first cross-attention calculation: taking Q1 as Query, K1 and V1 as Key and Value, performing cross-attention calculation to obtain intermediate results K2 and V2;

[0021] 63) second cross-attention calculation: taking Q2 as Query, the results K2 and V2 of the first cross-attention as Key and Value, performing cross-attention calculation to obtain N sequence hierarchical spatial attention feature vectors;

[0022] 7) perform time attention calculation to obtain motion modulation attention spatio-temporal feature pyramid fusion feature vector;

[0023] The time attention calculation includes the following steps:

[0024] 71) arrange the N sequence hierarchical spatial attention feature vectors in time sequence to form a time sequence; let the current time be t, then the N sequence hierarchical spatial attention feature vectors arranged in time sequence from bottom to top represent the feature vectors of time t, t-1, t-2,..., t-N+1 in turn;

[0025] 72) First cross-attention calculation: take the feature vector at time t-N+2 as Query, the feature vector at time t-N+1 as Key and Value, and perform cross-attention calculation to obtain intermediate results K2, V2;

[0026] 73) Second cross-attention calculation: take the feature vector at time t-N+3 as Query, and the results K2, V2 of the first cross-attention calculation as Key and Value, and perform cross-attention calculation;

[0027] 74) Repeat the cross-attention calculation until the feature vector at time t is taken as the last Query, and the cross-attention calculation is performed with the results of the last cross-attention calculation to obtain the motion modulation attention spatio-temporal feature pyramid fusion feature vector;

[0028] 8) Decode the motion modulation attention spatio-temporal feature pyramid fusion feature vector, and predict the target detection result through the feedforward neural network.

[0029] Further, in step 1), N frames of video sequence images are obtained at one time, and each frame of video sequence image is subjected to feature extraction through a CNN backbone network to obtain feature maps of three scales of 1 / 8, 1 / 16 and 1 / 32 respectively.

[0030] Further, in step 2), on each scale, the element-by-element difference between the feature map of the last frame of video sequence image and the feature map of the first frame of video sequence image is calculated to generate motion residual maps of three scales.

[0031] Further, the overall loss function of the motion modulation attention spatio-temporal feature pyramid-based network includes a classification loss, a bounding box loss and a motion residual guided loss;

[0032] The classification loss L cls is:

[0033] ;

[0034] The bounding box loss L is the probability that the predicted bounding box matches the real bounding box of the same class; ;

[0035] ;

[0036] If the class prediction is correct, then ; if the class prediction is incorrect, then ;

[0037] α is a weight coefficient for balancing positive and negative samples; is a focusing parameter;

[0038] boundary box loss L bbox is:

[0039] ;

[0040] wherein, is a boundary box of the predicted box ; is a boundary box of the real box ;

[0041] is an L1 loss; is a GIoU loss;

[0042] is a weight coefficient for balancing the L1 loss and the GIoU loss;

[0043] motion residual guiding loss L MR-guide is:

[0044] ;

[0045] wherein, ω i is a weight coefficient of the i-th scale motion residual guiding loss;

[0046] is a normalized mean square error function;

[0047] is a mean square error function;

[0048] and is a size of the feature map after down-sampling;

[0049] MRes i is a motion residual map;

[0050] is a response map with holes is down-sampled to the same scale as MRes i using the nearest neighbor interpolation method to obtain

[0051] ;

[0052] wherein, is a set of all integer pixel coordinates on the four boundary line segments of the j1-th real box b j1 ;

[0053] is a two-dimensional Gaussian distribution generated with the boundary pixel as the center;

[0054] ;

[0055] wherein, is the coordinate of the boundary pixel; (x2, y2) is the pixel coordinate on the response map; and σ2 is the standard deviation of the Gaussian distribution.

[0056] The overall loss function L MM-STFM is:

[0057] ;

[0058] wherein, λ cls , λ bbox , and λ mr-guide are weight coefficients for balancing different terms.

[0059] Compared with the prior art, the present application has the following beneficial effects:

[0060] The video target detection method of the present application adopts a motion modulation attention mechanism to make the model pay more attention to the motion area, and through capturing the time dependence, the robustness of the model to dynamic scene factors such as motion blur and occlusion is jointly enhanced, and the detection stability is improved. The hierarchical spatial attention and the time attention can more effectively fuse the feature information of different scales and different times, and capture more rich spatio-temporal context relationships.

[0061] The present application effectively fuses the spatio-temporal features and motion information, fully utilizes the global context, optimizes the loss function, and introduces the motion residual guided loss to explicitly guide the network to learn high-quality motion features, which significantly improves the video target detection accuracy, especially in dynamic scenes and complex environments.

[0062] The present application ingeniously fuses the motion information, attention mechanism, Transformer architecture and DETR idea, and forms a complete and high-performance video target detection scheme, which has high academic value and application prospect. It can be widely applied in the fields of automatic driving, intelligent transportation, video monitoring, robot vision, etc., and provides key technical support for improving the intelligent level of related fields.

[0063] It should be understood that the content described in the summary section is not intended to limit the key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0064] Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments made with reference to the accompanying drawings:

[0065] Figure 1A workflow diagram of a video object detection method based on motion modulation attention spatiotemporal feature pyramid is shown in FIG. 1.

[0066] Figure 2 A feature pyramid schematic diagram is shown in FIG. 2.

[0067] Figure 3 A continuous cavity response map is shown in FIG. 3. DETAILED DESCRIPTION

[0068] The application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended for the purpose of interpretation of the application and are not limiting of the application. In addition, it should be noted that only parts related to the application are shown in the drawings for the purpose of description.

[0069] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0070] Reference should be made to Figures 1-3 The embodiments of the application provide a video object detection method based on motion modulation attention spatiotemporal feature pyramid, which comprises the following steps:

[0071] 1) CNN feature extraction;

[0072] N frames of video sequence images are acquired at one time, and each frame of video sequence image is subjected to feature extraction by a CNN backbone network to obtain three scale feature maps respectively; the three scale feature maps correspond to 1 / 8 scale, 1 / 16 scale and 1 / 32 scale of the video sequence image respectively;

[0073] 2) Motion feature extraction;

[0074] At each scale, the element-by-element difference between the feature map of the last frame of video sequence image and the feature map of the first frame of video sequence image is calculated to generate three scale motion residual maps;

[0075] 3) Position encoding;

[0076] The three scale feature maps and the motion residual maps are respectively added with position encoding to explicitly encode the spatial position information;

[0077] 4) Feature vector conversion;

[0078] The three scale feature maps of each frame of video sequence image added with position encoding and the three scale motion residual maps added with position encoding are respectively subjected to linear transformation by a linear layer to be converted into vector form;

[0079] 5) Motion information enhancement;

[0080] The feature vectors after linear transformation of three scales of each frame of video sequence images are respectively multiplied with the motion residual vectors after linear transformation of the corresponding scales in an element-by-element manner to enhance the motion information of the feature vectors;

[0081] 6) hierarchical spatial attention calculation;

[0082] 61) scale correspondence;

[0083] Q1: feature vector after 1 / 16 scale modulation;

[0084] K1, V1: feature vector after 1 / 8 scale modulation;

[0085] Q2: feature vector after 1 / 32 scale modulation;

[0086] 62) first cross-attention calculation: taking Q1 as Query, K1 and V1 as Key and Value, performing cross-attention calculation to obtain intermediate results K2 and V2;

[0087] 63) second cross-attention calculation: taking Q2 as Query, the results K2 and V2 of the first cross-attention calculation as Key and Value, performing cross-attention calculation to obtain hierarchical spatial attention feature vectors of N sequences;

[0088] 7) time attention calculation;

[0089] 71) arranging the hierarchical spatial attention feature vectors of N sequences in time sequence to form a time sequence; assuming that the current time is t, the hierarchical spatial attention feature vectors of N sequences are arranged in time sequence from bottom to top to represent the feature vectors of time t, t-1, t-2,..., t-N+1 in turn;

[0090] 72) first cross-attention calculation: taking the feature vector at time t-N+2 as Query, the feature vector at time t-N+1 as Key and Value, performing cross-attention calculation to obtain intermediate results K2 and V2;

[0091] 73) second cross-attention calculation: taking the feature vector at time t-N+3 as Query, the results K2 and V2 of the first cross-attention calculation as Key and Value, performing cross-attention calculation;

[0092] 74) repeatedly performing cross-attention calculation until the feature vector at time t is taken as the last Query, and the result of the last cross-attention calculation is taken as Key and Value to perform cross-attention calculation, to obtain a motion modulation attention spatio-temporal feature pyramid fusion feature vector;

[0093] 8) decoding the feature vector based on the motion modulation attention spatio-temporal feature pyramid, and predicting the target detection result through a feedforward neural network;

[0094] wherein the overall loss function of the network based on the motion modulation attention spatio-temporal feature pyramid comprises a classification loss, a bounding box loss and a motion residual guidance loss;

[0095] wherein the classification loss L cls is:

[0096] ;

[0097] wherein, is the probability of the class of the predicted bounding box predicted being matched with the real bounding box ;

[0098] ;

[0099] wherein, if the class prediction is correct, then ; if the class prediction is incorrect, then ;

[0100] α is a weight coefficient for balancing positive and negative samples; is a focusing parameter;

[0101] the bounding box loss L bbox is:

[0102] ;

[0103] wherein, is the bounding box of the predicted bounding box ; is the bounding box of the real bounding box ;

[0104] is the L1 loss; is the GIoU loss;

[0105] is a weight coefficient for balancing the L1 loss and the GIoU loss;

[0106] the motion residual guidance loss L MR-guide is:

[0107] ;

[0108] wherein, ω i is a weight coefficient of the i-th scale motion residual guidance loss;

[0109] ​is the normalized mean square error function;

[0110] is the mean square error function;

[0111] and is the size of the down-sampled feature map;

[0112] MRes i is the motion residual map;

[0113] is the response map with holes is down-sampled to the same scale as MResi using the nearest-neighbor interpolation method;

[0114] In practical applications, the motion residual map MRes i and the target response map are both normalized to the range [0, 1] before calculating the MSE loss, to simplify the calculation process and achieve similar results;

[0115] ;

[0116] where, is the set of all integer pixel coordinates on the four boundary line segments of the j1th real box b j1 ;

[0117] is the two-dimensional Gaussian distribution generated with the boundary pixel as the center;

[0118] ;

[0119] where, is the coordinate of the boundary pixel; (x2, y2) is the pixel coordinate on the response map; σ2 is the standard deviation of the Gaussian distribution; here, an isotropic Gaussian distribution is used, and σ2 2 is used instead of σ x2 σ y2 , and σ2 2 is used instead of σ x2 2 , σ y2 2 to simplify the formula expression; after generating the response map, it is normalized to scale its value to the range [0, 1] to facilitate loss calculation with the motion residual map;

[0120] The overall loss function L MM-STFM is:

[0121] ;

[0122] where, λcls , λ bbox and λ mr-guide is a weight coefficient for balancing different terms.

[0123] In this embodiment, a convolutional neural network (CNN) is used to extract features of consecutive frames, and a feature map of the last frame of the input sequence is subtracted from a feature map of the first frame to obtain motion residual information and generate a motion residual map. Compared with the optical flow method, this method has smaller calculation amount and is more suitable for automatic driving scenarios with high real-time requirements. The motion residual map can provide motion information of targets in the scene, assist the spatio-temporal feature fusion module to focus on the motion area, and improve the detection capability of dynamic targets.

[0124] As shown in FIG. 1, wherein the “Positional Encoding” block and the “+” symbol represent the addition operation of the position encoding; the three small rectangles in the figure represent the results Embedding after the linear layer; Figure 1

[0125] The “xN” above the Level-wise Spatial Attention in FIG. 1 indicates that N frames of images are processed in parallel, instead of repeating the operation N times for each frame. After each frame of image passes through the Level-wise Spatial Attention module, a sequence of hierarchical spatial attention feature vectors is obtained. Therefore, for N frames of input, N sequence of hierarchical spatial attention feature vectors are obtained (corresponding to the 6 horizontal small rectangles in the “Temporal Attention” block in the figure, and N is taken as 6 for visualization of the drawing). Figure 1 The “Temporal Attention” module in the figure from bottom to top is the feature vector at time t, t-1,..., t-N+1.

[0126] The calculation of the Level-wise Spatial Attention is independent for each frame of image, so when the sequence is N, it needs to be done N times, which can be done in parallel. The innovation of the Level-wise Spatial Attention module lies in enhancing the spatial feature representation in dynamic scenes through multi-scale cross-attention mechanism and motion modulation.

[0127] The traditional FPN only fuses multi-scale features through top-down or horizontal connection, while the present application uses level cross-attention (such as 1 / 8 and 1 / 16 scale feature interaction) to enhance the semantic features of low-resolution levels with the detailed information of high-resolution levels;

[0128] ​The existing spatial attention (such as a non-local network) does not combine motion information, and the motion residual map is multiplied with the feature vector element by element in the application, the attention weight is dynamically adjusted, the model pays more attention to the motion area, and the background interference is reduced.

[0129] The hierarchical spatial attention is independently calculated for each frame of the video sequence, parallel processing is supported, the calculation efficiency is significantly improved, and the traditional frame-by-frame serial processing method is superior. The context information of the high-resolution layer is used to enhance the spatial feature representation of the current layer, and the motion information is used to guide the attention mechanism, so that the model pays more attention to the motion-related area.

[0130] The temporal attention module models the long-term temporal dependency relationship through the hierarchical cross-attention mechanism, and solves the limitations of existing methods (such as 3D convolution). The traditional time modeling method (such as C3D convolution) is difficult to capture long-term temporal dependencies due to the large number of parameters, and the application gradually aggregates historical frame information through step-by-step cross-attention, enhances temporal consistency, and introduces motion residual information to modulate the temporal attention, so that the model pays more attention to the temporal features related to the target motion trajectory, and improves the detection robustness in the occlusion or motion blur scene.

[0131] The final fusion feature vector (motion modulation attention spatio-temporal feature pyramid fusion feature vector) is input into the Transformer Decoder module, combined with Object Queries for decoding, and the final target detection result is predicted through the feedforward neural network (FFN).

[0132] In the application, the overall loss function of the network based on the motion modulation attention spatio-temporal feature pyramid includes a classification loss, a bounding box loss, and a motion residual guided loss.

[0133] The purpose of the classification loss is to measure the class prediction accuracy of the predicted box. In order to solve the common class imbalance problem in target detection and pay more attention to difficult samples, Focal Loss is used as the classification loss, and each matching pair found by the Hungarian matching algorithm is used. Focal Loss can effectively solve the class imbalance problem and make the model pay more attention to difficult samples, thereby improving the classification accuracy.

[0134] The purpose of the bounding box loss is to measure the position regression accuracy of the predicted box. In order to realize accurate bounding box positioning, a weighted combination of L1 loss and GIoU loss is used, and each matching pair found by the Hungarian matching algorithm is used.

[0135] The combination of L1 loss and GIoU loss can optimize the regression accuracy of the bounding box from different angles. L1 loss focuses on the absolute error of the size and center position of the bounding box, and GIoU loss focuses on the shape similarity of the bounding box. The combination of the two can achieve more accurate bounding box positioning.

[0136] The motion residual guidance loss can prompt the network to learn a motion residual map that more effectively indicates the location of the target. The motion residual map is obtained by calculating the difference between the feature maps of the first and last frames of the video sequence. The areas with larger values often correspond to the parts of the image that have motion changes. The edge region of the target is usually the area with more obvious motion changes. Therefore, it is expected that the motion residual map can highlight the edge information of the target and form a "hole" in the internal region of the target, thereby more effectively assisting subsequent feature fusion and target detection. To achieve this goal, the motion residual guidance loss is designed to explicitly constrain the output distribution of the motion residual map to approximate the target response map we expect.

[0137] To simulate the characteristics of the motion residual map, we propose a response map generation method for a continuous Gaussian distribution box. For each training image, first generate the corresponding target response map according to its real box. Unlike generating a Gaussian distribution directly at the center of the target, a Gaussian response is generated at each pixel point on the four continuous boundary line segments of each real box, and the Gaussian distributions of all boundary pixels are accumulated to finally obtain a response map with a hole , as shown in Figure 3 .

[0138] The motion residual guidance loss is achieved by calculating the normalized mean square error between each scale of the motion residual map MRes i and the down-sampled target response map with a hole at the corresponding scale. To align the response map and the motion residual map in scale, the high-resolution response map needs to be down-sampled to the same scale as MRes i using the nearest neighbor interpolation method to obtain . The motion residual guidance loss is the weighted sum of the NMSE losses of the three scales.

[0139] The motion residual guidance loss can most accurately simulate the continuous boundary and hole characteristics of the motion residual map through the response map of the continuous Gaussian distribution box. Through the constraint of the NMSE loss, the motion residual map learns to output responses related to the target position, improving the quality and discriminability of the motion features. By guiding the motion residual map to focus on the target position, the effectiveness of the motion features is improved, the background noise interference is reduced, and the adaptability of the network to dynamic scenes is enhanced.

[0140] The overall loss function of the present application introduces a motion residual guidance loss, and combines a Focal Loss classification loss and an L1+GIoU bounding box loss to form a complete and effective loss function scheme. The loss function can fully utilize motion information, guide the network to learn high-quality motion features, improve the precision and robustness of video object detection, and provide effective training guidance for an end-to-end video object detection model based on a motion modulation attention spatio-temporal feature pyramid (MA-STFP) architecture.

[0141] Embodiments

[0142] (1) Dataset:

[0143] ① KITTI dataset is an autonomous driving dataset, which contains images and video sequences in real road scenes and provides detailed annotations for object detection, object tracking and other tasks. In this embodiment, the tracking part of the KITTI dataset is used to evaluate the performance of video object detection. The KITTI tracking dataset contains 21 training sequences, which contain various types of moving objects such as vehicles, pedestrians, and cyclists, etc. The data of the vehicle class is used, and the overall scene complexity is high, which can effectively evaluate the detection performance of the model in real dynamic scenes.

[0144] ② ImageNet VID: ImageNet Video Detection (VID) dataset is a subset of ImageNet dataset, which is specifically used for video object detection task. ImageNet VID dataset contains 30 common object categories, and the data of the vehicle class is used in this embodiment.

[0145] ③ MOT Challenge: Although MOT Challenge is mainly used for multi-object tracking task, its dataset also contains a large number of video sequences and object annotations, including data with crowded scenes and complex target motion. The data of the vehicle class is used in this embodiment.

[0146] (2) Comparative method:

[0147] The representative object detection methods YOLO10, YOLO11, VideoYOLO, RT-DETR, and RT-DETRv2 are selected for comparison.

[0148] YOLO10 is the 10th version of the YOLO series. YOLO10 further improves the detection accuracy while maintaining real-time detection speed. Compared with the previous generations, YOLO10 introduces a more efficient feature extraction network and a more advanced loss function.

[0149] YOLO11 is the 11th version of the YOLO series, further improving the backbone and neck architecture of YOLO10, introducing PSA to enhance feature extraction capability through multi-head attention mechanism and feedforward neural network, and refining the architecture design and optimizing the training process to achieve faster processing speed while maintaining the balance between accuracy and performance.

[0150] VideoYOLO is a model designed specifically for object detection in video sequences, using 3D convolution to improve the stability and accuracy of detection.

[0151] RT-DETR is a real-time object detection model based on the Transformer architecture. It uses the self-attention mechanism of Transformer to capture global features in the image, achieving real-time performance while maintaining detection accuracy.

[0152] RT-DETRv2 is an improved version of RT-DETR, further optimizing the model architecture and training method. RT-DETRv2 introduces multi-scale feature fusion and dynamic attention mechanism, making the model perform better in handling small targets and overlapping targets.

[0153] (3) Evaluation indicators:

[0154] AP@0.5(%) and AP@0.5:0.95(%) are used for evaluation, AP@0.5 and AP@0.5:0.95 are commonly used evaluation indicators in object detection, based on Average Precision (AP) calculation. AP@0.5 represents the average precision at IoU (Intersection over Union) threshold of 0.5, representing the average value of precision at different recall rates. IoU is a measure of the overlap between the predicted box and the true box, with a value range of [0, 1]. IoU≥0.5 considers the detected target, this indicator has relatively loose requirements for positioning accuracy, the predicted box and the true box overlap degree reaches 0.5 or more. AP@0.5:0.95 calculates AP at multiple IoU thresholds (usually 0.5 to 0.95, step 0.05), then takes the average, considering 10 IoU thresholds from 0.5 to 0.95: [0.5, 0.55, 0.6,..., 0.9, 0.95], AP is calculated at each IoU threshold, and the average of these APs is taken.

[0155] (4) Experimental results and analysis:

[0156] Table 1 KITTI dataset results

[0157]

[0158] Table 2 ImageNet VID dataset results

[0159]

[0160] Table 3 MOT Challenge dataset results

[0161]

[0162] In terms of inference speed, the test inference speed is 32 fps on an NVIDIA 4090 GPU in FP32, with an input image size of 640*640, meeting the real-time requirement.

[0163] As can be seen from the experimental comparison results in Tables 1, 2 and 3, the AP@0.5 and AP@0.5:0.95 indicators of the MA-STFP method on the KITTI, ImageNet VID and MOT Challenge datasets are significantly better than those of the comparison methods (YOLOv10, YOLOv11, VideoYOLO, RT-DETR, RT-DETRv2).

[0164] The results are analyzed as follows:

[0165] Advantages of MA-STFP: The model has stronger target positioning robustness in dynamic scenes. Mainly due to the motion residual guided attention mechanism, the motion residual map is generated by the difference between the first and last frame features, and then the motion residual guided loss is used to constrain the matching between the motion residual map and the continuous Gaussian response map, which strengthens the extraction of target edge features, explicitly captures the motion area of the target, and reduces background interference. Hierarchical spatial attention (cross-scale fusion) and temporal attention (cross-time modeling) further enhance the target positioning ability in motion blur and occlusion scenes.

[0166] Limitations of comparison methods:

[0167] YOLO series: Relies on static image features, lacks temporal information modeling, and is easily affected by motion blur in dynamic scenes, leading to positioning deviation.

[0168] RT-DETR series: Although it uses Transformer to model the global context, it does not explicitly introduce motion information, and the spatiotemporal dependency modeling of the target in the motion scene is insufficient.

[0169] VideoYOLO: Uses 3D convolution to extract temporal features, but the parameter amount is large and the receptive field is limited, making it difficult to capture long temporal dependencies.

[0170] (5) Ablation experiment:

[0171] The influence of motion residual, hierarchical spatial attention and temporal attention on the final results is verified on the KITTI dataset, and the results are shown in Table 4.

[0172] Table 4 Ablation experiment results

[0173]

[0174] wherein no motion residual refers to not using the residual feature map to modulate the feature map of the level-wise spatial attention module, and also not using the motion residual guided loss function;

[0175] The level-wise spatial attention in the feature pyramid in the“Level-wise Spatial Attention” section in the“MA-STFP Overall Workflow Diagram” is not attention between levels, but is interpolation of the 1 / 32 scale and 1 / 16 scale features to 1 / 8 scale for weighted average, followed by position encoding and linear transformation. For example, if the video sequence is 6, the feature vectors of 6 frames of images will also be obtained to enter the“Temporal Attention” module; Figure 1 No temporal attention refers to the number of input sequence frames being 1, i.e., single-frame image input for object detection.

[0176] (6) Parameter sensitivity analysis:

[0177] Test the motion residual guided loss weight λ mr-guide Effect on experimental results: when λ mr-guide = 0.4, the AP@0.5:0.95 reaches the maximum value of 59.1%; when λ mr-guide <0.1, the performance decreases due to insufficient motion feature constraint; when λ mr-guide > 0.7, the classification and positioning loss weight is too small, and the performance also decreases.

[0178] In summary, the video object detection method of the present application integrates multi-scale motion residual information into the multi-scale feature vectors of each frame of image at the feature vector level, enhancing the motion information representation. Within each frame of image, through the level-crossing attention mechanism, the feature vectors of different scales are used for information fusion and enhancement, improving the spatial feature representation capability. In the time dimension, through the time-crossing attention mechanism, the feature vectors at different time are used for information fusion and enhancement, modeling the time dependence relationship. The motion information (motion residual) is not only used for initial feature vector enhancement, but also throughout the level spatial attention and time attention calculation process, through modulating the attention weight, guiding the network to pay more attention to the motion-related spatio-temporal region.

[0179]

[0180] ​The motion residual guiding loss function is innovatively designed to explicitly guide the network to learn high-quality motion features. By generating a response map of a continuous Gaussian distribution box and calculating the normalized mean square error between the motion residual map and the response map, the motion residual map can more accurately indicate the target position, highlight the target edge information, and suppress the background noise interference, effectively improving the discriminability of the motion features and significantly improving the detection performance.

[0181] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0182] The above is only the preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1.A method for video object detection based on spatiotemporal feature pyramid of motion modulated attention, characterized in that, Comprising the following steps: 1) CNN feature extraction, to obtain three scale feature maps; 2) motion feature extraction, to generate three scale motion residual maps; 3) position encoding, to add position encoding to the three scale feature maps and motion residual maps respectively, to explicitly encode spatial position information; 4) feature vector conversion, the three scale feature maps after adding position encoding, and the three scale motion residual maps after adding position encoding, are respectively linearly transformed by linear layers to convert them into vector form; 5) motion information enhancement, the three scale linearly transformed feature vectors of each frame of video sequence image are respectively element-wise multiplied with the linearly transformed motion residual vectors of the corresponding scale, to enhance the motion information of the feature vectors; 6) hierarchical spatial attention calculation is performed to obtain N sequence hierarchical spatial attention feature vectors; The hierarchical spatial attention calculation comprises the following steps: 61) scale correspondence; Q1: 1 / 16 scale modulated feature vector; K1, V1: 1 / 8 scale modulated feature vector; Q2: 1 / 32 scale modulated feature vector; 62) first cross-attention calculation: taking Q1 as Query, K1 and V1 as Key and Value, cross-attention calculation is performed to obtain intermediate results K2 and V2; 63) second cross-attention calculation: taking Q2 as Query, the results K2 and V2 of the first cross-attention as Key and Value, cross-attention calculation is performed to obtain N sequence hierarchical spatial attention feature vectors; 7) time attention calculation is performed to obtain motion modulation attention spatio-temporal feature pyramid fusion feature vectors; The time attention calculation comprises the following steps: 71) arrange the N sequence hierarchical spatial attention feature vectors in time sequence to form a time sequence; let the current time be t, then the N sequence hierarchical spatial attention feature vectors arranged in time sequence from bottom to top represent the feature vectors of time t, t-1, t-2,..., t-N+1 in turn; 72) first cross-attention calculation: taking the feature vector at time t-N+2 as Query, the feature vector at time t-N+1 as Key and Value, cross-attention calculation is performed to obtain intermediate results K2 and V2; 73) second cross-attention calculation: taking the feature vector at time t-N+3 as Query, the results K2 and V2 of the first cross-attention calculation as Key and Value, cross-attention calculation is performed; 74) repeat the cross-attention calculation until the feature vector at time t is the last Query, and cross-attention calculation is performed with the results of the last cross-attention calculation to obtain motion modulation attention spatio-temporal feature pyramid fusion feature vectors; 8) decode the motion modulation attention spatio-temporal feature pyramid fusion feature vectors, and predict the target detection results through a feedforward neural network. 2.The video object detection method based on motion modulated attention spatiotemporal feature pyramid according to claim 1, characterized in that, In the step 1), N frames of video sequence images are acquired at one time, and each frame of video sequence image is subjected to feature extraction through a CNN backbone network to obtain feature maps of three scales of 1 / 8, 1 / 16 and 1 / 32 respectively. 3.The video object detection method based on spatiotemporal feature pyramid of motion-modulated attention according to claim 2, characterized in that, In the step 2), the element-by-element difference between the feature map of the last frame of video sequence image and the feature map of the first frame of video sequence image is calculated at each scale to generate motion residual maps of three scales. 4.The method of claim 1, wherein, The overall loss function of the network based on the motion modulation attention spatiotemporal feature pyramid comprises a classification loss, a bounding box loss and a motion residual guided loss; wherein the classification loss L cls is: ； wherein, is a predicted bounding box predicted to match a ground truth bounding box category of probability; ； wherein, if the class prediction is correct, then ; and if the class prediction is incorrect, then ; a is a weight coefficient for balancing positive and negative samples; is a focusing parameter; Boundary box loss L bbox is: ； wherein, is a bounding box of the predicted box ; is a bounding box of the real box ; L1 loss for L1 loss; GIoU loss for GIoU loss; to balance the weight coefficients of L1 loss and GIoU loss; Motion residual guidance loss L MR-guide is: ； where ω i is the weight coefficient of the i-th scale motion residual guiding loss; is the normalized mean square error function; is the mean square error function; and is the size of the down-sampled feature map; MRes i is a motion residual map; Response map with holes Downsampled to the same scale as MRes using nearest neighbor interpolation i Obtained; ； wherein, is a set of all integer pixel coordinates on the four boundary line segments of the j1th real box b j1 . a two-dimensional Gaussian distribution centered at the boundary pixel; ； wherein, (x2, y2) is the pixel coordinate on the response map; and σ2is the standard deviation of the Gaussian distribution. Overall loss function L MM-STFM is: ； where λ cls , λ bbox , and λ mr-guide are weight coefficients that balance the different terms.

Images