An end-to-end video object detection method and system based on an attention mechanism

Abstract

The application relates to an end-to-end video target detection method and system based on an attention mechanism. The method comprises the following steps: dividing a video segment through adaptive key frame extraction of OpenCV, obtaining spatial enhancement features through feature fusion and self-attention enhancement; stacking the spatial enhancement features of each frame into a cross-frame feature cube and performing time sequence enhancement; initializing a cross-frame shared query base vector, combining time sequence position coding to generate a frame-specific query sequence, and decoding to obtain an initial detection result; obtaining a time sequence trajectory and dividing anchor point frames and frames to be corrected through inter-frame association; correcting the frames to be corrected by extracting anchor point frame features; taking the correction result and the anchor point frame result as feedback, enhancing the cross-frame feature cube through a feature mask, and iteratively executing to obtain a final detection result. Through adaptive frame extraction of OpenCV, anchor point frame-feature-to-frame-to-be-corrected feature optimization and feature mask enhancement iteration mechanism, the method improves the precision and cross-frame consistency of video target detection in a complex scene, and reduces the false alarm rate.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to an end-to-end video target detection method and system based on an attention mechanism. Background Technology

[0002] During investigations, video surveillance data has become a crucial resource for obtaining clues and reconstructing cases. With the widespread deployment of the "Skynet Project" and various security cameras, the amount of video data within the jurisdiction has grown exponentially. Furthermore, some civilian surveillance cameras exist offline, creating video silos and posing a significant challenge to rapid and effective analysis. Current video screening processes face the technical bottleneck of "finding targets in massive amounts of video," and manual frame-by-frame review or fast-forward playback is extremely time-consuming, easily leading to missed crucial clues due to visual fatigue. Traditional video analysis techniques, such as those based on background modeling and early CNN models, suffer from limited receptive fields, reliance on complex anchor boxes, and non-maximum suppression post-processing. In complex scenarios such as low light at night, dense crowds obscuring the view, and rapid target movement, the false alarm and false negative rates are high, making it difficult to meet the accuracy requirements of practical operations. Moreover, the analysis and response are delayed, often missing the golden window for arrest and interception. Summary of the Invention

[0003] Therefore, it is necessary to provide an end-to-end video target detection method and system based on an attention mechanism to address the above-mentioned technical problems, aiming to improve the target detection accuracy and tracking continuity in complex scenes, and enhance the real-time performance and intelligence level of video analysis.

[0004] Firstly, this application provides an end-to-end video object detection method based on an attention mechanism, comprising:

[0005] S1: An adaptive keyframe extraction strategy based on OpenCV is used to divide the input video into multiple video segments;

[0006] S2: Extract features from each frame of the video clip to obtain multi-scale feature maps. Fuse and flatten the multi-scale feature maps to obtain spatial feature sequences. Enhance the spatial feature sequences through self-attention calculation to obtain spatial enhanced feature sequences corresponding to each frame.

[0007] S3: Stack the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube. Perform temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequences corresponding to each frame image. Reassemble the temporal enhancement feature sequences to obtain the temporal enhancement feature cube.

[0008] S4: Initialize the cross-frame shared object query basis vector, and generate a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, use the features of the corresponding frame image in the temporal enhancement feature cube as the key and value, and the frame-specific query sequence of the corresponding frame image as the query, and decode through the decoding network to generate an initial detection result containing the target class probability, bounding box coordinates and identity embedding vector corresponding to each query vector in the frame-specific query sequence;

[0009] S5: Perform inter-frame association processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and divide the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected.

[0010] S6: Extract the features of the corresponding region of the anchor frame in the time-series trajectory to obtain the key-value pair set, and extract the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected. Aggregate the query vector to be corrected and the key-value pair set to obtain the correction query vector. Input the correction query vector into the decoding network to generate the correction detection result of the frame to be corrected.

[0011] S7: Using the correction detection results of the frame to be corrected in the time-series trajectory and the initial detection results of the anchor frame as feedback, generate a feature mask based on the cross-frame feature cube, enhance the cross-frame feature cube according to the feature mask, obtain an enhanced cross-frame feature cube, replace the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iterate through S3 to S6, and output the target detection results.

[0012] In one embodiment, S3 performs temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequence corresponding to each frame image, including:

[0013] S301: Define the current frame image as any frame image being processed in the video segment. For the current frame image, perform a linear transformation on its corresponding spatial enhancement feature sequence to generate the current frame query vector. If the current frame image is the first frame of the video segment, the forward key source is empty. Otherwise, determine the spatial enhancement feature sequences corresponding to each historical frame image from the first frame to the previous frame of the current frame image in the video segment as the forward key source.

[0014] S302: For each spatial location in the current frame query vector, predict the sampling offset and the first attention weight to obtain the sampling offset and the first attention weight corresponding to each spatial location; based on the sampling offset, perform feature sampling from the two-dimensional spatial location of the spatial enhancement feature sequence corresponding to each historical frame image through bilinear interpolation to obtain the original key-value features corresponding to each spatial location of each historical frame image; based on the first attention weight corresponding to each spatial location, weight the original key-value features to obtain the weighted key-value features corresponding to each spatial location of each historical frame image; concatenate the weighted key-value features corresponding to each spatial location of each historical frame image to generate a forward key-value sequence;

[0015] S303: Calculate the second attention weights of the current frame query vector and the forward key value sequence, and perform a weighted summation of the value features in the forward key value sequence based on the second attention weights to obtain the forward attention output; if the forward key value source is empty, then set the forward attention output to the zero vector.

[0016] S304: If the current frame image is the last frame of the video segment, then set the reverse key source to empty; otherwise, determine the spatial enhancement feature sequence corresponding to each future frame image from the next frame to the last frame in the video segment as the reverse key source; repeat the processing steps of S302 and S303 to obtain the reverse attention output; if the reverse key source is empty, then set the reverse attention output to the zero vector.

[0017] S305: Adaptive weighted fusion of forward attention output, backward attention output and spatial enhancement feature sequence corresponding to the current frame image is performed by a preset learnable gating network to obtain the temporal enhancement feature sequence corresponding to the current frame image;

[0018] S306: Perform S301 to S305 sequentially on each frame of the video clip to obtain the temporal enhancement feature sequence corresponding to each frame.

[0019] In one embodiment, the decoding network is a Transformer decoder structure comprising a multi-head cross-attention layer, a feedforward network layer, and an output projection layer; wherein the number of heads in the multi-head cross-attention layer and the dimension of the hidden layer are preset learnable parameters;

[0020] In S4, for each frame image, the features of the corresponding frame image in the temporal enhancement feature cube are used as keys and values, and the frame-specific query sequence of the corresponding frame image is used as the query. The decoding network decodes the data to generate initial detection results containing the target class probability, bounding box coordinates, and identity embedding vector corresponding to each query vector in the frame-specific query sequence. These results include:

[0021] Layer normalization is performed on the frame-specific query sequence corresponding to each frame image to obtain a normalized query sequence; linear projection is performed on the features of the corresponding frame images in the temporal enhancement feature cube to generate key feature matrix and value feature matrix;

[0022] The normalized query sequence is input into the multi-head cross-attention layer. Using the key feature matrix as the key and the value feature matrix as the value, multi-head parallel attention calculation is performed to obtain the multi-head attention output. The multi-head attention output is concatenated to obtain the concatenated attention output. The concatenated attention output is linearly projected to obtain the cross-attention features.

[0023] The cross-attention features are input into the feedforward network layer, and then linear transformation, activation function mapping, and quadratic linear transformation are performed sequentially to obtain the feedforward output. The feedforward output is then subjected to residual connections to obtain residual connection features. Finally, the residual connection features are subjected to layer normalization to obtain the decoding features.

[0024] The decoded features are input to the projection layer and processed based on three parallel linear projection branches. The first branch outputs the target category probability distribution corresponding to each query vector, the second branch outputs the bounding box coordinate offset corresponding to each query vector, and the third branch outputs the identity embedding vector corresponding to each query vector.

[0025] The bounding box coordinate offset is calibrated by coordinate regression to obtain the final bounding box coordinates; the target class probability distribution is normalized by the softmax function to obtain the target class probability; the target class probability, bounding box coordinates and identity embedding vector are integrated to generate the initial detection results corresponding to each frame image.

[0026] In one embodiment, step S5 performs inter-frame correlation processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and divides the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected, including:

[0027] For the initial detection results corresponding to each frame image, the maximum value of the target category probability corresponding to each query vector is extracted, and the maximum value is used as the detection confidence of the query vector; the detection information corresponding to the query vectors whose detection confidence is greater than the first confidence threshold in the initial detection results corresponding to each frame image is taken as valid detection; wherein, each valid detection includes the corresponding bounding box coordinates, target category label and identity embedding vector; the target category label is the category corresponding to the maximum value of the target category probability corresponding to the query vector;

[0028] Initialize the trajectory set. Each trajectory in the trajectory set contains a target identifier, an associated frame identifier sequence, a historical bounding box sequence, Kalman filter state parameters, a historical category label sequence, and cumulative appearance features. Among them, the target identifier is used to distinguish different trajectories, the associated frame identifier sequence is used to record the frame image identifiers associated with the trajectory, the cumulative appearance features are initialized to a zero vector, the historical category label sequence is initialized to an empty sequence, and the Kalman filter state parameters contain position, velocity, and acceleration information.

[0029] The existing trajectories in the trajectory set are recorded as existing trajectories. For each valid detection in the current frame image, the matching cost between the valid detection and each existing trajectory in the trajectory set is calculated. The matching cost includes appearance cost, motion cost, and category cost.

[0030] Each matching cost is constructed into a cost matrix; the Hungarian algorithm is used to solve the cost matrix to obtain the optimal matching result between the effective detection and the existing trajectory;

[0031] Based on the optimal matching result, the trajectory set is updated through effective detection, and the existing trajectory that has not successfully matched any effective detection of consecutive preset frame images is taken as the target trajectory. The target trajectory is removed from the trajectory set to obtain the updated trajectory set. The updated trajectory set is used as the trajectory set for the next iteration.

[0032] By integrating the associated frame identifier sequence, target category label, and historical bounding box sequence information of each target trajectory, the temporal trajectory of the target is obtained;

[0033] For each target trajectory, based on the associated frame identifier sequence of each target trajectory, the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory is extracted; based on the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory, the graph confidence of each frame image associated with the target trajectory is calculated; the frame images with graph confidence greater than the second confidence threshold are marked as anchor frames, and the frame images with graph confidence less than the third confidence threshold are marked as frames to be corrected; wherein, the second confidence threshold is greater than the first confidence threshold, and the third confidence threshold is less than the first confidence threshold.

[0034] In one embodiment, the matching cost is calculated through the following steps:

[0035] Based on the validly detected identity embedding vector and the cumulative appearance features of the corresponding existing trajectory, the cosine distance is calculated as the appearance cost; if the cumulative appearance features of the existing trajectory are zero vectors, then a preset initial value is used as the appearance cost.

[0036] The target position of the current frame image is predicted by the Kalman filter state parameters, and the predicted position is obtained. The Mahalanobis distance between the predicted position and the coordinates of the effectively detected bounding box is calculated as the motion cost.

[0037] If the existing historical category label sequence of the trajectory is not empty, then the latest target category label in the historical category label sequence is extracted as the historical category label; it is determined whether the effective detected target category label is consistent with the historical category label. If they are consistent, the category cost is set to the preset minimum value. If they are inconsistent, the category cost is set to the preset infinite value. If the historical category label sequence is empty, the category cost is set to the preset minimum value.

[0038] The matching cost is obtained by weighted summation of appearance cost, motion cost, and category cost.

[0039] In one embodiment, step S6 involves extracting features from the corresponding region of the anchor frame in the time-series trajectory to obtain a set of key-value pairs, extracting the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected, and aggregating the query vector to be corrected and the set of key-value pairs to obtain the corrected query vector, including:

[0040] For each target trajectory contained in the time-series trajectory, obtain the set of anchor point frames and the set of frames to be corrected corresponding to each target trajectory;

[0041] For each anchor frame in the anchor frame set, the feature map corresponding to the anchor frame is extracted from the temporal enhancement feature cube; based on the bounding box coordinates of the effective detections associated with the target trajectory in the anchor frame, a rectangular region is cropped on the feature map with the bounding box center as the reference and the width and height expanded by a preset ratio; the spatial position feature vectors within the rectangular region are flattened to obtain the local key vector and local value vector corresponding to the anchor frame, and local key-value pairs are constructed based on the local key vector and local value vector;

[0042] Spatial dimension concatenation is performed on the local key-value pairs of each anchor frame corresponding to the same target trajectory to obtain a global anchor key matrix and a global anchor value matrix. The global anchor key matrix and the global anchor value matrix are combined to obtain the set of key-value pairs corresponding to the target trajectory.

[0043] For each frame to be corrected in the set of frames to be corrected, read the identity embedding vector corresponding to the query vector that matches the target trajectory from the initial detection result corresponding to the frame to be corrected, and use the identity embedding vector corresponding to the query vector that matches the target trajectory as the query vector to be corrected.

[0044] Linear projection is performed on the query vector to be corrected to obtain the projected query vector. Layer normalization is performed on the projected query vector to obtain the normalized corrected query vector. Layer normalization is performed on the global anchor key matrix to obtain the normalized key matrix.

[0045] Calculate the third attention weight between the normalized modified query vector and the normalized key matrix, and normalize the third attention weight using the softmax function to obtain the normalized attention weight;

[0046] Based on the normalized attention weights, the global anchor value matrix is ​​weighted and summed to obtain the aggregated features; the aggregated features are then residually connected to the normalized modified query vector to obtain the residual connected features; the residual connected features are then layer-normalized to obtain the modified query vector.

[0047] Secondly, this application also provides an end-to-end video object detection system based on an attention mechanism, comprising:

[0048] The image frame sampling module is used to perform step 1, which includes: using an OpenCV-based adaptive keyframe extraction strategy to divide the input video and obtain multiple video segments;

[0049] The spatial feature enhancement module is used to perform step 2, which includes: extracting features from each frame of the video clip to obtain a multi-scale feature map, fusing and flattening the multi-scale feature map to obtain a spatial feature sequence, and enhancing the spatial feature sequence through self-attention calculation to obtain the spatial enhanced feature sequence corresponding to each frame of the image.

[0050] The temporal feature modeling module is used to execute step 3, which includes: stacking the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube; performing temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequences corresponding to each frame image; and recombining the temporal enhancement feature sequences to obtain the temporal enhancement feature cube.

[0051] The frame-specific query decoding module is used to execute step 4, which includes: initializing the cross-frame shared object query basis vector, generating a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, using the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as queries, decoding is performed through a decoding network to generate an initial detection result containing the target class probability, bounding box coordinates and identity embedding vector corresponding to each query vector in the frame-specific query sequence;

[0052] The trajectory association and frame classification module is used to execute step 5, which includes: performing inter-frame association processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and dividing the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected.

[0053] The trajectory awareness correction module is used to execute step 6, which includes: extracting features of the corresponding region of the anchor frame in the time-series trajectory to obtain a key-value pair set, extracting the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected, aggregating the query vector to be corrected and the key-value pair set to obtain the correction query vector, and inputting the correction query vector into the decoding network to generate the correction detection result of the frame to be corrected.

[0054] The iterative feature optimization module is used to execute step 7, which includes: using the correction detection results of the frame to be corrected in the time-series trajectory and the initial detection results of the anchor frame as feedback, generating a feature mask based on the cross-frame feature cube, enhancing the cross-frame feature cube according to the feature mask to obtain an enhanced cross-frame feature cube, replacing the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iteratively executing steps 3 to 6, and outputting the target detection result.

[0055] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the first aspect.

[0056] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the first aspect.

[0057] The aforementioned end-to-end video object detection method and system based on an attention mechanism first employs an OpenCV-based adaptive keyframe extraction strategy to segment video segments, resolving the information redundancy or omission issues caused by fixed-interval sampling and improving the accuracy and timeliness of key information capture. Secondly, through multi-scale feature extraction, spatial self-attention enhancement, and temporal feature cube modeling, deep aggregation of cross-frame contextual information is achieved, enhancing the discriminative power and robustness of target features in complex scenes. Furthermore, based on cross-frame shared query basis vectors, frame-specific query sequences are generated and decoded. Combined with trajectory association to segment anchor frames and frames to be corrected, key-value aggregation and iterative optimization mechanisms address the identity drift problem caused by occlusion and deformation, improving the continuity of target tracking and the stability of detection results. Finally, using the corrected detection results as feedback, feature masks are generated to enhance cross-frame feature cubes, achieving a closed-loop iteration of detection and feature optimization, and improving the adaptive capability of the end-to-end system. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of an end-to-end video object detection method based on an attention mechanism is provided as an exemplary embodiment of the present invention;

[0060] Figure 2 A flowchart of a method for temporal enhancement of cross-frame feature cubes is provided as an exemplary embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of an end-to-end video target detection system based on an attention mechanism, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] In one embodiment, such as Figure 1 As shown, an end-to-end video object detection method based on an attention mechanism is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0064] S1: An adaptive keyframe extraction strategy based on OpenCV is used to divide the input video into multiple video segments.

[0065] Specifically, surveillance videos often contain numerous redundant still images that, while lacking valid target information, consume significant backend computing resources. Therefore, the input video (including offline-stored civilian surveillance videos) can be parsed using OpenCV's multi-format video decoding interface to obtain a sequence of consecutive frames. Based on preset inter-frame difference thresholds and motion pixel ratio thresholds, a sliding window is used to filter the frame sequence, dividing a preset number of consecutive frames that meet the keyframe criteria into a video segment. This ensures the continuity of target motion within the segment while avoiding computational burden caused by excessively long segments. This step significantly reduces the computational load on the backend feature extraction and detection modules, retaining the core frames containing target information from the video.

[0066] S2: Extract features from each frame of the video clip to obtain multi-scale feature maps. Fuse and flatten the multi-scale feature maps to obtain spatial feature sequences. Enhance the spatial feature sequences through self-attention calculation to obtain spatial enhanced feature sequences corresponding to each frame.

[0067] Specifically, targets may exhibit different sizes, poses, and occlusion states within video frames. Single-scale features are insufficient to comprehensively capture target information, while self-attention mechanisms can model dependencies between arbitrary spatial locations within an image, enabling the association of global contextual information. For example, features can first be extracted from each frame using a convolutional neural network to obtain multi-scale feature maps at different levels. Then, these feature maps are upsampled to the same size, and multi-scale feature fusion is achieved through channel concatenation. A flattening operation transforms the two-dimensional feature maps into a one-dimensional spatial feature sequence, adapting it to the input format of self-attention computation. Finally, a multi-head self-attention mechanism enhances the spatial feature sequence by calculating the attention weights of each feature location with all other locations. Weighted aggregation of global features based on these weights yields a spatially enhanced feature sequence. Based on this process, multi-scale fusion can compensate for the shortcomings of single-scale features, and self-attention enhancement establishes global spatial dependencies, enabling the feature sequence to more comprehensively and accurately express the spatial attributes of the target, thus improving the ability to capture target features in complex backgrounds and partially occluded scenes.

[0068] S3: Stack the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube. Perform temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequences corresponding to each frame image. Reassemble the temporal enhancement feature sequences to obtain the temporal enhancement feature cube.

[0069] Specifically, the motion of targets in videos is continuous, and the spatial features of a single frame cannot reflect the target's trajectory and inter-frame relationships. Cross-frame feature cubes, however, can integrate spatial features from multiple frames, providing a data foundation for temporal dependency modeling. The temporal enhancement process can capture the dynamic relationships between consecutive frames, strengthening the feature representation of the target's motion and improving the robustness of detecting fast-moving targets. For example, spatial enhancement feature sequences from each frame in the same video segment can be stacked in the third dimension (frame dimension) according to frame order, forming a cross-frame feature cube with the shape [number of frames × feature length × feature dimension], achieving structured integration of multi-frame spatial features. Subsequently, the cross-frame feature cube is processed by a temporal enhancement model to model the feature dependencies between the current frame and its preceding and following frames, uncovering the temporal patterns of target motion and generating temporal enhancement feature sequences for each frame. Finally, all temporal enhancement feature sequences are reassembled in the original frame order to obtain the temporal enhancement feature cube, providing comprehensive features that fuse spatial and temporal information for subsequent frame-specific query decoding.

[0070] S4: Initialize the cross-frame shared object query basis vector, and generate a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, use the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as queries, and decode through a decoding network to generate an initial detection result containing the target class probability, bounding box coordinates and identity embedding vector corresponding to each query vector in the frame-specific query sequence.

[0071] Specifically, object query basis vectors, as a pre-defined number of learnable vectors, can share parameters across frames, ensuring consistency in the detection features of the same target across different frames. Temporal position encoding can be used to distinguish query sequences from different frames, avoiding confusion of query features between frames. The decoding network can establish a mapping relationship between query sequences and image features through a cross-attention mechanism, directly outputting target category, bounding box, and identity embedding vectors without anchor box generation or non-maximum suppression (NMS) post-processing, achieving end-to-end detection. For example, a set of learnable object query basis vectors with a dimension of [number of queries × hidden layer dimension] can be initialized first. This vector is continuously optimized through the training process, adaptively capturing common features of different types of targets. Then, for each frame image, the object query basis vectors are superimposed with the temporal position encoding corresponding to that frame to generate a frame-specific query sequence, so that the query vector carries frame temporal information. Using the features of the corresponding frames in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence as queries, a Transformer-based decoding network is input for cross-attention computation. Multi-head parallel attention models the fine-grained association between the query and image features, followed by a feedforward network layer to enhance feature representation. Finally, through the output projection layer, the target class probability distribution, bounding box coordinate offset, and identity embedding vector corresponding to each query vector are output. Regression calibration of the coordinate offset and softmax normalization of the class probability distribution are then performed to integrate the initial detection results for each frame. This process eliminates the complex parameter design and post-processing of traditional anchor box mechanisms, improving detection inference speed. Furthermore, by sharing object query basis vectors across frames, cross-frame consistency in target detection is ensured, providing reliable identity feature support for subsequent inter-frame association.

[0072] S5: Perform inter-frame correlation processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and divide the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected.

[0073] Specifically, the initial detection results of a single frame may exhibit confidence fluctuations, and some frames may have lower detection accuracy due to factors such as lighting and occlusion. Inter-frame correlation can leverage the continuity of the target's appearance, motion, and category to generate a complete trajectory. Based on the detection confidence, highly reliable anchor frames and frames requiring optimization can be selected. For example, based on the identity embedding vector (appearance features), bounding box coordinates (motion features), and target category label (category features) in the initial detection results, the matching cost of effective detection results between adjacent frames can be calculated. The correlation of inter-frame detection results is achieved through the principle of cost minimization. The continuously correlated detection results are aggregated into a temporal trajectory of the target, containing detection information and frame identifiers for the target in different frames. Subsequently, the detection confidence of each trajectory-correlated frame is extracted. Frames with confidence scores higher than a preset threshold are marked as anchor frames as feature reference benchmarks, while frames with confidence scores lower than another preset threshold are marked as frames requiring correction, allowing for optimization through anchor frame features.

[0074] S6: Extract features of the corresponding region of the anchor frame in the time-series trajectory to obtain a key-value pair set, and extract the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected. Aggregate the query vector to be corrected and the key-value pair set to obtain the corrected query vector. Input the corrected query vector into the decoding network to generate the correction detection result of the frame to be corrected.

[0075] Specifically, the target features of anchor frames possess high reliability and high discriminability. By extracting features from the target-corresponding regions in anchor frames to construct a set of key-value pairs, a high-quality feature library can be formed. However, the query vectors of the frames to be corrected suffer from insufficient feature representation due to scene interference. By aggregating anchor frame features through cross-attention, the query vectors can be enhanced and optimized, improving their target feature discriminability. For example, for each trajectory's anchor frame set, based on the bounding box coordinates of the target in the anchor frames, the target region can be cropped on the feature map corresponding to the temporal enhancement feature cube to avoid feature information loss. The feature vectors of the cropped region are flattened, and local key vectors and local value vectors are constructed respectively. After concatenating the local key-value pairs of all anchor frames in spatial dimension, a global anchor key matrix and value matrix corresponding to the trajectory can be formed, i.e., a set of key-value pairs. Subsequently, the identity embedding vector corresponding to the query vector matching the trajectory in the frame to be corrected is extracted as the query vector to be corrected. The query vector and key matrix are standardized through linear projection and layer normalization. The attention weights between them are calculated, and the global anchor value matrix is ​​weighted and summed based on the weights to obtain aggregated features. The aggregated features are then residually connected to the query vector to be corrected and normalized to obtain the enhanced corrected query vector. This enhanced corrected query vector is then input into the decoding network to regenerate the target class probability and bounding box coordinates of the frame to be corrected, i.e., the corrected detection result.

[0076] S7: Using the correction detection results of the frame to be corrected in the time-series trajectory and the initial detection results of the anchor frame as feedback, generate a feature mask based on the cross-frame feature cube, enhance the cross-frame feature cube according to the feature mask, obtain an enhanced cross-frame feature cube, replace the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iterate through S3 to S6, and output the target detection results.

[0077] Specifically, the initial cross-frame feature cube may contain redundant or noisy features. Based on existing anchor frames and corrected detection results, a feature mask can be generated to enhance effective target features and suppress background noise features. Through iterative updates of the feature cube, subsequent detection processes can be based on better feature inputs, improving detection accuracy. For example, based on the final detection results of each frame (initial detection results of anchor frames and corrected detection results of frames to be corrected), effective target regions and background regions in the video can be identified. A feature mask with the same dimensions as the cross-frame feature cube can be generated, assigning high weights to features corresponding to effective target regions and low weights to features corresponding to background regions to achieve feature enhancement. The enhanced cross-frame feature cube replaces the original stacked cross-frame feature cube, and the temporal enhancement, decoding detection, and correlation correction processes from S3 to S6 are re-executed. The number of iterations can be preset according to the actual detection accuracy requirements. After reaching the maximum number of iterations, the final detection result containing target category, bounding box coordinates, and structured data (such as time, location, target attributes, and screenshots) can be output. This structured data can be directly connected to the data platform, supporting analysis functions such as image search and collision analysis, transforming unstructured video data into usable clues and significantly shortening the analysis response time.

[0078] The above method first segments the video using an OpenCV-based adaptive keyframe extraction strategy, eliminating redundant still images and reducing the computational load on the backend. Second, it enhances spatial features through self-attention and strengthens cross-frame temporal correlation through a bidirectional temporal attention model, overcoming the limitation of the receptive field in traditional CNNs and improving the robustness of feature extraction in complex scenes. Initial detection results are generated based on cross-frame shared object query basis vectors and a Transformer decoding network, abandoning the traditional anchor box mechanism, simplifying the detection process, and improving inference efficiency. Combining three-cost inter-frame correlation and cross-attention aggregation correction, target trajectory generation and low-confidence frame correction are achieved, improving detection accuracy. Finally, feature masks are generated based on detection results to iteratively enhance cross-frame feature cubes, continuously optimizing detection performance and ensuring the stability and accuracy of target detection in complex scenes.

[0079] In one embodiment, such as Figure 2As shown, in the end-to-end video object detection method S3 based on an attention mechanism provided in this application, temporal enhancement is performed on the cross-frame feature cube to obtain the temporally enhanced feature sequence corresponding to each frame image, which may include:

[0080] S301: Define the current frame image as any frame image being processed in the video segment. For the current frame image, perform a linear transformation on its corresponding spatial enhancement feature sequence to generate the current frame query vector. If the current frame image is the first frame of the video segment, the forward key source is empty. Otherwise, determine the spatial enhancement feature sequences corresponding to each historical frame image from the first frame to the previous frame of the current frame image in the video segment as the forward key source.

[0081] Specifically, the spatially enhanced feature sequence is a one-dimensional feature sequence obtained after multi-scale feature fusion and self-attention enhancement, which can fully characterize the target spatial attributes of the current frame image. For the spatially enhanced feature sequence corresponding to the current frame image, a fully connected linear transformation layer can be used for feature mapping to generate a current frame query vector adapted for temporal attention computation. If the current frame image is the starting frame of a video segment, since there is no preceding historical frame information, the forward key source can be determined as an empty set. If the current frame image is not the starting frame of a video segment, the spatially enhanced feature sequences corresponding to all historical frames within the video segment from the first frame to the frame preceding the current frame can be uniformly used as the forward key source, providing historical feature support for forward temporal attention computation.

[0082] S302: For each spatial location in the current frame query vector, predict the sampling offset and the first attention weight to obtain the sampling offset and the first attention weight corresponding to each spatial location; based on the sampling offset, perform feature sampling from the two-dimensional spatial location of the spatial enhancement feature sequence corresponding to each historical frame image through bilinear interpolation to obtain the original key-value features corresponding to each spatial location of each historical frame image; based on the first attention weight corresponding to each spatial location, weight the original key-value features to obtain the weighted key-value features corresponding to each spatial location of each historical frame image; concatenate the weighted key-value features corresponding to each spatial location of each historical frame image to generate a forward key-value sequence.

[0083] Specifically, for each independent spatial location within the query vector of the current frame, a two-layer perceptron network can be used to predict the sampling offset and the first attention weight of the corresponding spatial location. The sampling offset is used to locate the feature coordinates associated with the current spatial location in the historical frame feature sequence, and the first attention weight is used to characterize the contribution of historical frame features to the current spatial location. Based on the sampling offset, a bilinear interpolation algorithm can be used to perform feature sampling operations on the two-dimensional spatial locations of the spatial enhancement feature sequences corresponding to each historical frame image, obtaining the original key-value features corresponding to each spatial location of each historical frame. The bilinear interpolation ensures the continuity and integrity of the sampled features, avoiding information loss during the feature sampling process. By performing a weighted operation on the original key-value features according to the first attention weight corresponding to each spatial location, the weighted key-value features corresponding to each spatial location of each historical frame can be obtained. Furthermore, by concatenating the weighted key-value features of all historical frames in frame order, a forward key-value sequence for forward attention calculation can be generated.

[0084] S303: Calculate the second attention weights of the current frame query vector and the forward key value sequence, and perform a weighted summation of the value features in the forward key value sequence based on the second attention weights to obtain the forward attention output; if the forward key value source is empty, then set the forward attention output to the zero vector.

[0085] Specifically, a scaled dot product attention mechanism can be used to calculate the second attention weight between the current frame query vector and the forward key-value sequence. Based on the second attention weight, the value features in the forward key-value sequence can be weighted and summed to obtain the forward attention output. Scaled dot product attention can balance the impact of feature dimensions on the calculation of attention weights, improving the stability of temporal feature modeling. If the source of the forward key values ​​is an empty set, i.e., the current frame is the starting frame of the video segment, the forward attention output is directly assigned a zero vector and does not participate in subsequent feature fusion.

[0086] S304: If the current frame image is the last frame of the video segment, then set the reverse key source to empty; otherwise, determine the spatial enhancement feature sequence corresponding to each future frame image from the next frame to the last frame in the video segment as the reverse key source; repeat the processing steps of S302 and S303 to obtain the reverse attention output; if the reverse key source is empty, then set the reverse attention output to the zero vector.

[0087] Specifically, if the current frame is the last frame of a video segment, since there is no information about subsequent future frames, the reverse key source can be set to an empty set. If the current frame is not the last frame of a video segment, the spatial enhancement feature sequence corresponding to all future frames from the current frame to the last frame within the video segment can be determined as the reverse key source. Following the feature sampling, weighting, and attention calculation process in S302 and S303, the reverse key source is processed to obtain the reverse attention output. This reverse attention calculation can capture the temporal dependency between the current frame and future frames, compensating for the feature loss in unidirectional temporal modeling. If the reverse key source is an empty set, i.e., the current frame is the last frame of a video segment, the reverse attention output can be assigned a zero vector.

[0088] S305: Adaptively weightedly fuse the forward attention output, the backward attention output, and the spatial enhancement feature sequence corresponding to the current frame image through a preset learnable gating network to obtain the temporal enhancement feature sequence corresponding to the current frame image.

[0089] Specifically, a learnable gating network composed of a multilayer perceptron and activation functions can be constructed. This gating network can adaptively learn feature fusion weights through training, and can perform adaptive weighted fusion of forward attention output, backward attention output, and spatial enhancement feature sequence corresponding to the current frame image. In this process, the fusion ratio of each branch feature can be dynamically adjusted according to the target motion state and occlusion of the video scene through the gating mechanism, which strengthens effective temporal features and suppresses noise features, and finally generates the temporal enhancement feature sequence corresponding to the current frame image. This sequence integrates spatial features and bidirectional temporal features, and can accurately represent the target's motion trajectory and inter-frame correlation attributes.

[0090] S306: Perform S301 to S305 sequentially on each frame of the video clip to obtain the temporal enhancement feature sequence corresponding to each frame.

[0091] Specifically, according to the frame order of the video segment, the temporal enhancement processing flow from S301 to S305 can be executed sequentially on each frame image in the video segment. After traversal, the temporal enhancement feature sequence corresponding to all frame images in the video segment can be obtained, providing a feature foundation with both spatial accuracy and temporal continuity for subsequent query-based decoding detection and inter-frame correlation.

[0092] In one embodiment, the end-to-end video object detection method S4 based on an attention mechanism provided in this application, for each frame image, uses the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as the query, and decodes it through a decoding network to generate an initial detection result containing the target class probability, bounding box coordinates, and identity embedding vector corresponding to each query vector in the frame-specific query sequence, which may include:

[0093] Layer normalization is performed on the frame-specific query sequence corresponding to each frame image to obtain a normalized query sequence; linear projection is performed on the features of the corresponding frame images in the temporal enhancement feature cube to generate key feature matrix and value feature matrix;

[0094] The normalized query sequence is input into the multi-head cross-attention layer. Using the key feature matrix as the key and the value feature matrix as the value, multi-head parallel attention calculation is performed to obtain the multi-head attention output. The multi-head attention output is concatenated to obtain the concatenated attention output. The concatenated attention output is linearly projected to obtain the cross-attention features.

[0095] The cross-attention features are input into the feedforward network layer, and then linear transformation, activation function mapping, and quadratic linear transformation are performed sequentially to obtain the feedforward output. The feedforward output is then subjected to residual connections to obtain residual connection features. Finally, the residual connection features are subjected to layer normalization to obtain the decoding features.

[0096] The decoded features are input to the projection layer and processed based on three parallel linear projection branches. The first branch outputs the target category probability distribution corresponding to each query vector, the second branch outputs the bounding box coordinate offset corresponding to each query vector, and the third branch outputs the identity embedding vector corresponding to each query vector.

[0097] The bounding box coordinate offset is calibrated by coordinate regression to obtain the final bounding box coordinates; the target class probability distribution is normalized by the softmax function to obtain the target class probability; the target class probability, bounding box coordinates and identity embedding vector are integrated to generate the initial detection results corresponding to each frame image.

[0098] Specifically, a learnable parameter initialization method can be used to construct a cross-frame shared object query basis vector. This object query basis vector is a fixed-dimensional vector set, with parameter weights shared across all frames of the same video segment. Through continuous iterative optimization during model training, it can adaptively learn the general features of multiple target types such as pedestrians, vehicles, and non-motorized vehicles, ensuring the consistency of detection features for the same target across different frames. Based on the object query basis vector, a corresponding temporal position code can be assigned to each frame of the video segment. The temporal position code is used to distinguish the temporal features of different frames, avoiding confusion between inter-frame query features. By superimposing and fusing the temporal position code with the object query basis vector, a frame-specific query sequence corresponding to each frame can be generated. This sequence carries both the general target features shared across frames and the temporal information of a single frame, providing a foundation for subsequent inter-frame target association and consistency detection.

[0099] Specifically, the decoding network can adopt a Transformer decoder structure, which sequentially includes a multi-head cross-attention layer, a feedforward network layer, and an output projection layer. The number of attention heads and the dimension of the hidden layers in the multi-head cross-attention layer are preset, learnable parameters that can be adaptively adjusted according to the scene complexity and accuracy requirements of video object detection. Layer normalization is performed on the frame-specific query sequences corresponding to each frame image to eliminate differences in numerical distribution within the feature sequences, improving the stability of subsequent attention calculations and the model convergence speed, resulting in a normalized query sequence. For the temporal enhancement features corresponding to the current frame image in the temporal enhancement feature cube, feature mapping transformation can be performed through two sets of independent fully connected linear projection layers to generate dimension-matched key feature matrices and value feature matrices. The key feature matrix is ​​used for calculating attention weights, and the value feature matrix is ​​used for weighted feature aggregation, providing standardized input data for multi-head cross-attention calculations.

[0100] Furthermore, the normalized query sequence is input into a multi-head cross-attention layer, using the key feature matrix as the key and the value feature matrix as the value. Multiple attention heads can perform cross-attention calculations in parallel, independently modeling the fine-grained spatial association between the query sequence and image features, capturing the correspondence between the query vector and the target feature. The outputs of all attention heads are concatenated and fused, then integrated through a linear projection layer to obtain the cross-attention feature. This feature, incorporating multi-head attention information, comprehensively characterizes the association strength between the query and image features. The cross-attention feature is input into a feedforward network layer, where linear transformations are used to enhance the feature dimension, activation functions are used to achieve non-linear feature mapping, and quadratic linear transformations are used to restore the feature dimension, completing high-order abstraction and enhancement of the features, resulting in the feedforward output. Residual connection between the feedforward output and the cross-attention feature preserves the original feature information and alleviates the gradient vanishing problem in deep networks. Layer normalization is then performed on the residually connected features to obtain stable decoded features. These decoded features integrate query information, image spatial features, and temporal features, providing feature support for subsequent detection outputs.

[0101] The decoded features are input to the output projection layer. The output projection layer can have three independent parallel linear projection branches. The first branch predicts the target category and outputs the probability distribution of the target category for each query vector. The second branch predicts the target localization and outputs the bounding box coordinate offset for each query vector. The third branch extracts the target identity features and outputs the identity embedding vector for each query vector. This identity embedding vector is used for subsequent inter-frame target matching and trajectory generation. Finally, the bounding box coordinate offset is calibrated using a preset regression function to obtain the final bounding box coordinates that match the image pixel coordinates. The target category probability distribution is normalized using a softmax function to obtain the confidence probability of each target category. Integrating the target category probability, bounding box coordinates, and identity embedding vector generates the initial detection results for each frame. This result eliminates the need for anchor box generation and non-maximum suppression post-processing, achieving end-to-end target detection output, effectively improving detection inference efficiency, and providing a unified identity feature basis for subsequent inter-frame association.

[0102] In one embodiment, the end-to-end video target detection method S5 based on an attention mechanism provided in this application performs inter-frame correlation processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and divides the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected, which may include:

[0103] For the initial detection results corresponding to each frame image, the maximum value of the target category probability corresponding to each query vector is extracted, and the maximum value is used as the detection confidence of the query vector; the detection information corresponding to the query vectors whose detection confidence is greater than the first confidence threshold in the initial detection results corresponding to each frame image is taken as valid detection; wherein, each valid detection includes the corresponding bounding box coordinates, target category label and identity embedding vector; the target category label is the category corresponding to the maximum value of the target category probability corresponding to the query vector;

[0104] Initialize the trajectory set. Each trajectory in the trajectory set contains a target identifier, an associated frame identifier sequence, a historical bounding box sequence, Kalman filter state parameters, a historical category label sequence, and cumulative appearance features. Among them, the target identifier is used to distinguish different trajectories, the associated frame identifier sequence is used to record the frame image identifiers associated with the trajectory, the cumulative appearance features are initialized to a zero vector, the historical category label sequence is initialized to an empty sequence, and the Kalman filter state parameters contain position, velocity, and acceleration information.

[0105] The existing trajectories in the trajectory set are recorded as existing trajectories. For each valid detection in the current frame image, the matching cost between the valid detection and each existing trajectory in the trajectory set is calculated. The matching cost includes appearance cost, motion cost, and category cost.

[0106] Each matching cost is constructed into a cost matrix; the Hungarian algorithm is used to solve the cost matrix to obtain the optimal matching result between the effective detection and the existing trajectory;

[0107] Based on the optimal matching result, the trajectory set is updated through effective detection, and the existing trajectory that has not successfully matched any effective detection of consecutive preset frame images is taken as the target trajectory. The target trajectory is removed from the trajectory set to obtain the updated trajectory set. The updated trajectory set is used as the trajectory set for the next iteration.

[0108] By integrating the associated frame identifier sequence, target category label, and historical bounding box sequence information of each target trajectory, the temporal trajectory of the target is obtained;

[0109] For each target trajectory, based on the associated frame identifier sequence of each target trajectory, the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory is extracted; based on the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory, the graph confidence of each frame image associated with the target trajectory is calculated; the frame images with graph confidence greater than the second confidence threshold are marked as anchor frames, and the frame images with graph confidence less than the third confidence threshold are marked as frames to be corrected; wherein, the second confidence threshold is greater than the first confidence threshold, and the third confidence threshold is less than the first confidence threshold.

[0110] Specifically, for the initial detection results corresponding to each frame of the image, the probability distribution of the target category corresponding to each query vector can be traversed, the maximum value can be extracted, and this value can be used as the detection confidence score corresponding to the current query vector. This detection confidence score is used to characterize the reliability of the target detection result corresponding to the current query. Subsequently, the detection information corresponding to the query vectors in the initial detection results of each frame of the image with a detection confidence score greater than the first confidence score threshold can be determined as valid detections. Valid detections include three core types of information: bounding box coordinates, target category label, and identity embedding vector. The target category label is determined by the target category corresponding to the maximum value in the probability distribution of the target category corresponding to the query vector. The target category covers types such as pedestrians, vehicles, and non-motorized vehicles in the monitoring scene.

[0111] As an illustration, a trajectory set for storing target tracking states can be initialized first to adapt to the trajectory creation requirements of the video clip's starting frame. Each trajectory in this set includes a target identifier, a sequence of associated frame identifiers, a sequence of historical bounding boxes, Kalman filter state parameters, a sequence of historical category labels, and cumulative appearance features. The target identifier uniquely distinguishes the tracking trajectories of different targets. The sequence of associated frame identifiers records the image identifiers of all frames associated with the trajectory. The sequence of historical bounding boxes stores the target's location information in each associated frame. The Kalman filter state parameters contain the target's position, velocity, and acceleration information, used for continuous prediction and updating of the target's motion state. The cumulative appearance features are initialized to a zero vector, and the historical category label sequence is initialized to an empty sequence.

[0112] Specifically, existing trajectories can be defined as those that have been created and are continuously updated in the trajectory set. For each valid detection in the current frame image, the matching cost between the valid detection and each existing trajectory in the trajectory set is calculated. This matching cost consists of three parts: appearance cost, motion cost, and category cost. The appearance cost is obtained by calculating the cosine distance between the identity embedding vector of the valid detection and the cumulative appearance features of the corresponding existing trajectory. The identity embedding vector represents the unique appearance attributes of the target, and the cumulative appearance features are used to fuse the target appearance information from historically associated frames of the trajectory. If the cumulative appearance features of an existing trajectory are zero, a preset initial value can be used as the current appearance cost. The motion cost is obtained by predicting the target position in the current frame image using Kalman filter state parameters, and then calculating the Mahalanobis distance between the predicted position and the bounding box coordinates of the valid detection. This calculation process, using Kalman filtering, can accurately model the uniform and uniformly accelerated motion states of the target, adapting to scenarios where targets move rapidly in surveillance videos. During the category cost calculation, if the historical category label sequence of an existing trajectory is not empty, the latest target category label in the sequence can be extracted as the historical category label. Then, it is determined whether the effectively detected target category label matches the historical category label. If they match, the category cost is set to a preset minimum value; otherwise, it is set to a preset infinite value. If the historical category label sequence is empty, the category cost can be directly set to the preset minimum value. The final matching cost is obtained by weighted summing of the calculated appearance cost, motion cost, and category cost. The calculation formula is as follows: ,in For the total matching cost, , , These are the weighted coefficients for appearance cost, motion cost, and category cost, respectively. For the sake of appearance, For the price of sports, As a category cost, the weighting coefficients can be adaptively adjusted according to the complexity of the monitoring scenario.

[0113] Specifically, the matching costs between all valid detections in the current frame and all existing trajectories in the trajectory set can be arranged in an ordered manner according to their correspondences to construct a two-dimensional cost matrix. The row dimension of this cost matrix is ​​consistent with the total number of valid detections in the current frame, and the column dimension is consistent with the total number of existing trajectories in the trajectory set. Each element in the matrix uniquely corresponds to a combination of valid detections and existing trajectories, and the element value is the total matching cost obtained by weighted summation of the combination. The value directly reflects the degree of matching correlation between the two. Using this two-dimensional cost matrix as input, the Hungarian algorithm can be used to perform optimal matching in the bipartite graph. This algorithm takes minimizing the global total matching cost as its core optimization objective. It can establish a strict one-to-one correspondence between valid detections and existing trajectories, ensuring that a single valid detection can only match one existing trajectory, and a single existing trajectory can only match one valid detection. This avoids the problems of duplicate matching, cross-matching, and missed matching caused by multiple targets occlusion and positional overlap in dense target scenes. Finally, it outputs the optimal matching result set of valid detections and existing trajectories.

[0114] Furthermore, the trajectory set can be updated based on the optimal matching result set. That is, the current frame image contains multiple valid detections, and the trajectory set contains multiple existing trajectories. After the Hungarian algorithm completes the one-to-one optimal matching between valid detections and existing trajectories, there are only three processing scenarios: First, for existing trajectories that are successfully matched, the corresponding valid detections can be used to update the trajectory attributes; second, for valid detections that do not match any existing trajectory, new trajectories can be initialized and added to the trajectory set; third, for existing trajectories that do not match any valid detection in the current frame image, the number of consecutive unmatched frames can be incremented by one. When the number of consecutive unmatched frames reaches a preset threshold, it can be determined that the target has left the monitoring screen, the trajectory is marked as a terminated target trajectory and removed from the trajectory set, and the trajectories that are still in the effective tracking state are retained, resulting in an updated trajectory set suitable for processing the next frame image.

[0115] Indicatively, for existing trajectories that have been matched, the current frame image identifier can be appended to the associated frame identifier sequence, the coordinates of the currently detected bounding box can be merged into the historical bounding box sequence, the identity embedding vector of the currently detected trajectories can be weighted and fused with the original cumulative appearance features of the trajectory, and the position, velocity, and acceleration in the Kalman filter state parameters can be corrected based on the coordinates of the currently detected bounding box. The target category label of the currently detected trajectories can be added to the historical category label sequence. For existing trajectories that have not been matched, only their motion state can be predicted and updated using Kalman filtering, without modifying other trajectory attributes. All removed terminating target trajectories are structurally integrated, and the associated frame identifier sequence, target category label, and historical bounding box sequence of each trajectory are extracted in frame order. After data stitching, a complete temporal trajectory of the target within the video segment can be generated. This temporal trajectory continuously records key information about the target's cross-frame motion and can directly generate a visualized spatiotemporal trajectory, providing a complete reconstruction of the target's movement path for video analysis.

[0116] Specifically, for each target trajectory, the detection confidence of the trajectory in each associated frame image can be extracted based on its associated frame identifier sequence. The graph confidence of the trajectory in the corresponding frame image is calculated based on the detection confidence of each associated frame. This graph confidence is used to comprehensively characterize the overall detection reliability of the target trajectory in the current frame. Subsequently, frames with graph confidence values ​​greater than a second confidence threshold (greater than a first confidence threshold) can be marked as anchor frames. Anchor frames can serve as a high-reliability reference benchmark for target features. Frames with graph confidence values ​​less than a third confidence threshold (less than a first confidence threshold) can be marked as frames to be corrected. This indicates that the frames to be corrected are frames with insufficient detection accuracy due to interference from lighting, occlusion, etc., facilitating subsequent optimization and correction of the detection results of the frames to be corrected using anchor frame features, thereby improving the overall stability of target detection in complex monitoring scenarios.

[0117] In one embodiment, the end-to-end video object detection method S6 based on an attention mechanism provided in this application extracts features of the corresponding region of the anchor frame in the temporal trajectory to obtain a key-value pair set, extracts the identity embedding vector of the query vector corresponding to the frame to be corrected in the temporal trajectory to obtain the query vector to be corrected, and aggregates the query vector to be corrected and the key-value pair set to obtain the corrected query vector, which may include:

[0118] For each target trajectory contained in the time-series trajectory, obtain the set of anchor point frames and the set of frames to be corrected corresponding to each target trajectory;

[0119] For each anchor frame in the anchor frame set, the feature map corresponding to the anchor frame is extracted from the temporal enhancement feature cube; based on the bounding box coordinates of the effective detections associated with the target trajectory in the anchor frame, a rectangular region is cropped on the feature map with the bounding box center as the reference and the width and height expanded by a preset ratio; the spatial position feature vectors within the rectangular region are flattened to obtain the local key vector and local value vector corresponding to the anchor frame, and local key-value pairs are constructed based on the local key vector and local value vector;

[0120] Spatial dimension concatenation is performed on the local key-value pairs of each anchor frame corresponding to the same target trajectory to obtain a global anchor key matrix and a global anchor value matrix. The global anchor key matrix and the global anchor value matrix are combined to obtain the set of key-value pairs corresponding to the target trajectory.

[0121] For each frame to be corrected in the set of frames to be corrected, read the identity embedding vector corresponding to the query vector that matches the target trajectory from the initial detection result corresponding to the frame to be corrected, and use the identity embedding vector corresponding to the query vector that matches the target trajectory as the query vector to be corrected.

[0122] Linear projection is performed on the query vector to be corrected to obtain the projected query vector. Layer normalization is performed on the projected query vector to obtain the normalized corrected query vector. Layer normalization is performed on the global anchor key matrix to obtain the normalized key matrix.

[0123] Calculate the third attention weight between the normalized modified query vector and the normalized key matrix, and normalize the third attention weight using the softmax function to obtain the normalized attention weight;

[0124] Based on the normalized attention weights, the global anchor value matrix is ​​weighted and summed to obtain the aggregated features; the aggregated features are then residually connected to the normalized modified query vector to obtain the residual connected features; the residual connected features are then layer-normalized to obtain the modified query vector.

[0125] Specifically, the temporal trajectory is generated through inter-frame association processing and trajectory filtering. Each target trajectory forms a unique mapping relationship with a single target within the video segment, recording the cross-frame motion trajectory, category attributes, and appearance feature changes of the target from entering the monitoring area to leaving the monitoring range. For each target trajectory, based on the detection confidence of each frame recorded during the inter-frame association process, an anchor frame set can be selected. After preprocessing with OpenCV (denoising and illumination equalization) and Transformer feature encoding, the target features in this set have high recognizability and high reliability. However, the frame images in the set to be corrected are often affected by complex scene interference such as dense crowds, low light at night, blurring due to rapid target movement, and sudden changes in camera angle, resulting in missing or biased feature representations in the initial detection. Therefore, the high-confidence features of the anchor frame set can be used for targeted optimization to compensate for feature defects in complex scenes and ensure the overall accuracy of target detection.

[0126] Indicatively, for each anchor frame in the anchor frame set, the feature map at the corresponding index position can be accurately located and extracted from the temporal enhancement feature cube based on the frame identifier of the anchor frame in the video segment. The dimensional structure of the temporal enhancement feature cube is [total number of frames in the video segment × feature sequence length × feature dimension], and the dimension of the feature map corresponding to the anchor frame is [feature sequence length × feature dimension]. This integrates the spatial detail features of the target, global context features, and temporal motion correlation features, providing data support for subsequent feature optimization. Based on the bounding box coordinates (x1, y1, x2, y2) corresponding to the effective detection matched by the target trajectory in the anchor frame, the geometric center of the bounding box ((x1+x2) / 2, (y1+y2) / 2) is used as a fixed reference point. The pixel range is expanded along the horizontal direction (x-axis) and the vertical direction (y-axis) on the feature map by a preset proportion. This expansion proportion can be determined based on the statistical distribution of the target size in the monitoring scene, ensuring that the cropping area can completely cover the main features of the target while retaining a certain range of related background features around the target.

[0127] Subsequently, the feature vectors of all spatial locations within the cropped rectangular region can be subjected to one-dimensional flattening in row-major order, transforming the two-dimensional spatial feature matrix into a one-dimensional feature sequence. The local key vectors and local value vectors use the same flattening logic, i.e., the dimension of the local key vector can be set to [expanded feature length × key vector dimension] for subsequent attention weight calculation, and the dimension of the local value vector can be set to [expanded feature length × value vector dimension] for weighted aggregation of feature information, which together can constitute the local key-value pair corresponding to the anchor frame.

[0128] Specifically, for all anchor frames corresponding to the same target trajectory, the local key vectors of each anchor frame can be concatenated sequentially along the spatial dimension (anchor frame dimension) to generate a global anchor key matrix. Its dimensional structure is [number of anchor frames × extended feature length × key vector dimension]. This matrix integrates the target's appearance features and scene association features across different anchor frames, forming a multi-view, highly reliable feature association library. Furthermore, the local value vectors of each anchor frame can be concatenated along the same spatial dimension to generate a global anchor value matrix. Its dimensional structure is [number of anchor frames × extended feature length × value vector dimension]. This matrix stores the complete feature information of the target in each anchor frame. Combining the global anchor key matrix and the global anchor value matrix, a target trajectory-specific key-value pair set can be formed based on the target's cross-frame consistency features. This ensures that during subsequent attention aggregation, the query vector of the frame to be corrected can accurately match suitable high-confidence features, providing a reliable data foundation for feature optimization.

[0129] Specifically, for each frame in the set of frames to be corrected, a query vector consistent with the current target trajectory identifier can be selected from the initial detection results of that frame based on the target identifier assigned during the inter-frame association process. This query vector is generated by fusing the object query basis vector of the DETR model with the temporal position encoding, and its corresponding identity embedding vector dimension is [hidden layer dimension], which can uniquely represent the appearance attributes of the target. This identity embedding vector is directly used as the query vector to be corrected. Due to scene interference in the frames to be corrected, the query vector to be corrected may have missing or weakened representations of key target features. By focusing on the key-value pair set of anchor frames, key feature information can be supplemented, and the discriminative power of target attributes can be enhanced. For example, a single hidden layer fully connected neural network can be used as a linear projection layer to transform the feature dimension of the query vector to be corrected. The input dimension of the linear projection layer is the original dimension (hidden layer dimension) of the query vector to be corrected, and the output dimension is consistent with the key vector dimension of the global anchor key matrix. The projection process achieves feature mapping through matrix multiplication, and its formula can be:

[0130]

[0131] in, For projection query vector, The weight matrix of the linear projection layer. The query vector to be corrected. As a bias term, the weight matrix is ​​initialized using the Xavier initialization method, with the bias term initialized as a zero vector to ensure the stability and effectiveness of the projection process. Layer normalization is performed on the projected query vector, calculating across all elements of the feature sequence; the formula is as follows: ,in The mean of the projected query vector. For variance, To prevent extremely small values ​​where the denominator is zero, normalization yields a normalized modified query vector with a stable numerical distribution. Similarly, layer-wise normalization can be performed on the global anchor key matrix along the feature dimension. The mean and variance of the key vector corresponding to each anchor frame are calculated, and the same normalization formula is used to obtain the normalized key matrix. This ensures that the query features and key features maintain consistency in numerical distribution, avoiding attention calculation deviations caused by dimensional differences or numerical shifts.

[0132] Furthermore, the third attention weight, representing the strength of their association, can be obtained by calculating the dot product between the normalized query vector and the normalized key matrix. The dot product calculation involves element-wise multiplication followed by summation, accurately reflecting the similarity between the query vector and the key vectors of each anchor frame. The softmax function is used to normalize the third attention weight, mapping the weight values ​​to the [0,1] interval to ensure the total weight sum is 1. The calculation formula is as follows:

[0133]

[0134] in, To normalize attention weights, To normalize and correct the query vector, For the normalized bond matrix, The dimension of the bond vector in the normalized bond matrix, divided by It can alleviate the problem of softmax gradient vanishing caused by excessively large dot product results when the feature dimension is high, making the weight allocation more reasonable.

[0135] Based on normalized attention weights, each element in the global anchor value matrix is ​​multiplied by its corresponding normalized attention weight. Then, the weighted value vectors of all anchor frames are summed element-wise to obtain aggregated features. These features integrate the high-confidence features from all anchor frames that have the highest correlation with the query vector to be corrected, compensating for the feature gaps in the query vector. Element-wise residual connections are then performed between the aggregated features and the normalized corrected query vector (i.e., addition is performed dimension-wise). This preserves the original effective information in the query vector while incorporating high-quality features from the anchor frames, effectively mitigating the gradient vanishing problem during deep network training, resulting in residual connection features. Layer normalization is then applied again to the residual connection features to eliminate numerical offsets and noise interference generated during feature fusion, ultimately yielding a corrected query vector with dimensions consistent with the normalized corrected query vector. The corrected query vector retains the real-time feature information of the target in the frame to be corrected, and integrates the highly reliable appearance and motion features of the anchor frame, which significantly improves the recognizability and stability of the target features, provides feature support for the accurate detection of the frame to be corrected, and effectively reduces the false alarm rate and false alarm rate in complex scenarios.

[0136] To illustrate, the modified query vector is input into the decoding network based on the Transformer structure in the above embodiment. Through multiple rounds of processing, the modified detection result of the frame to be modified can be obtained. Compared with the initial detection result, the target feature recognition is higher and the positioning is more accurate, effectively correcting the detection deviation caused by complex scenes.

[0137] In one embodiment, in the end-to-end video target detection method S7 based on an attention mechanism provided in this application, the corrected detection results of all frames to be corrected in the temporal trajectory and the initial detection results of the anchor frames can be used as feedback information to construct a structured feature library of the detection results. This feature library contains the target category label, bounding box coordinates, identity embedding vector, and detection confidence of each frame. Subsequently, based on the dimensional structure of the cross-frame feature cube [total number of frames in the video segment × feature sequence length × feature dimension], a feature mask with the same dimension can be generated. The generation logic of the feature mask is as follows: for the feature map corresponding to each frame in the cross-frame feature cube, the effective target region on the feature map is determined according to the bounding box coordinates in the detection result of that frame. High weight coefficients are assigned to the feature positions corresponding to the effective target regions, and low weight coefficients are assigned to the feature positions corresponding to the background regions and invalid detection regions. The weight coefficients are adaptively learned and optimized through the training process to ensure that the effective target features are enhanced and the background noise features are suppressed. The calculation formula of the feature mask can be:

[0138]

[0139] in, This represents the generated feature mask. This represents the target region weighting coefficient. An indicator matrix representing the valid target region, where elements in the target region are 1 and elements in the non-target region are 0. An indicator matrix representing the background region, where elements in the background region are 1 and elements in the non-background region are 0.

[0140] Specifically, the generated feature mask is multiplied element-wise with the original cross-frame feature cube. The feature values ​​of the effective target region are enhanced by multiplying with high-weight coefficients, while background noise features are weakened by multiplying with low-weight coefficients, ultimately resulting in an enhanced cross-frame feature cube. This enhanced cross-frame feature cube retains the temporal correlation and spatial structure of the original features, highlights the feature expression of the effective target, and reduces the impact of noise interference on subsequent detection processes. Therefore, the enhanced cross-frame feature cube can replace the initial cross-frame feature cube obtained by stacking frame dimensions as input data for a new round of detection. The complete processing steps S3 to S6 of this method are iteratively executed: the enhanced cross-frame feature cube is temporally enhanced again to generate a more accurate temporally enhanced feature sequence and temporally enhanced feature cube; a frame-specific query sequence is generated based on the object query basis vector and temporal position encoding; a new round of initial detection results is obtained through a decoding network; the temporal trajectory and anchor frame, and the set of frames to be corrected are updated through inter-frame correlation processing; and query vector correction and correction detection result generation are performed on the new frames to be corrected.

[0141] As an illustration, the iteration process can be terminated when the maximum number of iterations is reached or the improvement in detection accuracy between two adjacent iterations is less than a preset accuracy threshold. The final output includes target detection results from all frames. This result integrates the target category, bounding box coordinates, identity embedding vector, and temporal trajectory information for each frame, transforming unstructured detection results into structured data, such as the target's appearance time, monitoring location, target attributes (category, appearance features), and keyframe screenshots. This provides rapid and accurate technical support for scenarios such as video retrieval and process monitoring, effectively shortening the analysis and response time.

[0142] Based on the same inventive concept, this application also provides an attention-based end-to-end video target detection system for implementing the attention-based end-to-end video target detection method described above. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more examples of an attention-based end-to-end video target detection system provided below can be found in the limitations of the attention-based end-to-end video target detection method described above, and will not be repeated here.

[0143] In one exemplary embodiment, such as Figure 3 As shown, an end-to-end video object detection system 400 based on an attention mechanism is provided, comprising:

[0144] The image frame sampling module 401 is used to perform step 1, which includes: using an OpenCV-based adaptive keyframe extraction strategy to divide the input video and obtain multiple video segments;

[0145] The spatial feature enhancement module 402 is used to perform step 2, which includes: extracting features from each frame of the video clip to obtain a multi-scale feature map, fusing and flattening the multi-scale feature map to obtain a spatial feature sequence, and enhancing the spatial feature sequence through self-attention calculation to obtain the spatial enhancement feature sequence corresponding to each frame of the video clip.

[0146] The temporal feature modeling module 403 is used to execute step 3, which includes: stacking the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube; performing temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequences corresponding to each frame image; and recombining the temporal enhancement feature sequences to obtain the temporal enhancement feature cube.

[0147] The frame-specific query decoding module 404 is used to execute step 4, which includes: initializing the cross-frame shared object query basis vector, generating a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, using the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as queries, decoding is performed through a decoding network to generate an initial detection result containing the target class probability, bounding box coordinates and identity embedding vector corresponding to each query vector in the frame-specific query sequence;

[0148] The trajectory association and frame classification module 405 is used to execute step 5, which includes: performing inter-frame association processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and dividing the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected.

[0149] The trajectory perception correction module 406 is used to execute step 6, which includes: extracting features of the corresponding region of the anchor frame in the time-series trajectory to obtain a key-value pair set, extracting the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected, aggregating the query vector to be corrected and the key-value pair set to obtain the correction query vector, and inputting the correction query vector into the decoding network to generate the correction detection result of the frame to be corrected.

[0150] The iterative feature optimization module 407 is used to execute step 7, which includes: using the correction detection result of the frame to be corrected in the time-series trajectory and the initial detection result of the anchor frame as feedback, generating a feature mask based on the cross-frame feature cube, enhancing the cross-frame feature cube according to the feature mask to obtain an enhanced cross-frame feature cube, replacing the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iteratively executing steps 3 to 6, and outputting the target detection result.

[0151] In one exemplary embodiment, the present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the end-to-end video object detection method based on an attention mechanism according to this application. A multi-core processor is preferred to improve the parallel processing capability of the system. The memory provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of data and computational tasks.

[0152] In one exemplary embodiment, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of an attention-based end-to-end video object detection method according to the present application. The computer-readable storage medium may include: a read-only memory, a random access memory, a solid-state drive, or an optical disk, etc.

[0153] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An end-to-end video object detection method based on an attention mechanism, characterized in that, The method includes: S1: An adaptive keyframe extraction strategy based on OpenCV is used to divide the input video into multiple video segments; S2: Extract features from each frame of the video segment to obtain a multi-scale feature map, fuse and flatten the multi-scale feature map to obtain a spatial feature sequence, and enhance the spatial feature sequence through self-attention calculation to obtain a spatial enhanced feature sequence corresponding to each frame image. S3: Stack the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube; perform temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequence corresponding to each frame image; reassemble each temporal enhancement feature sequence to obtain the temporal enhancement feature cube; S4: Initialize the cross-frame shared object query basis vector, and generate a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, use the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as queries, and decode through a decoding network to generate an initial detection result containing the target category probability, bounding box coordinates and identity embedding vector corresponding to each query vector in the frame-specific query sequence; S5: Perform inter-frame association processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and divide the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected. S6: Extract the features of the corresponding region of the anchor frame in the time-series trajectory to obtain a key-value pair set, and extract the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected. Aggregate the query vector to be corrected and the key-value pair set to obtain the corrected query vector. Input the corrected query vector into the decoding network to generate the correction detection result of the frame to be corrected. S7: Using the correction detection results of the frame to be corrected in the time-series trajectory and the initial detection results of the anchor frame as feedback, generate a feature mask based on the cross-frame feature cube, enhance the cross-frame feature cube according to the feature mask to obtain an enhanced cross-frame feature cube, replace the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iterate through S3 to S6, and output the target detection result.

2. The method according to claim 1, characterized in that, In step S3, temporal enhancement is performed on the cross-frame feature cube to obtain the temporal enhancement feature sequence corresponding to each frame image, including: S301: Define the current frame image as any frame image being processed in the video segment. For the current frame image, perform a linear transformation on the corresponding spatial enhancement feature sequence to generate a current frame query vector. If the current frame image is the first frame of the video segment, the forward key source is empty. Otherwise, determine the spatial enhancement feature sequences corresponding to each historical frame image from the first frame to the frame preceding the current frame image in the video segment as the forward key source. S302: For each spatial position in the current frame query vector, perform sampling offset and first attention weight prediction to obtain the sampling offset and first attention weight corresponding to each spatial position; based on the sampling offset, perform feature sampling from the two-dimensional spatial position of the spatial enhancement feature sequence corresponding to each historical frame image through bilinear interpolation to obtain the original key-value features corresponding to each spatial position of each historical frame image; based on the first attention weight corresponding to each spatial position, weight the original key-value features to obtain the weighted key-value features corresponding to each spatial position of each historical frame image; concatenate the weighted key-value features corresponding to each spatial position of each historical frame image to generate a forward key-value sequence; S303: Calculate the second attention weights of the current frame query vector and the forward key value sequence, and perform a weighted summation of the value features in the forward key value sequence based on the second attention weights to obtain the forward attention output; if the forward key value source is empty, then set the forward attention output to a zero vector; S304: If the current frame image is the last frame of the video segment, then the reverse key source is set to empty; otherwise, the spatial enhancement feature sequence corresponding to each future frame image from the next frame to the last frame of the video segment is determined as the reverse key source; repeat the processing steps of S302 and S303 to obtain the reverse attention output; if the reverse key source is empty, then the reverse attention output is set to a zero vector; S305: Adaptively weightedly fuse the forward attention output, the backward attention output, and the spatial enhancement feature sequence corresponding to the current frame image through a preset learnable gating network to obtain the temporal enhancement feature sequence corresponding to the current frame image; S306: Perform S301 to S305 sequentially on each frame of the video segment to obtain the temporal enhancement feature sequence corresponding to each frame.

3. The method according to claim 1, characterized in that, The decoding network is a Transformer decoder structure comprising a multi-head cross-attention layer, a feedforward network layer, and an output projection layer; wherein the number of heads and the dimension of the hidden layer in the multi-head cross-attention layer are preset learnable parameters; In step S4, for each frame image, using the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as the query, a decoding network is used to decode and generate an initial detection result containing the target class probability, bounding box coordinates, and identity embedding vector corresponding to each query vector in the frame-specific query sequence, including: The frame-specific query sequences corresponding to each frame image are subjected to layer normalization to obtain normalized query sequences; the features of the corresponding frame images in the temporal enhancement feature cube are linearly projected to generate key feature matrices and value feature matrices. The normalized query sequence is input into the multi-head cross-attention layer. Using the key feature matrix as the key and the value feature matrix as the value, multi-head parallel attention calculation is performed to obtain multi-head attention output. The multi-head attention output is concatenated to obtain concatenated attention output. The concatenated attention output is linearly projected to obtain cross-attention features. The cross-attention features are input into the feedforward network layer, and linear transformation, activation function mapping, and quadratic linear transformation are performed sequentially to obtain the feedforward output; residual connections are performed on the feedforward output to obtain residual connection features; layer normalization is performed on the residual connection features to obtain the decoding features; The decoded features are input into the output projection layer and processed based on three parallel linear projection branches. The first branch outputs the target category probability distribution corresponding to each query vector, the second branch outputs the bounding box coordinate offset corresponding to each query vector, and the third branch outputs the identity embedding vector corresponding to each query vector. The bounding box coordinate offset is calibrated by coordinate regression to obtain the final bounding box coordinates; the target class probability distribution is normalized by the softmax function to obtain the target class probability; the target class probability, the bounding box coordinates, and the identity embedding vector are integrated to generate the initial detection result corresponding to each frame image.

4. The method according to claim 1, characterized in that, In step S5, inter-frame correlation processing is performed on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target. The frame images associated with each target trajectory in the temporal trajectory are divided into anchor frames and frames to be corrected, including: For each frame image, the maximum value of the target category probability corresponding to each query vector is extracted, and the maximum value is used as the detection confidence of the query vector. Detection information corresponding to query vectors whose detection confidence in the initial detection results of each frame image is greater than a first confidence threshold is considered as valid detection. Each valid detection includes the corresponding bounding box coordinates, target category label, and identity embedding vector. The target category label is the category corresponding to the maximum value of the target category probability corresponding to the query vector. An initial trajectory set is created, wherein each trajectory in the trajectory set includes a target identifier, an associated frame identifier sequence, a historical bounding box sequence, Kalman filter state parameters, a historical category label sequence, and cumulative appearance features; wherein, the target identifier is used to distinguish different trajectories, the associated frame identifier sequence is used to record the frame image identifiers associated with the trajectory, the cumulative appearance features are initialized to a zero vector, the historical category label sequence is initialized to an empty sequence, and the Kalman filter state parameters include position, velocity, and acceleration information; The existing trajectories in the trajectory set are recorded as existing trajectories. For each valid detection in the current frame image, the matching cost between the valid detection and each existing trajectory in the trajectory set is calculated. The matching cost includes appearance cost, motion cost, and category cost. The matching costs are constructed into a cost matrix; the Hungarian algorithm is used to solve the cost matrix to obtain the optimal matching result between the effective detection and the existing trajectory; Based on the optimal matching result, the trajectory set is updated by the effective detection, and the existing trajectory that does not match any of the effective detections of consecutive preset frame images is taken as the target trajectory. The target trajectory is removed from the trajectory set to obtain the updated trajectory set. The updated trajectory set is used as the trajectory set for the next iteration. By integrating the associated frame identifier sequence, target category label, and historical bounding box sequence information of each target trajectory, the temporal trajectory of the target is obtained; For each target trajectory, based on the associated frame identifier sequence of each target trajectory, the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory is extracted; based on the detection confidence of the effective detection corresponding to each frame image associated with the target trajectory, the graph confidence of each frame image associated with the target trajectory is calculated; the frame image with the graph confidence greater than the second confidence threshold is marked as the anchor frame, and the frame image with the graph confidence less than the third confidence threshold is marked as the frame to be corrected; wherein, the second confidence threshold is greater than the first confidence threshold, and the third confidence threshold is less than the first confidence threshold.

5. The method according to claim 4, characterized in that, The matching cost is calculated through the following steps: Based on the effective detected identity embedding vector and the cumulative appearance features of the corresponding existing trajectory, the cosine distance is calculated as the appearance cost; If the cumulative appearance features of the existing trajectory are zero vectors, then a preset initial value is used as the appearance cost. The target position of the current frame image is predicted by the Kalman filter state parameters to obtain the predicted position, and the Mahalanobis distance between the predicted position and the coordinates of the effectively detected bounding box is calculated as the motion cost. If the historical category label sequence of the existing trajectory is not empty, then the latest target category label in the historical category label sequence is extracted as the historical category label; it is determined whether the effective detected target category label is consistent with the historical category label. If they are consistent, the category cost is set to a preset minimum value. If they are inconsistent, the category cost is set to a preset infinite value. If the historical category label sequence is empty, then the category cost is set to the preset minimum value; The matching cost is obtained by weighted summation of the appearance cost, the motion cost, and the category cost.

6. The method according to claim 4, characterized in that, In step S6, features are extracted from the corresponding regions of anchor frames in the time-series trajectory to obtain a set of key-value pairs. The identity embedding vector corresponding to the query vector of the frame to be corrected in the time-series trajectory is extracted to obtain the query vector to be corrected. The query vector to be corrected and the set of key-value pairs are then aggregated to obtain the corrected query vector, including: For each target trajectory included in the time-series trajectory, obtain the anchor point frame set and the frame set to be corrected corresponding to each target trajectory; For each anchor frame in the anchor frame set, the feature map corresponding to the anchor frame is extracted from the temporal enhancement feature cube; based on the bounding box coordinates of the effective detections associated with the target trajectory in the anchor frame, a rectangular region is cropped on the feature map with the bounding box center as the reference and the width and height extended by a preset ratio; the spatial position feature vectors within the rectangular region are flattened to obtain the local key vector and local value vector corresponding to the anchor frame, and local key-value pairs are constructed based on the local key vector and the local value vector; Spatial dimension concatenation is performed on the local key-value pairs of each anchor frame corresponding to the same target trajectory to obtain a global anchor key matrix and a global anchor value matrix. The global anchor key matrix and the global anchor value matrix are combined to obtain the set of key-value pairs corresponding to the target trajectory. For each frame to be corrected in the set of frames to be corrected, read the identity embedding vector corresponding to the query vector that matches the target trajectory from the initial detection result corresponding to the frame to be corrected, and use the identity embedding vector corresponding to the query vector that matches the target trajectory as the query vector to be corrected. Linear projection is performed on the query vector to be corrected to obtain the projected query vector. Layer normalization is performed on the projected query vector to obtain the normalized corrected query vector. Layer normalization is performed on the global anchor key matrix to obtain the normalized key matrix. Calculate the third attention weight between the normalized modified query vector and the normalized key matrix, and normalize the third attention weight using the softmax function to obtain the normalized attention weight; Based on the normalized attention weights, the global anchor value matrix is ​​weighted and summed to obtain aggregated features; the aggregated features are residually connected to the normalized corrected query vector to obtain residual connected features; the residual connected features are layer-normalized to obtain the corrected query vector.

7. An end-to-end video target detection system based on an attention mechanism, characterized in that, The system includes: The image frame sampling module is used to perform step 1, which includes: using an OpenCV-based adaptive keyframe extraction strategy to divide the input video and obtain multiple video segments; A spatial feature enhancement module is used to perform step 2, which includes: extracting features from each frame image in the video segment to obtain a multi-scale feature map; fusing and flattening the multi-scale feature map to obtain a spatial feature sequence; and enhancing the spatial feature sequence through self-attention calculation to obtain a spatial enhancement feature sequence corresponding to each frame image. The temporal feature modeling module is used to execute step 3, which includes: stacking the spatial enhancement feature sequences corresponding to each frame image in the frame dimension to generate a cross-frame feature cube; performing temporal enhancement on the cross-frame feature cube to obtain the temporal enhancement feature sequence corresponding to each frame image; and recombining the temporal enhancement feature sequences to obtain the temporal enhancement feature cube. A frame-specific query decoding module is used to execute step 4, which includes: initializing a cross-frame shared object query basis vector; generating a frame-specific query sequence based on the object query basis vector and the temporal position encoding corresponding to each frame image; for each frame image, using the features of the corresponding frame image in the temporal enhancement feature cube as keys and values, and the frame-specific query sequence of the corresponding frame image as the query, decoding is performed through a decoding network to generate an initial detection result containing the target class probability, bounding box coordinates, and identity embedding vector corresponding to each query vector in the frame-specific query sequence; The trajectory association and frame classification module is used to execute step 5, which includes: performing inter-frame association processing on the initial detection results corresponding to each frame image to obtain the temporal trajectory of the target, and dividing the frame images associated with each target trajectory in the temporal trajectory into anchor frames and frames to be corrected. The trajectory awareness correction module is used to execute step 6, which includes: extracting features of the corresponding region of the anchor frame in the time-series trajectory to obtain a key-value pair set, and extracting the identity embedding vector of the query vector corresponding to the frame to be corrected in the time-series trajectory to obtain the query vector to be corrected, aggregating the query vector to be corrected and the key-value pair set to obtain the correction query vector, and inputting the correction query vector into the decoding network to generate the correction detection result of the frame to be corrected; The iterative feature optimization module is used to execute step 7, which includes: taking the correction detection result of the frame to be corrected in the time-series trajectory and the initial detection result of the anchor frame as feedback, generating a feature mask based on the cross-frame feature cube, enhancing the cross-frame feature cube according to the feature mask to obtain an enhanced cross-frame feature cube, replacing the cross-frame feature cube obtained by stacking with the enhanced cross-frame feature cube, iteratively executing steps 3 to 6, and outputting the target detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images