Video target detection method and system based on hybrid Transform-Mama
By using a hybrid Transformer-Mamba architecture, combining a spatially adaptive deformable Transformer encoder and a temporally cascaded bidirectional Mamba encoder, the problem of balancing efficiency and accuracy in video object detection is solved, realizing a highly efficient video object detection method applicable to fields such as intelligent security, autonomous driving, and video content analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-24
AI Technical Summary
Existing video object detection methods suffer from efficiency bottlenecks when processing long video sequences. The Transformer architecture has high computational complexity, while the Mamba model has limited ability to learn contextual information and generalize, making it difficult to achieve a balance between accuracy and efficiency in video object detection.
A hybrid Transformer-Mamba architecture is adopted, which generates token feature sequences through a shared feature extractor. By combining a spatially adaptive deformable Transformer encoder and a temporally cascaded bidirectional Mamba encoder, efficient modeling of intra-frame long-range dependencies and cross-frame temporal dependencies is achieved. Furthermore, an entangled Mamba-Transformer decoder is used to enrich the instance-level contextual information of the target query.
It achieves an ideal balance between computational efficiency and detection accuracy, improving the real-time performance and robustness of video target detection, and is suitable for multiple industrial fields with high real-time and accuracy requirements.
Smart Images
Figure CN121725296A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of target detection, and in particular to a video target detection method and system based on a mixed Transformer-Mamba. BACKGROUND
[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.
[0003] As one of the core tasks in the field of computer vision, the core goal of video target detection is to realize real-time classification and accurate positioning of target objects in video sequences. This technology provides indispensable technical support for a series of key fields such as visual navigation, safe driving, intelligent monitoring, and human-computer interaction, and has high academic research value and industrial application prospect. Compared with the static image target detection task, the core advantage of video target detection lies in its ability to fully exploit and utilize the additional information contained in the temporal dimension of the video. These information not only includes the motion trajectory clues of the target objects, but also covers the consistent semantic features across frames. Through effective fusion of these temporal information, the representation dimension of the target features can be significantly enriched, thereby laying a foundation for achieving superior detection performance.
[0004] In recent years, with the rapid development of deep learning technology, the Transformer architecture has shown significant advantages in capturing long-range context dependencies due to its unique self-attention mechanism, which has driven breakthroughs in the field of video target detection. Existing video target detection methods based on Transformer usually integrate multiple spatio-temporal Transformer modules (wherein the spatial Transformer module can be selectively configured according to actual needs), to fully exploit attention-related information from the spatio-temporal domain of the video. This process effectively enhances the representation ability of the target object's appearance features, enabling related detection methods to perform excellently in various public datasets and practical application scenarios.
[0005] However, the inherent quadratic computational complexity of the Transformer architecture poses a serious efficiency bottleneck when processing long video sequences. When the number of video frames or feature map size increases, the computational load and memory occupancy will increase dramatically, making it difficult to meet the needs of real-time video detection scenarios. Although a series of efficient Transformer variant architectures such as MobileViT, TinyViT, etc. have been proposed in the industry, trying to alleviate the efficiency problem through simplifying the model structure and optimizing the attention calculation method, these solutions often need to sacrifice part of the long-range context modeling ability in exchange for efficiency improvement, resulting in a difficult ideal balance between detection accuracy and processing efficiency, limiting their application in long sequence video detection tasks.
[0006] To solve the efficiency problem of long sequence processing, a new state space model Mamba emerges as the times require, which gradually stands out as a potential alternative to Transformer. The core advantage of Mamba model lies in the innovative state space architecture design, which has linear computational complexity relative to sequence length. This feature enables it to effectively model cross-frame temporal dependencies while efficiently processing long video sequences, significantly reducing the computational cost and memory overhead of long sequence processing. With this advantage, Mamba has shown excellent performance in language modeling, long sequence classification and other fields, providing a new technical approach for long sequence data processing.
[0007] Although Mamba performs outstandingly in efficiency and scalability, its application potential in the specific field of video object detection has not been fully explored and verified. More importantly, compared with Transformer, Mamba has obvious limitations in learning the comprehensiveness of context information and multi-task generalization ability due to its fixed-size hidden state design, which makes it difficult to fully capture the complex context relationships between objects and backgrounds, objects and objects when applied to video object detection tasks alone, which may lead to decreased detection accuracy.
[0008] In summary, the two architectures of Transformer and Mamba show distinct complementary features in the task of video object detection: Transformer has strong long-range context modeling ability but is inefficient, while Mamba has efficient long sequence processing ability but is limited in context learning and generalization performance. In the current field of video object detection, there is no technical solution that can effectively integrate the advantages of both and avoid their respective defects. SUMMARY
[0009] To solve the technical problems in the background art, the present application provides a video object detection method and system based on hybrid Transformer-Mamba, which combines the powerful global attention modeling capability of Transformer and the advantage of Mamba in efficiently modeling cross-frame temporal dependencies in long sequences. Under the premise of overcoming the inherent limitations of both in context learning efficiency and computational complexity, a better balance between accuracy, robustness and real-time performance of video object detection is achieved.
[0010] To achieve the above purpose, the technical scheme adopted by the present application is as follows: The first aspect of the present application provides a video object detection method based on hybrid Transformer-Mamba.
[0011] A video object detection method based on hybrid Transformer-Mamba includes: Based on all frame images in the video to be detected, a shared feature extractor is used to generate a token feature sequence. The token feature sequence is fused with the positional encoding and then input into a spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames. The spatial encoder features of all frames are then concatenated and input into a temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The spatiotemporal encoder features of all frames are input into the target query entangled Mamba-Transformer decoder. Through query-feature interaction and fine-grained alignment, the instance-level context information of the target query is enriched to obtain the spatiotemporal decoder features. The spatiotemporal decoder features are input into a shared feedforward network for classification and bounding box regression to obtain the target detection results for each frame.
[0012] Furthermore, based on all frame images in the video to be detected, a shared feature extractor is used to generate token features; the method includes: based on all frame images in the video to be detected, a shared feature extractor is used to extract image features of each frame image in parallel; and each frame feature map is converted into a token feature sequence through projection and flattening layers.
[0013] Furthermore, the process of employing a spatially adaptive deformable Transformer encoder includes: After fusing the token feature sequence with the positional encoding, the result is input into a multi-scale adaptive deformable attention module to obtain the attention features. The multi-scale adaptive deformable attention module is represented by the following formula:
[0014]
[0015] in, Indicates the query element. The dimension is Query characteristics; Indicates the query element The normalized coordinates of the reference point, where and These represent the top left and bottom right corners of the frame image, respectively. Indicates the first Feature maps at various scales; , and These represent the index attention head, feature scale, and sampling point, respectively. Indicates the first The first scale and the first the attention weight of the i-th sampling point in the j-th attention head, and is normalized by and denote the learnable projection weights, where denote the sampling offset of the i-th sampling point in the j-th attention head and the k-th feature scale; and denote the total number of attention heads and feature scales, respectively; is the total number of sampling points of the query element denote the function of rounding to the nearest integer, denote the predefined minimum and maximum number of sampling points, respectively; The output of the multi-scale adaptive deformable attention module is passed through an element-wise addition and normalization operation, and then through a feedforward network to generate spatial encoder features.
[0016] Further, the spatial encoder features of all frames are spliced and input into a temporal cascaded bidirectional Mamba encoder to generate spatio-temporal encoder features of all frames; the method comprises: The spatial encoder features of all frames are subjected to layer normalization to obtain two information streams; one information stream is input into a temporal priority forward state space model after linear projection and one-dimensional convolution processing to capture forward long-range temporal dependencies, thereby obtaining first stream features; the other information stream is subjected to linear projection and activation function processing to obtain second stream features; the first stream features and the second stream features are combined by Hadamard product operation to generate forward feature representation; The same operation sequence is cascaded to sequentially perform layer normalization, linear projection and one-dimensional convolution on the forward feature representation, and then capture backward long-range temporal dependencies in the reverse spatial order through a temporal priority backward state space model, thereby finally outputting spatio-temporal encoder features.
[0017] Further, the temporal priority forward state space model is expressed by the following formula:
[0018]
[0019] wherein, denotes the spatial encoder feature; Indicates the characteristics of the first flow; This represents the forward feature representation; It represents the Hadamardi (or Hadama) stack; This is the activation function.
[0020] Furthermore, by integrating the spatiotemporal encoder features of all frames into the target query entangled Mamba-Transformer decoder, and through query-feature interaction and fine-grained alignment, the instance-level contextual information of the target query is enriched, resulting in spatiotemporal decoder features. The method includes: Multi-head self-attention layer processing is applied to the target query, enabling each target query to aggregate contextual information from other queries; The output of the multi-head self-attention layer and the spatiotemporal encoder features generated by the temporally concatenated bidirectional Mamba encoder are jointly input into the multi-scale adaptive deformable attention layer to enhance the target query by collecting contextual information from the encoder features. By utilizing a cascaded bidirectional Mamba layer, an informative long-range context in both the forward and backward directions is integrated to obtain a representation of the target query. The target query representation is input into the feedforward network to generate spatiotemporal decoder features.
[0021] A second aspect of the present invention provides a video target detection system based on the hybrid Transformer-Mamba.
[0022] A video object detection system based on the hybrid Transformer-Mamba includes: The encoding module is configured to: generate a token feature sequence based on all frame images in the video to be detected using a shared feature extractor; fuse the token feature sequence with the positional encoding and input it into a spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames; and concatenate the spatial encoder features of all frames and input them into a temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The decoding module is configured to: input the spatiotemporal encoder features of all frames into the target query entangled Mamba-Transformer decoder, enrich the instance-level context information of the target query through query-feature interaction and fine-grained alignment, and obtain the spatiotemporal decoder features; The classification module is configured to input the spatiotemporal decoder features into a shared feedforward network for classification and bounding box regression, thereby obtaining the target detection results for each frame.
[0023] A third aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video object detection method based on the hybrid Transformer-Mamba described in the first aspect above.
[0024] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the video object detection method based on the hybrid Transformer-Mamba described in the first aspect above.
[0025] The fifth aspect of the present invention provides a computer program product or computer program.
[0026] This invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the hybrid Transformer-Mamba-based video object detection method as described in the first aspect above.
[0027] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a high-efficiency hybrid video object detection model by integrating the Transformer and Mamba architectures, effectively overcoming the limitations of existing technologies. The Transformer excels at long-range context modeling and can achieve fine-grained intra-frame spatial feature aggregation; while the Mamba model possesses the advantage of linear computational complexity in handling long sequences, efficiently modeling cross-frame temporal dependencies. This invention cleverly utilizes the powerful spatial context awareness of the Transformer and the efficient temporal modeling capabilities of Mamba to achieve an ideal balance between computational efficiency and detection accuracy. Attached Figure Description
[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0029] Figure 1 This is a flowchart illustrating a video target detection method based on the hybrid Transformer-Mamba as shown in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another embodiment of the video target detection method based on the hybrid Transformer-Mamba as shown in this invention. Figure 3 This is an internal structure diagram of the adaptive deformable Transformer encoder shown in an embodiment of the present invention; Figure 4 This is an internal structure diagram of a timing-cascaded bidirectional Mamba encoder shown in an embodiment of the present invention; Figure 5 This is an internal structure diagram of the entangled Mamba-Transformer decoder shown in an embodiment of the present invention; Figure 6 This is a structural diagram of a video target detection system based on the hybrid Transformer-Mamba as shown in an embodiment of the present invention; Figure 7 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation
[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0031] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0033] As described in the background section, existing video object detection methods cannot simultaneously achieve both detection accuracy and processing efficiency. To address this issue, this invention provides a video object detection method and system based on the hybrid Transformer-Mamba architecture. The following describes the solution of this invention in detail through several embodiments.
[0034] Figure 1 This is a flowchart illustrating a video target detection method based on the hybrid Transformer-Mamba as shown in an embodiment of the present invention; see reference. Figure 1 The method includes: Based on all frame images in the video to be detected, a shared feature extractor is used to generate a token feature sequence. The token feature sequence is fused with the positional encoding and then input into a spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames. The spatial encoder features of all frames are then concatenated and input into a temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The spatiotemporal encoder features of all frames are input into the target query entangled Mamba-Transformer decoder. Through query-feature interaction and fine-grained alignment, the instance-level context information of the target query is enriched to obtain the spatiotemporal decoder features. The spatiotemporal decoder features are input into a shared feedforward network for classification and bounding box regression to obtain the target detection results for each frame.
[0035] This invention first develops a spatially adaptive deformable Transformer encoder to model intra-frame long-range dependencies and perform intra-frame feature aggregation, significantly enhancing the spatial feature representation of the target. Second, it introduces a temporally concatenated bidirectional Mamba encoder, capable of modeling inter-frame long-range dependencies and performing inter-frame feature aggregation with linear complexity, effectively improving the temporal feature representation of the target. Finally, it designs an entangled Mamba-Transformer decoder to fully explore the interaction between the target query and spatiotemporal features, achieving fine-grained query-feature alignment and injecting more instance-level target-related information into the target query. Extensive experiments were conducted on the ImageNet VID and EPIC-KITCHENS-55 datasets, and the results validate the effectiveness of the proposed model.
[0036] Figure 2 This is a flowchart illustrating another embodiment of the video object detection method based on the hybrid Transformer-Mamba method of the present invention; see also... Figure 2 The video target detection method based on the hybrid Transformer-Mamba includes: The video to be detected is acquired, and all frame images in the video are input into the trained target detection model in batches to obtain the target detection results of all frame images. The trained object detection model extracts feature maps for each frame of image using a shared feature extractor. After projection and flattening, it generates token features, which are then fused with positional encodings and input into a spatially adaptive deformable Transformer encoder to achieve intra-frame long-range dependency modeling and feature aggregation, outputting spatial encoder features. The spatial encoder features from multiple frames are concatenated and input into a temporally cascaded bidirectional Mamba encoder to achieve cross-frame long-range dependency modeling and inter-frame feature aggregation, generating spatiotemporal encoder features. The spatiotemporal encoder features and the object query are input together into an entangled Mamba-Transformer decoder to enrich the instance-level contextual information of the object query through query-feature interaction and fine-grained alignment. Finally, a shared feedforward network is used for classification and bounding box regression to obtain the object detection results for each frame.
[0037] In some embodiments, the training process of the trained object detection model includes: Construct a training set, which consists of video frame images with known target detection labels; The training set is input into the object detection model to train the model. Training is stopped when the loss function value of the model no longer decreases, or when the number of iterations reaches the set number, and the trained object detection model is obtained.
[0038] In some embodiments, the training set refers to: given a video dataset, the dataset contains several videos, and each video contains several video frames. The video target detection method of the present invention uses continuous... It takes frames (default setting is 25 frames) as input and outputs the detection results of all input frames at once.
[0039] In some embodiments, the trained object detection model includes: The shared feature extractor receives each frame of the image to be detected at its input end, extracts the image features of each frame in parallel, and outputs the feature map corresponding to each frame. Then, the feature map of each frame is converted into a token feature sequence through projection and flattening layers, and fused with the spatial location code provided by the location coding generation module to form a token input containing location information.
[0040] The spatially adaptive deformable Transformer encoder receives the aforementioned token feature sequence fused with position encoding at its input end. It is used to independently perform intra-frame long-range dependency modeling and spatial feature aggregation on each frame of the image, and outputs the spatial encoder features of each frame.
[0041] The temporally cascaded bidirectional Mamba encoder receives a temporal sequence at its input, which is composed of spatial encoder features from multiple frames spliced together in chronological order. This sequence is used to model long-range dependencies across frames with linear computational complexity, perform efficient inter-frame feature aggregation, and output encoder features that fuse spatiotemporal information.
[0042] The entangled Mamba-Transformer decoder receives spatiotemporal encoder features from a temporally cascaded bidirectional Mamba encoder and a target query vector generated by a deformable DETR at its input. This module deeply explores the interaction between the query and features by alternately using Mamba and Transformer mechanisms, achieving fine-grained query-feature alignment. This enriches the instance-level contextual information of the target query and outputs the final spatiotemporal decoder features. Finally, a shared feedforward network is used for classification and bounding box regression to obtain the target detection results for each frame.
[0043] In some embodiments, such as Figure 3 As shown, the spatially adaptive deformable Transformer encoder includes: The labeled features and corresponding location codes of each frame are first integrated through element-wise addition, and then input into a multi-scale adaptive deformable attention module (MADAttn) to effectively capture long-range spatial dependencies within each frame, while aggregating intra-frame attention features. The formula for MADAttn is defined as follows: (1) in, Indicates the query element. The dimension is The query characteristics. Indicates the query element The normalized coordinates of the reference point, where and These represent the top left and bottom right corners of the frame image, respectively. Indicates the first Feature maps at various scales. , and These represent the index attention head, feature scale, and sampling point, respectively. Indicates the first The first scale and the first The first one in the attention. The attention weights of each sampling point are, and through Normalize. and Represents the learnable projective weights, where . Normalized coordinates Rescale to the On the input feature map at each scale. Indicates the first The first scale and the first The first one in the attention. The sampling offset of each sampling point. Due to It is a decimal, calculated using bilinear interpolation. . and These represent the total number of attention heads and feature scales, respectively. It is the query element The total number of sampling points. For adaptive prediction. In query features A lightweight multilayer perceptron is used, and the sampling intensity score is obtained through the sigmoid function. The score is linearly mapped to discrete values through the following transformation: (2) in, A function that rounds to the nearest integer. and These represent the predefined minimum and maximum number of sampling points, respectively. For example... Figure 4 As shown, The output is processed through element-wise addition and normalization, and then a spatial encoder feature is generated through a simple feedforward network. .
[0044] In some embodiments, such as Figure 4 As shown, the time-cascaded bidirectional Mamba encoder includes: Spatial encoder features output by a spatial adaptive deformable Transformer encoder First, the data undergoes layer normalization and is then split into two information streams. The first stream is processed by linear projection and one-dimensional convolution before being input into a temporally prioritized forward state space model to capture long-range temporal dependencies. The second stream is processed by linear projection and activation functions. Subsequently, the Hadamard product of these two streams is combined to generate a forward feature representation. : (3) (4) in, It represents the Hadamah accumulation. The activation function is then used. Next, the same sequence of operations is cascaded: layer normalization, linear projection, and one-dimensional convolution. This is then processed by a temporally prioritized backward state-space model (TB-SSM) to capture the backward long-range temporal dependencies in the reverse spatial order, ultimately outputting the spatiotemporal encoder features. .
[0045] The core of the temporally cascaded bidirectional Mamba encoder is a novel time-first bidirectional scanning algorithm, comprising forward scanning (implemented by TF-SSM) and backward scanning (implemented by TB-SSM). This algorithm proceeds forward along the temporal dimension at fixed spatial coordinates: after all features at a certain fixed spatial coordinate have been processed, the scan moves to the next spatial coordinate, repeating until all features have been traversed. For ease of explanation, the spatial encoder features are denoted as... ,in The temporal order of the index frames, Index the spatial location of features within each frame. Forward scan from Beginning, in a fixed spatial location Advancing along the upper timeline: After scanning all frames at this spatial location, the forward scan moves to the next spatial location. ,scanning This process continues in this order-priority manner until all spatial locations are covered. In contrast, backward scanning starts from the last spatial location. The first frame Begin by proceeding chronologically: When spatial position After all frames have been scanned, the backward scan moves to Continue scanning in a time-first manner until the end. .
[0046] In some embodiments, such as Figure 5 As shown, the entangled Mamba-Transformer decoder includes: First, the target query is processed by a multi-head self-attention layer, enabling communication and interaction between queries, allowing each target query to aggregate contextual information from other queries. Then, the output of the multi-head self-attention layer and the spatiotemporal encoder features generated by the temporally concatenated bidirectional Mamba encoder are input into a multi-scale adaptive deformable attention layer, which enhances the target query by collecting contextual information from the encoder features. Subsequently, a concatenated bidirectional Mamba layer is used to further refine the representation of the target query by integrating informative long-range context from both the forward and backward directions. This process can be expressed as the following formula: (5) (6) (7) in Indicates the target query. For the reference point coordinate tensor, These are encoder features. This indicates a bullish self-attention strategy. For multi-scale adaptive deformable attention operations, their calculation is shown in Equation (1). This represents a cascaded bidirectional Mamba operation, its structure is as follows: Figure 5 As shown, calculation details are given in formulas (3)-(4). Output of CBi-Mamba The data is fed into a feedforward network to generate spatiotemporal decoder features. With the help of a carefully designed entangled Mamba-Transformer decoder, each target query can simultaneously leverage the advantages of attention and Mamba to extract rich, target-related instance-level contextual information from the encoder features.
[0047] The technology proposed in this invention has profound application value and broad socio-economic benefits in multiple industrial fields with high requirements for real-time performance and accuracy. For example, in the field of intelligent security and monitoring, this technology can achieve accurate and real-time detection and tracking of multiple targets and long-term activities in surveillance videos, significantly improving the early warning and handling capabilities of public safety incidents. In autonomous driving and intelligent transportation systems, this technology can provide more reliable and comprehensive target perception information to the decision-making system by efficiently analyzing the temporal dynamic scene around the vehicle, thereby significantly improving driving safety and traffic efficiency. In the fields of video content analysis and human-computer interaction, this technology can improve the understanding and recognition accuracy of actions and behaviors, promoting the upgrading of interactive experiences and the intelligentization of content production. In addition, in industrial scenarios such as industrial vision inspection and robot vision, this technology can also improve the robustness and efficiency of inspection of products or components on dynamic production lines. Overall, the application of this invention will strongly promote the comprehensive progress of related industries in terms of intelligence level, operational efficiency, system security, and user experience.
[0048] The above combination Figure 1 , Figure 2 The video target detection method based on hybrid Transformer-Mamba provided in the embodiments of the present invention has been described in detail. Next, the video target detection system based on hybrid Transformer-Mamba provided in the embodiments of the present invention will be described in conjunction with the accompanying drawings.
[0049] Figure 6 This is a schematic diagram of the structure of a video target detection system based on the hybrid Transformer-Mamba as shown in an embodiment of the present invention. (Refer to...) Figure 6 The system described in this invention includes: The encoding module is configured to: generate a token feature sequence based on all frame images in the video to be detected using a shared feature extractor; fuse the token feature sequence with the positional encoding and input it into a spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames; and concatenate the spatial encoder features of all frames and input them into a temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The decoding module is configured to: input the spatiotemporal encoder features of all frames into the target query entangled Mamba-Transformer decoder, enrich the instance-level context information of the target query through query-feature interaction and fine-grained alignment, and obtain the spatiotemporal decoder features; The classification module is configured to input the spatiotemporal decoder features into a shared feedforward network for classification and bounding box regression, thereby obtaining the target detection results for each frame.
[0050] In some embodiments, the encoding module is specifically configured to: extract image features of each frame image in parallel using a shared feature extractor based on all frame images in the video to be detected; and convert each frame feature map into a token feature sequence through a projection and flattening layer.
[0051] In some embodiments, the process of employing a spatially adaptive deformable Transformer encoder includes: fusing the token feature sequence with the positional encoding, and then inputting the fused sequence into a multi-scale adaptive deformable attention module to obtain attention features; the multi-scale adaptive deformable attention module is represented by the following formula:
[0052]
[0053] in, Indicates the query element. The dimension is Query characteristics; Indicates the query element The normalized coordinates of the reference point, where and These represent the top left and bottom right corners of the frame image, respectively. Indicates the first Feature maps at various scales; , and These represent the index attention head, feature scale, and sampling point, respectively. Indicates the first The first scale and the first The first one in the attention. The attention weights of each sampling point are, and through Perform normalization; and Represents the learnable projective weights, where ; Normalized coordinates Rescale to the On the input feature map at each scale; Indicates the first The first scale and the first The first one in the attention. Sampling offset of each sampling point; and These represent the total number of attention heads and feature scales, respectively. It is the query element The total number of sampling points; A function that rounds to the nearest integer. and These represent the predefined minimum and maximum number of sampling points, respectively; the output of the multi-scale adaptive deformable attention module is processed through element-wise addition and normalization, and then a feedforward network is used to generate spatial encoder features.
[0054] In some embodiments, the encoding module is further configured to: normalize the spatial encoder features of all frames to obtain two information streams; process one information stream through linear projection and one-dimensional convolution, and input it into a temporally prioritized forward state space model to capture forward long-range temporal dependencies, resulting in a first stream feature; process the other information stream through linear projection and activation function to obtain a second stream feature; combine the first and second stream features through a Hadamard product operation to generate a forward feature representation; cascade the same sequence of operations, performing layer normalization, linear projection, and one-dimensional convolution on the forward feature representation in sequence, and then capture backward long-range temporal dependencies in the reverse spatial order through a temporally prioritized backward state space model, finally outputting the spatiotemporal encoder features.
[0055] In some embodiments, the time-first forward state space model is represented by the following formula:
[0056]
[0057] in, Indicates spatial encoder features; Indicates the characteristics of the first flow; This represents the forward feature representation; It represents the Hadamardi (or Hadama) stack; This is the activation function.
[0058] In some embodiments, the decoding module is specifically configured to: process the target query with a multi-head self-attention layer, enabling each target query to aggregate contextual information from other queries; input the output of the multi-head self-attention layer and the spatiotemporal encoder features generated by the temporally concatenated bidirectional Mamba encoder into a multi-scale adaptive deformable attention layer, thereby enhancing the target query by collecting contextual information from the encoder features; utilize the concatenated bidirectional Mamba layer to obtain a target query representation by integrating informative long-range context in both the forward and backward directions; and input the target query representation into a feedforward network to generate spatiotemporal decoder features.
[0059] According to embodiments of the present invention, the hybrid Transformer-Mamba-based video target detection system can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the hybrid Transformer-Mamba-based video target detection system are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.
[0060] See Figure 7 The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement the corresponding steps in the embodiment of the video target detection method based on the hybrid Transformer-Mamba.
[0061] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.
[0062] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described embodiment of the video target detection method based on hybrid Transformer-Mamba.
[0063] This embodiment provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding steps in the above-described embodiment of the video object detection method based on Hybrid Transformer-Mamba.
[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0068] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A video target detection method based on hybrid Transformer-Mamba, characterized in that, include: Based on all frame images in the video to be detected, a shared feature extractor is used to generate a token feature sequence. The token feature sequence is fused with the positional encoding and then input into a spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames. The spatial encoder features of all frames are then concatenated and input into a temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The spatiotemporal encoder features of all frames are input into the target query entangled Mamba-Transformer decoder. Through query-feature interaction and fine-grained alignment, the instance-level context information of the target query is enriched to obtain the spatiotemporal decoder features. The spatiotemporal decoder features are input into a shared feedforward network for classification and bounding box regression to obtain the target detection results for each frame.
2. The video target detection method based on hybrid Transformer-Mamba according to claim 1, characterized in that, Based on all frame images in the video to be detected, a shared feature extractor is used to generate token features. The method includes: based on all frame images in the video to be detected, a shared feature extractor is used to extract the image features of each frame image in parallel; and the feature map of each frame is converted into a token feature sequence through projection and flattening layers.
3. The video target detection method based on hybrid Transformer-Mamba according to claim 1, characterized in that, The process of using a spatially adaptive deformable Transformer encoder includes: After fusing the token feature sequence with the positional encoding, the result is input into a multi-scale adaptive deformable attention module to obtain the attention features. The multi-scale adaptive deformable attention module is represented by the following formula: in, Indicates the query element. The dimension is Query characteristics; Indicates the query element The normalized coordinates of the reference point, where and These represent the top left and bottom right corners of the frame image, respectively. Indicates the first Feature maps at various scales; , and These represent the index attention head, feature scale, and sampling point, respectively. Indicates the first The first scale and the first The first one in the attention. The attention weights of each sampling point are, and through Perform normalization; and Represents the learnable projective weights, where ; Normalized coordinates Rescale to the On the input feature map at each scale; Indicates the first The first scale and the first The first one in the attention. Sampling offset of each sampling point; and These represent the total number of attention heads and feature scales, respectively. It is the query element The total number of sampling points; A function that rounds to the nearest integer. and These represent the predefined minimum and maximum number of sampling points, respectively. The output of the multi-scale adaptive deformable attention module is processed through element-wise addition and normalization, and then a feedforward network is used to generate spatial encoder features.
4. The video target detection method based on hybrid Transformer-Mamba according to claim 1, characterized in that, The spatial encoder features of all frames are concatenated and then input into a temporally cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames; the method includes: The spatial encoder features of all frames are normalized by layers to obtain two information streams. One information stream is processed by linear projection and one-dimensional convolution, and then input into the temporally prioritized forward state space model to capture forward long-range temporal dependencies, resulting in the first stream feature. The other information stream is processed by linear projection and activation function to obtain the second stream feature. The first and second stream features are combined by Hadamard product operation to generate the forward feature representation. The same sequence of operations is cascaded to perform layer normalization, linear projection and one-dimensional convolution on the forward feature representation in turn. Then, the backward long-range temporal dependencies in the reverse spatial order are captured by the time-first backward state space model, and finally the spatiotemporal encoder features are output.
5. The video target detection method based on hybrid Transformer-Mamba according to claim 4, characterized in that, The time-first forward state-space model is represented by the following formula: in, Indicates spatial encoder features; Indicates the characteristics of the first flow; This represents the forward feature representation; It represents the Hadamardi (or Hadama) stack; This is the activation function.
6. The video target detection method based on hybrid Transformer-Mamba according to claim 1, characterized in that, The spatiotemporal encoder features of all frames are input into the target query entangled Mamba-Transformer decoder. Through query-feature interaction and fine-grained alignment, the instance-level context information of the target query is enriched to obtain the spatiotemporal decoder features. The method includes: Multi-head self-attention layer processing is applied to the target query, enabling each target query to aggregate contextual information from other queries; The output of the multi-head self-attention layer and the spatiotemporal encoder features generated by the temporally concatenated bidirectional Mamba encoder are jointly input into the multi-scale adaptive deformable attention layer to enhance the target query by collecting contextual information from the encoder features. By utilizing a cascaded bidirectional Mamba layer, an informative long-range context in both the forward and backward directions is integrated to obtain a representation of the target query. The target query representation is input into the feedforward network to generate spatiotemporal decoder features.
7. A video target detection system based on hybrid Transformer-Mamba, characterized in that, include: The encoding module is configured to: generate a token feature sequence based on all frame images in the video to be detected using a shared feature extractor; fuse the token feature sequence with the positional encoding; and input the fused token feature sequence into the spatial adaptive deformable Transformer encoder to obtain the spatial encoder features of all frames. The spatial encoder features of all frames are concatenated and then input into the temporal cascaded bidirectional Mamba encoder to generate the spatiotemporal encoder features of all frames. The decoding module is configured to: enrich the instance-level context information of the target query by query-feature interaction and fine-grained alignment in the spatiotemporal encoder features of all frames and the target query input entangled Mamba-Transformer decoder to obtain spatiotemporal decoder features; The classification module is configured to input the spatiotemporal decoder features into a shared feedforward network for classification and bounding box regression, thereby obtaining the target detection results for each frame.
8. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the video target detection method based on hybrid Transformer-Mamba as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the video object detection method based on hybrid Transformer-Mamba as described in any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the video target detection method based on hybrid Transformer-Mamba as described in any one of claims 1-6.