Online video target detection method
By introducing causal space-time Mamba module and category-aware Memory Bank in the video object detection model, the problem of high complexity of video object detection and insufficient utilization of space-time information in the prior art is solved, and more efficient detection performance and more accurate target positioning are achieved.
Patent Information
- Application Number
- CN202411868233.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-22
AI Technical Summary
The existing video object detection methods are complex and have high computational cost when processing video frames, and cannot effectively use space-time information for joint enhancement, resulting in insufficient detection performance, especially in the case of background changes and target diversity, which is prone to detection errors or failures.
The video object detection model based on Faster R-CNN is adopted to add the causal space-time Mamba module (CSTMB) and the category-aware Memory Bank (CAMB) to perform feature enhancement at the frame level and candidate region level. CSTMB performs feature joint enhancement in the spatiotemporal dimension through causal time-first bidirectional 3D scanning, while CAMB performs generalized enhancement of candidate region features through class-aware memory database.
The combined enhancement of video frame features in the time and space dimensions is achieved, detection performance is improved, computing complexity and storage space occupation is reduced, and target positioning more accurately and adapting to background changes and target diversity.
Smart Images
Figure CN119863606A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to an online video target detection method. Background Art
[0002] As a key task in the field of computer vision, image object detection has made significant progress in recent years. However, in practical application scenarios such as video surveillance and vehicle assisted driving, the demand for video object detection is more extensive and urgent. However, due to the temporal continuity of video frames, the movement of the camera or the target will cause blurring of the picture, and the scale and appearance of the target in the video are diverse, and the background may also change dramatically. Direct single-frame detection based on the image object detection model is prone to detection errors or even failures.
[0003] Therefore, how to use the rich temporal information in the video to enhance the feature representation of the above-mentioned degraded frames to improve detection performance is a key research direction. Specifically, this type of model effectively aggregates the features of the support frames in the video into the feature representation of the current frame, thereby achieving feature enhancement of the current frame. The levels of feature enhancement can be divided into frame-level enhancement and candidate region-level enhancement. Regarding the source range of the support frames, there are neighboring frames and global frames. Existing methods often have the following implementations: Frame-level enhancement is based on optical flow, transformer integration modules or GCN to obtain spatiotemporal information. Candidate region-level enhancement is based on attention, and the samples used for enhancement often come from support frames or memory banks. And the support frames mostly contain neighboring frames from the future or extracted global frames.
[0004] However, the existing methods mainly have the following problems:
[0005] (1) The above frame-level enhancement methods based on optical flow and GCN have high complexity and computational cost, while the integrated module of transformer uses a spatiotemporal separation method to enhance the image without realizing spatiotemporal joint enhancement.
[0006] (2) For candidate region-level enhancement, samples from supporting frames require additional computation; existing memory banks cover a wider range of samples to ensure performance, which takes up too much space and does not consider the category balance and diversity of the samples.
[0007] (3) Using only single-level feature enhancement cannot more effectively improve detection performance.
[0008] (4) Using neighboring frames from the future or extracted global frames as support frames adds non-causal information, making it impossible to achieve online reasoning and further put it into practical application. Summary of the invention
[0009] The technical problem to be solved by the present invention is to provide an online video object detection method, which can solve the above technical problems.
[0010] The technical solution adopted by the present invention to solve the technical problem is: to provide an online video target detection method, comprising the following steps:
[0011] Get online video streaming;
[0012] Construct a video target detection model based on Faster R-CNN to perform target detection on the video stream frame by frame to obtain target detection results; the video target detection model includes:
[0013] A partitioning module, used for partitioning an input video frame into a plurality of image blocks;
[0014] A feature extraction network, used for performing multi-level feature extraction on each image block to obtain a plurality of first feature blocks;
[0015] A CSTMB module, configured to use the first feature block of a supporting frame to perform feature enhancement on the first feature block of a current frame at the same spatial position to obtain a second feature block, wherein the supporting frame is a set number of historical video frames sampled according to a set step length based on the current frame;
[0016] A region proposal network, configured to generate a candidate region feature according to the second feature block of the current frame;
[0017] The CAMB module is used to construct a candidate region feature memory library based on the target category and use the samples in the library to perform generalized enhancement operations on the candidate region features;
[0018] The prediction module is used to predict the target position and target category and the corresponding target classification confidence according to the enhanced candidate area features as the target detection result.
[0019] Furthermore, the using the first feature block of the supporting frame to perform feature enhancement on the first feature block of the current frame at the same spatial position includes:
[0020] Arranging the first feature blocks of the current frame and each of its supporting frames in forward time order to obtain a first feature block sequence;
[0021] After the first feature block sequence is processed by a normalization layer, it is projected into a feature block sequence x and a feature block sequence z through different linear layers respectively;
[0022] The feature block sequence x is passed through the first convolution layer and the first activation function in sequence to obtain the feature block sequence x f ', using the first S6 model to scan the feature block sequence x in a forward time order along a forward space orderf ', obtain the positive block features y of the current frame and its supporting frames f ;
[0023] The feature block sequence x is passed through the second convolution layer and the second activation function in sequence to obtain the feature block sequence x b ', using the second S6 model to scan the feature block sequence x in a forward time order and a reverse space order b ', get the reverse block features y of the current frame and its supporting frames b ;
[0024] The feature block sequence z is respectively combined with the forward block feature sequence y after passing through the third activation function f and the reverse block feature sequence y b Multiply, add the two multiplication results and then project them to the original dimension through a linear layer, and add the projected result to the first feature block sequence to obtain a second feature block sequence.
[0025] Furthermore, the first activation function, the second activation function, and the third activation function are all SiLU functions.
[0026] Furthermore, the memory library contains samples of different target categories, and the total number of samples of each target category does not exceed a set upper limit.
[0027] Furthermore, the memory bank is updated by the following method:
[0028] Selecting a set number of candidate region features with the highest confidence level of the target classification as preferred region features;
[0029] Performing weighted sums on the preferred area features having the same target category based on the target classification confidence to obtain a plurality of category candidate area samples;
[0030] The samples in the library corresponding to the target category are updated using the category candidate area samples.
[0031] Furthermore, the category candidate region feature is calculated by the following formula:
[0032]
[0033] Among them, for the current target category c, is the candidate region sample of the category, is the weight, is the target classification confidence, N c is the number of the preferred regional features belonging to category c, is a feature belonging to category c in the preferred area.
[0034] Furthermore, the updating of samples in the library corresponding to the target category by using the category candidate region samples includes:
[0035] If the total number of samples of the current category candidate area samples corresponding to the target category does not exceed the set upper limit, the candidate samples are put into the memory library, otherwise they are updated according to the similarity between the category candidate area samples and the samples in the library.
[0036] Furthermore, the updating according to the similarity between the category candidate region samples and the samples in the library includes:
[0037] Calculating the cosine similarity between the candidate region samples of the category and the samples in each library corresponding to the target category respectively;
[0038] The category candidate area samples are fused with the samples in the library with the highest cosine similarity and greater than the set threshold, and the obtained fused samples are used to replace the samples in the library. If the cosine similarities are not greater than the set threshold, the memory library is updated in a first-in-first-out manner.
[0039] Furthermore, the fusion sample is expressed as:
[0040]
[0041] in, is the candidate region sample of the category, is the fusion sample, is the sample in the library with the highest cosine similarity to the candidate sample and greater than the set threshold, w m is the weight of the samples in the library.
[0042] Beneficial Effects
[0043] Due to the adoption of the above technical scheme, the present invention has the following advantages and positive effects compared with the prior art: on the basis of FasterR-CNN, the present invention adds a causal spatiotemporal Mamba module (CSTMB) and a category-aware Memory Bank (CAMB) to perform frame-level enhancement and candidate region-level enhancement respectively, and the causal spatiotemporal Mamba module performs a causal time-priority bidirectional 3D scan on the input spatiotemporal feature blocks, and realizes the joint enhancement of video frame features in the time and space dimensions with a single module; the input spatiotemporal feature blocks are arranged as a sequence in positive order under the time and space axes, and the bidirectional 3D scanning is performed in the positive direction along the time axis in the time dimension, and the first and last spatial positions of the first frame (i.e., the first time position in the sequence) are used as the starting positions in the spatial dimension, respectively, in this way, the input feature block sequence is scanned in forward and reverse directions in the spatiotemporal dimension, and causal scanning of the same spatial position is realized, so that it can more accurately locate Objective: The Category-Aware MemoryBank (CAMB) stores enhanced candidate region feature samples by category, and uses the category candidate region features to update samples. Only a single category candidate region feature is updated category by category each time, thereby achieving information-rich and quantity-balanced sample storage, and making the memory bank occupy only a small amount of space while still having good performance. CAMB samples are obtained by weighted summing of high-confidence samples of the same category according to classification confidence, and the updating method is a combination of similarity weighted summing and first-in-first-out, which ensures the timeliness and diversity of stored samples. The present invention achieves better multi-level feature aggregation, and achieves online enhancement without using non-causal features, which meets the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a schematic diagram of the MMVOD architecture of an embodiment of the present invention;
[0045] Figure 2 Schematic diagram of scanning direction of CSTMB according to an embodiment of the present invention;
[0046] Figure 3 It is a schematic diagram of the CSTMB architecture of an implementation mode of the present invention. DETAILED DESCRIPTION
[0047] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0048] An embodiment of the present invention relates to an online video object detection method, comprising the following steps:
[0049] Get online video streaming;
[0050] A video object detection model (MMVOD) based on FasterR-CNN is constructed to perform object detection on the video stream frame by frame to obtain object detection results.
[0051] like Figure 1 As shown in the figure, the MMVOD model adds a causal spatiotemporal Mamba module (CSTMB) and a category-aware MemoryBank (CAMB) to the FasterR-CNN with SwinTransformer as the backbone network for frame-level enhancement and candidate region-level enhancement respectively.
[0052] Among them, CSTMB takes the image block feature tokens of the support frame and the current frame through the backbone network SwinTransformer as input, performs bidirectional 3D scanning on the input spatiotemporal features in a causal time-first manner, and realizes the joint spatiotemporal enhancement of features with a single module; CAMB takes the candidate region features generated by the enhanced current frame features through RPN as input, takes out all samples stored by category from the memory bank to enhance the input features, and then generates the final position and classification results from the enhanced candidate region features. According to the classification confidence, the highest number of samples and the corresponding classification scores are selected to generate category candidate region features, and the category candidate regions are used to update the memory bank category by category in a similarity fusion or first-in-first-out manner, thereby reducing the calculation of the candidate region features generated by the support frame, and the above update method ensures the category balance, diversity and timeliness of the samples, and saves richer support candidate region features with smaller memory usage, ensuring model performance.
[0053] The Causal Spatiotemporal Mamba Module (CSTMB) and the Category-Aware MemoryBank (CAMB) are further explained below.
[0054] 1. Causal Space-Time Mamba Module (CSTMB)
[0055] The input image contains the current frame I t and support frame I t-i ,I t-2i (where i is the sampling step of the supported frame), the total input I∈R T×C×H×W (T=3,C=3). After 2D Patch Embedding, the backbone network Swin Transformer extracts features and converts the output of each stage into (H s ,W s With C sThe feature map size and number of channels of each stage are respectively input into CSTMB to enhance the spatiotemporal features of the current frame, and finally only the enhanced features of each stage of the current frame are output.
[0056] 1. Enhanced spatiotemporal features
[0057] This method extends the bidirectional Mamba module to achieve 3D video frame-level spatiotemporal enhancement. In order to better explore the displacement of the target, a causal time priority method is used for scanning, such as Figure 2 For the convenience of explanation, the feature blocks of each frame are marked in sequence according to time and space, as shown in (t x ,s y ), where t x Indicates that the feature block belongs to time frame x, s y Represents that the feature block is located at the spatial position y in the above frame. The forward scan scans the feature blocks at the same spatial position in the forward time order, that is, starting from (t0, s0), then scanning (t1, s0), (t2, s0), and then continuing to scan (t0, s0), (t1, s0), (t2, s0), and so on to scan each spatial position. The reverse scan is also scanned in the forward time order, but the last spatial position of the first frame is used as the starting position (t0, s3), and finally scans to (t0, s0), (t1, s0), (t2, s0). This scanning method realizes causal scanning of the same spatial position, which is more helpful to accurately locate the target. For compatibility with SwinTransformer, no additional position embedding is added.
[0058] CSTMB architecture Figure 3 As shown, σ represents the nonlinear activation function, which is the SiLU function here. The specific calculation process is as follows: the input block sequence T first passes through the normalization layer, and then passes through different linear layers to project into sequences x and z, as shown in Formula 1.
[0059]
[0060] Then construct the forward and reverse of sequence x (the construction result of the reverse sequence is as described in the forward and reverse scans, that is, only flipping in the spatial dimension) sequence x f ,x b , and then calculate x′=SiLU(conv1d(x)), then x′ f ,x′ b Send it to the S6 model to get the corresponding output y f ,y bThe S6 model is constructed by incorporating selective scanning into the discrete state space model (SSM), which has the ability to perceive the input context and can achieve adaptive weight adjustment. The calculation process is shown in Formula 2. is the state transfer matrix, is the projection parameter and Δ is the time scale parameter.
[0061]
[0062] S6 output y is multiplied by SiLU(z) to obtain y′, and the reverse sequence y′ is converted to b The order is restored to the forward direction (i.e., it is flipped only in the spatial dimension again). The output y′ of the two branches f ,y′ b The addition is then projected to the original dimension through the linear layer, and finally added to the input sequence T to obtain the enhanced feature block sequence T′. This process is shown in Formula 3.
[0063] T′=linear T (y f ·SiLU(z)+y b ·SiLU(z))+T(3)
[0064] 2. Category-Aware Memory Bank (CAMB)
[0065] In order to enhance the candidate region features of the current frame (where N represents the number of candidate region features in the current frame), first obtain all samples in the memory bank (where N m is the total number of samples stored in the memory bank). Then, Q is enhanced with K by means of the generalized enhancement operation, and the enhanced result is returned. Finally, the category candidate region features used to update the memory bank are obtained from the enhanced candidate region features and the corresponding classification confidence, and are updated category by category in a similarity fusion combined with a first-in-first-out method.
[0066] 1. Feature enhancement of candidate regions
[0067] The candidate region feature enhancement uses the generalized enhancement operation in MAMBA (Multi-level Aggregation via Memory Bank for Video Object Detection) to perform two-level enhancement. Given the candidate region feature of the current frame and all samples in the memory The generalized enhancement operation is performed by calculating all samples k in K j The weighted summation obtains the relational features to enhance each q i Query sample q i The mth relation feature The calculation is shown in formula 4, where w ij is based on Calculate the similarity of the elements in the similarity matrix W, represents the linear transformation matrix, and M represents the number of relationship features.
[0068]
[0069] The calculation formula of the similarity matrix W is shown in Formula 5, where W Q and W K represents the linear transformation matrix, and d is the feature dimension.
[0070]
[0071] Then the generalized enhancement operation concatenates the relation features from M attention heads. Finally, the original feature q i and aggregated relational features f R (q i ,K) add to obtain enhanced features
[0072] The generalized enhancement operation can be performed level by level. Specifically, for the features sent to the next level Features enhanced by this level After a ReLu activation function and a fully connected layer, it is obtained, that is Two levels of enhancement are used in this method.
[0073] 2. Update by category
[0074] The memory is initialized to an empty list that can store background categories and the maximum number of each category is set to len. To reduce memory usage, only len=10 is used. The enhanced candidate region features are temporarily saved. Then, based on the final detection results, only the 75 enhanced features with the highest classification confidence are selected for sample update in the memory to ensure the high quality of the stored samples. These samples are classified, and the samples of the same category are weighted and summed according to the classification confidence to obtain the category candidate region features. As a candidate sample, the calculation formula is shown in Formula 6. Where c is the category to which the sample belongs, N c is the number of samples of the current category. In each update process, a fixed number (i.e., a single) of category candidate region features are stored in the memory bank to ensure a more balanced distribution of category samples. Updating can obtain features that contain richer information, allowing the memory bank to achieve good performance even with limited storage capacity.
[0075]
[0076] get Afterwards, if the number of samples of this category currently stored in the memory bank has not reached the upper limit, it will be directly stored. Otherwise, it needs to be updated by similarity fusion or first-in-first-out. Specifically, first calculate the category candidate region features The cosine similarity between all corresponding category samples in storage, if there is a sample with the highest similarity greater than the threshold 0.8 Update according to formula 7. The hyperparameter w m It is the weight of retaining the features of the previous samples during the fusion process, and its value is 0.4. If it does not exist, it will be updated in a first-in-first-out manner. Updating in this way ensures the timeliness of the stored samples while also ensuring their diversity.
[0077]
Claims
1. An online video object detection method, characterized in that: The following steps are involved: Get online video streaming; Construct a video target detection model based on Faster R-CNN to perform target detection on the video stream frame by frame to obtain target detection results; the video target detection model includes: A partitioning module, used for partitioning an input video frame into a plurality of image blocks; A feature extraction network, used for performing multi-level feature extraction on each image block to obtain a plurality of first feature blocks; A CSTMB module, configured to use the first feature block of a supporting frame to perform feature enhancement on the first feature block of a current frame at the same spatial position to obtain a second feature block, wherein the supporting frame is a set number of historical video frames sampled according to a set step length based on the current frame; A region proposal network, configured to generate a candidate region feature according to the second feature block of the current frame; The CAMB module is used to construct a candidate region feature memory library based on the target category and use the samples in the library to perform generalized enhancement operations on the candidate region features; The prediction module is used to predict the target position and target category and the corresponding target classification confidence according to the enhanced candidate area features as the target detection result.
2. The method according to claim 1, characterized in that The using the first feature block of the supporting frame to perform feature enhancement on the first feature block of the current frame at the same spatial position includes: Arranging the first feature blocks of the current frame and each of its supporting frames in forward time order to obtain a first feature block sequence; After the first feature block sequence is processed by a normalization layer, it is projected into a feature block sequence x and a feature block sequence z through different linear layers respectively; The feature block sequence x is passed through the first convolution layer and the first activation function in sequence to obtain the feature block sequence x f ', using the first S6 model to scan the feature block sequence x in a forward time order along a forward space order f ', obtain the positive block features y of the current frame and its supporting frames f ; The feature block sequence x is passed through the second convolution layer and the second activation function in sequence to obtain the feature block sequence x b ', using the second S6 model to scan the feature block sequence x in a forward time order and a reverse space order b ', get the reverse block features y of the current frame and its supporting frames b ; The feature block sequence z is respectively combined with the forward block feature sequence y after passing through the third activation function f and the reverse block feature sequence y b Multiply, add the two multiplication results and then project them to the original dimension through a linear layer, and add the projected result to the first feature block sequence to obtain a second feature block sequence.
3. The method according to claim 2, characterized in that The first activation function, the second activation function, and the third activation function are all SiLU functions.
4. The method according to claim 1, characterized in that The memory library contains samples of different target categories, and the total number of samples of each target category does not exceed a set upper limit.
5. The method according to claim 4, characterized in that The memory is updated by: Selecting a set number of candidate region features with the highest confidence level of the target classification as preferred region features; Performing weighted sums on the preferred area features having the same target category based on the target classification confidence to obtain a plurality of category candidate area samples; The samples in the library corresponding to the target category are updated using the category candidate area samples.
6. The method according to claim 5, characterized in that The category candidate region features are calculated by the following formula Among them, for the current target category c, is the candidate region sample of the category, is the weight, is the target classification confidence, N c is the number of the preferred regional features belonging to category c, is a feature belonging to category c in the preferred area.
7. The method according to claim 5, characterized in that The updating of samples in a library corresponding to the target category by using the category candidate region samples includes: If the total number of samples of the current category candidate area samples corresponding to the target category does not exceed the set upper limit, the candidate samples are put into the memory library, otherwise they are updated according to the similarity between the category candidate area samples and the samples in the library.
8. The method according to claim 7, characterized in that The updating according to the similarity between the candidate region samples of the category and the samples in the library includes: Calculating the cosine similarity between the candidate region samples of the category and the samples in each library corresponding to the target category respectively; The category candidate area samples are fused with the samples in the library with the highest cosine similarity and greater than the set threshold, and the obtained fused samples are used to replace the samples in the library. If the cosine similarities are not greater than the set threshold, the memory library is updated in a first-in-first-out manner.
9. The method according to claim 8, characterized in that The fusion sample is expressed as: in, is the candidate region sample of the category, is the fusion sample, is the sample in the library with the highest cosine similarity to the candidate sample and greater than the set threshold, w m is the weight of the samples in the library.