Video target detection method and system based on multi-scale perception diffusion

Abstract

The application provides a video target detection method and system based on multi-scale perception diffusion, and relates to the field of target detection. The method comprises the following steps: obtaining a plurality of frame images of a video to be detected; inputting the plurality of frame images into a trained target detection model, wherein the target detection model comprises a diffusion query module and a multi-scale perception module in parallel; inputting frame features and noise boxes into the diffusion query module to generate initial query features, and gradually refining the initial query features through an iterative optimizer at diffusion time steps to obtain diffusion query features related to semantics; extracting scale coding features based on scale branches and attention branches arranged in parallel by the multi-scale perception module; decoding the diffusion query features and the scale coding features based on a space-time Transformer decoder to obtain decoding features of the frame images; and identifying the decoding features to obtain target detection results of the frame images. Through the collaborative design of multiple modules, the accuracy and robustness of video target detection are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a video target detection method and system based on multi-scale perceptual diffusion. Background Technology

[0002] Video object detection is one of the core tasks in computer vision, aiming to locate and identify target objects in real time and accurately from continuous video frames. It is widely used in key scenarios such as intelligent surveillance, autonomous driving, and virtual reality, and its technical performance directly determines the reliability and intelligence level of downstream applications. With the development of deep learning, Transformer-based video object detection methods have become mainstream due to their powerful global feature modeling capabilities. By capturing the spatiotemporal correlation between frames, they effectively improve the robustness of detection in dynamic scenes.

[0003] While the Transformer architecture has made significant progress in this field in recent years, the detection requirements in complex real-world scenarios place higher demands on technical details. Current mainstream methods mostly focus on global dependency modeling, but neglect the optimization of key aspects such as target query initialization, scale adaptability, and dynamic noise suppression. As a result, detection accuracy is easily affected in scenes with rapid movement, variable scale, or complex backgrounds, making it difficult to meet the requirements of high-precision applications.

[0004] Specifically, existing technologies have three core shortcomings: First, the target query initialization stage lacks effective association with frame-level features, resulting in the initial query vector being disconnected from image semantics and affecting the efficiency of subsequent feature interaction; Second, the traditional Transformer attention mechanism lacks scale guidance and cannot specifically capture the dynamic changes in spatiotemporal features of targets at different scales, leading to poor detection performance for small targets or targets with abrupt scale changes; Third, in complex motion scenes, the dynamic noise distribution in the feature space is not effectively modeled, and background interference easily obscures key target features, causing detection box offsets or missed detections, thus restricting overall detection performance. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a video target detection method and system based on multi-scale perceptual diffusion. By combining a diffusion model with frame features to generate robust and diverse diffusion query features, and extracting scale-encoded features that fuse multi-scale information based on parallel processing, the two types of features are then decoded by a spatiotemporal Transformer decoder to fully exploit the spatiotemporal information of frame features and improve the accuracy of video target detection.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a video target detection method based on multi-scale perceptual diffusion, comprising: Acquire multiple frame images of the video to be detected; The multiple frame images are input into a trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module. The diffusion query module generates initial query features using frame features and noise boxes as input, and refines them step by step over the diffusion time step using an iterative optimizer to obtain semantically relevant diffusion query features. The multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches. The diffusion query features and scale-encoded features are decoded using a spatiotemporal Transformer decoder to obtain the decoded features of the frame images. The decoded features are identified to obtain the target detection results of the frame image.

[0007] Secondly, the present invention provides a video target detection system based on multi-scale perceptual diffusion, comprising: The video acquisition unit is configured to acquire multiple frame images of the video to be detected; The target detection unit is configured to input the multiple frame images into a trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module. The diffusion query module generates initial query features using frame features and noise boxes as input, and refines them step by step over the diffusion time step using an iterative optimizer to obtain semantically relevant diffusion query features. The multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches. The diffusion query features and scale-encoded features are decoded using a spatiotemporal Transformer decoder to obtain the decoded features of the frame images. The decoded features are identified to obtain the target detection results of the frame image.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video target detection method based on multi-scale perceptual diffusion described in the first aspect.

[0009] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the video target detection method based on multi-scale perceptual diffusion described in the first aspect.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention combines frame features with noise boxes through a diffusion query module, and refines the diffusion time step step gradually through an iterative optimizer to obtain semantically relevant diffusion query features. This effectively enhances the diversity and robustness of target queries, solving the problem of insufficient correlation between traditional initialization queries and frame-level features, making queries more closely match real target features. A multi-scale perception module extracts scale-encoded features using parallel scale and attention branches, fully integrating spatiotemporal features at different scales. This overcomes the lack of scale guidance in traditional Transformer attention, accurately capturing dynamic changes in targets at multiple scales. A spatiotemporal Transformer decoder decodes both types of features, deeply mining the spatiotemporal information of frame features. In complex motion scenes, this effectively suppresses dynamic noise in the feature space, reducing false negatives and false positives caused by background interference, ultimately improving detection accuracy and providing more reliable target detection support for intelligent monitoring, autonomous driving, and other scenarios.

[0011] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 A main flowchart of a video target detection method based on multi-scale perceptual diffusion provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a video target detection method based on multi-scale perceptual diffusion, provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the internal structure of the diffusion query module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the internal structure of the diffusion cascade module provided in an embodiment of the present invention; Figure 5 A schematic diagram of a multi-scale sensing module provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the internal structure of the spatiotemporal Transformer decoder provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Example 1 like Figure 1 As shown, this embodiment discloses a video target detection method based on multi-scale perceptual diffusion, including the following steps: S1: Acquire multiple frame images of the video to be detected; S2: Input the multiple frame images into the trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module; the diffusion query module generates initial query features using frame features and noise boxes as input, and refines them step by step over the diffusion time step through an iterative optimizer to obtain semantically relevant diffusion query features; the multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches; the diffusion query features and scale-encoded features are decoded based on a spatiotemporal Transformer decoder to obtain the decoded features of the frame images; S3: Identify the decoded features to obtain the target detection results of the frame image.

[0016] Next, combined Figure 2 This embodiment provides a detailed description of a video target detection method based on multi-scale perceptual diffusion.

[0017] The Transformer architecture has been widely adopted for video object detection tasks due to its powerful long-range modeling capabilities. Combining the Transformer structure with spatiotemporal modeling can effectively fuse features from multiple frames, achieving more robust object localization. However, existing methods still suffer from insufficient utilization of multi-level semantic information when aggregating spatiotemporal contextual information. Furthermore, targets in videos may exhibit inconsistent representations due to motion blur or lighting variations, impacting cross-frame detection performance.

[0018] In the field of generative modeling, diffusion models have achieved significant breakthroughs in recent years, particularly in image generation and feature modeling. By adding noise to the data and then progressively denoising and reconstructing it, this approach has been introduced into video object detection tasks to improve the stability, coherence, and diversity of queries.

[0019] To leverage the advantages of Transformer in modeling global dependencies while effectively overcoming problems in existing technologies such as unstable target representation, poor adaptability to scale changes, and insufficient cross-frame consistency modeling, this invention proposes a video target detection method, system, medium, and device based on multi-scale perceptual diffusion. This method aims to more fully and comprehensively aggregate spatiotemporal information, thereby improving the accuracy and robustness of video target detection.

[0020] First, a diffusion query module is proposed. This module generates target queries by combining a diffusion model with frame features and models temporal uncertainties through the diffusion process, thereby improving the diversity and robustness of target queries and effectively solving the problem of insufficient correlation between traditional query initialization methods and frame-level features. Second, a multi-scale perception module is designed. This module combines an attention mechanism with multi-head convolutional blocks with different kernel sizes, providing a different receptive field for each head. This design can capture and fuse multi-scale features across frames, which is crucial for detecting objects of different sizes and fast motion.

[0021] The target detection model proposed in this invention includes a backbone network group, a diffusion query module, a multi-scale perception module, a spatiotemporal Transformer decoder, and a feedforward network group. The video to be detected is input into the backbone network group for frame division, resulting in multiple frame images. The frame features are then input into the diffusion query module and the multi-scale perception module to extract diffusion query features and scale-encoded features, respectively. These features are fused in the spatiotemporal Transformer decoder to obtain decoded features, which are then input into the feedforward network in the feedforward network group to obtain the target detection result for each frame image.

[0022] I. Backbone Network Group The backbone network group consists of several backbone networks used to extract features from several frames of images. One backbone network corresponds to one frame of image, and the processed data yields the frame features for all frames.

[0023] As one implementation method, this embodiment uses ResNet-101 as the backbone network and Deformable DETR as the baseline network.

[0024] II. Diffuse Query Module The diffusion query module is a diffusion-driven adaptive query module that utilizes the ideas of the diffusion model, such as... Figure 3 As shown, the diffusion query module includes a feature locator and a diffusion cascade module connected in sequence. The feature locator extracts initial query features from frame features and noise boxes, and the diffusion cascade module extracts diffusion query features from the initial query features.

[0025] First, the diffusion model as a whole includes forward and reverse processes.

[0026] ① The forward diffusion process constructs a Markov chain by gradually adding noise, and Gaussian noise is added to the original data samples. The formula is as follows: (1) in, It is the diffusion time step. It is a fixed variance plan that controls the size of the step size. It is the cth Noise data of the step, It is the identity matrix, representing the covariance matrix of the noise. To directly obtain... get It can handle Gaussian noise. Perform sampling and calculation: (2) in During the training process, the neural network Trained to predict in a given noisy input and time step In cases where Gaussian noise is added, the following simplification objective is typically used: (3) ② In reverse reasoning, the diffusion model starts from pure Gaussian noise. ~ Begin by iteratively applying a learnable inverse process. To generate clean samples .

[0027] (1) Feature locator First, for a given backbone network, the extracted first... Frame features ,in, , and These represent the dimensions, height, and width of the frame feature, respectively.

[0028] Frame features and the first The data samples of the frame are initialized as a set of noise boxes generated based on Gaussian noise, which are then input into the feature locator of the diffusion query module to generate the initial query.

[0029] The feature locator is built based on a region feature extraction function, and the initial query... By from the noise box Internal feature extraction and generation, initialization method is: (4) in, , It is the number of target queries. It is the feature dimension. This represents the region feature extraction function. This represents the global average pooling operation for the extracted regional features.

[0030] Traditional methods of capturing query features based on frame features may result in a lack of diversity and randomness in the query features, making it difficult to effectively capture targets in complex scenes. Introducing noise can overcome this limitation, giving the initial query features more variability. In the iterative optimizer of the subsequent diffusion cascade module, the model gradually optimizes the initial query features using self-attention mechanisms, instance relationship interaction modules, etc., removing the adverse effects of noise while retaining its increased diversity. This allows the query features to better adapt to different targets and scenes, improving the accuracy of target detection.

[0031] (2) Diffusion cascade module This embodiment uses a diffusion cascade module to generate the target query required by the video target detection model. For example... Figure 4 As shown, the main idea is to gradually refine the initial query using an optimizer constructed from an iterative denoising network, aligning it more closely with the true target distribution. The resulting target query serves as a set of learnable vectors, which interact with frame features through a Transformer structure to obtain detection information for each target.

[0032] Since the initial queries originate from arbitrary regions, they typically lack rich semantics and location clues about the target. To compensate for this, this embodiment employs an iterative optimizer to progressively enrich their instance-level representations.

[0033] The iterative optimizer internally comprises a self-attention mechanism, a feature locator, an instance relationship interaction module, a scaling and offset module, and a decoder, all connected in sequence. The complete processing flow through one iterative optimizer is defined as one stage. Through multi-stage processing that progressively optimizes the initial query, the query is made closer to the true bounding box until a preset number of optimization stages (S stages) is reached.

[0034] Through multiple stages, the iterative optimizer updates each target query by aggregating contextual information from other target queries and focusing on the region within its own predicted bounding box, thereby achieving a richer representation. Specifically, within the same stage, the target queries do not operate in isolation. Taking a frame of an image at a certain stage as an example, assuming it corresponds to two target queries, "cat" and "car," the iterative optimizer allows the "cat" query to utilize a self-attention mechanism to acquire contextual information such as location and category association carried by the "car" query. This information acquisition is crucial, enabling the "cat" query to more accurately determine its relative position in the image, clarify its semantic attributes, and effectively avoid misjudgments caused by a lack of surrounding target references. Similarly, the "car" query also utilizes relevant information from the "cat" query. Based on this, the initial query, guided by the diffusion time step, is gradually refined through operations such as instance-level interactions. After multiple stages of iterative optimization, the query can gradually evolve from a state of pure noise to a representation consistent with the true target features, thereby enhancing its representational ability.

[0035] Specifically, in terms of form, within the self-attention mechanism, at the stage The calculation of multi-stage multi-head self-following applied to the initial query is as follows: (5) (6) in, Indicates the first Phase 1 The target query for the frame. Figure 4 middle, These are the initial query features. It refers to the number of heads of attention. , , , It is a learnable projection matrix. Attention weights are calculated using the scaled dot product between the projected query and the key; the output is a weighted sum of the projected values. Each attention head independently projects the input onto the dimension. , and The query, keywords, and value vectors. All header results are concatenated and then... The projection is applied to the final representation of the target query features after self-attention optimization.

[0036] Furthermore, the target query features optimized by self-attention will be combined with the original input features of the self-attention mechanism, i.e., the first... Phase 1 Frame target query Perform an element-wise additive residual join operation. This operation preserves the basic information in the original target query, avoids the loss of key features due to changes in the self-attention mechanism, and simultaneously overlays the contextual information optimized by self-attention to form the target query features after residual fusion.

[0037] Then, in the feature locator, within the predicted bounding box... and frame features The RoIAlign layer is applied to extract fine-grained region features.

[0038] To integrate region features into the target query representation, the instance relation interaction module employs a specialized operation called dynamic interaction. In this step, each target query focuses on region features extracted based on its previously predicted bounding boxes. These region features are modulated through parameters dynamically generated from each target query via a fully connected layer. Furthermore, the diffusion time step... Encoding is performed through linear transformation to generate time-dependent embeddings, which are used to normalize the target query and implement the multi-step reverse process in the diffusion model. To clearly and concisely represent the update process, this embodiment... The stage further refines the target query as follows: (7) in, express Phase 1 Frame target query, Indicates the first The iterative refinement operation of the stage, here This represents the target query features after residual fusion. It is the first Phase 1 Predicted bounding boxes for each frame, Indicates the first Frame features of a frame This indicates the current diffusion time step.

[0039] Subsequently, the output of the instance relationship interaction module, refined from the target query, and the target query features after residual fusion are subjected to an element-wise residual join operation. The resulting target query features after instance relationship interaction residual fusion are fed in parallel into the scaling and offset module and the next-stage iterative optimizer, for further optimization of spatial correction and instance representation, respectively. Scaling and offset dynamically update the query position and size at each iteration stage by regressing the translation and scale factor of the predicted bounding box using a multilayer perceptron. The features after scaling and offset operations are then used as input to a decoder equipped with fully connected layers for further processing. At the end of each iterative optimization stage, the final prediction head processes the optimized target query. To generate corresponding predictions, including prediction boxes. and classification scores Then, in the next optimization stage, the predicted boxes... Used as a spatial reference to extract new regional features. The final stage target query (represented as a diffuse target query). It is fed into the spacetime Transformer decoder module.

[0040] In this embodiment, in each round (stage s), the self-attention mechanism allows different target queries to communicate and share information; then, the feature locator combines the predicted bounding boxes and frame features to redetermine the region of interest; the instance relationship interaction module analyzes the relationships between different targets; scaling and offset adjust the features according to the diffusion time step; and the decoder outputs the optimized target query. After each round of optimization, predicted bounding boxes and classification scores are obtained, where the predicted bounding boxes serve as a reference for the next round of optimization. Finally, high-quality diffusion query features are obtained and sent to the spatiotemporal Transformer decoder module.

[0041] Among them, multi-module interaction enables the model to gradually delve deeper into the feature information of the target query, focusing on different levels of information at different stages, from preliminary feature extraction to detailed feature enhancement, continuously improving the understanding and representation of the target; residual connection avoids the loss of features during the transmission process, ensures the integrity of information, allows the model to effectively utilize the basic information of the initial query, and integrates the new information after optimization, prevents gradient vanishing, enhances the stability and learning ability of the model, and improves the overall detection performance.

[0042] III. Multi-scale sensing module The multi-scale sensing module is a multi-scale sensing Transformer encoder, such as... Figure 5 As shown, the multi-scale perception module includes two parallel branches: a scale branch and an attention branch. Frame features are processed through the scale branch and the attention branch respectively, and the processed features are input into the concatenated module in parallel. The output features of the concatenated module are fed into a 1×1 convolutional layer for dimensionality reduction, finally obtaining the scale-encoded features.

[0043] (1) Scale branch The scale branch consists of a scale fusion module, a feedforward network, and a flattening layer connected in sequence.

[0044] The scale fusion module includes a hybrid depthwise separable convolution submodule, a multi-scale feature aggregation submodule, an element-wise multiplication submodule, and a feature fusion submodule. The hybrid depthwise separable convolution submodule performs multi-scale convolution processing on the input features to extract spatial features at different scales; the multi-scale feature aggregation submodule performs cross-scale fusion of the spatial features to obtain fused features; the element-wise multiplication submodule performs element-wise multiplication of the input features and the fused features to obtain modulated features; and the feature fusion submodule adds the input features and the modulated features element-wise to obtain the first multi-scale feature.

[0045] The scale fusion module is used to explore feature dependencies at multiple scales. Frame features are input into the scale fusion module, first passed to a mapping layer consisting of linear layers, and then mixed with a depthwise separable convolutional submodule.

[0046] The mixed depthwise separable convolutional submodule first processes the input features Divided along the channel dimension A head, represented as For each head, use the kernel size The convolution is processed, where With each head increasing by 2 units, the depthwise separable convolutional submodule MixedC has the following formula: (8) in, Indicates that it is applied to a kernel size of The Size The feature depth can be separable convolution. This multi-head design allows each head to focus on spatial patterns at different scales, capturing fine-grained features and obtaining spatial features at different scales.

[0047] Next, the spatial features are input into the multi-scale feature aggregation submodule. This submodule selects one channel from each convolutional head to form a feature group, and uses an inverse bottleneck structure to fuse features within each group. This process effectively breaks down the granularity differences of features generated by multi-head hybrid convolutions and recombines them, thereby improving feature diversity. Simultaneously, the module employs an intelligent grouping strategy, achieving efficient feature aggregation with only a small computational overhead. The multi-scale feature aggregation submodule (MSA) can be formulated as follows: (9) (10) in, and The weight matrix is ​​the result of pointwise convolution. This represents the characteristics after a linear transformation. Indicates by the first The first in the head Spatial features obtained by applying depthwise convolution to each channel. , In and These represent the number of headers and groups, respectively. Indicates the first A depthwise convolutional head.

[0048] After multi-scale feature aggregation, the output processed features are denoted as... These features are used to dynamically adjust tensors through element-wise multiplication. .

[0049] For input features Output features of the scale fusion module Represented as: (11) (12) (13) in, This represents element-wise multiplication. and This is the weight matrix for the linear layer. Features It can adaptively adjust based on the input content, thus achieving dynamic modulation. Unlike computation... The attention matrix employs a self-attention mechanism, which preserves the channel dimension, enabling targeted modulation in both spatial and channel dimensions simultaneously after element-wise multiplication. This feature not only enhances feature representation capabilities but also exhibits higher memory efficiency when processing high-resolution images.

[0050] Finally, the original input features are processed by element-wise addition. Combined with aggregated features to obtain .

[0051] Subsequently, residual connections, feedforward networks (FFN), and flattening operations are applied to further refine and reshape the features. .

[0052] (2) Attention Branch The attention branch consists of a flattened layer, a multi-head attention mechanism, and a feedforward network connected in sequence.

[0053] For attention branches, this embodiment specifically employs multi-head self-attention blocks to enhance the modeling of remote dependencies.

[0054] Specifically, each block integrates a flattening operation, after which the multi-head attention (MHA) formula is formulated as follows: (14) (15) in, , Indicates the first The learnable weight matrix corresponding to each size. It is a linear transformation matrix that concatenates the outputs of all attention heads and then maps them back to the original feature dimensions. Residual connections and a feedforward network are then applied to obtain the second multi-scale feature.

[0055] Finally, the first and second multi-scale features are concatenated, input into the concatenation module, and then... Convolution is used to process the data to obtain the output of the final multi-scale perceptual module, i.e., scale-encoded features.

[0056] IV. Spatiotemporal Transformer Decoder The diffusion query features and scale-encoded features are input into the spatiotemporal Transformer decoder to obtain the decoded features.

[0057] like Figure 6 As shown, the spatiotemporal Transformer decoder includes Two parallel spatial branches and one temporal branch; first, the diffusion query features and scale-encoded features are input into... The system processes the features in parallel spatial branches, each branch consisting of a spatial self-attention network, a spatial deformable attention network, and a feedforward network connected in sequence. The processed features are then input into the temporal branches, which include a temporal deformable attention network and a feedforward network. Finally, the system outputs the decoded features.

[0058] Specifically, firstly, the diffused query features of each frame are input into the spatial self-attention mechanism to explore the dependencies between the query features, thus obtaining the spatial self-attention features. The calculation formula for the spatial self-attention mechanism (SpaSA) is as follows: (16) The meanings of each parameter are explained in Formula 6. Subsequently, the features processed by the spatial self-attention mechanism and the scale-encoded features are input into the spatial deformable attention mechanism. The purpose is to allow the target query to interact with the encoded features, obtaining spatial interactive fusion features. The calculation of the spatial deformable attention mechanism (SpaMA) is explained in Formulas 5 and 6.

[0059] The spatial interaction fusion features are input into the feedforward network, and linear transformation and nonlinear activation processing are performed to obtain the feedforward processed features.

[0060] In this module, the diffusion query features are processed separately first, allowing the model to focus on the dependencies within the diffusion query features and fully explore their semantic and structural information. Subsequently, scale-encoded features are integrated, which can accurately introduce information of the scale dimension, enabling the model to more comprehensively understand the target features at different scales, enhance the perception and encoding capabilities of multi-scale targets, improve the richness and accuracy of decoded features, and provide better feature support for subsequent detection tasks.

[0061] After multi-channel feedforward processing, the features are input into the temporal branch, where the temporally deformable attention mechanism is similar to the spatially deformable attention mechanism. Finally, the feedforward network outputs the final encoded features.

[0062] In this embodiment, after constructing the object detection model, the model is trained using a training set, where the images in the training set are video frame images labeled with object detection tags. The training set is then input into the object detection model to train it. Training is stopped when the loss function value of the model no longer decreases or the number of iterations reaches a set number, resulting in a well-trained object detection model.

[0063] The training set refers to a given video dataset containing several videos, each containing several video frames. The video target detection method in this embodiment uses continuous... Frame (default setting) The system takes one frame as input and outputs the detection results of all input frames at once.

[0064] The video target detection method based on multi-scale perceptual diffusion disclosed in this embodiment, through key technology improvements, effectively integrates features at different scales, achieving full extraction of spatial and temporal information from frame images and improving the accuracy of target detection. Its overall architecture diagram is shown below. Figure 1 As shown, given an input video sequence, each video frame is first processed by a backbone network with shared weights to extract corresponding frame features. Then, the frame features are input to a diffusion query module to extract diffusion query features, and to a multi-scale perception module to extract scale-encoded features. Next, the scale-encoded features and diffusion query features are input to a spatiotemporal Transformer decoder module for decoding to obtain decoded features. Finally, the decoded features of each frame are input to a feedforward network for recognition and localization, yielding the target detection result for each frame.

[0065] The video target detection method based on multi-scale perceptual diffusion disclosed in this embodiment has significant application value in fields such as intelligent surveillance, virtual reality, and autonomous driving. In intelligent surveillance, it can enhance dynamic target recognition capabilities; in virtual reality, it can optimize the interactive experience; and in autonomous driving, it can improve the reliability of environmental perception. Furthermore, this technology is also applicable to scenarios such as drone inspection and industrial robots, promoting the improvement of the accuracy and efficiency of intelligent applications.

[0066] Example 2 This embodiment provides a video target detection system based on multi-scale perceptual diffusion, including: The video acquisition unit is configured to acquire multiple frame images of the video to be detected; The target detection unit is configured to input the multiple frame images into a trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module. The diffusion query module generates initial query features using frame features and noise boxes as input, and refines them step by step over the diffusion time step using an iterative optimizer to obtain semantically relevant diffusion query features. The multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches. The diffusion query features and scale-encoded features are decoded using a spatiotemporal Transformer decoder to obtain the decoded features of the frame images. The decoded features are identified to obtain the target detection results of the frame image.

[0067] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a video target detection method based on multi-scale perceptual diffusion as described in Embodiment 1 above.

[0068] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the video target detection method based on multi-scale perceptual diffusion as described in Embodiment 1 above.

[0069] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video target detection method based on multi-scale perceptual diffusion, characterized in that, include: Acquire multiple frame images of the video to be detected; The multiple frame images are input into the trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module. The diffusion query module takes frame features and noise boxes as input to generate initial query features, and then refines them step by step over the diffusion time step through an iterative optimizer to obtain semantically related diffusion query features. The multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches; the spatiotemporal Transformer decoder decodes the diffusion query features and scale-encoded features to obtain the decoded features of the frame image; the diffusion query module includes a feature locator and a diffusion cascade module; the feature locator extracts initial query features from frame features and noise boxes based on a region feature extraction function, and the diffusion cascade module extracts diffusion query features from the initial query features; The diffusion process includes a forward diffusion process and a backward inference process. The backward inference process optimizes through multi-stage iterative steps, enabling the initial query features to aggregate contextual information from other target queries under the guidance of the diffusion time step and focus on the region within its own prediction box to gradually capture detailed features, thereby obtaining the target query enhanced diffusion query features. The decoded features are identified to obtain the target detection results of the frame image.

2. The video target detection method based on multi-scale perceptual diffusion as described in claim 1, characterized in that, Multiple frame images are extracted from the video to be detected using several backbone networks.

3. The video target detection method based on multi-scale perceptual diffusion as described in claim 1, characterized in that, The diffusion cascade module includes a self-attention mechanism, a feature locator, an instance relationship interaction module, a scaling and offset module, and a decoder connected in sequence. The first Phase 1 The target query input of the frame is processed by a self-attention mechanism to obtain the target query features optimized by self-attention. Among them, the first Phase 1 The target query of the frame includes the initial query features; The residual join operation, which involves adding elements of the target query features optimized by self-attention and the original target query, yields the target query features after residual fusion. The predicted bounding box and frame features are input into the feature locator to obtain the region features; Input the target query features after fusing regional features and residuals into the instance relationship interaction module to obtain a refined target query; The refined target query and the target query features after residual fusion are subjected to a residual join operation that adds elements together to obtain the target query features after instance relationship interaction residual fusion, along with parallel input scaling and offset modules and the iterative optimizer for the next stage. The query position and size are dynamically updated in each iteration stage by using the translation amount and scale factor of the multilayer perceptron regression prediction box in the scaling and offset module. The feature input, after scaling and offset operations, is equipped with a decoder in a fully connected layer; through multi-stage iterative optimization, the diffuse query features are obtained.

4. The video target detection method based on multi-scale perceptual diffusion as described in claim 1, characterized in that, The scale branch includes a scale fusion module, a feedforward network, and a flattening layer connected in sequence. In the scale fusion module, the input frame features are processed by multi-scale convolution to extract spatial features of different scales through the hybrid depth separable convolution submodule; the spatial features are fused across scales through the multi-scale feature aggregation submodule to obtain fused features; the frame features and the fused features are multiplied element-wise through the element-wise multiplication submodule to obtain modulation features; and the frame features and modulation features are added element-wise through the feature fusion submodule to obtain the first multi-scale feature.

5. The video target detection method based on multi-scale perceptual diffusion as described in claim 1, characterized in that, The spatiotemporal Transformer decoder includes A parallel spatial branch and a temporal branch connected to it; input the diffusion query features and scale-encoded features into... The spatial branches are processed in parallel. Each branch includes a spatial self-attention mechanism, a spatial deformable attention mechanism, and a feedforward network connected in sequence. The processed features are then input into the temporal branch, which contains a temporal deformable attention mechanism and a feedforward network, for further processing. Finally, the decoded features are obtained.

6. The video target detection method based on multi-scale perceptual diffusion as described in claim 5, characterized in that, The diffusion query features and scale encoding features are input into The process involves processing parallel spatial branches, each branch comprising a spatial self-attention mechanism, a spatial deformable attention mechanism, and a feedforward network connected sequentially. Specifically, it includes: The diffusion query features are input into the spatial self-attention mechanism to obtain spatial self-attention features; Spatial self-attention features and scale-encoded features are input into a spatial deformable attention mechanism to perform interaction between target query and encoded features, resulting in spatial interaction fusion features. The spatial interaction fusion features are input into the feedforward network, and linear transformation and nonlinear activation processing are performed to obtain the feedforward processed features.

7. A video target detection system based on multi-scale perceptual diffusion, characterized in that, include: The video acquisition unit is configured to acquire multiple frame images of the video to be detected; The target detection unit is configured to input the multiple frame images into a trained target detection model, which includes a parallel diffusion query module and a multi-scale perception module. The diffusion query module takes frame features and noise boxes as input to generate initial query features, and then refines them step by step over the diffusion time step through an iterative optimizer to obtain semantically related diffusion query features. The multi-scale perception module extracts scale-encoded features based on parallel scale branches and attention branches; the spatiotemporal Transformer decoder decodes the diffusion query features and scale-encoded features to obtain the decoded features of the frame image; the diffusion query module includes a feature locator and a diffusion cascade module; the feature locator extracts initial query features from frame features and noise boxes based on a region feature extraction function, and the diffusion cascade module extracts diffusion query features from the initial query features; The diffusion process includes a forward diffusion process and a backward inference process. The backward inference process optimizes through multi-stage iterative steps, enabling the initial query features to aggregate contextual information from other target queries under the guidance of the diffusion time step and focus on the region within its own prediction box to gradually capture detailed features, thereby obtaining the target query enhanced diffusion query features. The decoded features are identified to obtain the target detection results of the frame image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the video target detection method based on multi-scale perceptual diffusion as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the video target detection method based on multi-scale perceptual diffusion as described in any one of claims 1-6.

Images