Unmanned aerial vehicle video target detection method and system based on space-time correlation

Abstract

The invention discloses an unmanned aerial vehicle video target detection method and system based on space-time correlation, and the method comprises the steps: S01, inputting the video image data of an unmanned aerial vehicle into a feature extraction network, and obtaining an initial feature sequence; s02, embedding the initial feature sequence into a spatial position, performing deep convolution processing, performing space-time modeling by using a state-space equation, and fusing with the proportionally adjusted initial feature through residual connection to form a space-time feature; s03, inputting the spatial-temporal features into an attention network with residual connection, setting different weights according to appearance similarity, and fusing the spatial-temporal features through the residual connection; s04, performing feature fusion on the initial features and the spatio-temporal features after foreground attention processing; and S05, inputting the space-time fusion feature to a detection head to obtain a target detection result and outputting the target detection result. The method has the advantages of being simple in implementation method, few in needed computing resources, high in real-time performance and detection precision, high in expandability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle visual target detection, and in particular to a method and system for unmanned aerial vehicle video target detection based on spatiotemporal correlation. Background Art

[0002] Unmanned aerial vehicles (UAVs) can perform a variety of tasks such as searching for targets, reconnaissance and surveillance, and are widely used in aerial photography and surveying, disaster monitoring, power inspection and other fields. Due to the instability of UAV platforms and the complexity of the environment, higher requirements are placed on the perception capabilities of UAVs during mission execution. UAV video target detection is to identify and track targets in the video through video data captured by the camera onboard the UAV. However, unlike traditional target detection problems, due to the movement of the UAV itself and the complexity of the working environment, the difficulty of target detection tasks based on UAV platforms will increase exponentially, making UAV video target detection face huge challenges.

[0003] Traditional target detection usually uses a target detection network based on a single-frame image to detect the target, that is, taking a single-frame image as the target, the single-frame image is input into the target detection network to detect the target information. However, unlike a single-frame image, a video contains various appearance features of the target under the influence of different environments and rich motion information. There may also be drastic environmental changes between different video frames. The target detection network based on a single-frame image is difficult to detect targets with similar appearance and small targets in complex environments, and it is also difficult to accurately detect targets with serious appearance degradation. It is also difficult to balance computing resources, real-time performance, detection accuracy, detection speed and robustness. It is not suitable for UAV platforms, and there are also problems such as long implementation cycle, poor network generalization performance, and insufficient scalability. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a drone video target detection method and system based on spatiotemporal correlation, which has a simple implementation method, requires few computing resources, has high real-time performance and detection accuracy, and has strong scalability.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for detecting target in drone video based on spatiotemporal association, comprising the following steps:

[0007] Step S01. Input the continuous video image data detected by the camera carried by the drone into the feature extraction network for feature extraction to obtain an initial feature sequence output;

[0008] Step S02. After embedding the initial feature sequence into a spatial position and performing deep convolution processing, the features of each frame are spatiotemporally modeled using a state space equation to learn the spatiotemporal correlation between features of different frames, and then fused with the initial features adjusted by a specified ratio through a residual connection to form a spatiotemporal feature that retains both temporal and spatial information;

[0009] Step S03. Input the spatiotemporal features into an attention network with residual connections, and the attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing;

[0010] Step S04. Fusing the initial features with the spatiotemporal features after foreground attention processing in two dimensions, time and space, to obtain spatiotemporal fusion features;

[0011] Step S05: Input the spatiotemporal fusion features into the detection head, and the detection head performs target detection according to the spatiotemporal fusion features to obtain a target detection result output.

[0012] Further, step S02 includes:

[0013] Step S201. After embedding the initial feature sequence into a spatial position, perform a deep convolution process to obtain a result after the deep convolution;

[0014] Step S202: After the result of the deep convolution passes through a linear layer, the state space equation is used to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames;

[0015] Step S203: multiply the result of the features learned using the state space equation after layer normalization with the result of the initial feature sequence after passing through the linear layer;

[0016] Step S204: superimpose the multiplication result of step S203 and the result of the initial feature sequence adjusted by a specified ratio, and obtain the final extracted spatiotemporal features after layer normalization of the superimposed result.

[0017] Furthermore, the state space equation is used to perform spatiotemporal modeling of each frame feature according to the following formula:

[0018]

[0019] y t =Ch t

[0020]

[0021] Where x(t) represents the input image sequence, h(t) represents the intermediate potential state, y(t) represents the feature sequence obtained after mapping, A, B, C are trainable parameter matrices, and Δ represents the sampling time scale;

[0022] Sampling global convolution operations to achieve parallel computing:

[0023]

[0024] Among them, ⊙ represents the convolution operation, and K represents the size of the convolution kernel.

[0025] Furthermore, in step S02, the spatiotemporal feature map c′ after foreground attention processing is obtained according to the following formula: i :

[0026]

[0027] Among them, c i represents the spatiotemporal features extracted in step S01, f i represents the i-th feature in the initial feature sequence, t represents the time interval between different frames and the current frame, ⊙ represents the cosine similarity measure, and × represents matrix multiplication.

[0028] Furthermore, in step S04, the initial features are fused with the spatiotemporal features after foreground attention processing in both time and space dimensions by using a deep separable convolution to obtain a fused feature, and the fused feature is connected to the initial feature using a residual connection to retain the initial feature, and finally the spatiotemporal fused feature is obtained. The deep separable convolution mainly includes two processes, namely, deep convolution and point convolution. First, the deep convolution performs convolution on each input channel independently, and then the output of the deep convolution is combined by point-by-point convolution.

[0029] Furthermore, the expression of fused features obtained by using depthwise separable convolution for feature fusion is:

[0030] f i fuse =PWConv(DWConv(Concat(c′ i , f i )))+f i

[0031] Among them, DWConv is depth convolution, PWConv is point convolution, Concat is connection according to channel dimension, c i represents the spatiotemporal feature of the i-th foreground after attention processing, f i represents the i-th feature in the initial feature sequence, f i fuse represents the i-th fusion feature.

[0032] Furthermore, the detection head uses a deformable convolution to perform a convolution operation, and the deformable convolution adaptively calculates the offset Δp of the convolution kernel by using a convolution layer for calculating an offset and a convolution layer for calculating a weight value. n and the corresponding weight Δm n , according to the convolution kernel offset Δp n and the corresponding weight Δm n Calculate the pixel at each position, p 0 The pixel y at t-l→t (p 0 ) is calculated as:

[0033]

[0034] Among them, N represents the size of the convolution kernel, Yes 0 +p i The corresponding weight at the position, p 0 represents the original pixel position, p i represents the fixed convolution kernel displacement, Δp n represents the offset value calculated by the convolutional network, tI->t represents the distance from the tlth frame to the tth frame, x(p 0 +p i +Δp n ) represents the input image p 0 +p i +Δp n The pixel at position, tl→t represents the transition from the tlth frame to the tth frame.

[0035] A UAV video target detection system based on spatiotemporal correlation, comprising:

[0036] The feature extraction network is used to input the continuous video image data detected by the camera carried by the drone for feature extraction to obtain the initial feature sequence;

[0037] A spatiotemporal feature extraction module is used to embed the initial feature sequence into a spatial position and perform deep convolution processing, then use the state space equation to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames, and then fuse them with the initial features adjusted by a specified ratio through a residual connection to form a spatiotemporal feature that retains both temporal and spatial information;

[0038] A foreground attention module is used to input the spatiotemporal features into an attention network with residual connections, and the attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing;

[0039] A spatiotemporal fusion module, used to fuse the initial features with the spatiotemporal features after foreground attention processing in two dimensions of time and space to obtain spatiotemporal fusion features;

[0040] The target detection module is used to input the spatiotemporal fusion features into the detection head, and the detection head performs target detection according to the spatiotemporal fusion features to obtain a target detection result output.

[0041] Furthermore, the spatiotemporal feature extraction module includes a position embedding unit, a deep convolution unit, a state-space equation unit, a first ratio adjustment unit, two linear layers, and two normalization layers. The position embedding unit inputs an initial feature sequence extracted by the feature extraction network, and after embedding the spatial position, the deep convolution unit performs deep convolution. After the result of the deep convolution passes through the first linear layer, the state-space equation unit uses the state-space equation to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames; the features output by the state-space equation unit pass through the first normalization layer and are multiplied by the result of the initial features after passing through the second linear layer, and the multiplied result and the initial features are superimposed after the result adjusted according to the specified ratio by the ratio adjustment unit, and the superimposed result passes through the second normalization layer to obtain the final extracted feature map retaining the time and space information.

[0042] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1. The present invention uses a continuous video data sequence as network input, uses a feature extraction network to extract basic features respectively to form an initial feature sequence, then embeds the initial feature sequence into a spatial position and performs deep convolution processing, and uses state-space equations to model the spatiotemporal features in the video. It can effectively aggregate spatiotemporal features from the video, enabling the network to robustly and efficiently cope with perception tasks in complex environments. At the same time, the spatiotemporal features are input into an attention network with residual connections. The attention mechanism based on appearance similarity measurement can effectively suppress background interference introduced from other frames in the process of spatiotemporal modeling, which is conducive to the network paying more attention to areas related to foreground targets. The residual connection can also realize the fusion of appearance features and motion features, making full use of the rich feature information in the video to enhance the effect of target detection, so that both detection speed and accuracy can be taken into account.

[0045] 2. The present invention constructs a lightweight video target detection network that supports cross-view information modeling, which can be applied to UAV video target detection, taking into account computing resources, real-time performance, detection accuracy, detection speed and robustness, and effectively solving the problems faced by UAV platforms in perception tasks, such as target appearance degradation, drastic environmental changes, and computing resource limitations. It can not only accurately detect targets with severe appearance degradation, but also accurately detect targets and small targets with similar appearance, thereby improving the robustness to environmental changes and meeting the real-time requirements of UAV platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the structure principle of traditional target detection networks based on single-frame images and video images.

[0047] Figure 2 Schematic diagram of the structure of the drone video target detection system based on spatiotemporal association in this embodiment.

[0048] Figure 3 It is a schematic diagram of the principle of using depthwise separable convolution for feature fusion in this embodiment.

[0049] Figure 4 Schematic diagram of the principle of embedding deformable convolution in the detection head in this embodiment. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0051] For ease of understanding, the relevant technical background of the present invention is first introduced by way of example.

[0052] Traditionally, the target detection network structure based on a single frame image is used. Figure 1 As shown in (a). The network consists of two parts: the backbone and the detection head. The backbone extracts high-level feature representations with semantic information from the raw data by stacking multiple convolutional layers. These features can capture the local structure and global context of the input data, providing better feature information for subsequent tasks. The detection head usually includes one or more fully connected layers, which are usually connected to the output of the backbone, and use the features extracted by the backbone for final classification and regression of the detection box position. After the target is detected, some post-processing steps, such as non-maximum suppression (NMS), are required to eliminate overlapping bounding boxes and improve the accuracy of the detection results.

[0053] The network structure of video-based object detection is as follows: Figure 1As shown in (b). The network consists of a backbone, a neck, and a detection head. The backbone and the detection head are consistent with the target detection network based on a single frame image. The neck is located between the backbone and the detection head. It is usually composed of an attention network with strong global context learning capabilities or a recurrent neural network with temporal information learning capabilities to model the motion information in the video and aggregate it into the appearance features of the current frame, thereby improving the target detection effect of the current frame. In order to enable the video-based detection network to meet the real-time requirements of the drone platform while having the ability to cope with drastic changes in the environment and changes in target deformation and scale caused by drone movement, it is necessary to design a neck network with cross-view global context modeling and accurate local learning capabilities.

[0054] The target detection network structure based on a single-frame image is difficult to accurately detect targets with severe appearance degradation. Since the target detection network based on a single-frame image usually mines more effective features from the image by constructing a deeper network and fusing multi-dimensional features, however, due to the limited information provided by a single-frame image, severe deformation and occlusion make it difficult for the network to learn effective features, resulting in poor performance of this method in detecting targets with severe appearance degradation in complex environments.

[0055] If the target detection network based on a single-frame image is directly extended to the field of video target detection to enhance the target detection effect by utilizing the rich feature information in the video, it will be difficult to balance the computing resources, real-time performance, detection accuracy, detection speed, and robustness. For example, in the prior art, a video target detection algorithm FGFA (Flow-guided Feature Aggregation) models the motion deviation between multiple frames through optical flow, and uses a bilinear interpolation method to align the target features of adjacent frames to the corresponding spatial position of the target features of the current frame. However, due to the movement of the drone itself and the complexity of the environment, the above algorithm requires a lot of computing resources to obtain accurate optical flow. Another way is to use video sequences to provide auxiliary information, that is, to use video sequences as network input, design a neural network with cross-view modeling capabilities, and make predictions based on multi-frame fusion features. For example, a recurrent convolutional neural network (LSTM) is used to realize the transmission and interaction of features in video sequences. However, directly aggregating multi-frame features will introduce a lot of background interference, and due to the inherent limitations of LSTM, it will be difficult to model long-term feature associations.

[0056] In addition, the use of self-attention mechanism to establish associations between different positions can capture dependencies in the sequence. For example, the neural network model based on the Transformer architecture introduces the Transformer self-attention mechanism, which enables the network to learn the global context information in the image and establish associations between different positions, thereby better understanding the image content. However, when processing image data, the traditional Transformer model needs to divide the image into small blocks and process it due to the large size of the image, which will result in large computational and memory overhead. The amount of computation will increase exponentially with the increase in the scale of the input image, making it difficult to meet the real-time requirements of high-speed movement of drones.

[0057] A system architecture to which the present invention is applicable is as follows Figure 2 As shown in the figure, the overall structure is divided into five parts: feature extraction network, spatiotemporal extraction module, foreground attention module, spatiotemporal fusion module and detection head, which can form a plug-and-play cross-view spatiotemporal association network. Using the above architecture, the feature extraction network inputs the continuous video image data detected by the camera carried by the drone for feature extraction to obtain the initial feature sequence. The spatiotemporal feature extraction module embeds the initial feature sequence into the spatial position and performs deep convolution processing. The state space equation is used to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames. The learned features are fused with the initial features adjusted by the specified ratio through residual connections to form spatiotemporal features that retain both temporal and spatial information. The foreground attention module inputs the spatiotemporal features into the attention network with residual connections. The attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing. The spatiotemporal fusion module fuses the initial features with the spatiotemporal features after foreground attention processing in both time and space dimensions to obtain spatiotemporal fusion features. Finally, the spatiotemporal fusion features are input into the detection head, and the detection head performs target detection based on the spatiotemporal fusion features to obtain the target detection result output.

[0058] The above-mentioned system of the present invention can construct a lightweight video target detection network that supports cross-view information modeling. By using a continuous video data sequence as the network input, a feature extraction network is first used to extract the common features or basic features of each frame to form an initial feature sequence, and then the initial feature sequence is embedded in the spatial position and then deep convolution processing is performed. The state-space equation is used to model the spatiotemporal features in the video, which can effectively aggregate the spatiotemporal features from the video, enabling the network to robustly and efficiently cope with perception tasks in complex environments. The spatiotemporal features are then input into an attention network with residual connections. The attention mechanism based on appearance similarity measurement can effectively suppress background interference introduced from other frames during the spatiotemporal modeling process, which is conducive to the network paying more attention to areas related to foreground targets. The residual connection can also realize the fusion of appearance features and motion features, which can effectively fuse features of different modalities and make full use of the rich feature information in the video to enhance the effect of target detection, so that both detection speed and accuracy can be taken into account.

[0059] See also Figure 2 In (a), the spatiotemporal feature extraction module in this embodiment includes a position embedding unit, a deep convolution unit, a state-space equation unit, a first ratio adjustment unit, two linear layers, and two normalization layers. The position embedding unit inputs the initial feature sequence extracted by the feature extraction network, and after embedding the spatial position, the deep convolution unit performs deep convolution. After the deep convolution result passes through the first linear layer, the state-space equation unit uses the state-space equation to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames; the features output by the state-space equation unit pass through the first normalization layer and are multiplied with the results of the initial features after passing through the second linear layer, and the multiplication result and the initial features are superimposed after the first ratio adjustment unit adjusts the results according to the specified ratio, and the superimposed results pass through the second normalization layer to obtain the final extracted feature map retaining the time and space information. This embodiment adopts the above structure and constructs a spatiotemporal feature extraction module with linear computational complexity according to the state-space equation. The spatiotemporal features are aggregated from different image frames using a neural network module with linear complexity. This can make full use of the rich motion and appearance information in the video. While maintaining the linear computational complexity, it provides the network with global perception across views, which can effectively improve the applicability of the network under conditions of limited computing resources.

[0060] See also Figure 2In (b), the foreground attention module in this embodiment includes a similarity measurement unit, a matrix multiplication unit, a residual connection unit, a second ratio adjustment unit and a layer normalization layer. The similarity measurement unit receives the initial features extracted by the feature extraction network and the spatiotemporal features extracted by the spatiotemporal feature extraction module to calculate the similarity measurement. The matrix multiplication unit receives the output results of the foreground embedding and the similarity measurement unit to perform matrix multiplication operations. The residual connection unit receives the output results of the matrix multiplication unit and the spatiotemporal features adjusted by the second ratio adjustment unit according to the specified ratio, and fuses them by residual connection. The fused results are layer normalized to obtain the spatiotemporal features after foreground attention processing. The foreground attention module in this embodiment adopts an attention network with residual connections. The network can pay more attention to the area related to the foreground target, suppress background interference, and use a deep residual network to fuse the appearance features and motion features to improve the network's perception ability in complex environments.

[0061] See also Figure 2 In (c), the spatiotemporal fusion module in this embodiment includes a deep convolution unit, a point convolution unit, a residual connection unit, a third ratio adjustment unit and a layer normalization layer. The spatiotemporal features after foreground attention processing output by the foreground attention module and the initial features extracted by the feature extraction network are successively subjected to deep convolution by the deep convolution unit and point convolution by the point convolution unit, and then output to the residual connection unit. The residual connection unit fuses the convolution operation result with the initial feature after adjustment by the third ratio adjustment unit by using a residual connection, and obtains the final spatiotemporal fusion feature after layer normalization.

[0062] The present invention can take into account computing resources, real-time performance, detection accuracy, detection speed and robustness, and effectively solve the problems faced by UAV platforms in perception tasks, such as target appearance degradation, drastic environmental changes and computing resource limitations. It can not only accurately detect targets with severe appearance degradation, but also accurately detect targets and tiny targets with similar appearances, improve the robustness to environmental changes, and meet the real-time requirements of UAV platforms.

[0063] The present invention will be further described below in conjunction with specific embodiments.

[0064] like Figure 1 As shown, the steps of the drone video target detection method based on spatiotemporal association in this embodiment include:

[0065] Step S01. Basic feature extraction: input the continuous video image data detected by the camera carried by the drone into the feature extraction network for feature extraction to obtain the initial feature sequence output.

[0066] In this embodiment, the main function of the feature extraction network is to accurately extract common features or basic features from the input image. The feature extraction network specifically includes a convolution layer, a pooling layer, and an activation function layer. Through the synergy of each layer, it is possible to deeply mine key information in the image, including but not limited to edge details, texture features, and shape contours of objects in the image.

[0067] Optionally, the feature extraction network can adopt the feature extraction network in the YOLOv7 network. Considering that the UAV platform has extremely high requirements for real-time performance, the YOLO series belongs to a one-stage target detection network, which can well balance detection speed and accuracy. Compared with the previous YOLO series algorithms, YOLOv7 introduces an extended efficient layer aggregation network (ELAN), which enhances the feature extraction capability, and because of the use of a more complex label allocation strategy, it can improve the efficiency and accuracy of training, and has significant improvements in accuracy and speed. By adopting the feature extraction network of the YOLOv7 network, the shallow features of the image can be efficiently obtained to meet the real-time requirements of the UAV platform in practical applications.

[0068] It is understandable that the feature extraction network can of course also adopt other types of networks according to actual needs.

[0069] Step S02. Spatiotemporal feature extraction: After the initial feature sequence is embedded in the spatial position and subjected to deep convolution processing, the features of each frame are spatiotemporally modeled using the state-space equation to learn the spatiotemporal correlation between features of different frames. The learned features are fused with the initial features adjusted by a specified ratio through a residual connection to form spatiotemporal features that retain both temporal and spatial information.

[0070] Specifically, when performing spatiotemporal feature extraction in this embodiment, the image embedding spatial position prior is first input, and then the C×H×W feature map is flattened and converted into a shape of L×N×C (where L=(H×W) / N) through feature dimensionality transformation. Subsequently, feature maps of relevant areas in other frames are selected, and the corresponding features of the current frame are enhanced to compensate for the target appearance degradation caused by factors such as drone movement. In order to achieve effective local information modeling, deep convolution is used to process the features of each frame after position embedding, and the state-space equation (SSM) is used to learn the spatiotemporal correlation between different frames. Figure 2As shown in the state space equation section, the scanning operation is to flatten the features from the upper left corner to the lower right corner (feature center) and from the lower right corner to the upper left corner (feature center), and then through the interaction between the features of different frames, each frame can learn the features that are beneficial to itself from other frames (through the back propagation of the error to guide the network). In addition, by residually connecting the features after the above processing with the initial features after proportional adjustment, the loss of feature information during transmission can be minimized to generate a C-channel feature map that retains both temporal and spatial information. Through the above method, spatiotemporal features can be aggregated from different image frames, effectively improving the applicability of the network under conditions of limited computing resources.

[0071] The state-space model (SSM) represents a class of sequence models in deep learning, which can map a one-dimensional sequence x(t)∈RL to y(t)∈RL through an intermediate latent state h(t)∈RN:

[0072] h′(t)=Ah(t)+Bx(t) (1)

[0073] y(t)=Ch(t) (2)

[0074] Among them, A, B, and C are trainable parameter matrices.

[0075] For image sequences, it is necessary to convert the above mathematical expressions into discrete forms, and convert the continuous time parameters (A, B) into discrete corresponding expressions within the specified sampling time scale Δ∈R>0:

[0076]

[0077] The final mathematical expression of the discretization model is as follows:

[0078]

[0079] y t =Ch t (6)

[0080] Among them, x(t) represents the input image sequence, h(t) represents the intermediate potential state, y(t) represents the feature sequence obtained after mapping, A, B, C are trainable parameter matrices, and Δ represents the sampling time scale.

[0081] In order to improve the computational efficiency, the iterative process described in the above formulas (5) and (6) can be further accelerated by using global convolution operations to implement parallel computing:

[0082]

[0083] Among them, ⊙ represents the convolution operation, and K∈RL represents the size of the convolution kernel.

[0084] In this embodiment, the input feature sequence is modeled in time and space according to the above formula (7) (8) by using the state space equation. The feature output after the state space equation is specifically y, and h is the intermediate feature that realizes the transfer of features from the previous frame to the current frame. Therefore, the final y is equivalent to a new feature containing past information, so that the time and space correlation between different frames can be learned. By using the above method to generate the output of the sequence using convolution, the computational efficiency and scalability can be improved.

[0085] The state-space equation can significantly improve the model's ability to process sequence inputs while maintaining linear complexity. However, the ability to capture sequence context is limited by static parameterization. To solve this problem, the Mamba network directly calculates B, C, and Δ parameters from the input sequence through convolution operations, which can achieve richer, sequence-aware parameterization. The Mamba model can achieve linear scalability of sequence length by utilizing efficient state-space equations. The present invention uses the above-mentioned state-space equations based on the Mamba model to realize information interaction and spatiotemporal modeling between input sequences. At the same time, in order to avoid the problem of information loss during the vertical transmission of features and to ensure efficient computational efficiency, this embodiment uses a residual connection with proportional adjustment to retain the original feature information. The proportional adjustment factor is a pre-defined hyperparameter. By multiplying the initial input features by the fixed proportional adjustment factor, it can help balance the signal strength on different paths and ensure that information can flow effectively in the network.

[0086] Since image sequences are different from input sequences such as speech text, image sequences rely on spatial position encoding. Directly converting the image sequence into a one-dimensional form will destroy the spatial position encoding of the image itself. In this embodiment, the initial feature sequence extracted by the feature network is used as the input of the spatiotemporal extraction module, and in the feature flattening process, flattening from the upper left corner to the lower right corner (feature center) and from the lower right corner to the upper left corner (feature center) are used to minimize the impact of damage to the spatial position encoding information.

[0087] Optionally, the specific steps of performing spatiotemporal feature extraction by the spatiotemporal feature extraction module in this embodiment are:

[0088] Step S201. After embedding the initial feature sequence into a spatial position, perform a deep convolution process to obtain a result after the deep convolution;

[0089] Step S202: After the result of the deep convolution passes through a linear layer, the state space equation is used to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames;

[0090] Step S203: multiply the result of the features learned using the state space equation after layer normalization with the result of the initial feature sequence after passing through the linear layer;

[0091] Step S204: superimpose the multiplication result of step S203 and the result of the initial feature sequence adjusted by a specified ratio, and obtain the final extracted spatiotemporal features after layer normalization of the superimposed result.

[0092] Step S03. Foreground attention processing: The spatiotemporal features are input into an attention network with residual connections. The attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing.

[0093] Since the spatiotemporal extraction step may introduce background interference, this embodiment passes the extracted spatiotemporal features through a foreground attention module to retain useful information as much as possible and suppress background interference.

[0094] In this embodiment, the foreground attention processing is performed before feature fusion, and the influence of appearance similarity on weight allocation is considered. Different weights are assigned to other frames according to their contribution to the degraded target in the current frame. This can effectively suppress background interference and make the network pay more attention to the areas related to the detected targets.

[0095] In the foreground attention processing, this embodiment not only considers the influence of appearance similarity on weight allocation, but also considers the influence of time interval, so that the detection network can effectively filter out the background and learn richer feature representations. Optionally, the cosine similarity measure between the feature maps of different frames and the feature map of the current frame is used to represent the appearance similarity weight, that is, the cosine similarity measure is used to represent the contribution of other frames to the degraded target in the current frame, and the forgetting function is used to simulate the influence of time interval to help the network ignore information from distant time. The calculation expression is:

[0096]

[0097] Among them, c i represents the spatiotemporal features extracted in step S01, c′ i represents the spatiotemporal feature of the i-th foreground after attention processing, f i represents the i-th feature in the initial feature sequence, t represents the time interval between different frames and the current frame, ⊙ represents the cosine similarity measure, and × represents matrix multiplication.

[0098] As shown in the above formula, f i ⊙c i That is the cosine similarity matrix obtained, and then c iPixel-by-pixel multiplication can obtain optimized features. Combined with the forgetting function to simulate the influence of time intervals, the influence of time intervals can also be considered, thereby effectively filtering out the background and learning richer feature representations.

[0099] In a specific application embodiment, during the training process, the calculated hidden feature h is saved as an intermediate vector to the next training batch, so that during each batch training, there is no need to wait for the previous frame to be calculated before calculating the current frame. This is equivalent to obtaining an intermediate feature with a dimension of (b, c, h, w) each time to perform a new calculation with the current input feature (b, c, h, w), where B is the batch, c is the number of channels, and H and w are the feature sizes.

[0100] Step S04: Fuse the initial features with the spatiotemporal features after foreground attention processing in two dimensions, time and space, to obtain spatiotemporal fusion features.

[0101] The feature fusion module can fuse features of different forms or scales, enhance the representation ability of features, and provide richer information for detection tasks in complex environments. Different from target detection in ordinary scenes, the working environment of drones is more complex, which will increase the difficulty of feature fusion. Considering that the direct application of 1×1 convolution will lead to incorrect fusion and introduce a lot of interference, and too large a convolution kernel will lead to a significant increase in the amount of calculation. This embodiment uses deep separable convolution to design a feature fusion module, aggregates features in two dimensions of time and space, and uses the residual structure to better retain the original features. Specifically, the deep separable convolution is mainly divided into two processes, namely deep convolution and point convolution, such as Figure 3 As shown in the figure, by using depth-wise separable convolution, the image sequence and the processed feature map are fused in both time and space dimensions. The depth-wise convolution first performs convolution on each input channel independently, and point-by-point convolution is used to combine the outputs of the depth-wise convolution. In this way, the amount of computation and the number of parameters can be greatly reduced while maintaining similar performance.

[0102] Optionally, the computational expression for feature fusion using depthwise separable convolution is:

[0103] f i fuse =PWConv(DWConv(Concat(c′ i , f i )))+f i (10)

[0104] Among them, DWConv is the depth convolution, PWConv is the point convolution, Concat is the connection according to the channel dimension, and c′ i represents the spatiotemporal feature of the i-th foreground after attention processing, fi represents the i-th feature in the initial feature sequence, f i use represents the i-th fusion feature.

[0105] Step S05: Input the spatiotemporal fusion features into the detection head, and the detection head performs target detection based on the spatiotemporal fusion features to obtain target detection result output.

[0106] The detection head is used to generate candidate target bounding boxes on the feature map. Through preset rules and learning models, it frames the area that may contain the target as accurately as possible. At the same time, it predicts the category of the target in the generated candidate box and calculates the probability of the category to which the target in each box belongs, so as to accurately determine the target category. In addition, the detection head is also responsible for accurate bounding box regression of the candidate box, and continuously optimizes the position and size of the box to better fit the actual boundary of the target. The detection head can fuse multi-scale prediction results from different feature layers to effectively process targets of different sizes. Finally, the detection head integrates and outputs the category prediction and bounding box information after the above processing to form the final target detection result.

[0107] Optionally, since there are a large number of tiny targets in the images collected by the drone platform, in order to solve the problem of a large number of tiny targets in the videos collected by the drone platform, this embodiment further uses deformable convolution to replace general convolution in the detection head to perform convolution operation, such as Figure 4 As shown, the deformable convolution adaptively calculates the offset Δp of the convolution kernel by using the convolution layer that calculates the offset and the convolution layer that calculates the weight value. n and the corresponding weight Δm n , according to the convolution kernel offset Δp n and the corresponding weight Δm n By calculating the pixels at each position and using deformable convolution, the network's performance in detecting tiny targets and targets with deformed appearance can be effectively enhanced.

[0108] Optionally, using a deformable convolution approach, p 0 The pixel y at t-l→t (p 0 ) is calculated as:

[0109]

[0110] Where N represents the size of the convolution kernel (3×3), Yes 0 +p i The corresponding weight at the position. p i ={(-1,-1),(-1,0)…(1,1)}, Yes 0 +p iThe corresponding weight at the position, p 0 represents the original pixel position, p i represents the fixed convolution kernel displacement, Δp n represents the offset value calculated by the convolutional network, tl->t ​​represents the distance from the tlth frame to the tth frame, x(p 0 +p i +Δp n ) represents the input image p 0 +p i +Δp n The pixel at position tl→t represents the tlth frame to the tth frame, and bilinear interpolation is used to realize p 0 +p i +Δp n process.

[0111] Optionally, during the training of the network, the training set data of VisDrone2019-DET and VisDrone2019-VID can be used. Specifically, the feature and detection networks are first pre-trained on VisDrone2019-DET, and then the feature and detection networks are initialized according to the pre-trained weights, and the entire model is fine-tuned on VisDrone2019-VID. At the same time, the model is trained and tested using a single NVIDIA RTX3090 GPU. The training process has a total of 90 epochs, and the first two epochs are dedicated to warm-up. In addition, a stochastic gradient descent (SGD) optimizer with an initial learning rate of 1×10-4 and a cosine learning rate schedule is used; the learning rate of the last epoch is ×1280 pixels to ensure that small objects are fully represented and detected, and the Gaussian Wasserstein Distance (GWD) loss function, which is more conducive to small object detection, can also be used to guide the training process of the network.

[0112] The present invention can also adopt a modular concept and integrate the design of all modules into a large module, which can be easily applied to different single-frame image target detection networks, enhance the ability to obtain additional information from video sequences, and enable the same excellent performance in an extremely lightweight network framework.

[0113] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0114] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0115] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for detecting target in drone video based on spatiotemporal association, characterized in that the steps include: Step S01. Input the continuous video image data detected by the camera carried by the drone into the feature extraction network for feature extraction to obtain an initial feature sequence output; Step S02. After embedding the initial feature sequence into a spatial position and performing deep convolution processing, the features of each frame are spatiotemporally modeled using a state space equation to learn the spatiotemporal correlation between features of different frames, and then fused with the initial features adjusted by a specified ratio through a residual connection to form a spatiotemporal feature that retains both temporal and spatial information; Step S03. Input the spatiotemporal features into an attention network with residual connections, and the attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing; Step S04. Fusing the initial features with the spatiotemporal features after foreground attention processing in two dimensions, time and space, to obtain spatiotemporal fusion features; Step S05: Input the spatiotemporal fusion features into the detection head, and the detection head performs target detection according to the spatiotemporal fusion features to obtain a target detection result output.

2. The method for detecting target in drone video based on spatiotemporal association according to claim 1, characterized in that: Step S02 includes: Step S201. After embedding the initial feature sequence into a spatial position, perform a deep convolution process to obtain a result after the deep convolution; Step S202: After the result of the deep convolution passes through a linear layer, the state space equation is used to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames; Step S203: multiply the result of the features learned using the state space equation after layer normalization with the result of the initial feature sequence after passing through the linear layer; Step S204: superimpose the multiplication result of step S203 and the result of the initial feature sequence adjusted by a specified ratio, and obtain the final extracted spatiotemporal features after layer normalization of the superimposed result.

3. The method for detecting target in drone video based on spatiotemporal association according to claim 1, characterized in that: The state-space equations are used to model the spatiotemporal features of each frame as follows: the t =Ch t Where x(t) represents the input image sequence, h(t) represents the intermediate potential state, y(t) represents the feature sequence obtained after mapping, A, B, C are trainable parameter matrices, and Δ represents the sampling time scale; Sampling global convolution operations to achieve parallel computing: Among them, ⊙ represents the convolution operation, and K represents the size of the convolution kernel.

4. The method for detecting target in drone video based on spatiotemporal association according to claim 1, characterized in that: In step S02, the spatiotemporal feature map c′ after foreground attention processing is obtained according to the following formula: i : Among them, c i represents the spatiotemporal features extracted in step S01, f i represents the i-th feature in the initial feature sequence, t represents the time interval between different frames and the current frame, ⊙ represents the cosine similarity measure, and × represents matrix multiplication.

5. The method for detecting target in drone video based on spatiotemporal association according to claim 1, characterized in that: In step S04, the initial features are fused with the spatiotemporal features after foreground attention processing in time and space by using depthwise separable convolution to obtain fused features, and the fused features are connected to the initial features using residuals to retain the initial features, and finally the spatiotemporal fused features are obtained. The depthwise separable convolution mainly includes two processes: depthwise convolution and pointwise convolution. First, the depthwise convolution performs convolution on each input channel independently, and then the outputs of the depthwise convolution are combined by pointwise convolution.

6. The method for detecting target in drone video based on spatiotemporal association according to claim 5, characterized in that: The expression of fused features obtained by using depth-separable convolution for feature fusion is: f i fuse =PWConv(DWConv(Concat(c′ i ,f i )))+f i Among them, DWConv is the depth convolution, PWConv is the point convolution, Concat is the connection according to the channel dimension, and c′ i represents the spatiotemporal feature of the i-th foreground after attention processing, f i represents the i-th feature in the initial feature sequence, f i fuse represents the i-th fusion feature.

7. The method for detecting target in drone video based on spatiotemporal association according to any one of claims 1 to 6, characterized in that: The detection head uses deformable convolution to perform convolution operations. The deformable convolution adaptively calculates the offset Δp of the convolution kernel by using a convolution layer that calculates an offset and a convolution layer that calculates a weight value. n and the corresponding weight Δm n , according to the convolution kernel offset Δp n and the corresponding weight Δm n Calculate the pixels at each position, the pixel y at p0 t-l→t The calculation expression of (p0) is: Among them, N represents the size of the convolution kernel, It is p0+p i The corresponding weight at the position, p0 represents the original pixel position, p i represents the fixed convolution kernel displacement, Δp n represents the offset value calculated by the convolutional network, tI->t represents the distance from the tIth frame to the tth frame, x(p0+p i +Δp n ) indicates that p0+p in the input image i +Δp n The pixel at position, tl→t represents the transition from the tlth frame to the tth frame.

8. A drone video target detection system based on spatiotemporal correlation, characterized in that: include: The feature extraction network is used to input the continuous video image data detected by the camera carried by the drone for feature extraction to obtain the initial feature sequence; A spatiotemporal feature extraction module is used to embed the initial feature sequence into a spatial position and perform deep convolution processing, then use the state space equation to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames, and then fuse them with the initial features adjusted by a specified ratio through a residual connection to form a spatiotemporal feature that retains both temporal and spatial information; A foreground attention module is used to input the spatiotemporal features into an attention network with residual connections, and the attention network with residual connections sets different weights for the spatiotemporal features of each frame according to the appearance similarity between different frames and the current frame, and fuses the initial features with the features after the attention mechanism through residual connections to obtain the spatiotemporal features after foreground attention processing; A spatiotemporal fusion module, used to fuse the initial features with the spatiotemporal features after foreground attention processing in two dimensions of time and space to obtain spatiotemporal fusion features; The target detection module is used to input the spatiotemporal fusion features into the detection head, and the detection head performs target detection according to the spatiotemporal fusion features to obtain a target detection result output.

9. The drone video target detection system based on spatiotemporal association according to claim 8, characterized in that: The spatiotemporal feature extraction module includes a position embedding unit, a deep convolution unit, a state-space equation unit, a first ratio adjustment unit, two linear layers, and two normalization layers. The position embedding unit inputs an initial feature sequence extracted by a feature extraction network, and after embedding the spatial position, the deep convolution unit performs deep convolution. After the result of the deep convolution passes through the first linear layer, the state-space equation unit uses the state-space equation to perform spatiotemporal modeling on the features of each frame to learn the spatiotemporal correlation between the features of different frames. The features output by the state-space equation unit pass through the first normalization layer and are multiplied with the results of the initial features after passing through the second linear layer. The multiplied results and the initial features are superimposed after the ratio adjustment unit adjusts the results according to the specified ratio. The superimposed results pass through the second normalization layer to obtain the final extracted feature map retaining the time and space information.

10. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 7.

Images