Unmanned aerial vehicle video target detection method and system based on space-time correlation

Abstract

The application discloses a kind of unmanned aerial vehicle video target detection method and system based on space-time association, the method steps include: S01.unmanned aerial vehicle video image data is input to feature extraction network, and initial feature sequence is obtained;S02.after initial feature sequence is embedded space position and is carried out depth convolution processing, space-time modeling is carried out using state space equation, then with the initial feature that is proportionally adjusted is fused by residual connection, and space-time feature is formed;S03.unmanned aerial vehicle video image data is input to feature extraction network, and initial feature sequence is obtained;S04.feature fusion is carried out to initial feature and foreground attention processing after space-time feature;S05.time-space fusion feature is input to detection head, and target detection result output is obtained.The application has the advantages of simple implementation method, less required computing resources, real-time, high detection accuracy, strong scalability and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle (UAV) vision target detection, and particularly relates to a UAV video target detection method and system based on space-time correlation. BACKGROUND

[0002] An unmanned aerial vehicle (UAV) can perform various tasks such as searching for targets, reconnaissance, and surveillance, and is widely used in aerial photography, disaster monitoring, power inspection, and other fields. Due to the instability of the UAV platform and the complexity of the environment, the perception ability of the UAV during task execution is required to be higher. UAV video target detection is to identify and track targets in video data captured by a camera carried by the UAV. However, unlike traditional target detection problems, due to the motion of the UAV itself and the complexity of the working environment, the difficulty of the target detection task based on the UAV platform will increase exponentially, making UAV video target detection face great challenges.

[0003] Traditional target detection usually uses a single-frame image-based target detection network to detect targets, that is, a single-frame image is taken as a target, and the single-frame image is input into the target detection network to detect target information. However, unlike a single-frame image, a video contains various appearance features of a target under the influence of different environments and rich motion information, and there may be dramatic environmental changes between different video frames. A single-frame image-based target detection network is difficult to detect similar appearance targets and small targets in a complex environment, accurately detect targets with severe appearance degradation, and balance computing resources, real-time performance, detection accuracy, detection speed, and robustness, and is not suitable for a UAV platform. It also has problems such as long implementation cycle, poor network generalization performance, and insufficient scalability. SUMMARY

[0004] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides a UAV video target detection method and system based on space-time correlation, which has the advantages of simple implementation method, less required computing resources, high real-time performance and detection accuracy, and strong scalability.

[0005] To solve the above technical problems, the technical solution provided by the present application is as follows:

[0006] A UAV video target detection method based on space-time correlation, comprising the following steps:

[0007] Step S01. Input continuous video image data detected by a camera carried by a UAV into a feature extraction network for feature extraction to obtain an initial feature sequence output;

[0008] Step S02. After embedding the initial feature sequence in spatial positions and performing deep convolution processing, the state space equation is used to model the space-time correlation between different frame features, and the initial features adjusted by a specified proportion are fused through residual connection to form space-time features that retain time and space information.

[0009] Step S03. The space-time features are input into the attention network with residual connection, and the attention network with residual connection sets different weights for each frame space-time feature according to the appearance similarity between different frames and the current frame, and the initial features and the features after attention mechanism are fused through residual connection to obtain foreground attention processed space-time features.

[0010] Step S04. The initial features and the foreground attention processed space-time features are fused in time and space dimensions to obtain space-time fusion features.

[0011] Step S05. The space-time fusion features are input into the detection head, and the detection head performs target detection according to the space-time fusion features to obtain target detection results.

[0012] Further, step S02 includes:

[0013] Step S201. After embedding the initial feature sequence in spatial positions, deep convolution processing is performed to obtain deep convolution results.

[0014] Step S202. The deep convolution results are processed through a linear layer, and the state space equation is used to model the space-time correlation between different frame features.

[0015] Step S203. The features learned by the state space equation are multiplied by the results of the initial feature sequence after the linear layer.

[0016] Step S204. The multiplication result of step S203 is superimposed with the result of the initial feature sequence after the specified proportion adjustment, and the superimposed result is normalized to obtain the final extracted space-time features.

[0017] Further, the state space equation is used to model the space-time correlation between different frame features according to the following formula:

[0018]

[0019] y t =Ch t

[0020]

[0021] Wherein, x(t) represents the input image sequence, h(t) represents the intermediate latent state, y(t) represents the feature sequence obtained after mapping, A, B, C are trainable parameter matrices, and Delta represents the sampling time scale.

[0022] The sampling global convolution operation realizes parallel computing:

[0023]

[0024] Wherein, represents the convolution operation, and K represents the size of the convolution kernel.

[0025] Further, in step S02, the spatio-temporal feature map c' after foreground attention processing is obtained according to the following formula i

[0026]

[0027] Wherein, c i represents the spatio-temporal feature extracted in step S01, f i represents the i-th feature in the initial feature sequence, t represents the time interval between different frames and the current frame, represents the cosine similarity measure, and represents matrix multiplication.

[0028] Further, in step S04, the initial feature and the spatio-temporal feature after foreground attention processing are fused in time and space dimensions by using a depth separable convolution to obtain a fused feature, and the fused feature and the initial feature are connected by a residual connection to retain the initial feature, and finally the spatio-temporal fused feature is obtained, wherein the depth separable convolution mainly includes two processes of depth convolution and point convolution, the depth convolution is first independently performed on each input channel, and then the output of the depth convolution is combined by point-by-point convolution.

[0029] Further, the expression of the fused feature obtained by using the depth separable convolution for feature fusion is:

[0030] f i fuse = PWConv(DWConv(Concat(c′ i , f i )) + f i

[0031] Wherein, DWConv is depth convolution, PWConv is point convolution, Concat is connection in the channel dimension, c i represents the i-th spatio-temporal feature after foreground attention processing, f i represents the i-th feature in the initial feature sequence, and f i fuse represents the i-th fused feature. ​

[0032] Further, the detection head uses deformable convolution for convolution operation, and the deformable convolution adaptively calculates the offset Δp of the convolution kernel by using a convolution layer for calculating the offset and a convolution layer for calculating the weight value n and the corresponding weight Δm n to calculate the pixel at each position, the pixel y at p0 n and the corresponding weight Δm n p0 t-l→t The calculation expression of (p0) is:

[0033]

[0034] where N represents the size of the convolution kernel, is the corresponding weight at p0+p i p0 represents the original pixel position, p i represents the fixed convolution kernel displacement, Δp n represents the offset value calculated by the convolution network, t-I->t represents the t-lth frame to the tth frame, x(p0+p i +Δp n ) represents the pixel at p0+p i +Δp n in the input image, t-l→t represents the t-lth frame to the tth frame.

[0035] A UAV video target detection system based on space-time correlation comprises:

[0036] a feature extraction network configured to input continuous video image data detected by a camera carried by a UAV to perform feature extraction and obtain an initial feature sequence;

[0037] a space-time feature extraction module configured to embed the initial feature sequence in a spatial position, perform deep convolution processing, use a state space equation to model each frame feature to learn the space-time correlation between different frame features, and then fuse the initial feature after a specified proportion adjustment with the space-time feature through residual connection to form a space-time feature that retains both time and space information;

[0038] a foreground attention module configured to input the space-time feature into an attention network with residual connection, set different weights for each frame space-time feature according to the appearance similarity between different frames and the current frame by the attention network with residual connection, and fuse the initial feature with the feature after the attention mechanism through residual connection to obtain a foreground attention processed space-time feature;

[0039] The spatio-temporal fusion module is configured to fuse the initial feature and the spatio-temporal feature after the foreground attention processing in both time and space dimensions to obtain a spatio-temporal fusion feature.

[0040] The target detection module is configured to input the spatio-temporal fusion feature into a detection head, and the detection head is configured to perform target detection according to the spatio-temporal fusion feature to obtain a target detection result output.

[0041] Further, the spatio-temporal feature extraction module comprises a position embedding unit, a deep convolution unit, a state space equation unit, a first proportion adjusting unit, two linear layers and two layer normalization layers. The position embedding unit is configured to input an initial feature sequence extracted by a feature extraction network, embed a spatial position, and then perform deep convolution by the deep convolution unit. After the deep convolution, the result is input into the state space equation unit to perform spatio-temporal modeling on each frame feature by using a state space equation to learn the spatio-temporal correlation between different frame features. The feature output by the state space equation unit is multiplied by the result after the initial feature passes through a second linear layer after the first layer normalization layer. The multiplication result is superimposed with the result after the initial feature passes through the proportion adjusting unit according to a specified proportion adjustment. The superimposed result is input into the second layer normalization layer to obtain a final extracted feature map which retains time and space information.

[0042] An electronic device comprises a processor and a memory. The memory is configured to store a computer program. The processor is configured to execute the computer program to perform the above method.

[0043] Compared with the prior art, the present application has the following advantages:

[0044] 1. The present application uses a feature extraction network to extract a basic feature to form an initial feature sequence by taking a continuous video data sequence as a network input. Then, the initial feature sequence is embedded with a spatial position and then subjected to deep convolution processing. A state space equation is used to model the spatio-temporal feature in the video. The spatio-temporal feature can be effectively aggregated from the video, which enables the network to robustly and efficiently cope with perception tasks in complex environments. The spatio-temporal feature is input into an attention network with residual connection. The attention mechanism based on appearance similarity measurement can effectively suppress the background interference introduced from other frames in the spatio-temporal modeling process, which is conducive to the network paying more attention to the region related to the foreground target. The residual connection can also realize the fusion of appearance features and motion features, fully utilize the rich feature information in the video to enhance the target detection effect, and make it possible to balance the detection speed and accuracy.

[0045] 2、The application can be applied to unmanned aerial vehicle video target detection by constructing a lightweight video target detection network supporting cross-view information modeling, and can balance computing resources, real-time performance, detection accuracy, detection speed and robustness, effectively solving the problems of target appearance degradation, severe environmental changes and computing resource limitations in the perception task of the unmanned aerial vehicle platform, accurately detecting not only targets with serious appearance degradation, but also targets with similar appearance and small targets, improving the robustness to environmental changes, and meeting the real-time requirements of the unmanned aerial vehicle platform. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a schematic diagram of a traditional single-frame image-based and video image-based target detection network structure principle.

[0047] Figure 2 is a structural schematic diagram of the unmanned aerial vehicle video target detection system based on space-time association in the embodiment.

[0048] Figure 3 is a schematic diagram of feature fusion using deep separable convolution in the embodiment.

[0049] Figure 4 is a schematic diagram of embedding deformable convolution in the detection head in the embodiment. DETAILED DESCRIPTION

[0050] The application will be further described below in conjunction with the drawings and specific preferred embodiments of the present application, but the protection scope of the present application is not limited thereto.

[0051] For the sake of understanding, first, the related technical background involved in the present application is exemplarily introduced.

[0052] The traditional single-frame image-based target detection network structure is shown in Figure 1 (a). The network consists of two parts, backbone and detection head. The backbone part extracts high-level feature representations with semantic information from the original data by stacking multiple convolution layers. These features can capture the local structure and global context of the input data, providing better feature information for subsequent tasks. The detection head part usually includes one or more fully connected layers, usually connected to the output of the backbone part, using the features extracted by the backbone for final classification and regression of bounding box position. After detecting the target, some post-processing steps such as non-maximum suppression (NMS) are needed to eliminate overlapping bounding boxes and improve the accuracy of the detection results.

[0053] The video-based target detection network structure is shown in Figure 1The network consists of a backbone, a neck and a head, the backbone and the head part are consistent with the single-frame image-based target detection network, the neck is located between the backbone and the head, and is usually composed of an attention network with strong global context learning ability or a recurrent neural network with time sequence information learning ability, so as to model the motion information in the video and aggregate into the appearance features of the current frame, thereby improving the target detection effect of the current frame. In order to enable the video-based detection network to meet the real-time requirements of the unmanned aerial vehicle platform while having the ability to cope with the target deformation and scale changes caused by the dramatic changes of the environment and the unmanned aerial vehicle motion, a neck network with cross-view global context modeling and precise local learning ability needs to be designed.

[0054] The single-frame image-based target detection network structure is difficult to accurately detect the target with serious appearance degradation, since the single-frame image-based target detection network usually mines more effective features from the image by constructing a deeper network and fusing multi-dimensional features. However, due to the limited information provided by the single-frame image, severe deformation and occlusion make it difficult for the network to learn effective features, resulting in poor performance of this method in detecting targets with serious appearance degradation in complex environments.

[0055] If the single-frame image-based target detection network is directly extended to the field of video target detection to utilize the rich feature information in the video to enhance the target detection effect, it will be difficult to balance the computing resources, real-time performance, detection accuracy, detection speed and robustness. For example, one of the existing video target detection algorithms FGFA (Flow-guided Feature Aggregation) models the motion deviation between multiple frames through optical flow, and uses a bilinear interpolation method to align the target features of adjacent frames to the corresponding spatial positions of the target features of the current frame. However, due to the motion of the unmanned aerial vehicle itself and the complexity of the environment, the above algorithm requires a large amount of computing resources to obtain accurate optical flow. Another way is to use the video sequence to provide auxiliary information, that is, to use the video sequence as the network input, to design a neural network with cross-view modeling capability, and to make a prediction according to the multi-frame fused features. For example, a recurrent convolutional neural network (LSTM) is used to realize the transmission and interaction of features in the video sequence. However, directly aggregating multi-frame features will introduce a lot of background interference, and due to the inherent limitations of LSTM, it will be difficult to model long-term feature association.

[0056] In addition, the self-attention mechanism is used to establish the correlation between different positions, which can capture the dependency in the sequence, for example, a neural network model based on the Transformer architecture, which can learn the global context information in the image by introducing the self-attention mechanism of the Transformer, and establish the correlation between different positions, so as to better understand the image content. However, the traditional Transformer model has a large size when processing image data, and needs to be divided into small blocks for processing, which will cause large computational and memory overhead, and the amount of calculation will increase exponentially with the increase of the input image size, which is difficult to meet the real-time requirements of high-speed motion of unmanned aerial vehicles.

[0057] A system architecture suitable for the present application is shown in Figure 2 The overall structure is divided into five parts: feature extraction network, space-time extraction module, foreground attention module, space-time fusion module and detection head, which can form a plug-and-play cross-view space-time correlation network. Using the above architecture, the feature extraction network inputs the continuous video image data detected by the camera carried by the unmanned aerial vehicle to extract features, and obtains an initial feature sequence. The space-time feature extraction module embeds the initial feature sequence in the spatial position and performs deep convolution processing, and then uses the state space equation to model the space-time of each frame feature to learn the space-time correlation between different frame features. The learned features and the initial features adjusted by a specified proportion are fused through residual connection to form space-time features that retain both time and space information. The foreground attention module inputs the space-time features into the attention network with residual connection, and sets different weights for each frame space-time feature according to the appearance similarity between different frames and the current frame. The initial features and the features after attention mechanism are fused through residual connection to obtain the foreground attention processed space-time features. The space-time fusion module fuses the initial features and the foreground attention processed space-time features in time and space dimensions to obtain space-time fusion features. Finally, the space-time fusion features are input into the detection head, and the detection head detects the target according to the space-time fusion features to obtain the target detection result output.

[0058] The system can construct a lightweight video object detection network supporting cross-view information modeling, and the network input is a continuous video data sequence. The general features or basic features of each frame are extracted by a feature extraction network to form an initial feature sequence. The initial feature sequence is embedded in a spatial position and then subjected to deep convolution processing. A state space equation is used to model the space-time features in the video, which can effectively aggregate the space-time features from the video and enable the network to robustly and efficiently handle perception tasks in complex environments. The space-time features are input into an attention network with residual connection. The attention mechanism based on appearance similarity measurement can effectively suppress the background interference introduced from other frames in the space-time modeling process, which helps the network pay more attention to the regions related to the foreground object. The residual connection can also realize the fusion of appearance features and motion features, effectively fuse features of different modalities, fully utilize the rich feature information in the video to enhance the object detection effect, and balance the detection speed and accuracy.

[0059] Referring to Figure 2 In the embodiment, the space-time feature extraction module includes a position embedding unit, a deep convolution unit, a state space equation unit, a first proportional adjustment unit, and two linear layers and two layer normalization layers. The position embedding unit inputs the initial feature sequence extracted by the feature extraction network, embeds the space position, and then performs deep convolution by the deep convolution unit. After deep convolution, the result is subjected to the first linear layer, and then the state space equation unit uses the state space equation to model the space-time features of each frame to learn the space-time correlation between different frame features. The features output by the state space equation unit are subjected to the first layer normalization layer, multiplied by the result after the second linear layer of the initial features, and then the multiplied result is superimposed with the result after the initial features are adjusted by the first proportional adjustment unit according to a specified proportion. The superimposed result is subjected to the second layer normalization layer to obtain the final extracted feature map that retains the time and space information. In the embodiment, the above structure is adopted to construct a space-time feature extraction module with linear computational complexity according to the state space equation. The neural network module with linear complexity is used to aggregate space-time features from different image frames, which can fully utilize the rich motion and appearance information in the video, maintain linear computational complexity, provide global perception across views for the network, and effectively improve the applicability of the network under limited computing resources.

[0060] Referring to Figure 2In the embodiment, the foreground attention module includes a similarity measurement unit, a matrix multiplication unit, a residual connection unit, a second proportional adjustment unit, and a layer normalization layer. The similarity measurement unit calculates similarity measurements based on the initial features extracted by the feature extraction network and the spatiotemporal features extracted by the spatiotemporal feature extraction module. The matrix multiplication unit performs matrix multiplication on the foreground embedding and the output of the similarity measurement unit. The residual connection unit fuses the output of the matrix multiplication unit and the result of the spatiotemporal features adjusted by the second proportional adjustment unit according to a specified proportion through residual connection. The fused result is normalized by the layer normalization layer to obtain the spatiotemporal features after foreground attention processing. The foreground attention module in the embodiment uses an attention network with residual connection. The network can pay more attention to the region related to the foreground target and suppress background interference. The appearance features and motion features are fused using a deep residual network to improve the perception ability of the network in a complex environment.

[0061] Referring to Figure 2 In the embodiment, the spatiotemporal fusion module includes a deep convolution unit, a point convolution unit, a residual connection unit, a third proportional adjustment unit, and a layer normalization layer. The spatiotemporal features after foreground attention processing output by the foreground attention module and the initial features extracted by the feature extraction network are sequentially subjected to deep convolution by the deep convolution unit and point convolution operation by the point convolution unit, and then output to the residual connection unit. The residual connection unit fuses the convolution operation result and the features adjusted by the third proportional adjustment unit using residual connection. The final spatiotemporal fusion features are obtained after layer normalization.

[0062] The present application can balance the computing resources, real-time performance, detection accuracy, detection speed, and robustness, effectively solve the problems of target appearance degradation, severe environmental changes, and computing resource limitations in the perception task of the unmanned aerial vehicle platform, accurately detect targets with serious appearance degradation, accurately detect targets with similar appearance and small targets, improve the robustness to environmental changes, and meet the real-time requirements of the unmanned aerial vehicle platform.

[0063] The present application will be further described below in conjunction with specific embodiments.

[0064] As Figure 1 shown, the steps of the unmanned aerial vehicle video target detection method based on spatiotemporal correlation in the embodiment include:

[0065] Step S01. Basic feature extraction: input the continuous video image data detected by the camera carried by the unmanned aerial vehicle into the feature extraction network for feature extraction to obtain an initial feature sequence output.

[0066] In this embodiment, the main role of the feature extraction network is to accurately extract general features or basic features from the input image. The feature extraction network specifically includes convolutional layers, pooling layers, and activation function layers, etc. Through the synergistic effect of each layer, key information in the image can be deeply mined, including but not limited to edge details, texture features, and object shape contours in the image, etc.

[0067] Optionally, the feature extraction network can adopt the feature extraction network in the YOLOv7 network. Considering the extremely high real-time requirement of the unmanned aerial vehicle platform, the YOLO series belongs to a one-stage target detection network, which can well balance the detection speed and accuracy. And compared with the previous YOLO series algorithm, YOLOv7 introduces an extended efficient layer aggregation network (ELAN), enhances the feature extraction capability, and due to the adoption of a more complex label assignment strategy, can improve the efficiency and accuracy of training, and has significant improvement in accuracy and speed. By adopting the feature extraction network of the YOLOv7 network, the shallow features of the image can be efficiently obtained, meeting the real-time requirements of the unmanned aerial vehicle platform in actual application.

[0068] It can be understood that the feature extraction network can of course also adopt other types of networks according to actual needs.

[0069] Step S02. Spatio-temporal feature extraction: after embedding the initial feature sequence in the spatial position and performing deep convolution processing, the state space equation is used to model the space-time of each frame feature to learn the space-time correlation between different frame features. The learned features are fused with the initial features adjusted by a specified proportion through residual connection to form spatio-temporal features that retain both time and space information.

[0070] Specifically, when the spatio-temporal feature extraction is performed in this embodiment, first, the image is embedded in the spatial position priori, and then the CxHxW feature is mapped and flattened through dimension transformation of the feature to the shape of LxNxC (where L=(HxW) / N). Subsequently, the features of the relevant regions in other frames are selected for mapping, and the corresponding features of the current frame are enhanced to compensate for the target appearance degradation caused by factors such as unmanned aerial vehicle movement. In order to realize effective local information modeling, the features of each frame are processed by deep convolution after position embedding, and the state space equation (SSM) is used to learn the space-time correlation between different frames. As shown in Figure 2The scanning operation is to flatten the features from the top-left corner to the bottom-right corner (feature center) and from the bottom-right corner to the top-left corner (feature center), and then through the interaction between the features of different frames, so that each frame can learn from other frames the features that are beneficial to itself (guided by the back propagation of errors to realize the network). In addition, by connecting the features processed through the above method with the initial features after scaling, the loss of feature information during transmission can be minimized to generate C-channel feature maps that retain both temporal and spatial information. Through the above method, temporal and spatial features can be aggregated from different image frames, effectively improving the applicability of the network under limited computing resources.

[0071] State space equation (SSM) represents a class of sequence models in deep learning, which can map one-dimensional sequence x(t) e RL to y(t) e RL through intermediate latent state h(t) e RN:

[0072] h'(t) = Ah(t) + Bx(t) (1)

[0073] y(t) = Ch(t) (2)

[0074] Where A, B, C are trainable parameter matrices.

[0075] For image sequences, the above mathematical expressions need to be converted into discrete form, and continuous time parameters (A, B) are converted into discrete corresponding expression form within a specified sampling time scale Δ e R>0:

[0076]

[0077] The final discrete model mathematical expression is as follows:

[0078]

[0079] y t =Ch t (6)

[0080] Where x(t) represents the input image sequence, h(t) represents the intermediate latent state, y(t) represents the feature sequence obtained after mapping, A, B, C are trainable parameter matrices, and Δ represents the sampling time scale.

[0081] In order to improve the calculation efficiency, the iterative process described in the above formulas (5) and (6) can be further accelerated by using global convolution operation to realize parallel calculation:

[0082]

[0083] Wherein, represents the convolution operation, K represents the size of the convolution kernel.

[0084] The embodiment uses the state space equation to model the input feature sequence in space-time according to the above formula (7) (8), and the output feature after the state space equation is y, h is an intermediate feature for realizing the transmission of the feature from the previous frame to the current frame, and thus the final y is equivalent to a new feature containing past information, so that the spatio-temporal correlation between different frames can be learned. By using the above method to generate the output of the sequence by convolution, the calculation efficiency and scalability can be improved.

[0085] The state space equation can significantly improve the ability of the model to process sequence input while maintaining linear complexity. However, the ability to capture sequence context is limited by static parameterization. To solve this problem, the Mamba network directly calculates the B, C, and Delta parameters from the input sequence through convolution operations, which can achieve more rich and sequence-aware parameterization. The Mamba model can achieve linear scalability of sequence length by using an efficient state space equation. The present application uses the above state space equation to realize information interaction and spatio-temporal modeling between input sequences based on the Mamba model. In order to avoid information loss during vertical transmission of features and ensure efficient calculation efficiency, the embodiment uses residual connection with proportional adjustment to preserve the original feature information. The proportional adjustment factor is a predefined hyperparameter. By multiplying the input initial feature by the fixed proportional adjustment factor, the signal strength on different paths can be balanced to ensure that information can flow effectively in the network.

[0086] Since the image sequence is different from the input sequence such as speech text, the image sequence relies on spatial position coding. Directly converting the image sequence into one-dimensional form will damage the spatial position coding of the image itself. The embodiment takes the initial feature sequence extracted by the feature network as the input of the space-time extraction module, and adopts left-top-to-right-bottom (feature center) flattening and right-bottom-to-left-top (feature center) flattening during feature flattening to minimize the impact of spatial position coding information damage.

[0087] Optionally, the specific steps of the space-time feature extraction module for space-time feature extraction in the embodiment are as follows:

[0088] Step S201. The initial feature sequence is embedded in the spatial position and then subjected to deep convolution processing to obtain a deep convolution result.

[0089] Step S202. The deep convolution result is subjected to linear layer processing and then subjected to space-time modeling of each frame feature using the state space equation to learn the spatio-temporal correlation between different frame features.

[0090] Step S203. Multiply the result after the feature sequence learned by the state space equation is normalized with the result after the initial feature sequence is linearly processed.

[0091] Step S204. Superimpose the result after step S203 is multiplied with the result after the initial feature sequence is adjusted by a specified proportion, and obtain the final extracted spatio-temporal feature after the superimposed result is normalized.

[0092] Step S03. Foreground attention processing: input the spatio-temporal feature into the attention network with residual connection, set different weights for the frame spatio-temporal features according to the appearance similarity between different frames and the current frame by the attention network with residual connection, and fuse the initial feature and the feature after the attention mechanism through the residual connection to obtain the spatio-temporal feature after the foreground attention processing.

[0093] Since the spatio-temporal extraction step will introduce background interference, the embodiment further processes the extracted spatio-temporal feature through the foreground attention module to retain as much useful information as possible and suppress the background interference.

[0094] The foreground attention processing in the embodiment is before the feature fusion, considers the influence of the appearance similarity on the weight distribution, and assigns different weights to the degraded target in the current frame according to the contribution degree of other frames, which can effectively suppress the background interference and make the network pay more attention to the region related to the detected target.

[0095] The embodiment not only considers the influence of the appearance similarity on the weight distribution in the foreground attention processing, but also considers the influence of the time interval, so that the detection network can effectively filter out the background and learn more rich feature representation. Alternatively, the cosine similarity measure between the feature mapping of different frames and the feature mapping of the current frame is used to represent the appearance similarity weight, that is, the cosine similarity measure is used to represent the contribution degree of other frames to the degraded target in the current frame, and the forgetting function is used to simulate the influence of the time interval, so as to help the network ignore the information from the remote time. The calculation expression is:

[0096]

[0097] wherein c i represents the spatio-temporal feature extracted in step S01, c′ i represents the i-th spatio-temporal feature after the foreground attention processing, f i represents the i-th feature in the initial feature sequence, t represents the time interval between different frames and the current frame, represents the cosine similarity measure, and represents matrix multiplication.

[0098] As shown in the above formula, f i ⊙c i is the obtained cosine similarity matrix, and then and c iThe pixel-by-pixel multiplication can obtain optimized features, the influence of the time interval can be considered by combining the forgetting function to simulate the influence of the time interval, and thus the background can be effectively filtered out and a more rich feature representation can be learned.

[0099] In a specific application embodiment, in the training process, the calculated hidden feature h is saved as an intermediate vector for the next training batch, so that the previous frame does not need to be calculated before the current frame is calculated during each batch training. That is, an intermediate feature with a dimension of (b, c, h, w) can be obtained each time to perform new calculation with the current input feature (b, c, h, w), where B is the batch, c is the number of channels, and H and w are the feature sizes.

[0100] Step S04. The initial features are fused with the foreground attention processed spatio-temporal features in the time and space dimensions to obtain spatio-temporal fusion features.

[0101] The feature fusion module can fuse features of different forms or scales, enhance the representation ability of the features, and provide more rich information for the detection task in a complex environment. Unlike target detection in ordinary scenes, the working environment of the unmanned aerial vehicle is relatively complex, which increases the difficulty of feature fusion. Considering that directly applying a 1x1 convolution will lead to incorrect fusion and introduce a large amount of interference, and a too large convolution kernel will significantly increase the computational load. In this embodiment, a feature fusion module is designed by using a depth separable convolution, which aggregates features in the time and space dimensions, and uses a residual structure to better preserve the original features. Specifically, the depth separable convolution mainly includes two processes, namely depth convolution and point convolution, as shown in FIG. 8. The image sequence and the processed feature map are fused in the time and space dimensions by using the depth separable convolution. The depth convolution is first used to independently convolve each input channel, and the point convolution is used to combine the output of the depth convolution. In this way, the computational load and the number of parameters can be greatly reduced while maintaining similar performance. Figure 3

[0102] Optionally, the calculation expression of the feature fusion by using the depth separable convolution is as follows:

[0103] f i fuse = PWConv(DWConv(Concat(c′ i , f i )) + f i (10)

[0104] where DWConv is the depth convolution, PWConv is the point convolution, Concat is the connection in the channel dimension, c′ i represents the i-th foreground attention processed spatio-temporal feature, and f​i f represents the i-th feature in the initial feature sequence. i use This represents the i-th fusion feature.

[0105] Step S05. Input the spatiotemporal fusion features into the detection head, and the detection head performs target detection based on the spatiotemporal fusion features to obtain the target detection result output.

[0106] The detection head generates candidate bounding boxes for targets on the feature map. Through preset rules and learning modes, it accurately defines regions that may contain targets. Simultaneously, it predicts the class of targets within the generated candidate boxes, calculating the probability of each target belonging to its class, thus accurately determining the target category. Furthermore, the detection head performs precise bounding box regression on the candidate boxes, continuously optimizing the box position and size to better match the true boundaries of the targets. The detection head can fuse multi-scale prediction results from different feature layers to effectively handle targets of varying sizes. Finally, the detection head integrates and outputs the processed class predictions and bounding box information to form the final target detection result.

[0107] Optionally, since there are a large number of tiny targets in the images acquired by the UAV platform, this embodiment further uses deformable convolution instead of general convolution in the detection head to perform convolution operations, addressing the problem of numerous tiny targets in the videos acquired by the UAV platform. Figure 4 As shown, deformable convolution adaptively calculates the kernel offset Δp by using convolutional layers that compute the offset and convolutional layers that compute the weight values. n and the corresponding weight Δm n To adjust according to the kernel offset Δp n and the corresponding weight Δm n Calculating the pixels at each location and using deformable convolution can effectively enhance the network's performance in detecting small targets and targets with deformable appearances.

[0108] Optionally, a deformable convolution method is used, where the pixel y at p0... t-l→t The expression for calculating (p0) is:

[0109]

[0110] Where N represents the size of the convolution kernel (3×3), It is p0+p i The corresponding weight at the position. i ={(-1,-1), (-1,0)…(1,1)}, It is p0+p i The corresponding weights at the positions, where p0 represents the original pixel position, p irepresents the fixed convolution kernel displacement, Δp n represents the offset value calculated by the convolution network, t-l→t represents the t-lth frame to the tth frame, x(p0+p i +Δp n ) represents the pixel at the position p0+p i +Δp n in the input image, t-l→t represents the t-lth frame to the tth frame, and the process of p0+p i +Δp n is realized by using bilinear interpolation.

[0111] Optionally, in the training process of the network, the training set data of VisDrone2019-DET and VisDrone2019-VID can be used. Specifically, first, the feature and detection network are pre-trained on VisDrone2019-DET, then the feature and detection network are initialized according to the pre-trained weights, and the entire model is fine-tuned on VisDrone2019-VID. At the same time, the model is trained and tested using a single NVIDIA RTX3090 GPU. The training process has a total of 90 epochs, and the first two epochs are dedicated to warm-up. In addition, a stochastic gradient descent (SGD) optimizer with an initial learning rate of 1×10-4 is used, and a cosine learning rate schedule is adopted; the learning rate of the last epoch is ×1280 pixels to ensure sufficient representation and detection of small targets, and further, a Gaussian Wasserstein Distance (GWD) loss function that is more conducive to small target detection can be used to guide the training process of the network.

[0112] The application can also adopt the modular idea, integrate the design of all modules into a large module, which can be conveniently applied to different single-frame image target detection networks, enhance the ability to obtain additional information from video sequences, and enable excellent performance in an extremely lightweight network framework.

[0113] The embodiment further provides an electronic device, including a processor and a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program to execute the method as described above.

[0114] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0115] Those skilled in the art will understand that the above embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more blocks or any combination thereof for information, data, signals, work, instructions or concepts.

[0116] The above merely preferred embodiments of the present application and not intended to limit the present application in any form. Although the present application has been disclosed with the preferred embodiments as above, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, according to the technical essence of the present application, should fall within the scope of protection of the technical solution of the present application.

Claims

1. A method for detecting a target in a UAV video based on spatio-temporal correlation, characterized in that the steps of Comprising: Step S01. Input the continuous video image data detected by the camera carried by the unmanned aerial vehicle into a feature extraction network for feature extraction to obtain an initial feature sequence output; Step S02. After embedding the initial feature sequence in a spatial position and performing deep convolution processing, use a state space equation to model the space-time correlation between different frame features, and then fuse the initial feature after adjusting the proportion by a specified proportion through residual connection to form a space-time feature that retains time and space information at the same time; Step S03. Input the space-time feature into an attention network with residual connection, set different weights for each frame space-time feature according to the appearance similarity between different frames and the current frame by the attention network with residual connection, and fuse the initial feature and the feature after the attention mechanism through residual connection to obtain a foreground attention processing space-time feature; Step S04. Fuse the initial feature and the foreground attention processing space-time feature in the time and space dimensions to obtain a space-time fusion feature; Step S05. Input the space-time fusion feature into a detection head, and perform target detection according to the space-time fusion feature by the detection head to obtain a target detection result output; Step S02 comprises: Step S201. Perform deep convolution processing on the initial feature sequence after embedding it in a spatial position to obtain a deep convolution result; Step S202. Use a state space equation to model the space-time correlation between different frame features after linear layer processing of the deep convolution result; Step S203. Multiply the feature learned by the state space equation after layer normalization with the result after linear layer processing of the initial feature sequence; Step S204. Superimpose the multiplication result of step S203 with the result after adjusting the proportion of the initial feature sequence by a specified proportion, and obtain the final extracted space-time feature after layer normalization of the superimposed result. 2.The spatio-temporal correlation based UAV video target detection method of claim 1, wherein, Use the state space equation to model the space-time correlation between different frame features according to the following formula: wherein, denotes an input image sequence, denotes an intermediate latent state, denotes a resulting feature sequence after mapping, A, B, C are trainable parameter matrices, and Δ denotes a sampling time scale; The sampling global convolution operation realizes parallel computing: wherein, denotes a convolution operation, K denotes the size of the convolution kernel. 3.The UAV video target detection method based on spatio-temporal correlation according to claim 1, characterized in that, In step S03, the spatio-temporal feature map after foreground attention processing is obtained according to the following formula : wherein, c i denotes the spatio-temporal features extracted in step S01, f i denotes the i-th feature in the initial feature sequence, i denotes the time interval between different frames and the current frame, denotes the cosine similarity measure, denotes the matrix multiplication.​ 4.The spatio-temporal correlation based UAV video target detection method of claim 1, wherein, In step S04, the initial feature and the foreground attention processing space-time feature are fused in the time and space dimensions to obtain a fusion feature by using a depth separable convolution, and the fusion feature and the initial feature are connected by a residual connection to retain the initial feature, and finally the space-time fusion feature is obtained. The depth separable convolution mainly includes two processes of depth convolution and point convolution, which first performs convolution on each input channel independently by depth convolution, and then combines the output of depth convolution by point-by-point convolution. 5.The UAV video target detection method based on spatio-temporal correlation according to claim 4, characterized in that, The expression of the fusion feature obtained by using the depth separable convolution for feature fusion is: Wherein, DWConv is a deep convolution, PWConv is a point convolution, Concat is a connection in the channel dimension, represents the i th spatial-temporal feature map after foreground attention processing, f i represents the i th feature in the initial feature sequence, represents the i th fusion feature.

6. The UAV video target detection method based on space-time correlation according to any one of claims 1-5, characterized in that, The detection head uses deformable convolution for convolution operation, and the deformable convolution adaptively calculates the offset of the convolution kernel by using a convolution layer for calculating the offset and a convolution layer for calculating the weight value and the corresponding weight to calculate the pixel at each position according to the offset of the convolution kernel and the corresponding weight , the pixel at each position The calculation expression is: wherein, denotes the size of the convolution kernel, is the corresponding weight at position, denotes the original pixel position, denotes the fixed convolution kernel shift, denotes the offset value calculated by the convolution network, denotes the pixel at position in the input image, denotes the first t - l frame to the first t frame.

7. A spatio-temporal correlation based UAV video target detection system for use in the method of any one of claims 1-6. Comprising: A feature extraction network for inputting continuous video image data detected by a camera carried by an unmanned aerial vehicle for feature extraction to obtain an initial feature sequence; The spatio-temporal feature extraction module is configured to embed the initial feature sequence in a spatial position and perform deep convolution processing, use a state space equation to model the spatio-temporal correlation between different frame features, and fuse the initial feature after a specified proportion adjustment with the frame features through residual connection to form spatio-temporal features that retain time and space information. The foreground attention module is configured to input the spatio-temporal features into an attention network with residual connection, set different weights for the frame spatio-temporal features according to the appearance similarity between different frames and the current frame by the attention network with residual connection, and fuse the initial feature with the feature after the attention mechanism through residual connection to obtain the spatio-temporal features after foreground attention processing. The spatio-temporal fusion module is configured to fuse the initial feature with the spatio-temporal features after foreground attention processing in the time and space dimensions to obtain spatio-temporal fusion features. The target detection module is configured to input the spatio-temporal fusion features into a detection head, perform target detection according to the spatio-temporal fusion features by the detection head, and output a target detection result.

8. The spatio-temporal correlation based UAV video target detection system of claim 7, wherein, The spatio-temporal feature extraction module includes a position embedding unit, a deep convolution unit, a state space equation unit, a first proportion adjustment unit, two linear layers, and two layer normalization layers. The position embedding unit inputs the initial feature sequence extracted by the feature extraction network, embeds the spatial position, and then performs deep convolution by the deep convolution unit. After the deep convolution, the result is processed by the first linear layer, and then the state space equation unit uses the state space equation to model the spatio-temporal correlation between different frame features. The feature output by the state space equation unit is multiplied by the result after the second linear layer of the initial feature after the first layer normalization layer. The multiplied result is superimposed with the result after the initial feature is adjusted by the proportion adjustment unit according to the specified proportion, and the superimposed result is processed by the second layer normalization layer to obtain the final extracted feature map that retains time and space information.

9. An electronic device comprising a processor and a memory for storing a computer program, characterized in that The processor is configured to execute the computer program to perform the method of any one of claims 1-6.

Images