Multi-view cooperative tracking method and device for fast moving target under low illumination condition

Abstract

The application discloses a kind of low-illumination condition fast moving target multi-view cooperative tracking method and device, the method includes low-illumination condition fast moving target multi-view tracking algorithm and unmanned aerial vehicle cluster cooperative tracking algorithm;Multi-source environment perception is carried out using millimeter wave radar sensor, RGB vision sensor and infrared vision sensor carried by unmanned aerial vehicle;Multi-source perception data is normalized in time and space dimensions by three space-time heterogeneous alignment algorithm;Multi-source fusion perception data after space-time normalization is input into multi-source fusion perception feature calculation network to calculate the world coordinates of the tracked target under the current frame;The calculation result is compared with the calculation result of last frame, to judge the change of the position and moving direction of the tracked target in world coordinate system;The world coordinate information of the tracked target is shared through cluster communication, and the trajectory of unmanned aerial vehicle cluster is formulated and updated using multi-agent reinforcement learning, to realize unmanned aerial vehicle cluster cooperative tracking.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target tracking technology, and in particular to a multi-view collaborative tracking method and device for fast-moving targets under low-light conditions. Background Technology

[0002] In recent years, drone technology has developed rapidly. Its integration with artificial intelligence technologies such as deep learning and reinforcement learning has enabled drones to perform a variety of tasks. Furthermore, due to their small size, flexibility, and autonomous intelligence, drones are widely used in civilian and military fields, such as location mapping and path planning. Among the various applications of drones, target tracking is the core of most tasks. However, drone target tracking tasks are constrained by the complexity of the scene and the variability of the target being tracked, making target feature extraction and model building difficult and posing a significant challenge to tracking performance.

[0003] Traditional target tracking technologies mostly use visual data as the data source for target tracking. However, in low-light scenarios, such data often suffers from defects such as blurred visual information and insufficient information contained in target features, which seriously affects the final tracking performance. On the other hand, UAV-based target tracking methods mostly use single UAVs to track related targets. In complex environments such as underground caves, the target is easily lost due to obstacles and sudden changes in terrain. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a method and apparatus for multi-view cooperative tracking of fast-moving targets under low-light conditions. In low-light scenarios, a drone swarm is deployed, and each drone is equipped with a corresponding multi-source fusion sensing device as a source of multi-view fusion data. After any agent in the drone swarm completes target localization, a multi-agent reinforcement learning algorithm is used to achieve cooperative tracking, thereby achieving the goal of multi-view cooperative tracking of fast-moving targets under low-light conditions.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide the following solutions:

[0006] On the one hand, a multi-view cooperative tracking method for fast-moving targets under low illumination conditions is provided, the method including a multi-view tracking algorithm for fast-moving targets under low illumination conditions and a UAV swarm cooperative tracking algorithm;

[0007] The multi-view tracking algorithm for fast-moving targets under low-light conditions includes the following steps:

[0008] Multi-source environmental perception is achieved by using sensors carried by drones, including millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors.

[0009] The multi-source sensing data received from different sensors is normalized in the temporal and spatial dimensions by using a three-temporal heterogeneous alignment algorithm;

[0010] The spatiotemporally normalized multi-source fusion sensing data is input into the multi-source fusion sensing feature calculation network to calculate the world coordinates of the tracked target in the current frame.

[0011] The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system.

[0012] The drone swarm cooperative tracking algorithm includes the following steps:

[0013] By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

[0014] Preferably, the three-spatial-temporal heterogeneous alignment algorithm specifically includes:

[0015] First, the multi-source sensing data collected by the sensors is preprocessed; for the sensing data from the millimeter-wave radar sensor, invalid data, including stationary targets and empty targets, is removed; for the sensing data from the RGB vision sensor and the infrared vision sensor, image denoising is performed to highlight target features.

[0016] Next, the sensor data from the denoised RGB vision sensor and the infrared vision sensor are spatiotemporally aligned. Since the viewing directions of the two sensors are relatively fixed on the same drone, the SIFT operator from the machine learning method is used for alignment.

[0017] Based on this, the perception data of the millimeter-wave radar sensor is calibrated using the perception data of the RGB vision sensor as a reference. For temporal heterogeneity, radar-vision time calibration is performed based on the sampling frequency of the millimeter-wave radar perception data. For spatial heterogeneity, a unified coordinate system is established according to the actual deployment locations of the two sensors, and the transformation relationship between their respective coordinate systems and the world coordinate system is constructed. Finally, the spatiotemporal heterogeneous alignment of the three types of perception data is completed.

[0018] Preferably, after data preprocessing is completed, the scale normalization of multi-source sensing data is performed using a three-spatial-temporal heterogeneous alignment algorithm; the sensing data of infrared vision sensor and millimeter-wave radar sensor are aligned respectively based on the spatial scale and temporal dimension of the sensing data of RGB vision sensor.

[0019] For aligning the perception data of the infrared vision sensor and the RGB vision sensor, since both are fixed relative to the drone body and the fluctuation of the viewing angle is small, the SIFT operator is used for calibration. The spatial parameters of the RGB vision sensor perception data are fixed, and the infrared vision sensor perception data in the same time dimension is used as input to output pixel-aligned infrared vision sensor perception data. In the time dimension, the two are aligned using the same sampling frequency.

[0020] For aligning the perception data of the millimeter-wave radar sensor and the RGB vision sensor, the time calibration of the two sensors uses the sampling frequency of the RGB vision sensor as the time reference. Each sampling sends a pulse frequency, and the central control unit, upon receiving this pulse frequency, determines the frame number in the millimeter-wave radar sensor sampling as the sampling result, thus achieving time calibration. For spatial calibration, a unified coordinate system is required, defining the radar video as solving for a feature matrix [x, y, 1]. T The radar physical coordinates of four pairs of targets were collected and calibrated using the left side of the video. Then, the calibration relationship between the radar and the world coordinate system was established. The transformation relationship calculation formula is as follows:

[0021]

[0022] Where (x,y) are the radar-detected target coordinates, (x',y') are the transformed world coordinates, Δ is the scaling relationship between the relevant coordinate axes, α and β represent the translation amount of the corresponding coordinate axes, and θ represents the coordinate system rotation angle.

[0023] Preferably, the multi-source fusion sensing feature calculation network is a convolutional neural network, which includes feature extraction modules, feature fusion modules, and target location prediction modules for each sensor.

[0024] The feature extraction module extracts target features from the sensor data of each sensor, sends the extracted features to the feature fusion module to obtain multi-source fused sensing features, and finally calculates the target's position in the current frame based on the target position prediction module. It also transforms the target's position in the current world coordinate system based on the relative position of the drone's sensors that captured the target.

[0025] Preferably, in the feature extraction module, a feature extraction network is built for the sensor data of the three types of sensors. The lightweight CSPDarknet53 network is used to extract features from the RGB vision sensor data and the infrared vision sensor data, and the lightweight VGG-13 network is used to extract features from the millimeter-wave radar sensor data. The millimeter-wave radar sensor input sensor data is converted into radar features of the same size as the RGB vision sensor data.

[0026] In the feature fusion module, the features of the three types of sensor data are further processed and concatenated into a composite vector. The calculation formula is as follows:

[0027] F fusion =σ(f radar +f RGB +f IR )

[0028] f * =W2·τ(W1·τ(W0(AvgPool(F* ) )))

[0029] In the formula, σ and τ represent the Sigmoid activation function and the ReLU activation function, respectively, W0, W1, and W2 are the network weight parameters, and F... * For the feature map, f radar For the extracted radar features, f RGB For the extracted RGB visual features, f IR For the extracted infrared visual features, F fusion For fusion features, f * For splicing feature maps;

[0030] The stitched feature map is sent to the target position prediction module to determine the target position; then, based on the transformation relationship between the RGB visual sensor coordinate system and the world coordinate system, the current position of the target in the world coordinate system is calculated.

[0031] Preferably, the UAV swarm cooperative tracking algorithm specifically includes:

[0032] For any drone in the drone swarm, after completing the target localization, the world coordinates of the target and its own position information are shared through the drone swarm communication function;

[0033] Then, a multi-agent reinforcement learning method is used to make other drones in the drone swarm move closer to the coordinates of the tracked target, thus achieving collaborative tracking of the drone swarm.

[0034] Preferably, the use of multi-agent reinforcement learning to make other drones in the drone swarm move closer together around the coordinates of the tracked target specifically means:

[0035] By using a multi-agent reinforcement learning method, other drones in the drone swarm that have not yet located the target move closer to the target location. Drones that have completed target location keep the target centered within their respective perception range, ultimately achieving collaborative tracking of the drone swarm.

[0036] On the other hand, a multi-view cooperative tracking device for fast-moving targets under low illumination conditions is provided, including a multi-view tracking algorithm module for fast-moving targets under low illumination conditions and a UAV swarm cooperative tracking algorithm module.

[0037] The multi-view tracking algorithm module for fast-moving targets under low-light conditions is used for:

[0038] Multi-source environmental perception is achieved by using sensors carried by drones, including millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors.

[0039] The multi-source sensing data received from different sensors is normalized in the temporal and spatial dimensions by using a three-temporal heterogeneous alignment algorithm;

[0040] The spatiotemporally normalized multi-source fusion sensing data is input into the multi-source fusion sensing feature calculation network to calculate the world coordinates of the tracked target in the current frame.

[0041] The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system.

[0042] The drone swarm cooperative tracking algorithm module is used for:

[0043] By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

[0044] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described multi-view cooperative tracking method for fast-moving targets under low-light conditions.

[0045] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described multi-view cooperative tracking method for fast-moving targets under low-light conditions.

[0046] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0047] In this embodiment of the invention, a multi-view collaborative tracking method for fast-moving targets under low-light conditions is provided. In low-light scenarios, feature extraction is performed on multi-source fusion sensor data of a UAV swarm to determine the world coordinate system position of the target, and coordinate information is shared through swarm communication to collaboratively track the high-speed moving target.

[0048] The aforementioned multi-view collaborative tracking method for fast-moving targets under low-light conditions fuses and extracts features from the output data of millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors. After feature extraction, a multi-source fusion perception feature computing network is used to determine the world coordinate system position of the target. Subsequently, the UAV target tracking results are shared through cluster communication, and a multi-agent reinforcement learning algorithm is used for trajectory planning and updating. This improves the quality of target feature information extraction, reduces perception and communication losses, and enhances the target tracking effect in low-light scenarios.

[0049] This invention proposes a multi-view collaborative tracking method for fast-moving targets under low-light conditions, which can perform multi-angle collaborative observation of fast-moving objects in underground weak texture and low-light environments, thereby improving the tracking quality and efficiency of UAV swarms. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the multi-view collaborative tracking method for fast-moving targets under low illumination conditions provided in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of the three-spatial-temporal heterogeneous alignment algorithm provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the multi-source fusion sensing feature computing network provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the UAV swarm collaborative tracking algorithm provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Embodiments of this invention provide a multi-view cooperative tracking method for fast-moving targets under low-light conditions, applicable to scenarios with weak illumination (such as underground caves). Reference Figure 1As shown, the method includes a multi-view tracking algorithm for fast-moving targets under low illumination conditions and a collaborative tracking algorithm for UAV swarms.

[0057] The multi-view tracking algorithm for fast-moving targets under low-light conditions includes the following steps:

[0058] Multi-source environmental perception is achieved using sensors onboard the UAV, including millimeter-wave radar, RGB vision, and infrared vision sensors. A three-time-space heterogeneous alignment algorithm is used to normalize the multi-source perception data received from different sensors in both time and space dimensions. The spatiotemporally normalized multi-source fusion perception data is then input into a multi-source fusion perception feature calculation network to calculate the world coordinates of the tracked target in the current frame. The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system.

[0059] The drone swarm cooperative tracking algorithm includes the following steps:

[0060] By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

[0061] In this embodiment of the invention, a drone swarm is used to track high-speed moving targets under low-light conditions within a defined area. A radar-visual fusion algorithm is used to combine data from two types of sensors: millimeter-wave radar and airborne cameras (RGB visual sensor and infrared visual sensor). The fused feature parameters are extracted and compared with the feature parameters of the previous frame to determine the position and movement direction changes of the tracked target. Then, the perception information is fused into the communication information and transmitted in the drone swarm, reducing data transmission loss while improving the swarm's positioning and deployment capabilities.

[0062] Furthermore, such as Figure 2 As shown, the three-spatial-temporal heterogeneous alignment algorithm specifically includes:

[0063] First, the multi-source sensing data collected by the sensors is preprocessed; for the sensing data from the millimeter-wave radar sensor, invalid data, including stationary targets and empty targets, is removed; for the sensing data from the RGB vision sensor and the infrared vision sensor, image denoising is performed to highlight target features.

[0064] Next, the sensor data from the denoised RGB vision sensor and the infrared vision sensor are spatiotemporally aligned. Since the viewing directions of the two sensors are relatively fixed on the same drone, the SIFT operator from the machine learning method is used for alignment.

[0065] Based on this, the perception data of the millimeter-wave radar sensor is calibrated using the perception data of the RGB vision sensor as a reference. For temporal heterogeneity, radar-vision time calibration is performed based on the sampling frequency of the millimeter-wave radar perception data. For spatial heterogeneity, a unified coordinate system is established according to the actual deployment locations of the two sensors, and the transformation relationship between their respective coordinate systems and the world coordinate system is constructed. Finally, the spatiotemporal heterogeneous alignment of the three types of perception data is completed.

[0066] Specifically, after data preprocessing is completed, the scale normalization of multi-source sensing data is performed using a three-spatial-temporal heterogeneous alignment algorithm; the sensing data of infrared vision sensor and millimeter-wave radar sensor are aligned respectively based on the spatial scale and temporal dimension of the sensing data of RGB vision sensor. Figure 2 (a) is a schematic diagram of the alignment process between infrared visual perception data and RGB visual perception data, and (b) is a schematic diagram of the alignment process between visual perception data and millimeter-wave radar perception data.

[0067] For aligning the perception data of infrared and RGB vision sensors, since both are fixed relative to the drone body and their viewing angles fluctuate relatively little, the SIFT operator is used for calibration. The spatial parameters of the RGB vision sensor data are fixed, and the infrared vision sensor data at the same time dimension is used as input, resulting in pixel-aligned infrared vision sensor data. Specifically, the spatial parameters of the RGB vision sensor data are fixed, and the infrared vision sensor data at the same time dimension is used as input. Different Gaussian kernels are selected to generate image clusters. These image clusters are subtracted from each other and downsampled to calculate a multi-scale Gaussian difference map. Then, pixels on the Gaussian difference map are compared with surrounding pixels, and the maximum value is used to replace the pixel, obtaining keypoints. Using the keypoint as the center, all gradient directions within the circle are superimposed onto eight directions, and the maximum value is selected as the principal direction of the keypoint. Based on this, a 128-dimensional vector is used to describe the keypoint features, and the Euclidean distance between these vectors is used for feature matching. Finally, the infrared vision perception data after feature matching is output. In the time dimension, both sensors use the same sampling frequency for alignment.

[0068] For aligning the perception data of the millimeter-wave radar sensor and the RGB vision sensor, the time calibration of the two sensors uses the sampling frequency of the RGB vision sensor as the time reference. Each sampling sends a pulse frequency, and the central control unit, upon receiving this pulse frequency, determines the frame number in the millimeter-wave radar sensor sampling as the sampling result, thus achieving time calibration. For spatial calibration, a unified coordinate system is required, defining the radar video as solving for a feature matrix [x, y, 1]. TThe radar physical coordinates of four pairs of targets were collected and calibrated using the left side of the video. Then, the calibration relationship between the radar and the world coordinate system was established. The transformation relationship calculation formula is as follows:

[0069]

[0070] Where (x,y) are the radar-detected target coordinates, (x',y') are the transformed world coordinates, Δ is the scaling relationship between the relevant coordinate axes, α and β represent the translation amount of the corresponding coordinate axes, and θ represents the coordinate system rotation angle.

[0071] Furthermore, the multi-source fusion sensing feature calculation network is a convolutional neural network, which includes feature extraction modules, feature fusion modules, and target location prediction modules for each sensor.

[0072] The feature extraction module extracts target features from the sensor data of each sensor, sends the extracted features to the feature fusion module to obtain multi-source fused sensing features, and finally calculates the target's position in the current frame based on the target position prediction module. It also transforms the target's position in the current world coordinate system based on the relative position of the drone's sensors that captured the target.

[0073] Figure 3 This is a schematic diagram of the multi-source fusion sensing feature computing network. After completing the spatiotemporal alignment operation, the three types of sensing data are sent into the multi-source fusion sensing feature computing network.

[0074] In the feature extraction module, feature extraction networks are built for the three types of sensor data. The lightweight CSPDarknet53 network is used to extract features from the RGB vision sensor data and the infrared vision sensor data, and the lightweight VGG-13 network is used to extract features from the millimeter-wave radar sensor data. The millimeter-wave radar sensor input data is converted into radar features of the same size as the RGB vision sensor data.

[0075] In the feature fusion module, the features of the three types of sensor data are further processed and concatenated into a composite vector. The calculation formula is as follows:

[0076] F fusion =σ(f radar +f RGB +f IR )

[0077] f * =W2·τ(W1·τ(W0(AvgPool(F)) * ))))

[0078] In the formula, σ and τ represent the Sigmoid activation function and the ReLU activation function, respectively, W0, W1, and W2 are the network weight parameters, and F... * For the feature map, f radar For the extracted radar features, f RGB For the extracted RGB visual features, f IR For the extracted infrared visual features, F fusion For fusion features, f * For splicing feature maps;

[0079] The stitched feature map is sent to the target position prediction module to determine the target position; then, based on the transformation relationship between the RGB visual sensor coordinate system and the world coordinate system, the current position of the target in the world coordinate system is calculated.

[0080] Furthermore, Figure 4 This is a schematic diagram of the UAV swarm cooperative tracking algorithm in an embodiment of the present invention. The UAV swarm cooperative tracking algorithm specifically includes:

[0081] For any drone in a drone swarm, after completing target localization, the drone swarm communication function is used to share the world coordinates of the target and its own position information. Then, a multi-agent reinforcement learning method is used to make the other drones in the drone swarm move closer to the coordinates of the tracked target, so as to achieve collaborative tracking of the drone swarm.

[0082] Specifically, the use of multi-agent reinforcement learning to make other drones in the drone swarm move closer together around the coordinates of the tracked target refers to:

[0083] By employing multi-agent reinforcement learning, other drones in the swarm that have not yet located the target move closer to the target location. Drones that have already located the target then keep it centered within their respective perception ranges. Figure 4 As shown in (a), the final goal is to achieve collaborative tracking of the UAV swarm, as follows: Figure 4 As shown in (b).

[0084] In summary, the embodiments of the present invention provide a multi-view collaborative tracking method for fast-moving targets under low-light conditions. In low-light scenarios, feature extraction is performed on multi-source fusion sensor data of a UAV swarm to determine the world coordinate system position of the target, and coordinate information is shared through swarm communication to collaboratively track high-speed moving targets.

[0085] The aforementioned multi-view collaborative tracking method for fast-moving targets under low-light conditions fuses and extracts features from the output data of millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors. After feature extraction, a multi-source fusion perception feature computing network is used to determine the world coordinate system position of the target. Subsequently, the UAV target tracking results are shared through cluster communication, and a multi-agent reinforcement learning algorithm is used for trajectory planning and updating. This improves the quality of target feature information extraction, reduces perception and communication losses, and enhances the target tracking effect in low-light scenarios.

[0086] The aforementioned multi-view collaborative tracking method for fast-moving targets under low illumination conditions can perform multi-angle collaborative observation of fast-moving objects in underground weak texture and low illumination environments, thereby improving the tracking quality and efficiency of UAV swarms.

[0087] Accordingly, embodiments of the present invention also provide a multi-view cooperative tracking device for fast-moving targets under low illumination conditions, including a multi-view tracking algorithm module for fast-moving targets under low illumination conditions and a UAV swarm cooperative tracking algorithm module;

[0088] The multi-view tracking algorithm module for fast-moving targets under low-light conditions is used for:

[0089] Multi-source environmental perception is achieved by using sensors carried by drones, including millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors.

[0090] The multi-source sensing data received from different sensors is normalized in the temporal and spatial dimensions by using a three-temporal heterogeneous alignment algorithm;

[0091] The spatiotemporally normalized multi-source fusion sensing data is input into the multi-source fusion sensing feature calculation network to calculate the world coordinates of the tracked target in the current frame.

[0092] The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system.

[0093] The drone swarm cooperative tracking algorithm module is used for:

[0094] By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

[0095] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0096] This invention also provides an electronic device, which may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) and one or more memories, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the steps of the above-described multi-view cooperative tracking method for fast-moving targets under low-light conditions.

[0097] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned multi-view cooperative tracking method for fast-moving targets under low-light conditions. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0099] The use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0100] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0101] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-view cooperative tracking method for fast-moving targets under low-light conditions, characterized in that, The method includes a multi-view tracking algorithm for fast-moving targets under low illumination conditions and a cooperative tracking algorithm for UAV swarms; The multi-view tracking algorithm for fast-moving targets under low-light conditions includes the following steps: Multi-source environmental perception is achieved by using sensors carried by drones, including millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors. The multi-source sensing data received from different sensors is normalized in the temporal and spatial dimensions by using a three-temporal heterogeneous alignment algorithm; The three-spatial-temporal heterogeneous alignment algorithm specifically includes: preprocessing the multi-source sensing data collected by the sensors; after preprocessing, aligning the sensing data of the infrared vision sensor and the millimeter-wave radar sensor respectively, based on the spatial scale and temporal dimension of the sensing data of the RGB vision sensor. For the alignment of the perception data of the infrared vision sensor and the RGB vision sensor, the SIFT operator is used for calibration. The spatial parameters of the perception data of the RGB vision sensor are fixed, and the perception data of the infrared vision sensor in the same time dimension is used as input to output pixel-aligned infrared vision sensor perception data. In the time dimension, the two are aligned using the same sampling frequency. For the alignment of perception data between the millimeter-wave radar sensor and the RGB vision sensor, the time calibration of the two sensors uses the sampling frequency of the RGB vision sensor as the time reference. Each sampling sends a pulse frequency, and the central control unit, upon receiving this pulse frequency, determines the frame number in the millimeter-wave radar sensor sampling as the sampling result. For the spatial calibration of the two sensors, the radar video is defined as solving a feature matrix [x,y,1]. T The radar physical coordinates and video of the acquired target are used for calibration to establish the calibration relationship between the radar and the world coordinate system. The transformation formula is as follows: Where (x,y) are the coordinates of the radar-detected target, (x... ' ,y ' ) represents the transformed world coordinates, Δ represents the scaling relationship between the relevant coordinate axes, α and β represent the translation amount of the corresponding coordinate axes, and θ represents the rotation angle of the coordinate system; The spatiotemporally normalized multi-source fusion sensing data is input into the multi-source fusion sensing feature calculation network to calculate the world coordinates of the tracked target in the current frame. The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system. The drone swarm cooperative tracking algorithm includes the following steps: By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

2. The multi-view cooperative tracking method for fast-moving targets under illumination conditions according to claim 1, characterized in that, The multi-source fusion sensing feature calculation network is a convolutional neural network, which includes feature extraction modules, feature fusion modules, and target location prediction modules for each sensor. The feature extraction module extracts target features from the sensor data of each sensor, sends the extracted features to the feature fusion module to obtain multi-source fused sensing features, and finally calculates the target's position in the current frame based on the target position prediction module. It also transforms the target's position in the current world coordinate system based on the relative position of the drone's sensors that captured the target.

3. The multi-view cooperative tracking method for fast-moving targets under illumination conditions according to claim 2, characterized in that, In the feature extraction module, feature extraction networks are built for the three types of sensor data. The lightweight CSPDarknet53 network is used to extract features from the RGB vision sensor data and the infrared vision sensor data, and the lightweight VGG-13 network is used to extract features from the millimeter-wave radar sensor data. The millimeter-wave radar sensor input data is converted into radar features of the same size as the RGB vision sensor data. In the feature fusion module, the features of the three types of sensor data are further processed and concatenated into a composite vector. The calculation formula is as follows: F fusion =σ(f radar +f RGB +f IR ) f * =W2·τ(W1·τ(W0(AvgPool(F * )))) In the formula, σ and τ represent the Sigmoid activation function and the ReLU activation function, respectively, W0, W1, and W2 are the network weight parameters, and F... * For the feature map, f radar For the extracted radar features, f RGB For the extracted RGB visual features, f IR For the extracted infrared visual features, F fusion For fusion features, f * For splicing feature maps; The stitched feature map is sent to the target position prediction module to determine the target position; then, based on the transformation relationship between the RGB visual sensor coordinate system and the world coordinate system, the current position of the target in the world coordinate system is calculated.

4. The multi-view cooperative tracking method for fast-moving targets under illumination conditions according to claim 1, characterized in that, The drone swarm cooperative tracking algorithm specifically includes: For any drone in the drone swarm, after completing the target localization, the world coordinates of the target and its own position information are shared through the drone swarm communication function; Then, a multi-agent reinforcement learning method is used to make other drones in the drone swarm move closer to the coordinates of the tracked target, thus achieving collaborative tracking of the drone swarm.

5. The multi-view cooperative tracking method for fast-moving targets under illumination conditions according to claim 4, characterized in that, The aforementioned use of multi-agent reinforcement learning to make other drones in a drone swarm move closer together around the coordinates of the tracked target specifically refers to: By using a multi-agent reinforcement learning method, other drones in the drone swarm that have not yet located the target move closer to the target location. Drones that have completed target location keep the target centered within their respective perception range, ultimately achieving collaborative tracking of the drone swarm.

6. A multi-view cooperative tracking device for fast-moving targets under low-light conditions, characterized in that, The apparatus is used to implement the method as described in any one of claims 1-5, and the apparatus includes a multi-view tracking algorithm module for fast-moving targets under low illumination conditions and a UAV swarm cooperative tracking algorithm module. The multi-view tracking algorithm module for fast-moving targets under low-light conditions is used for: Multi-source environmental perception is achieved by using sensors carried by drones, including millimeter-wave radar sensors, RGB vision sensors, and infrared vision sensors. The multi-source sensing data received from different sensors is normalized in the temporal and spatial dimensions by using a three-temporal heterogeneous alignment algorithm; The spatiotemporally normalized multi-source fusion sensing data is input into the multi-source fusion sensing feature calculation network to calculate the world coordinates of the tracked target in the current frame. The calculation results are compared with those of the previous frame to determine the changes in the position and direction of movement of the tracked target in the world coordinate system. The drone swarm cooperative tracking algorithm module is used for: By sharing the world coordinates of the tracked target through cluster communication, multi-agent reinforcement learning is used to formulate and update the trajectory of the UAV cluster, ensuring that the tracked target is at the center of the UAV's perception range, thus achieving collaborative tracking by the UAV cluster.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the multi-view cooperative tracking method for fast-moving targets under low light conditions as described in any one of claims 1-5.

8. A computer-readable storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement the multi-view cooperative tracking method for fast-moving targets under low-light conditions as described in any one of claims 1-5.

Images