CN-YOLOv5-based multi-modal data fusion anti-unmanned aerial vehicle high-precision detection method

Abstract

The invention relates to a multi-modal data fusion anti-unmanned aerial vehicle high-precision detection method based on CN-YOLOv5. The invention relates to the technical field of unmanned aerial vehicle detection, and the method comprises the steps: deploying a sensor, and carrying out the data collection; a CN-YOLOv5 model is built, and the model is trained; receiving data according to the trained model, judging whether the unmanned aerial vehicle exists or not, if yes, determining the type, position and speed information of the unmanned aerial vehicle, performing real-time tracking, and generating the flight path of the unmanned aerial vehicle. According to the invention, high-precision detection is realized. Through matching fusion of multi-modal data and a feature fusion method of an attention mechanism and optimization in the model, advantage complementation of the modal data is fully utilized, so that the model can more accurately extract and analyze the target features of the unmanned aerial vehicle, the detection accuracy is remarkably improved, the false alarm rate and the missing report rate are effectively reduced, and the detection efficiency is improved. And different types of unmanned aerial vehicles in different flight states can be accurately identified.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle detection, and is a high-precision anti-unmanned aerial vehicle detection method based on CN-YOLOv5 and multi-modal data fusion. Background Art

[0002] In recent years, with the rapid development of drone technology, drones have been widely used in many fields, but at the same time, they have also brought a series of security risks, such as unauthorized intrusion into sensitive areas, interference with normal flight order, etc. Therefore, anti-drone detection technology has become increasingly important.

[0003] Current anti-UAV technology has many shortcomings: detection based solely on single-mode data (such as visual images, radar data, etc.) is easily affected by environmental factors, target feature complexity, etc., resulting in low detection accuracy, high false alarm and missed alarm rates. For example, under complex lighting conditions, relying solely on visual images may not be able to clearly identify drones; and radar detection may miss detections of some small drones with low reflective cross-sections. Some existing deep learning-based target detection models cannot fully extract key features when dealing with relatively small and highly maneuverable targets such as drones, making it difficult to accurately locate and identify them, especially in multi-target, complex background scenes.

[0004] Although some multimodal detection methods have attempted to fuse multiple sensor data to improve detection performance, there are still deficiencies in feature fusion and fully mining the effective information in multimodal data to achieve high-precision detection and stable tracking. Traditional feature fusion methods may not be able to adaptively optimize fusion according to the characteristics of different modal data and target characteristics, and fail to effectively guide the model to focus on the more critical feature areas for drone detection and tracking, resulting in the need to improve overall detection accuracy and tracking stability, making it difficult to meet the needs of high-precision anti-drone detection and tracking in complex actual scenarios.

[0005] To overcome the above problems, a multimodal fusion detection method based on CN-YOLOv5 can achieve high-precision detection of drones. Compared with traditional anti-drone technology, this system can adaptively match and fuse key features of different modalities, and the detection accuracy has been significantly improved. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention proposes a high-precision anti-UAV detection method based on CN-YOLOv5 multimodal data fusion. This method uses deep learning technology, takes the YOLOv5 target detection model as a template, integrates the attention mechanism of the convolutional attention module (CBAM) and the feature aggregation method of the neural architecture search feature pyramid network (NAS-FPN) into the original neural network framework, and constructs an improved CN-YOLOv5 model. By effectively matching and fusing multimodal data such as visual images and videos, infrared images and videos, radar data, radio frequency signals, and audio signals, deep reasoning and learning are performed. The method fully mines the feature information in each modal data that is valuable for UAV detection and tracking, overcomes the limitations of the prior art, realizes high-precision detection of UAVs and stable and accurate tracking, and ensures the safety of sensitive areas.

[0007] The present invention provides a high-precision anti-UAV detection method based on CN-YOLOv5 multimodal data fusion, including the following technical solutions:

[0008] A multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5, the method comprising the following steps:

[0009] Step 1: Deploy sensors and collect data;

[0010] Step 2: Build the CN-YOLOv5 model and train the model;

[0011] Step 3: Based on the trained model, after receiving the data, determine whether there is a drone. If so, determine its category, location, and speed information, and track it in real time to generate the drone's flight trajectory.

[0012] Preferably, the step 1 is specifically:

[0013] Deploy visual image and video acquisition equipment: plan the location and number of cameras according to the airport runway, apron and surrounding airspace range and key area distribution to ensure full airspace coverage and overlapping fields of view; set the camera frame rate to 30 frames per second to collect images, and store the data in the monitoring data center according to timestamps and camera numbers to provide visual basis for subsequent detection;

[0014] Deploy infrared image and video acquisition equipment: Infrared thermal imaging equipment and visual cameras are installed in coordination and aimed at the same airspace; their parameters are adjusted according to the airport ambient temperature and monitoring distance so that the infrared image clearly presents the thermal characteristics of the drone. The collected data is stored in the monitoring data center in time sequence for subsequent fusion analysis;

[0015] Deployment of radar sensors: The airport is equipped with primary and backup air traffic control radars and low-altitude blind spot radars, which are reasonably deployed and positioned according to the airspace range and geographical boundaries, and the scanning cycle and beam width parameters are set to obtain radar echo data in real time, including various parameters and acquisition time, and transmit them to the radar data processing server for synchronization with other modal data;

[0016] Deploy radio frequency signal monitoring equipment: Arrange radio frequency signal detection equipment combining directional and omnidirectional antennas on the roof of the terminal building and at the runway entrance. Set the working frequency band to cover the common drone frequency band. After pre-processing the collected signals, store them in the radio frequency data storage server according to characteristics and time, to assist in determining the drone model and other information.

[0017] Deploy audio sensors: Place microphone arrays on both sides of the runway, on the apron, and near the terminal building. Determine parameters based on ambient noise and positioning accuracy. Convert the collected audio signals into digital data and store them in the audio data server according to timestamps and array configuration information to assist in detecting the status of drones.

[0018] Preferably, the step 2 is specifically:

[0019] After extracting the data, normalize the size, color correct, and denoise the visual and infrared images, and normalize the radar, radio frequency, and audio data; annotate the drone category, position coordinates of each mode, and related feature attributes to build a training data set;

[0020] Based on the common deep learning framework, a CN-YOLOv5 model integrating CBAM and NAS-FPN was built. The hyperparameters of the CBAM module were reasonably set according to the input feature map and other parameters. The hyperparameters of the NAS-FPN module were set according to the computing resources and fusion effect. At the same time, the model training parameters such as learning rate, batch size, and number of training rounds were determined. The labeled data set was divided into training set, validation set, and test set according to the ratio of 8:1:1. The model was trained with the optimizer, a comprehensive loss function was constructed, the parameters were updated through back propagation, the parameters were adjusted in time according to the performance indicators of the validation set, and the weight file was saved after meeting the standards.

[0021] Preferably, the convolutional attention module CBAM includes a channel attention module and a spatial attention module:

[0022] The formula of the channel attention module is as follows:

[0023]

[0024] The spatial attention module is expressed as follows:

[0025]

[0026] Preferably, the output M after channel attention modulation c(F) is convolved with the input feature map F to obtain the spatial attention module input F', and the spatial attention module output is convolved with F' to obtain the output F" of the entire CBAM:

[0027]

[0028] Preferably, the feature fusion process of NAS-FPN is as follows: the feature map with lower resolution is upsampled to make it consistent with the size of the feature map with higher resolution, and then added and fused. The two-dimensional upsampling formula is as follows:

[0029]

[0030] After upsampling With F i Add element by element to get the fused feature map F fusion , the formula is as follows:

[0031]

[0032] The formula used in multimodal data feature matching fusion is as follows: Represents the matching fusion feature map extracted from the lth layer:

[0033]

[0034] The multimodal data is put into the model for training, and the collected multimodal data is uniformly input into the CN-YOLOv5 model for reasoning and learning.

[0035] Preferably, the step 3 is specifically:

[0036] Each acquisition device transmits new data to the data processing center in real time, which is synchronized according to timestamps and pre-processed before being input into the trained model. After receiving the data, the model determines whether there is a drone. If so, it determines its category, integrates multi-modal positioning information and speed information, and uses Kalman filtering combined with the Hungarian algorithm for real-time tracking to generate the drone's flight trajectory. The detection and tracking results are displayed on the monitoring screen. In case of abnormalities, alarms are triggered and data is recorded to ensure the safety of the airport airspace.

[0037] A multi-modal data fusion anti-UAV high-precision detection system based on CN-YOLOv5, the system comprising:

[0038] Data collection module, which deploys sensors and collects data:

[0039] A model training module, wherein the model training module builds a CN-YOLOv5 model and trains the model;

[0040] The trajectory detection module determines whether there is a drone after receiving the data based on the trained model. If so, it determines its category, location and speed information, and performs real-time tracking to generate the drone flight trajectory.

[0041] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a multi-modal data fusion anti-UAV high-precision detection method based on CN-YOLOv5.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5 is implemented.

[0043] The present invention has the following beneficial effects:

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] The present invention achieves high-precision detection. Through the matching fusion of multimodal data, the attention mechanism in the model and the optimized feature fusion method, the complementary advantages of each modal data are fully utilized, so that the model can more accurately extract and analyze the target features of drones, significantly improve the accuracy of detection, effectively reduce the false alarm rate and missed alarm rate, and accurately identify drones of different types and different flight states.

[0046] The present invention has strong environmental adaptability. Whether in various complex natural environments such as day, night, sunny, rainy or snowy, or in the face of complex background interference (such as buildings and trees in urban environments), the synergy of multiple modal data enables the present invention to stably detect and track drones, and has good environmental adaptability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 Shown is a schematic diagram of the overall structure of the present invention;

[0049] Figure 2 Shown is a schematic diagram of the CN-YOLOv5 model structure of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below in conjunction with specific embodiments. Specific embodiment one:

[0052] according to Figure 1 to Figure 2 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention relates to a multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5.

[0053] A multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5, the method comprising the following steps:

[0054] Step 1: Deploy sensors and collect data:

[0055] Step 2: Build the CN-YOLOv5 model and train the model;

[0056] Step 3: Based on the trained model, after receiving the data, determine whether there is a drone. If so, determine its category, location, and speed information, and track it in real time to generate the drone's flight trajectory.

[0057] The present invention first collects multimodal data: visual images and videos: using high-definition cameras and other equipment to collect UAV appearance images and video clips of the flight process under different viewing angles and different lighting conditions, so as to obtain the appearance feature information of the UAV such as shape, color, texture, etc.; infrared images and videos: using infrared thermal imaging equipment to collect infrared images containing UAVs, especially at night or in low-light environments, infrared images can highlight the features such as the heating components of the UAV to assist in identification; radar data: using the radar system to obtain the UAV's distance, speed, direction and other motion information, and assist in judgment and positioning from the dynamic angle of the target; radio frequency signals: using radio frequency detection equipment to collect radio frequency signals for communication between the UAV and the remote control, and identifying the type of the UAV, communication frequency band and other related attributes based on the signal characteristics; audio signals: using microphones and other audio collection equipment to obtain the sound signals generated by the UAV during flight. The sounds emitted by UAVs of different types and states have certain characteristic differences, which can assist in detection.

[0058] Then, the target detection model is constructed and improved. The YOLOv5 target detection model is used as the basic framework, and two important mechanisms are integrated to construct the improved CN-YOLOV5. One of them is the attention mechanism. The convolutional attention mechanism module (CBAM) is embedded in the backbone network (Backbone) of the model. This mechanism enables the model to adaptively focus on the key areas in the input image or feature map, that is, the area where the drone is located and the part that plays a key role in its identification. For example, for visual images, it can focus on the contours, key components and other areas of the drone, suppressing the interference of irrelevant background information; when processing the features corresponding to the radar data, it pays more attention to the key data points that reflect the changes in the position and motion state of the drone. In this way, the model's ability to extract effective features is improved, and the ability to characterize drone targets is enhanced. The other is an improved feature aggregation method, which introduces the neural architecture search optimized feature pyramid network (NAS-FPN) to replace the FPN module in the Neck part of the original YOLOv5 model for feature fusion. Different from the traditional fixed-structure feature fusion method, NAS-FPN automatically explores and determines the optimal feature fusion architecture through neural architecture search technology. Compared with the traditional multimodal detection method, it can match and fuse the features extracted from different modal data through different levels of the model. It can reasonably and efficiently fuse the high-level features of high semantic information in visual images, the middle-level features reflecting thermal features in infrared images, and the bottom-level features reflecting position and motion in radar data, etc., fully explore the complementarity between the features of each modality, so that the fused features have multimodal advantages, providing strong support for subsequent accurate target detection and classification.

[0059] The present invention achieves high-precision detection. Through the matching fusion of multimodal data, the attention mechanism in the model and the optimized feature fusion method, the complementary advantages of each modal data are fully utilized, so that the model can more accurately extract and analyze the target features of drones, significantly improve the accuracy of detection, effectively reduce the false alarm rate and missed alarm rate, and accurately identify drones of different types and different flight states. Specific embodiment 2:

[0061] The difference between the second embodiment of the present application and the first embodiment is that:

[0062] The step 1 is specifically as follows:

[0063] Deploy visual image and video acquisition equipment: plan the location and number of cameras according to the airport runway, apron and surrounding airspace range and key area distribution to ensure full airspace coverage and overlapping fields of view; set the camera frame rate to 30 frames per second to collect images, and store the data in the monitoring data center according to timestamps and camera numbers to provide visual basis for subsequent detection;

[0064] Deploy infrared image and video acquisition equipment: Infrared thermal imaging equipment and visual cameras are installed in coordination and aimed at the same airspace; their parameters are adjusted according to the airport ambient temperature and monitoring distance so that the infrared image clearly presents the thermal characteristics of the drone. The collected data is stored in the monitoring data center in time sequence for subsequent fusion analysis;

[0065] Deployment of radar sensors: The airport is equipped with primary and backup air traffic control radars and low-altitude blind spot radars, which are reasonably deployed and positioned according to the airspace range and geographical boundaries, and the scanning cycle and beam width parameters are set to obtain radar echo data in real time, including various parameters and acquisition time, and transmit them to the radar data processing server for synchronization with other modal data;

[0066] Deploy radio frequency signal monitoring equipment: Arrange radio frequency signal detection equipment combining directional and omnidirectional antennas on the roof of the terminal building and at the runway entrance. Set the working frequency band to cover the common drone frequency band. After pre-processing the collected signals, store them in the radio frequency data storage server according to characteristics and time, to assist in determining the drone model and other information.

[0067] Deploy audio sensors: Place microphone arrays on both sides of the runway, on the apron, and near the terminal building. Determine parameters based on ambient noise and positioning accuracy. Convert the collected audio signals into digital data and store them in the audio data server according to timestamps and array configuration information to assist in detecting the status of drones. Specific embodiment three:

[0069] The difference between the third embodiment of the present application and the second embodiment is that:

[0070] The step 2 is specifically as follows:

[0071] After extracting the data, normalize the size, color correct, and denoise the visual and infrared images, and normalize the radar, radio frequency, and audio data; annotate the drone category, position coordinates of each mode, and related feature attributes to build a training data set;

[0072] Based on the common deep learning framework, a CN-YOLOv5 model integrating CBAM and NAS-FPN was built. The hyperparameters of the CBAM module were reasonably set according to the input feature map and other parameters. The hyperparameters of the NAS-FPN module were set according to the computing resources and fusion effect. At the same time, the model training parameters such as learning rate, batch size, and number of training rounds were determined. The labeled data set was divided into training set, validation set, and test set according to the ratio of 8:1:1. The model was trained with the optimizer, a comprehensive loss function was constructed, the parameters were updated through back propagation, the parameters were adjusted in time according to the performance indicators of the validation set, and the weight file was saved after meeting the standards. Specific embodiment four:

[0074] The difference between the fourth embodiment of the present application and the third embodiment is that:

[0075] The convolutional attention module CBAM includes a channel attention module and a spatial attention module:

[0076] The formula of the channel attention module is as follows:

[0077]

[0078] The spatial attention module is expressed as follows:

[0079] Specific embodiment five:

[0081] The difference between the fifth embodiment of the present invention and the fourth embodiment is that:

[0082] The output M after channel attention modulation c (F) is convolved with the input feature map F to obtain the spatial attention module input F', and the spatial attention module output is convolved with F' to obtain the output F" of the entire CBAM:

[0083] Specific embodiment six:

[0085] The difference between the sixth embodiment of the present invention and the fifth embodiment is that:

[0086] The feature fusion process of NAS-FPN is as follows: the feature map with lower resolution is upsampled to make it consistent with the size of the feature map with higher resolution, and then added and fused. The two-dimensional upsampling formula is as follows:

[0087]

[0088] After upsampling With F i Add element by element to get the fused feature map F fusion , the formula is as follows:

[0089]

[0090] The formula used in multimodal data feature matching fusion is as follows: Represents the matching fusion feature map extracted from the lth layer:

[0091]

[0092] The multimodal data is put into the model for training, and the collected multimodal data is uniformly input into the CN-YOLOv5 model for reasoning and learning. Specific embodiment seven:

[0094] The difference between the seventh embodiment of the present invention and the sixth embodiment is that:

[0095] The step 3 is specifically as follows:

[0096] Each acquisition device transmits new data to the data processing center in real time, which is synchronized according to timestamps and pre-processed before being input into the trained model. After receiving the data, the model determines whether there is a drone. If so, it determines its category, integrates multi-modal positioning information and speed information, and uses Kalman filtering combined with the Hungarian algorithm for real-time tracking to generate the drone's flight trajectory. The detection and tracking results are displayed on the monitoring screen. In case of abnormalities, alarms are triggered and data is recorded to ensure the safety of the airport airspace. Specific embodiment eight:

[0098] The difference between the eighth embodiment of the present invention and the seventh embodiment is that:

[0099] The present invention provides a multi-modal data fusion anti-UAV high-precision detection system based on CN-YOLOv5, which is characterized in that: the system comprises:

[0100] Data collection module, which deploys sensors and collects data:

[0101] A model training module, wherein the model training module builds a CN-YOLOv5 model and trains the model;

[0102] The trajectory detection module determines whether there is a drone after receiving the data based on the trained model. If so, it determines its category, location and speed information, and performs real-time tracking to generate the drone flight trajectory. Specific embodiment nine:

[0104] The difference between the ninth embodiment of the present invention and the eighth embodiment is that:

[0105] The present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a multi-modal data fusion anti-UAV high-precision detection method based on CN-YOLOv5.

[0106] The method comprises the following steps:

[0107] Step 1: Deploy sensors and collect data:

[0108] Step 2: Build the CN-YOLOv5 model and train the model;

[0109] Step 3: Based on the trained model, after receiving the data, determine whether there is a drone. If so, determine its category, location, and speed information, and track it in real time to generate the drone's flight trajectory. Specific embodiment ten:

[0111] The difference between the tenth embodiment of the present invention and the ninth embodiment is that:

[0112] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5 is implemented.

[0113] The method comprises the following steps:

[0114] Step 1: Deploy sensors and collect data:

[0115] Step 2: Build the CN-YOLOv5 model and train the model;

[0116] Step 3: Based on the trained model, after receiving the data, determine whether there is a drone. If so, determine its category, location, and speed information, and track it in real time to generate the drone's flight trajectory. Specific embodiment eleven:

[0118] The difference between the eleventh embodiment of the present invention and the tenth embodiment is that:

[0119] In view of the problems existing in the prior art, the present invention proposes a high-precision anti-UAV detection method based on CN-YOLOv5 multimodal data fusion. This method uses deep learning technology, takes the YOLOv5 target detection model as a template, integrates the attention mechanism of the convolutional attention module (CBAM) and the feature aggregation method of the neural architecture search feature pyramid network (NAS-FPN) into the original neural network framework, and constructs an improved CN-YOLOv5 model. By effectively matching and fusing multimodal data such as visual images and videos, infrared images and videos, radar data, radio frequency signals, and audio signals, deep reasoning and learning are performed. The method fully mines the feature information in each modal data that is valuable for UAV detection and tracking, overcomes the limitations of the prior art, realizes high-precision detection of UAVs and stable and accurate tracking, and ensures the safety of sensitive areas.

[0120] In order to achieve the above purpose, a high-precision anti-UAV detection method based on multimodal data fusion of CN-YOLOv5 is proposed. The specific implementation method is as follows:

[0121] First, multimodal data is collected: visual images and videos: high-definition cameras and other equipment are used to collect the appearance images of the drone under different viewing angles and lighting conditions, as well as video clips of the flight process, in order to obtain the shape, color, texture and other appearance feature information of the drone; infrared images and videos: infrared thermal imaging equipment is used to collect infrared images of the drone, especially at night or in low-light environments, as infrared images can highlight features such as the heating components of the drone to assist in identification; radar data: the radar system is used to obtain the distance, speed, direction and other motion information of the drone, to assist in judgment and positioning from the dynamic perspective of the target; radio frequency signals: radio frequency detection equipment is used to collect the radio frequency signals of communication between the drone and the remote control, and the type of the drone, communication frequency band and other related attributes are identified based on the signal characteristics; audio signals: the sound signals generated by the drone during flight are obtained through audio acquisition equipment such as microphones. The sounds emitted by drones of different types and states have certain characteristic differences, which can assist in detection.

[0122] Then, the target detection model is constructed and improved. The YOLOv5 target detection model is used as the basic framework, and two important mechanisms are integrated to construct the improved CN-YOLOV5. One of them is the attention mechanism. The convolutional attention mechanism module (CBAM) is embedded in the backbone network (Backbone) of the model. This mechanism enables the model to adaptively focus on the key areas in the input image or feature map, that is, the area where the drone is located and the part that plays a key role in its identification. For example, for visual images, it can focus on the contours, key components and other areas of the drone, suppressing the interference of irrelevant background information; when processing the features corresponding to the radar data, it pays more attention to the key data points that reflect the changes in the position and motion state of the drone. In this way, the model's ability to extract effective features is improved, and the ability to characterize drone targets is enhanced. The other is an improved feature aggregation method, which introduces the neural architecture search optimized feature pyramid network (NAS-FPN) to replace the FPN module in the Neck part of the original YOLOv5 model for feature fusion. Different from the traditional fixed-structure feature fusion method, NAS-FPN automatically explores and determines the optimal feature fusion architecture through neural architecture search technology. Compared with the traditional multimodal detection method, it can match and fuse the features extracted from different modal data through different levels of the model. It can reasonably and efficiently fuse the high-level features of high semantic information in visual images, the middle-level features reflecting thermal features in infrared images, and the bottom-level features reflecting position and motion in radar data, etc., fully explore the complementarity between the features of each modality, so that the fused features have multimodal advantages, providing strong support for subsequent accurate target detection and classification.

[0123] The convolutional attention module (CBAM) is mainly composed of a channel attention module and a spatial attention module.

[0124] The formula of the channel attention module is as follows:

[0125]

[0126] M c (F) represents the output of feature map F after the channel attention module, σ represents the ReLU activation function, W1 represents the weight matrix of the second fully connected layer, W0 represents the weight matrix of the first fully connected layer, It means to perform average pooling operation on the feature map F with the number of channels c. It means to perform the maximum pooling operation on the feature map F with the number of channels c.

[0127] The formula of the spatial attention module is as follows:

[0128]

[0129] M s (F) represents the output of feature map F after the spatial attention module, σ represents the Sigmoid activation function, and f 7×7 Indicates a convolution with a convolution kernel of 7×7. It means to perform average pooling operation on the feature map F modulated by the channel attention module. It means to perform the maximum pooling operation on the feature map F modulated by the channel attention module. right and Perform the Concat operation in the channel dimension.

[0130] The output M after channel attention modulation c (F) is convolved with the input feature map F to obtain the spatial attention module input F', and the spatial attention module output is convolved with F' to obtain the output F" of the entire CBAM:

[0131]

[0132] The feature fusion process of NAS-FPN is as follows: i and F j (i≠j) to fuse, we need to first upsample the feature map with lower resolution to make it consistent with the size of the feature map with higher resolution, and then perform addition fusion. Here we assume that F j The resolution is lower than F i , use the bilinear interpolation upsampling method to j Sampling and F i Feature maps of the same size w(m,n) is the weight interpolation, which is related to the coordinates of the original feature map and surrounding pixels. The two-dimensional upsampling formula is as follows:

[0133]

[0134] Then the upsampled With F i Add element by element to get the fused feature map F fusion , the formula is as follows:

[0135]

[0136] The formula used in multimodal data feature matching fusion is as follows: represents the matching fusion feature map extracted from the lth layer, D(·) represents the downsampling function, represents the fused feature map of layer l, represents the fusion weight of the a-th modality and the l-th layer, Represents the feature map extracted from the lth layer of the ath modality:

[0137]

[0138] Finally, the most critical thing is to put multimodal data into the model for training. The collected multimodal data are uniformly input into the CN-YOLOv5 model for reasoning and learning. In the training stage, a large number of labeled multimodal data samples containing drone targets are used to train the model for supervised learning. The annotation information covers the category of the drone (such as multi-rotor, fixed-wing and other specific types), the position in each modal data (such as pixel coordinates in visual images, spatial coordinates in radar data, etc.) and other related feature attributes. NAS-FPN in CN-YOLOv5 continuously adjusts its own parameters to learn the intrinsic relationship between the matching and fusion of each modal data and the drone target. For example, it can learn to analyze that when specific visual features and corresponding radar motion features and RF signal features appear at the same time, the drone can be detected with a high probability. In the reasoning stage, for the newly input multimodal data, the model comprehensively analyzes the features of each modality based on the knowledge learned in the training stage, accurately determines whether there is a drone target, and further outputs the specific category, location coordinates, speed and other detailed information of the drone, while realizing real-time tracking of the drone and generating its flight trajectory.

[0139] After the above steps, drones in the airspace can be detected and tracked in real time with high precision.

[0140] The present invention achieves high-precision detection. Through the matching fusion of multimodal data, the attention mechanism in the model and the optimized feature fusion method, the complementary advantages of each modal data are fully utilized, so that the model can more accurately extract and analyze the target features of drones, significantly improve the accuracy of detection, effectively reduce the false alarm rate and missed alarm rate, and accurately identify drones of different types and different flight states.

[0141] The present invention has strong environmental adaptability. Whether in various complex natural environments such as day, night, sunny, rainy or snowy, or in the face of complex background interference (such as buildings and trees in urban environments), the synergy of multiple modal data enables the present invention to stably detect and track drones, and has good environmental adaptability and robustness. Specific embodiment twelve:

[0143] The difference between the twelfth embodiment of the present invention and the eleventh embodiment is that:

[0144] In the monitoring area defined around the airport, the multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5 of the present invention is deployed and implemented in the following manner:

[0145] Step 1: Sensor deployment and data collection:

[0146] Deployment of visual image and video acquisition equipment: According to the airport runway, apron and surrounding airspace scope and key area distribution, the camera location and number are reasonably planned to ensure full coverage of the airspace and overlap of vision. The camera frame rate is set to about 30 frames per second to collect images, and the data is stored in the monitoring data center according to the timestamp and camera number to provide visual basis for subsequent detection.

[0147] Deployment of infrared image and video acquisition equipment: Infrared thermal imaging equipment and visual cameras are installed in coordination and aimed at the same airspace. The parameters are adjusted according to the airport ambient temperature, monitoring distance, etc., so that the infrared image clearly presents the thermal characteristics of the drone. The collected data is stored in the monitoring data center in time sequence for subsequent fusion analysis.

[0148] Radar sensor deployment: The airport is equipped with primary and backup air traffic control radars and low-altitude blind spot radars, which are reasonably deployed and positioned according to the airspace range and geographical boundaries, and set parameters such as scanning cycle and beam width. Radar echo data, including various parameters and acquisition time, are acquired in real time and transmitted to the radar data processing server for synchronization with other modal data.

[0149] Deployment of RF signal monitoring equipment: RF signal detection equipment combining directional and omnidirectional antennas is deployed at key locations such as the roof of the terminal building and the runway entrance. The working frequency band is set to cover the common drone frequency band. After pre-processing the collected signals, they are stored in the RF data storage server according to characteristics and time to assist in determining the drone model and other information.

[0150] Audio sensor deployment: Microphone arrays are arranged on both sides of the runway, on the apron and near the terminal building. Parameters are determined based on environmental noise and positioning accuracy. The collected audio signals are converted into digital data and stored in the audio data server according to timestamps and array configuration information to assist in detecting the status of the drone.

[0151] Step 2: Model training:

[0152] First, after extracting data from each server, normalize the size, color correct (visual image), and denoise the visual and infrared images, and normalize the radar, radio frequency, and audio data. Organize a professional team to annotate the drone category, the position coordinates of each mode, and related feature attributes to build a training data set. Next, build a CN-YOLOv5 model that incorporates CBAM and NAS-FPN based on a common deep learning framework, reasonably set the CBAM module hyperparameters based on input feature maps and other parameters, set the NAS-FPN module hyperparameters based on computing resources and fusion effects, and determine the model training parameters such as learning rate, batch size, and number of training rounds. Furthermore, divide the annotated data set into training set, validation set, and test set at a ratio of 8:1:1, train the model with an optimizer (such as Adam), build a comprehensive loss function, update the parameters through backpropagation, adjust the parameters in a timely manner based on the performance indicators of the validation set, and save the weight file after meeting the standards.

[0153] Step 3: Model Inference:

[0154] Each modality acquisition device transmits new data to the data processing center in real time, synchronizes according to the timestamp, and inputs the trained model after preprocessing. After receiving the data, the model determines whether there is a drone. If so, it determines its category, location (integrating multi-modal positioning information), speed and other information, uses Kalman filtering combined with the Hungarian algorithm for real-time tracking, generates the drone flight trajectory, and displays the detection and tracking results on the monitoring screen. In case of abnormality, an alarm is triggered and data is recorded to ensure the safety of the airport airspace.

[0155] Through the above process, this detection method can accurately detect, track and control drones in the complex environment of the airport, ensuring the normal operation of the airport and airspace safety.

[0156] The above is only a preferred implementation of a multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5. The protection scope of a multimodal data fusion anti-UAV high-precision detection method based on CN-YOLOv5 is not limited to the above embodiments. All technical solutions under this idea belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, several improvements and changes without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A high-precision anti-UAV detection method based on CN-YOLOv5 multimodal data fusion, characterized by: The method comprises the following steps: Step 1: Deploy sensors and collect data; Step 2: Build the CN-YOLOv5 model and train the model; Step 3: Based on the trained model, after receiving the data, determine whether there is a drone. If so, determine its category, location, and speed information, and track it in real time to generate the drone's flight trajectory.

2. The method according to claim 1, characterized in that: The step 1 is specifically as follows: Deploy visual image and video acquisition equipment: plan the location and number of cameras according to the airport runway, apron and surrounding airspace range and key area distribution to ensure full airspace coverage and overlapping fields of view; set the camera frame rate to 30 frames per second to collect images, and store the data in the monitoring data center according to timestamps and camera numbers to provide visual basis for subsequent detection; Deploy infrared image and video acquisition equipment: infrared thermal imaging equipment and visual cameras are installed in coordination and aimed at the same airspace; Adjust its parameters according to the airport ambient temperature and monitoring distance so that the infrared image can clearly show the thermal characteristics of the drone. The collected data is stored in the monitoring data center in time sequence for subsequent fusion analysis. Deployment of radar sensors: The airport is equipped with primary and backup air traffic control radars and low-altitude blind spot radars, which are reasonably deployed and positioned according to the airspace range and geographical boundaries, and the scanning cycle and beam width parameters are set to obtain radar echo data in real time, including various parameters and acquisition time, and transmit them to the radar data processing server for synchronization with other modal data; Deployment of radio frequency signal monitoring equipment: Radio frequency signal detection equipment combining directional and omnidirectional antennas is deployed on the roof of the terminal building and at the runway entrance. The working frequency band is set to cover the common drone frequency band. After pre-processing the collected signals, they are stored in the radio frequency data storage server according to characteristics and time to assist in determining the drone model and other information; Deploy audio sensors: Place microphone arrays on both sides of the runway, on the apron, and near the terminal building. Determine parameters based on ambient noise and positioning accuracy. Convert the collected audio signals into digital data and store them in the audio data server according to timestamps and array configuration information to assist in detecting the status of drones.

3. The method according to claim 2, characterized in that: The step 2 is specifically as follows: After extracting the data, normalize the size, color correct, and denoise the visual and infrared images, and normalize the radar, radio frequency, and audio data; annotate the drone category, position coordinates of each mode, and related feature attributes to build a training data set; Based on the common deep learning framework, a CN-YOLOv5 model integrating CBAM and NAS-FPN is built. The hyperparameters of the CBAM module are reasonably set according to the input feature map and other parameters. The hyperparameters of the NAS-FPN module are set according to the computing resources and fusion effect. At the same time, the model training parameters such as learning rate, batch size, and number of training rounds are determined. The labeled data set is divided into training set, validation set and test set according to the ratio of 8:1:

1. The model is trained with the optimizer, and a comprehensive loss function is constructed. The parameters are updated through back propagation. The parameters are adjusted in time according to the performance indicators of the validation set, and the weight file is saved after meeting the standards.

4. The method according to claim 3, characterized in that: The convolutional attention module CBAM includes a channel attention module and a spatial attention module: The formula of the channel attention module is as follows: The spatial attention module is expressed as follows:

5. The method according to claim 4, characterized in that: The output M after channel attention modulation c (F) is convolved with the input feature map F to obtain the spatial attention module input F', and the spatial attention module output is convolved with F' to obtain the output F" of the entire CBAM:

6. The method according to claim 5, characterized in that: The feature fusion process of NAS-FPN is as follows: the feature map with lower resolution is upsampled to make it consistent with the size of the feature map with higher resolution, and then added and fused. The two-dimensional upsampling formula is as follows: After upsampling With F i Add element by element to get the fused feature map F fusion , the formula is as follows: The formula used in multimodal data feature matching fusion is as follows: Represents the matching fusion feature map extracted from the lth layer: The multimodal data is put into the model for training, and the collected multimodal data is uniformly input into the CN-YOLOv5 model for reasoning and learning.

7. The method according to claim 5, characterized in that: The step 3 is specifically as follows: Each acquisition device transmits new data to the data processing center in real time, which is synchronized according to the timestamp and pre-processed before being input into the trained model. After receiving the data, the model determines whether there is a drone. If so, it determines its category, integrates multi-modal positioning information and speed information, and uses Kalman filtering combined with the Hungarian algorithm for real-time tracking to generate the drone's flight trajectory. The detection and tracking results are displayed on the monitoring screen. In case of abnormality, an alarm is triggered and data is recorded to ensure the safety of the airport airspace.

8. A multimodal data fusion anti-UAV high-precision detection system based on CN-YOLOv5, characterized by: The system comprises: Data collection module, which deploys sensors and collects data: A model training module, wherein the model training module builds a CN-YOLOv5 model and trains the model; The trajectory detection module determines whether there is a drone after receiving the data based on the trained model. If so, it determines its category, location and speed information, and performs real-time tracking to generate the drone flight trajectory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to claims 1-7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method of claims 1-7 is implemented.

Images