Aerial target detection method based on image recognition

Abstract

The invention relates to the technical field of aerial target detection, in particular to an aerial target detection method based on image recognition. Comprising the steps of image acquisition and preprocessing, target detection, target classification, target tracking, data storage, alarm and feedback and the like. According to the invention, technologies of deep learning, multispectral imaging and the like are combined, the defects of a traditional detection technology are effectively overcome, and the detection precision of low-altitude, low-speed and small targets is improved; the requirement of real-time monitoring is met while the detection precision is ensured; effective detection can be carried out at night and in a complex environment, and the adaptability of the system under different illumination conditions is improved; a target tracking algorithm is optimized, continuous and stable tracking of multiple targets can be realized, and target motion trails can be accurately output; the system integrates detection, identification, storage and alarm, and is suitable for various application scenes such as airport security, important facility protection, city management and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of aerial target detection, and in particular to an aerial target detection method based on image recognition. Background Art

[0002] In today's aerial target detection field, traditional technologies face numerous challenges and struggle to meet growing security and monitoring needs. Traditional aerial target detection technologies primarily rely on radar and radio detection. However, these technologies have significant limitations. For one thing, their ability to detect low-altitude, slow-moving, and small targets is limited. For example, in urban environments, small drones flying at low altitudes have a small radar cross-section, and their signals are easily blocked and interfered with by obstacles such as buildings, making them difficult to detect effectively by radar. Furthermore, targets such as birds, which fly relatively slowly in complex electromagnetic environments, are difficult to accurately identify using radio detection. Furthermore, traditional technologies struggle to achieve precise identification and are unable to accurately distinguish between different types of aerial targets, such as drones, birds, or other aircraft.

[0003] With the advancement of computer vision and deep learning technologies, image recognition-based aerial target detection technology has gradually gained popularity. However, existing image recognition-based aerial target detection technologies still face numerous challenges. Regarding multimodal data fusion, the feature fusion accuracy of visible and infrared images is insufficient, making it difficult to fully leverage the advantages of multimodal data. This results in low target detection rates in environments such as nighttime or complex lighting conditions. For small target detection, aerial targets such as drones and birds occupy a small portion of the image and have less distinct features, making existing detection algorithms prone to missed detections. In terms of balancing real-time performance with accuracy, complex deep learning algorithms are computationally intensive. While ensuring detection accuracy, they struggle to meet the demands of real-time monitoring and respond promptly to dangerous targets. Regarding target tracking, existing tracking algorithms are prone to tracking loss or misassociation when faced with fast-moving targets or those with similar appearances, resulting in the inability to continuously and accurately track the target. Furthermore, existing detection technologies have limited adaptability to the environment. Factors such as lighting changes and weather conditions can significantly impact detection performance, and they lack adaptive mechanisms to address these environmental changes.

[0004] For example, patent CN119717882A discloses a multi-sensor fusion drone detection method and system. While it uses multi-sensor fusion and a deep learning model for target detection, it does not mention adaptive noise covariance matrices to optimize trajectory prediction. Its detection capabilities for small drones or birds are weak, and its alarm mechanism does not consider the dynamic adjustment threshold of ambient lighting. Another example is patent CN119399653A, which discloses a drone target detection method based on an improved YOLOv5 algorithm. This method only uses the SE module to focus on channel attention, is not as comprehensive as this technology in terms of feature enhancement, and does not involve multimodal processing or dynamic adjustment of Kalman filter parameters.

[0005] In summary, existing aerial target detection technologies have many shortcomings. A new detection technology is urgently needed to overcome these issues. This technology aims to improve the accuracy, real-time performance, and environmental adaptability of aerial target detection, thereby meeting the practical application needs of areas such as airport bird strike prevention, drone intrusion prevention, and urban management. To address this, we propose an aerial target detection method based on image recognition. Summary of the Invention

[0006] The purpose of the present invention is to provide an aerial target detection method based on image recognition to solve the problems raised in the above background technology.

[0007] To solve the above technical problems, the present invention provides an aerial target detection method based on image recognition, comprising the following steps:

[0008] S1. Image acquisition and preprocessing: Synchronously deployed high-definition visible light cameras and infrared cameras collect dual-spectral image data of aerial targets in real time at a set frame rate, ensuring the continuity and timeliness of the acquired images. The images are also preprocessed to ensure the spatiotemporal consistency of multi-source data.

[0009] S2. Object Detection: Detect objects in images using a deep learning model. Using an improved YOLOv5 model as the backbone network, it inputs a fused multispectral image and outputs the object bounding box and confidence score. An attention mechanism is also introduced to optimize small object detection performance.

[0010] S3. Target Classification: Classify detected targets and identify their types. Use the trained SVM model to classify detected targets. When training the SVM model, a kernel function is used to map samples from low-dimensional space to high-dimensional space to improve classification accuracy. A classifier is also built based on ResNet-50 to further accurately distinguish target categories based on target morphology and motion characteristics.

[0011] S4. Target tracking: Combined with the DeepSORT algorithm, Kalman filtering and Hungarian matching are used to achieve continuous tracking of multiple targets and output motion trajectories;

[0012] S5. Data storage: The detection result information such as the target image, target type, time, location, type, etc. is stored in a cloud database in a certain format for subsequent query and analysis;

[0013] S6. Alarm and feedback: Based on the detection results, when an abnormal target is detected, the hierarchical alarm mechanism is triggered through the API, and the alarm signal or feedback signal is triggered according to the preset rules to notify the relevant personnel or system.

[0014] As a further improvement of the present technical solution, in S1, the camera continuously captures aerial images at a fixed frame rate. To ensure image quality, the camera is equipped with autofocus and autoexposure functions, which adjust parameters in real time according to ambient light and target distance. At the same time, image stabilization technology is used to reduce image blur caused by device vibration, ensuring that the captured image is clear and stable, providing reliable data for subsequent processing.

[0015] As a further improvement of the present technical solution, in S1, the process of preprocessing the image includes at least denoising, normalization, enhancement, registration, etc.; specifically, the process includes the following:

[0016] Image denoising: Use the non-local mean denoising algorithm to process the collected images to make the target outlines and features in the image clearer, providing more accurate data for subsequent target detection and recognition;

[0017] Image normalization: Mapping the pixel values ​​of an image to the range of 0-1 eliminates the impact of differences in pixel value ranges between different images. This allows subsequent deep learning models to learn image features more stably and efficiently when processing images, improving the accuracy and generalization ability of the model.

[0018] Image enhancement: For visible light images, contrast-limited adaptive histogram equalization is used to improve image quality in low illumination conditions. For infrared images, guided filtering is combined to suppress noise.

[0019] Image registration: A scale-invariant feature transformation algorithm is used to match feature points of visible and infrared images. A random sampling consensus algorithm is then used to eliminate mismatched points to ensure spatial alignment of the bispectral images. Finally, the homography matrix H is optimized using the least squares method to obtain an accurate image transformation relationship, enabling precise registration of visible and infrared images and providing reliable data for subsequent target detection and analysis based on fused images. The transformation matrix H is calculated as follows:

[0020]

[0021] in, is the homography matrix, which is solved by the least squares optimization method; is the coordinate of a pixel in the original image; It is the coordinate of the corresponding pixel point in the target image after transformation; the appropriate The matrix is ​​applied to each pixel coordinate of the original image, which can transform the original image to align with the target image space.

[0022] As a further improvement of this technical solution, in S2, the specific steps of target detection include the following:

[0023] S2.1. Model selection and improvement:

[0024] Selecting the backbone network: The improved YOLOv5 model is used as the backbone network for target detection. The YOLOv5 model has the advantages of fast speed and high accuracy in the target detection field, and is suitable for real-time detection of aerial targets.

[0025] At the same time, the attention mechanism is introduced: to address the problem that the traditional YOLO algorithm has insufficient accuracy in detecting small targets, the attention mechanism is introduced in the feature extraction stage of the improved YOLOv5 model;

[0026] S2.2, Multispectral Feature Fusion:

[0027] Backbone cross-modal feature fusion: A cross-modal feature fusion module is introduced into the Backbone part of YOLOv5 to fuse visible light and infrared image features. The calculation method is:

[0028]

[0029] in, Represents the fused feature map; is the adaptive weight, learned by the 1×1 convolutional layer; It represents the feature map obtained after the visible light image is processed by the neural network, which contains the characteristic information of the target in the visible light band; It represents the feature map obtained after the infrared image is processed by the neural network, reflecting the characteristics of the target in the infrared band;

[0030] Optimization of small target detection in the neck: The convolutional block attention module (CBAM) is embedded in the neck to enhance the feature response of small targets. The calculation formula is:

[0031]

[0032] in, It is the feature map output after being processed by the convolution block attention module; Sigmoid function, which maps the input value to between 0 and 1 and is used to normalize the weighted features; is a multilayer perceptron; represents the average pooling operation; Represents the maximum pooling operation; is the original feature map input to the convolutional block attention module; Represents element-by-element multiplication; by performing average pooling and maximum pooling on the feature map, and then using a multi-layer perceptron (MLP) for feature extraction and weighting, the characteristics of small targets are highlighted, thereby improving the small target detection performance;

[0033] Weighted feature pyramid for multi-spectral fusion: Based on FPN, feature maps at different levels are adaptively weighted and fused. The calculation method is:

[0034]

[0035] in, is the feature map output after weighted fusion; is the number of feature map levels involved in weighted fusion; Indicates the The weight of the feature map of each level is used to adjust the importance of the feature map of this level in the fusion process. Calculated; is a learnable scaling parameter optimized via back-propagation; For the This adaptive weighted fusion method can automatically assign weights based on the importance of feature maps at different levels to target detection, thereby improving feature utilization efficiency and detection accuracy.

[0036] S2.3, Dynamic confidence threshold adjustment: according to the ambient light intensity Adjust the detection threshold T, the formula is:

[0037]

[0038] in, The adjusted detection threshold changes dynamically according to the ambient light intensity and is used to determine whether the target detection result is reliable; The basic detection threshold is the initial detection threshold set under standard conditions; is the adjustment coefficient, which is used to control the influence of ambient light intensity on the detection threshold; The ambient light intensity is used to measure the intensity of light in the environment. In scenes with large changes in light intensity, the detection threshold is dynamically adjusted to avoid false detection or missed detection due to excessive or insufficient light.

[0039] S2.4. Lightweight model design: Replace standard convolution with depthwise separable convolution to reduce the amount of model computation. The computational reduction formula is:

[0040]

[0041] in, is the convolution kernel size, which indicates the size of the convolution kernel in the spatial dimension; is the number of input channels, that is, the number of channels of the feature map input to the convolutional layer; is the number of output channels, which is the number of channels of the feature map output after the convolution layer processing; The feature map size refers to the size of the feature map in the spatial dimension. Through lightweight design, the model's operating efficiency is improved while ensuring detection accuracy, making it more suitable for real-time detection tasks.

[0042] S2.5. Model input and output: The preprocessed and fused multispectral image is input into the improved YOLOv5 model. After a series of feature extraction, fusion, and calculations, the model outputs the target bounding box and confidence level. The target bounding box is used to determine the position and size of the target in the image, and the confidence level indicates the model's confidence that the detected target belongs to a certain category, providing basic data for subsequent target classification, tracking, and other operations.

[0043] As a further improvement to the present technical solution, in S2.1, when the improved YOLO algorithm is used for target detection, the traditional YOLO algorithm has the problem of insufficient accuracy when detecting small targets. In the feature extraction stage of the improved YOLOv5 model, an attention mechanism is introduced to enhance the attention to small target features by calculating the importance weights of different feature channels; the importance weights of different feature channels are calculated as follows:

[0044] First calculate the channel The global average pooling result of :

[0045]

[0046] in, Indicates channel The global average pooling result of ; Indicates the Feature maps of feature channels; and are the height and width of the feature map respectively; and Is the coordinate index of the pixel point on the feature map;

[0047] Then through the learnable weight matrix 、 , and the ReLU activation function and Sigmoid activation function Calculation Channel The attention weight :

[0048]

[0049] Finally, we get the feature map weighted by the attention mechanism :

[0050]

[0051] in, Represents element-wise multiplication, the attention weight With the original feature map The corresponding elements are multiplied to achieve the enhancement operation of the original feature map; in this way, the model's attention to small target features is enhanced, and the detection accuracy of low-altitude, slow-moving, and small targets is improved.

[0052] As a further improvement of this technical solution, in S3, the training and application process of the SVM model includes the following:

[0053] Select kernel function: When training the SVM model, a kernel function is used to map samples from a low-dimensional space to a high-dimensional space to improve classification accuracy.

[0054] Target classification: Use the trained SVM model to perform preliminary classification of detected aerial targets. The model calculates and judges in a high-dimensional space based on the characteristics of the input samples, and outputs the probability or category label of the target belonging to different categories, roughly classifying the targets into categories such as drones, birds, and aircraft.

[0055] As a further improvement of this technical solution, in S3, the process of building a classifier based on ResNet-50 to distinguish target categories includes:

[0056] Add an optical flow branch: When building a classifier based on ResNet-50 for motion feature extraction, add an optical flow branch to the input layer of ResNet-50 and use the Farneback dense optical flow algorithm to calculate the target's motion information. This algorithm is based on the assumptions of constant brightness and spatial consistency of the image. It calculates the displacement of pixels in adjacent frames to obtain information such as the target's motion speed and direction. The calculation formula is:

[0057]

[0058] in, Represents an image function, which is used to describe the pixel value information of an image at a certain moment and position; is the spatial coordinate of the pixel point on the image plane, Represents the horizontal coordinate, Represents the vertical coordinate; Represents time, used to identify the moment of image acquisition; 、 Represents the pixels at Direction and The amount of displacement in the direction; is the time interval, i.e., from time At the time The time difference between

[0059] Feature fusion and classification: Classification is performed by combining RGB image features with optical flow features. Features such as color and texture extracted from the RGB image are fused with motion features derived from the optical flow branch to form a more comprehensive and rich feature vector. These fused features are fed into a classifier built on ResNet-50. The classifier further extracts high-level features through multiple layers of convolution and pooling operations, and finally uses a fully connected layer for classification prediction. By integrating motion features, the classifier can more accurately distinguish dynamic targets, improving the recognition accuracy of drones, birds, and aircraft in mid-air, and making more precise judgments on target categories.

[0060] As a further improvement of this technical solution, in S4, target tracking specifically includes the following steps:

[0061] S4.1. Trajectory prediction optimization:

[0062] Introducing adaptive noise covariance matrix: In the Kalman filter process, an adaptive noise covariance matrix is ​​introduced to optimize trajectory prediction; in traditional Kalman filtering, process noise and observation noise It is usually set to a fixed value, but in actual aerial target tracking scenarios, the target's motion state is changeable, and it is difficult to accurately match the fixed noise value; therefore, a dynamic adjustment method is adopted, and the formula is:

[0063]

[0064] in, and are the coefficients used to adjust the process noise covariance matrix and the observation noise covariance matrix, and are the adjusted process noise and observation noise, respectively, and are the process noise and observation noise of the target’s previous trajectory position, respectively; and It will be adaptively adjusted according to the target movement speed;

[0065] Predicting target position: Based on the Kalman filter principle and the adjusted noise covariance matrix, the target's state estimate (position, velocity, and other information) at the previous moment is used to predict the target's position and state at the current moment. The prediction process involves the calculation of the state transfer equation and the observation equation. Through continuous iteration, a relatively accurate target prediction position is provided for subsequent data association.

[0066] S4.2. Data association improvements:

[0067] Calculate the cost matrix: In the Hungarian matching stage, the cost matrix is ​​calculated by combining appearance features and motion information; the calculation formula is:

[0068]

[0069] in, Represents the elements in the cost matrix, representing the The prediction target and The association cost between the detection targets;

[0070] Calculate the distance metric: Represents the intersection-over-union distance, which is used to measure the degree of overlap between the predicted target frame and the actual detected target frame; the higher the overlap, The smaller the value, the more likely it is that the two boxes represent the same target; It is the cosine distance of appearance features, which is obtained by calculating the cosine similarity between the appearance feature vectors (such as color, shape, etc.) of the target. The smaller the distance, the more similar the appearance features of the two targets are, and the higher the probability that they belong to the same target.

[0071] Adjust weights: The weight is adjustable and can be set according to the specific application scenario and target characteristics;

[0072] Matching targets: Based on the calculated cost matrix, the Hungarian matching algorithm is used to find the optimal match. Through a series of calculations and transformations, the algorithm finds the best correspondence between multiple predicted targets and detected targets, determines which predicted targets and which detected targets belong to the same target, and thus achieves accurate association and continuous tracking of multiple targets, outputting the target's motion trajectory.

[0073] As a further improvement of the present technical solution, in S5, the data storage process includes the following steps:

[0074] S5.1. Organize test result data:

[0075] Collect data: summarize various data generated by target detection, classification, tracking, etc.

[0076] Data Verification: Perform preliminary verification on the collected data to check its integrity and accuracy; ensure that the target image is not damaged or has no missing pixels, the target type information meets the preset classification standards, the time format is correct and continuous, and the location coordinates are within a reasonable range;

[0077] S5.2. Data format conversion:

[0078] Select storage format: Choose an appropriate storage format based on the cloud database requirements and data characteristics;

[0079] Data encoding: Encode the converted data to ensure the security and stability of the data during storage and transmission;

[0080] S5.3. Store in cloud database:

[0081] Connect to the database: Use the corresponding database connection tools and interfaces to establish a connection with the cloud database;

[0082] Create a data table (if necessary): If a data table for storing aerial target detection results does not exist in the cloud database, create a new data table based on the data structure and requirements;

[0083] Insert data: Insert the sorted and converted data into the data table of the cloud database.

[0084] As a further improvement of the present technical solution, in S6, the alarm and feedback process includes the following steps:

[0085] S6.1. Evaluation of test results:

[0086] Extract key information: Extract key information from the results of target detection, classification, and tracking, including target confidence, target type, and target location information;

[0087] Compare with preset rules: Compare the extracted key information with the preset alarm rules;

[0088] S6.2. Triggering of the hierarchical alarm mechanism:

[0089] Determine the threat level: Determine the threat level based on the target type and confidence level; use the following classification strategy:

[0090] If the confidence level is ≥ 0.9 and the target type is a drone, the threat level is considered high (High); when 0.7 ≤ confidence level < 0.9, the threat level is considered medium (Medium); in other cases (such as low confidence level or the target type is an ordinary bird), the threat level is considered low (Low);

[0091] Triggering alarm signals: triggering corresponding alarm signals based on the determined threat level; when the threat level is high, a strong alarm prompt is quickly sent to relevant personnel or systems through the API; when the threat level is medium, a normal alarm notification is sent; when the threat level is low, a record and prompt are made for subsequent observation;

[0092] S6.3. Feedback target information:

[0093] Organize feedback data: Organize the detailed information of the target into a format that is easy for the recipient to understand and use;

[0094] Send feedback information: Send the sorted target information to relevant personnel or systems through the API.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] 1. This image recognition-based aerial target detection method combines deep learning and multispectral imaging technologies to effectively address the shortcomings of traditional detection techniques. The improved YOLOv5 model, combined with the attention mechanism, enhances focus on small target features, improving the detection accuracy of low-altitude, slow-moving, and small targets. Furthermore, the SVM model is combined with a classifier built on ResNet-50, leveraging kernel functions and motion features to more accurately identify target types, such as drones, birds, and aircraft.

[0097] 2. This image recognition-based aerial target detection method uses GPU acceleration processing, combined with lightweight model design, to replace standard convolution with depthwise separable convolution, reducing the model's computational complexity. While maintaining detection accuracy, it also ensures that the system can detect and feedback target information in real time, meeting the needs of real-time monitoring.

[0098] 3. This image recognition-based aerial target detection method combines multispectral technology using visible light and infrared imaging, enabling effective detection at night and in complex environments. Image preprocessing operations such as denoising, enhancement, and registration, as well as dynamic adjustment of the detection threshold based on ambient light intensity, improve the system's adaptability to varying lighting conditions.

[0099] 4. This image recognition-based aerial target detection method optimizes the target tracking algorithm by introducing an adaptive noise covariance matrix into the Kalman filter and combining appearance features with motion information to calculate the cost matrix during the Hungarian matching phase. This enables continuous and stable tracking of multiple targets and accurately outputs target motion trajectories.

[0100] 5. This image recognition-based aerial target detection method integrates detection, identification, storage, and alarming. The data storage module stores the detection results in a formatted cloud database to facilitate subsequent query and analysis. The alarm and feedback module triggers a hierarchical alarm mechanism based on the detection results, promptly notifying relevant personnel or systems. It is suitable for a variety of application scenarios such as airport security, important facility protection, and urban management. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 This is an exemplary method flow chart of the present invention. DETAILED DESCRIPTION

[0102] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0103] Example

[0104] like Figure 1 As shown, this embodiment provides an aerial target detection method based on image recognition, comprising the following steps:

[0105] First, perform product configuration and application scenario settings before data collection, including:

[0106] Hardware configuration: The equipment includes high-definition cameras, infrared cameras, GPU-accelerated image processing units, storage devices, and alarm devices;

[0107] Software configuration: Image processing software developed in Python or C++, integrated with deep learning frameworks (such as TensorFlow and PyTorch) to achieve target detection, classification, and alarm functions;

[0108] Application scenarios can be:

[0109] Airport security: used to prevent birds from invading the airport's clear area and ensure aviation safety;

[0110] Important facilities and institutions: used to prevent drones from invading controlled areas;

[0111] Urban management: used to monitor drone activities over cities and maintain public safety.

[0112] S1. Image acquisition and preprocessing: Synchronously deployed high-definition visible light cameras and infrared cameras collect dual-spectral image data of aerial targets in real time at a set frame rate to ensure the continuity and timeliness of the acquired images. The images are also preprocessed to ensure the spatiotemporal consistency of multi-source data.

[0113] In this step, the camera continuously captures aerial images at a fixed frame rate. To ensure image quality, the camera is equipped with autofocus and autoexposure functions, adjusting parameters in real time based on ambient light and target distance. Image stabilization technology is also used to reduce image blur caused by device vibration, ensuring the captured image is clear and stable, providing reliable data for subsequent processing. The specific steps include the following:

[0114] Equipment selection and deployment: High-definition visible light cameras and infrared cameras are deployed simultaneously. These two cameras have their own advantages and can complement each other to obtain multispectral information of aerial targets. They are installed in appropriate locations according to a specific layout to ensure that they can cover the target monitoring area and achieve all-round and comprehensive image acquisition of aerial targets.

[0115] Frame rate setting: Specify the camera's acquisition frame rate. Set the HD camera to 30 frames per second and the infrared camera to 15 frames per second. This frame rate setting ensures the continuity of acquired images, enabling subsequent processing to consistently analyze the target's motion trajectory. It also balances device performance and data processing capacity, avoiding data redundancy or excessive device burden caused by excessive frame rates.

[0116] Functional configuration: To ensure image quality, the camera is equipped with autofocus and autoexposure functions. During the actual acquisition process, the camera will automatically adjust the focus and exposure parameters in real time according to the intensity of the ambient light and the distance to the target. For example, in an environment with frequent light changes, the automatic exposure function can respond quickly to ensure that the captured image always remains clear and bright, providing a reliable data basis for subsequent processing. At the same time, image stabilization technology is used to effectively reduce image blur caused by equipment vibration (such as in windy outdoor environments), further improving image stability and clarity.

[0117] Furthermore, the image preprocessing process includes at least denoising, normalization, enhancement, and registration, and specifically includes the following:

[0118] Image denoising: The collected images are processed using the non-local means denoising algorithm. This algorithm searches for similar image blocks within the image and uses the information from these blocks to suppress noise. It can effectively preserve image details while removing noise, avoiding image blurring caused by excessive denoising. This makes the outlines and features of objects in the image clearer, providing more accurate data for subsequent object detection and recognition.

[0119] Image normalization: Mapping the pixel values ​​of an image to the range of 0-1. This step can unify the scale of image data and eliminate the impact of differences in pixel value ranges between different images. This allows subsequent deep learning models to learn image features more stably and efficiently when processing images, improving the accuracy and generalization ability of the model.

[0120] Image enhancement: For visible light images, contrast-limited adaptive histogram equalization (CLAHE) technology is used to improve image quality in low-light conditions. CLAHE adaptively adjusts the image histogram to enhance image contrast, making targets that were previously difficult to see in low-light environments more clearly visible. For infrared images, guided filtering is combined with noise suppression. Guided filtering can smooth the image while preserving image edge information, effectively removing noise interference from infrared images, improving infrared image quality and providing a better foundation for multispectral image fusion and target detection.

[0121] Image registration: The scale-invariant feature transform (SIFT) algorithm is used to match the feature points of visible light and infrared images. The SIFT algorithm can extract stable feature points of images under different scales, rotations, and lighting conditions. Through these feature points, the correspondence between the visible light image and the infrared image can be established. Then, the random sampling consensus (RANSAC) algorithm is used to eliminate incorrect matching points. The RANSAC algorithm can effectively remove incorrect matching points caused by factors such as noise and occlusion through random sampling and model verification, ensuring the spatial alignment of the dual-spectral images. Finally, the homography matrix (Homography) H is optimized and solved by the least squares method to obtain an accurate image transformation relationship, achieving precise registration of the visible light image and the infrared image, and providing reliable data for subsequent target detection and analysis based on the fused image. The transformation matrix H is calculated as follows:

[0122]

[0123] in, is the homography matrix (Homography), which is solved by the least squares optimization method; is the coordinate of a pixel in the original image (such as a visible light image), Indicates the horizontal position of the pixel in the image. Indicates its position in the vertical direction; these coordinate values ​​are used to determine the position of a specific pixel in the original image and are the starting position identifier of the spatial transformation; It is the coordinate of the corresponding pixel point in the target image (such as infrared image) after transformation; and Respectively represent the new position of the pixel in the horizontal and vertical directions of the target image, which are based on the transformation matrix H to the original coordinates The result after transformation; through this transformation, the pixels in the original image can be accurately mapped to the corresponding positions in the target image, achieving spatial alignment of the images; Is a 3×3 matrix used to describe the projection transformation relationship between two planes; in the image registration scenario, The matrix integrates the geometric transformation information such as rotation, translation, and scaling between images; the appropriate Matrix, and applying it to each pixel coordinate of the original image, can achieve the effect of transforming the original image to align with the target image space; Matrix = Use the feature points extracted from the image (such as the feature points obtained by the SIFT algorithm) and their matching relationships to determine the values ​​of each element in the matrix.

[0124] S2. Object Detection: Detect objects in images using deep learning models (such as CNN, YOLO, Faster R-CNN, etc.). Use the improved YOLOv5 model as the backbone network, input the fused multispectral image, and output the object bounding box and confidence level. Also, introduce an attention mechanism to optimize small object detection performance.

[0125] In this step, the specific steps of target detection include the following:

[0126] S2.1. Model selection and improvement:

[0127] Backbone network selection: The improved YOLOv5 model is used as the backbone network for target detection. The YOLOv5 model has the advantages of fast speed and high accuracy in the field of target detection, and is suitable for real-time detection of aerial targets. At the same time, the attention mechanism is introduced: to address the problem that the traditional YOLO algorithm has insufficient accuracy in detecting small targets, the attention mechanism (CBAM module) is introduced in the feature extraction stage of the improved YOLOv5 model.

[0128] Among them, the traditional YOLO algorithm has the problem of insufficient accuracy when detecting small targets. In the feature extraction stage of the improved YOLOv5 model, the attention mechanism is introduced to enhance the focus on small target features by calculating the importance weights of different feature channels. The formula for calculating the importance weights of different feature channels is as follows:

[0129] First calculate the channel The global average pooling result of :

[0130]

[0131] in, Indicates channel The global average pooling result is used to reflect the The global feature information of each feature channel is the basis for subsequent calculation of attention weights; Indicates the The feature map of a feature channel is the feature representation of a specific channel obtained by the model during the feature extraction process, which contains the feature information related to the channel in the image; and They are the height and width of the feature map, respectively, which are used to determine the size of the feature map in the spatial dimension and determine the summation range during global average pooling calculation; and is the coordinate index of the pixel point on the feature map, Traverse the height direction of the feature map, Traverse the width direction and calculate the sum of all pixels ;

[0132] Then through the learnable weight matrix 、 , and the ReLU activation function and Sigmoid activation function Calculation Channel The attention weight :

[0133]

[0134] in, Indicates channel The attention weight is used to measure the The importance of a feature channel in the current image feature representation. The higher the weight, the greater the contribution of the channel to the target feature. The Sigmoid activation function maps the input value to the range of 0-1 and normalizes the weighted features so that they can be used as attention weights and multiplied element-by-element with the original feature map. 、 It is a learnable weight matrix that is continuously adjusted through back propagation during model training and is used to average the global pooling results. Perform weighted transformation to explore the relationship between different channel features; ReLU is the activation function, which is used to increase the nonlinear expression ability of the network. The results are transformed nonlinearly, so that the model can learn more complex feature representations;

[0135] Finally, we get the feature map weighted by the attention mechanism :

[0136]

[0137] in, It is the feature map after the weighting of the attention mechanism, which is the original feature map Based on the attention weight The result of weighted enhancement of the features of each channel highlights the features that are important for small target detection and improves the significance of small targets in the feature map; Represents element-wise multiplication, and the attention weight With the original feature map The corresponding elements are multiplied to achieve the enhancement operation of the original feature map; in this way, the model's attention to small target features is enhanced, and the detection accuracy of low-altitude, slow-moving, and small targets is improved.

[0138] S2.2, Multispectral Feature Fusion:

[0139] Backbone cross-modal feature fusion: The cross-modal feature fusion module (CMFF) is introduced in the Backbone part of YOLOv5 to fuse visible light and infrared image features. The calculation method is:

[0140]

[0141] in, The fused feature map combines the feature information of the visible light image and the infrared image. It is the result obtained after processing by the cross-modal feature fusion module and is used for subsequent tasks such as target detection. The adaptive weight is learned by the 1×1 convolution layer. This weight is used to adjust the contribution ratio of the visible light feature map and the infrared feature map in the fusion process, so that the model can automatically learn the optimal fusion weight according to different input images and task requirements; The feature map obtained after the visible light image is processed by the neural network contains the characteristic information of the target in the visible light band, such as color, texture, etc. The feature map obtained after the infrared image is processed by the neural network reflects the characteristics of the target in the infrared band. At night or in low-light environments, it can provide information complementary to the visible light characteristics, such as the thermal radiation characteristics of the object. This fusion method can automatically adjust the fusion ratio of visible light and infrared image features according to different scenarios and task requirements, making full use of the advantageous information of the two images.

[0142] Optimization of small target detection in the neck: The convolutional block attention module (CBAM) is embedded in the neck to enhance the feature response of small targets. The calculation formula is:

[0143]

[0144] in, The feature map output after being processed by the convolutional block attention module enhances the feature response of small targets, thereby improving the saliency of small targets in the feature map and making it more conducive to the detection of small targets. The Sigmoid function maps the input value to between 0 and 1, which is used to normalize the weighted features so that they can be used as attention weights and multiplied element-by-element with the original feature map. is a multi-layer perceptron, which is a neural network structure containing multiple neurons. In this formula, Used to perform further feature extraction and weighting processing on the pooled features to generate channel attention weights; Represents the average pooling operation, which performs the input feature map Perform average pooling to calculate the average value in a local area to obtain a value representing the characteristics of the area, thereby reducing the resolution of the feature map and highlighting the global features; Represents the maximum pooling operation, which performs the input feature map Perform maximum pooling and select the maximum value in a local area as the feature value representing the area. This also reduces the resolution of the feature map, but focuses more on retaining significant features. It is the original feature map input to the attention module of the convolution block and is the basis for the calculation of the attention mechanism; Represents element-by-element multiplication, which combines the weighted features of the attention mechanism with the original feature map The original feature map is enhanced by multiplying the corresponding elements. By performing average pooling and maximum pooling on the feature map, and then using a multi-layer perceptron (MLP) for feature extraction and weighting, the characteristics of small objects are highlighted, improving the detection performance of small objects.

[0145] Weighted Feature Pyramid (WFPN) for Multispectral Fusion: Based on FPN, adaptive weighted fusion of feature maps at different levels is performed; the calculation method is:

[0146]

[0147] in, The feature map output after weighted fusion integrates the information of feature maps at different levels, improving feature utilization efficiency and detection accuracy; is the number of feature map levels involved in weighted fusion, indicating that Feature maps at different levels participate in the final fusion calculation; Indicates the The weight of the feature map of each level is used to adjust the importance of the feature map of this level in the fusion process. Calculated; is a learnable scaling parameter that is optimized through back-propagation and determines the The weight of the feature map of each level is constantly adjusted during the training process. To optimize the effect of weighted fusion; For the The feature maps at different levels contain information of different scales and semantic levels, which are integrated through weighted fusion. This adaptive weighted fusion method can automatically assign weights based on the importance of feature maps at different levels to target detection, thereby improving feature utilization efficiency and detection accuracy.

[0148] S2.3, Dynamic confidence threshold adjustment: according to the ambient light intensity (Lux) Adjust the detection threshold T, the formula is:

[0149]

[0150] in, The adjusted detection threshold is used to determine whether the target detection result is reliable according to the dynamic change of the ambient light intensity; when the confidence of the detected target is greater than The test result is considered valid only when The basic detection threshold is the initial detection threshold set under the standard environment. Here , which is the benchmark value for dynamically adjusting the detection threshold; is the adjustment coefficient (the experimental optimization value is 0.1), which is used to control the influence of the ambient light intensity on the detection threshold. The larger the value, the more obvious the effect of ambient light intensity on the detection threshold; Lux is the ambient light intensity, which is used to measure the intensity of light in the environment. , the model can dynamically adjust the detection threshold according to changes in lighting conditions In scenes with large changes in light intensity, the detection threshold is dynamically adjusted to avoid false detection or missed detection due to excessive or insufficient light.

[0151] S2.4. Lightweight model design: Replace the standard convolution with depthwise separable convolution to reduce the model computational complexity. The computational complexity reduction formula is:

[0152]

[0153] in, The convolution kernel size indicates the size of the convolution kernel in the spatial dimension, which is usually an odd number, such as 3×3, 5×5, etc. The convolution kernel size affects the receptive field and computational complexity of the convolution operation. is the number of input channels, that is, the number of channels of the feature map input to the convolutional layer; for example, the number of input channels of RGB images is 3, and the number of input channels of multispectral images depends on the specific number of spectral bands; The number of output channels is the number of channels in the feature map output after the convolution layer processing. It is determined by the settings of the convolution layer and affects the ability of the convolution layer to extract features and the complexity of the subsequent network. The feature map size refers to the size of the feature map in the spatial dimension, usually expressed in height and width (such as ); The size of the feature map will change during the convolution operation according to factors such as the convolution kernel size and step size.

[0154] Through lightweight design, the model's operating efficiency is improved while ensuring detection accuracy, making it more suitable for real-time detection tasks.

[0155] S2.5. Model input and output: The preprocessed and fused multispectral image is input into the improved YOLOv5 model. After a series of feature extraction, fusion, and calculations, the model outputs the target bounding box and confidence level. The target bounding box is used to determine the position and size of the target in the image, and the confidence level indicates the model's confidence that the detected target belongs to a certain category, providing basic data for subsequent target classification, tracking, and other operations.

[0156] S3. Target Classification: Classify detected targets and identify their types (drones, birds, aircraft, etc.). Use the trained SVM model to classify detected targets. When training the SVM model, use a kernel function to map samples from low-dimensional space to high-dimensional space to improve classification accuracy. Construct a classifier based on ResNet-50 to further accurately distinguish between drones, birds, aircraft, and other categories based on target morphology and motion characteristics.

[0157] In this step, the training and application process of the SVM model includes the following:

[0158] Select kernel function: When training the SVM model, a kernel function is used to map samples in a low-dimensional space to a high-dimensional space to improve the accuracy of classification. For example, the radial basis kernel function is used, and its expression is:

[0159]

[0160] in, Represents the kernel function, which is a function used to map low-dimensional space samples to high-dimensional space; in SVM model training, this function calculates the sample vector and Similarity measurement in high-dimensional space; in this way, the data that may be linearly inseparable in low-dimensional space can be classified by finding a suitable hyperplane in high-dimensional space to improve the accuracy of classification; here and It is a sample vector representing two different samples in the training set. In the aerial target detection scenario based on image recognition, these sample vectors contain various feature information extracted from the aerial target image, such as the target's shape, texture, color, and other features. The vector representation is formed after digital processing and is used to describe the characteristic attributes of the aerial target. is the parameter of the kernel function, which controls the scope and shape of the radial basis kernel function; when When the value is large, the function is more sensitive to the distance between samples, which means that only samples with very close distances will have high similarity; when When the value is small, the function has a wider range of action, and samples that are farther away may also have certain similarities. In practical applications, by adjusting The value of can optimize the classification performance of the SVM model; Represents the sample vector and The Euclidean distance between two samples is used to measure the degree of difference between the two samples in the feature space. The larger the Euclidean distance, the greater the difference between the features of the two samples. The smaller the Euclidean distance, the more similar the features of the two samples. In the kernel function, the Euclidean distance is calculated to determine the similarity weight of the samples in the high-dimensional space. By adjusting The value of can optimize the classification performance of the SVM model.

[0161] Parameter adjustment optimization: by adjusting The classification performance of the SVM model is optimized by using the value of Take values ​​and analyze the classification accuracy, recall rate and other indicators of the model on the training set and validation set to determine the optimal The value makes the SVM model perform better in distinguishing different types of aerial targets;

[0162] Target classification: Use the trained SVM model to perform preliminary classification of detected aerial targets. The model calculates and judges in a high-dimensional space based on the characteristics of the input samples, and outputs the probability or category label of the target belonging to different categories, roughly classifying the targets into categories such as drones, birds, and aircraft.

[0163] Specifically, in support vector machines (SVMs), in addition to the radial basis kernel function, a variety of other kernel functions can be used instead. The following two common kernel functions are:

[0164] First, the polynomial kernel function (Polynomial Kernel), its formula is:

[0165]

[0166] in, and is a sample vector used to describe data characteristics; It is the coefficient of the kernel function, which controls the "tightness" of the kernel function and affects the degree of fit of the model to the data, similar to the radial basis kernel function. effect; is a bias term, which can increase the flexibility of the model and adjust the offset of the feature map space; is the degree of the polynomial, which determines the complexity of the polynomial kernel function. Common values ​​are positive integers such as 2 and 3. For example, when , is a quadratic polynomial kernel function; this kernel function maps low-dimensional data into a high-dimensional polynomial space and performs well when dealing with polynomial relationships between data features. In image recognition, when certain features of a target exhibit polynomial relationships, this kernel function can be used. It can capture these complex nonlinear relationships and improve the classification capabilities of the SVM model.

[0167] Second, the linear kernel function (Linear Kernel), the formula is:

[0168]

[0169] Among them, here and The linear kernel function is the simplest kernel function. It does not perform complex mapping transformations but simply calculates the inner product of two sample vectors. It is particularly advantageous when the data is linearly separable or has high-dimensional features, enabling effective classification without complex nonlinear transformations. In simple aerial target detection scenarios, where only a few key features of the target are of interest and these features exhibit a linear relationship, using a linear kernel function can simplify the calculation process while maintaining high computational efficiency and a certain level of classification accuracy, avoiding the overfitting problem often associated with complex kernel functions.

[0170] Furthermore, the process of building a classifier based on ResNet-50 to distinguish target categories includes:

[0171] Adding an optical flow branch: When building a classifier based on ResNet-50 for motion feature extraction, add an optical flow branch to the input layer of ResNet-50 and use the Farneback dense optical flow algorithm to calculate the target's motion information. This algorithm is based on the assumption of constant brightness and spatial consistency of the image. By calculating the displacement of pixels in adjacent frames, it obtains information such as the target's motion speed and direction. The calculation formula is:

[0172]

[0173] in, Represents an image function, which is used to describe the pixel value information of an image at a certain moment and position. Its value is related to the image's brightness, color and other attributes. In the context of aerial target detection based on image recognition, It describes the brightness, color and other attribute characteristics of each pixel in the image containing the aerial target and background at different times. It is the basic data for the entire optical flow calculation. By analyzing the image function at different times, the motion information of the target is obtained. In this aerial target detection scenario, this function describes the image characteristics of the aerial target and background at different times and positions. is the spatial coordinate of the pixel point on the image plane, Represents the horizontal coordinate, Represents the vertical coordinates. These two coordinates can uniquely determine the position of a pixel in the image on the two-dimensional plane. When calculating the optical flow, it is used to locate the initial position of each pixel. Represents time and is used to identify the moment of image acquisition. When continuously monitoring aerial targets, images collected at different times contain information about the target's different motion states. By recording images at different times and combining them with changes in pixel position, the target's motion trajectory and speed can be analyzed. 、 Represents the pixels at Direction and The displacement in the direction; these two parameters reflect the time interval The changes in the horizontal and vertical positions of the pixels in the image can be used to calculate the target's movement direction and speed. and are key motion information parameters, through which the target's motion direction and speed can be calculated; for example, if A positive value indicates that the pixel moves horizontally to the right; a negative value indicates that the pixel moves horizontally to the left. Likewise, positive values ​​indicate upward movement and negative values ​​indicate downward movement. is the time interval, i.e., from time At the time When analyzing the movement of aerial targets, and 、 Mutually correlated to calculate the target's speed and trajectory during the time period; 、 and The ratio of the target's average speed in the time period can be calculated; for example, the speed in the horizontal direction , the speed in the vertical direction .

[0174] Feature fusion and classification: Classification is performed by combining RGB image features with optical flow features. Features such as color and texture extracted from the RGB image are fused with motion features derived from the optical flow branch to form a more comprehensive and rich feature vector. These fused features are fed into a classifier built on ResNet-50. The classifier further extracts high-level features through multiple layers of convolution and pooling operations, and finally uses a fully connected layer for classification prediction. By integrating motion features, the classifier can more accurately distinguish dynamic targets, improving the recognition accuracy of drones, birds, and aircraft in mid-air, and making more precise judgments on target categories.

[0175] S4. Target tracking: Combined with the DeepSORT algorithm (or SORT algorithm), Kalman filtering and Hungarian matching are used to achieve continuous tracking of multiple targets and output motion trajectories.

[0176] In this step, target tracking specifically includes the following steps:

[0177] S4.1. Trajectory prediction optimization:

[0178] Introducing adaptive noise covariance matrix: In the Kalman filter process, an adaptive noise covariance matrix is ​​introduced to optimize trajectory prediction; in traditional Kalman filtering, process noise and observation noise It is usually set to a fixed value, but in actual aerial target tracking scenarios, the target's motion state is changeable, and it is difficult to accurately match the fixed noise value; therefore, a dynamic adjustment method is adopted, and the formula is:

[0179]

[0180] in, and are the coefficients used to adjust the process noise covariance matrix and the observation noise covariance matrix, It will be adjusted adaptively according to the target movement speed. Its main function is to control Time process noise covariance matrix Relative to the previous moment When the target moves faster, it means that the target state changes more complex and more difficult to predict. value, so that Increase, so that the Kalman filter algorithm can have a more adequate estimate of the uncertainty of the prediction model, and improve the model's adaptability to rapidly changing targets; and Similarly, it will change adaptively according to the target motion state. The moment-by-moment observation noise covariance matrix Relative to the previous moment When the target's speed changes, the reliability of the measurement data will also change. For example, when the target moves quickly, the measurement error may increase. Valuer Increasing allows the Kalman filter to more reasonably consider the measurement error when fusing the observation data, thus improving the filtering effect. and are the adjusted process noise and observation noise respectively. Process noise is used to describe the uncertainty of the system model itself. For example, in the aerial target detection scenario, the deviation between the actual motion and the model prediction is caused by the target maneuvering (sudden change of speed, direction, etc.). It is used to quantify this uncertainty. The size and distribution of the moment; Observation noise is used to reflect the error of the measurement data. For example, during the image acquisition process, due to factors such as camera accuracy and environmental interference, the observation data such as the target position obtained may have errors. This error is Quantify the size and distribution of moments. and are the process noise and observation noise of the target’s previous trajectory position, represent The process noise covariance matrix at time t is calculated Time process noise covariance matrix It reflects the degree of uncertainty of the system model at the previous moment and provides a reference for the prediction at the current moment; for The observation noise covariance matrix at time t is calculated It reflects the error of the measurement data at the previous moment.

[0181] Adaptive adjustment parameters: and It will be adjusted adaptively according to the target's movement speed; when the target's movement speed is faster, the target state changes more drastically, and the and The value of , makes the covariance of process noise and observation noise increase accordingly, so that the Kalman filter can adapt to the rapid changes of the target more flexibly and adjust the predicted trajectory in time; on the contrary, if the target moves slowly, reducing and The value of can reduce the noise covariance, ensure the stability of the prediction, and avoid excessive fluctuations in the prediction trajectory.

[0182] Predicting target position: Based on the Kalman filter principle and combined with the adjusted noise covariance matrix, the state estimate of the target at the previous moment (position, speed, and other information) is used to predict the position and state of the target at the current moment. The prediction process involves the calculation of the state transfer equation and the observation equation. Through continuous iteration, a more accurate target prediction position is provided for subsequent data association.

[0183] S4.2. Data association improvements:

[0184] Calculate the cost matrix: In the Hungarian matching stage, the cost matrix is ​​calculated by combining the appearance features (ReID) and motion information. The calculation formula is:

[0185]

[0186] in, Represents the elements in the cost matrix, representing the The prediction target and The smaller the value, the greater the possibility that the two targets belong to the same target, and they will be matched first in the Hungarian matching algorithm. The intersection-over-union distance is an indicator that measures the degree of overlap between two bounding boxes (the predicted target box and the detected target box). It is calculated by dividing the intersection area of ​​the two bounding boxes by the union area. The smaller the value of , the higher the degree of overlap between the two bounding boxes, which means they are more likely to be the same target; for example, when the two bounding boxes completely overlap, ; When the two bounding boxes do not have any overlap, . The cosine distance of the appearance feature is used to measure the similarity of the appearance features of two targets. It is obtained by calculating the cosine similarity between the target appearance feature vectors. The higher the cosine similarity, the more similar the appearance of the two targets is, and the smaller the corresponding cosine distance is. In practical applications, the appearance feature can be a color histogram, texture feature, or feature vector extracted by deep learning. The smaller the value, the greater the possibility that the two targets belong to the same target. It is an adjustable weight, with a value range of 0 to 1, which is set according to the specific application scenario and target characteristics; it is used to balance the intersection-over-union distance and cosine distance of appearance features Importance in cost matrix calculation; when When it is close to 1, it means that the target's motion continuity is more important, and the intersection-over-union distance plays a dominant role in the cost calculation. When it is close to 0, more attention is paid to the appearance features of the target, and the cosine distance of the appearance features has a greater impact on the cost.

[0187] Matching targets: Based on the calculated cost matrix, the Hungarian matching algorithm is used to find the optimal match. Through a series of calculations and transformations, the algorithm finds the best correspondence between multiple predicted and detected targets, determining which predicted and detected targets belong to the same target. This allows for accurate association and continuous tracking of multiple targets, and outputs the target's trajectory.

[0188] S5. Data storage: The detection result information such as the target image, target type, time, location, type, etc. is stored in a cloud database in a certain format for subsequent query and analysis.

[0189] In this step, the data storage process includes the following steps:

[0190] S5.1. Organize test result data:

[0191] Data collection: Aggregate various data generated by target detection, classification, and tracking. Target detection acquires images of the target, including visible light and infrared images, which fully record the target's appearance. Target classification acquires target type information, clarifying whether the target is a drone, bird, or aircraft. Target tracking extracts target time and location data, accurately recording the moment the target is detected. Location information represents the target's specific location in the image in the form of coordinates.

[0192] Data verification: Perform preliminary verification on the collected data to check its integrity and accuracy; ensure that the target image is not damaged or has no missing pixels, that the target type information meets the preset classification standards, that the time format is correct and continuous, and that the location coordinates are within a reasonable range; if any problems are found in the data, make corrections or supplements in a timely manner to ensure the quality of the stored data;

[0193] S5.2. Data format conversion:

[0194] Select a storage format: Choose an appropriate storage format based on the cloud database's requirements and data characteristics. For target images, common image formats such as JPEG and PNG may be used for compressed storage to reduce storage space. For structured data such as target type, time, and location, convert them to a format supported by the database. For example, convert time to a timestamp format and store location coordinates as specific numeric types.

[0195] Data encoding: Encode the converted data to ensure its security and stability during storage and transmission. For example, encrypt sensitive information to prevent data leakage; convert text data into character encoding to ensure compatibility across different systems and platforms.

[0196] S5.3. Store in cloud database:

[0197] Connect to the database: Use the appropriate database connection tools and interfaces to establish a connection to the cloud database. Depending on the type of cloud database (such as MySQL, MongoDB, etc.), configure the correct connection parameters, including server address, port number, username, password, etc., to ensure successful access to the database.

[0198] Create a data table (if necessary): If a data table for storing aerial target detection results does not already exist in the cloud database, create a new data table based on the data structure and requirements. Define the table's fields, such as the target image field (to store image paths or binary data), the target type field (set to string type to store target category names), the time field (select an appropriate time type), and the location field (defined based on the coordinate data type). Set primary keys and indexes to optimize data storage and query performance.

[0199] Insert data: Insert the sorted and converted data into the data table of the cloud database; insert the target image, target type, time, location and other data into the corresponding records one by one according to the field order of the data table; you can use the insert statements provided by the database (such as SQL's INSERT INTO statement or database-specific insert methods) to insert data in batches or one by one to ensure that the data is stored accurately in the database for subsequent query and analysis.

[0200] S6. Alarm and feedback: Based on the detection results, when an abnormal target is detected, a hierarchical alarm mechanism is triggered through the API, and an alarm signal or feedback signal is triggered according to the preset rules to notify relevant personnel or systems. For example, in an airport scenario, when a bird is detected entering a specific area of ​​the airport, an alarm message is immediately sent to the airport command center, and information such as the location and type of the target is fed back so that staff can take timely measures to remove the bird.

[0201] In this step, the alarm and feedback process includes the following steps:

[0202] S6.1. Evaluation of test results:

[0203] Extracting key information: Extracting key information from the results of target detection, classification, and tracking, including target confidence, target type, and target location information. For example, obtaining the target bounding box confidence output by the improved YOLOv5 model, the target type (drone, bird, aircraft, etc.) determined by the SVM model and the classifier built based on ResNet-50, and the real-time target location coordinates obtained by DeepSORT algorithm tracking.

[0204] Comparison with preset rules: The extracted key information is compared with preset alarm rules. In an airport scenario, the preset rules may specify that an alarm is triggered when a bird is detected entering a specific area of ​​the airport (such as a certain range around the runway). For drones, an alarm is also triggered if the drone enters a restricted flight area and the confidence level reaches a certain threshold.

[0205] S6.2. Triggering of the hierarchical alarm mechanism:

[0206] Determine threat level: Determine threat level based on target type and confidence level; Adopt a graded alarm strategy: Dynamically adjust alarm thresholds based on target threat level (e.g., drone vs. bird):

[0207]

[0208] in, Represents the alarm level, which is determined based on the target confidence and target type and is divided into three levels: High, Medium, and Low; Refers to the confidence level of target detection, which is the model's assessment of the confidence that the detected target belongs to a certain category (the value range is usually between 0 and 1, the closer the value is to 1, the more confident the model is about the accuracy of the detection result; the closer the value is to 0, the less reliable the detection result); Indicates the target category. In this aerial target detection scenario, this mainly includes drones, birds, and aircraft. Different categories pose different levels of threat to security, so the target category needs to be considered when determining the alarm level.

[0209] Taking drones as an example, the following classification strategy can be adopted:

[0210] If the confidence level is ≥ 0.9 and the target type is a drone, the threat level is considered high. In this case, the detected drone is highly reliable because it may be carrying dangerous goods. With high confidence, its intrusion poses a significant threat to airport security, important facilities, or urban public safety, requiring emergency response measures.

[0211] When 0.7 ≤ confidence level < 0.9, the threat level is considered medium. Within this range, although the accuracy of the target judgment is slightly lower than that of the high confidence level, it still requires attention and relevant personnel need to further confirm the target situation.

[0212] Other situations (such as low confidence or the target type is a common bird, etc.) are classified as low threat level (Low). At low alarm levels, records and prompts are generally made for subsequent observation;

[0213] Triggering an alarm signal: The corresponding alarm signal is triggered based on the determined threat level. When the threat level is high, a strong alarm prompt is quickly sent to relevant personnel or systems through the API, such as sending an emergency alert to the airport command center, which may be accompanied by sound and light prompts to remind staff to take immediate countermeasures. At a medium threat level, a general alarm notification is sent, and relevant personnel can be informed of the target dynamics via SMS, in-site messages, etc.; at a low threat level, records and prompts are made for subsequent observation.

[0214] S6.3. Feedback target information:

[0215] Organize feedback data: Organize detailed information about the target into a format that is easy for the recipient to understand and use. Detailed information about the target includes the target's exact location (expressed in coordinates or a specific geographic area), target type (specify whether it is a drone, bird, or aircraft), target trajectory (if tracked), and detection time.

[0216] Send feedback information: Send the organized target information to relevant personnel or systems through the API. For example, in an airport scenario, this information is fed back to the airport command center. Based on this information, staff can quickly understand the situation of abnormal targets and promptly arrange measures such as driving away birds and intercepting drones to ensure the safety of airport operations. In an urban management scenario, feedback is fed back to the monitoring system of the urban management department to help managers supervise drone activities over the city and maintain public safety.

[0217] Those skilled in the art will appreciate that the process of implementing all or part of the steps of the above embodiments may be accomplished by hardware, or by instructing related hardware through a program.

[0218] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting aerial targets based on image recognition, characterized in that: The steps include: S1. Image acquisition and preprocessing: Synchronously deployed high-definition visible light cameras and infrared cameras collect dual-spectral image data of aerial targets in real time at a set frame rate, ensuring the continuity and timeliness of the acquired images. The images are also preprocessed to ensure the spatiotemporal consistency of multi-source data. S2. Object Detection: Detect objects in the image using a deep learning model. Using an improved YOLOv5 model as the backbone network, the fused multispectral image is input and the object bounding box and confidence score are output. And introduce the attention mechanism to optimize the small target detection performance; S3, target classification: classify the detected targets and identify their types; The detected targets are classified using the trained SVM model. When training the SVM model, a kernel function is used to map samples from the low-dimensional space to the high-dimensional space. At the same time, a classifier is built based on ResNet-50 to further accurately distinguish target categories based on target morphology and motion characteristics. S4. Target tracking: Combined with the DeepSORT algorithm, Kalman filtering and Hungarian matching are used to achieve continuous tracking of multiple targets and output motion trajectories; S5. Data storage: Store the target detection result information in a cloud database in a certain format for subsequent query and analysis; S6. Alarm and feedback: Based on the detection results, when an abnormal target is detected, the hierarchical alarm mechanism is triggered through the API, and the alarm signal or feedback signal is triggered according to the preset rules to notify the relevant personnel or system.

2. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S1, the camera continuously captures aerial images at a fixed frame rate. To ensure image quality, the camera is equipped with autofocus and autoexposure functions, adjusting parameters in real time based on ambient light and target distance. Image stabilization technology is also used to reduce image blur caused by device vibration, ensuring that the captured images are clear and stable, providing reliable data for subsequent processing.

3. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S1, the image preprocessing process includes at least denoising, normalization, enhancement, and registration; specifically, the following steps: Image denoising: Use the non-local mean denoising algorithm to process the collected images to make the target outlines and features in the image clearer, providing more accurate data for subsequent target detection and recognition; Image normalization: Map the pixel values ​​of the image to the range of 0-1; Image enhancement: For visible light images, contrast-limited adaptive histogram equalization is used to improve image quality in low illumination conditions. For infrared images, guided filtering is combined to suppress noise. Image registration: A scale-invariant feature transformation algorithm is used to match feature points between visible and infrared images. A random sampling consensus algorithm is then used to eliminate mismatched points, ensuring spatial alignment of the bispectral images. Finally, the homography matrix H is optimized using the least squares method to obtain an accurate image transformation relationship, enabling precise registration of visible and infrared images and providing reliable data for subsequent target detection and analysis based on the fused images.

4. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S2, the specific steps of target detection include the following: S2.

1. Model selection and improvement: Select the backbone network: Use the improved YOLOv5 model as the backbone network for target detection; At the same time, the attention mechanism is introduced: to address the problem that the traditional YOLO algorithm has insufficient accuracy in detecting small targets, the attention mechanism is introduced in the feature extraction stage of the improved YOLOv5 model; S2.2 Multispectral feature fusion: Backbone cross-modal feature fusion: A cross-modal feature fusion module is introduced into the Backbone part of YOLOv5 to fuse visible light and infrared image features; Optimization of small target detection in the neck: The convolutional block attention module (CBAM) is embedded in the neck to enhance the feature response of small targets. Weighted feature pyramid for multi-spectral fusion: Based on FPN, feature maps at different levels are adaptively weighted and fused; S2.3, Dynamic confidence threshold adjustment: according to the ambient light intensity Adjust the detection threshold T; in scenes with large changes in light intensity, dynamically adjust the detection threshold to avoid false detection or missed detection due to excessive or insufficient light; S2.

4. Lightweight model design: Replace standard convolution with depthwise separable convolution to reduce model computational complexity. S2.

5. Model input and output: The preprocessed and fused multispectral image is input into the improved YOLOv5 model; after a series of feature extraction, fusion, and calculation, the model outputs the target bounding box and confidence level.

5. The method for detecting aerial targets based on image recognition according to claim 4, wherein: In S2.1, when the improved YOLO algorithm is used for target detection, an attention mechanism is introduced in the feature extraction stage of the improved YOLOv5 model to enhance the focus on small target features by calculating the importance weights of different feature channels. The importance weights of different feature channels are calculated using the following formula: First, based on the spatial dimension The original feature map , calculation channel The global average pooling result of ; Then through the learnable weight matrix 、 , and the ReLU activation function and Sigmoid activation function Calculation Channel The attention weight ; Finally, we get the feature map weighted by the attention mechanism , the attention weight With the original feature map The corresponding elements are multiplied to achieve the enhancement operation of the original feature map.

6. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S3, the training and application process of the SVM model includes the following: Select kernel function: When training the SVM model, a kernel function is used to map samples from low-dimensional space to high-dimensional space to improve classification accuracy. Target classification: Use the trained SVM model to perform preliminary classification of detected aerial targets; The model performs calculations and judgments in a high-dimensional space based on the characteristics of the input samples, and outputs the probability or category label that the target belongs to different categories.

7. The method for detecting aerial targets based on image recognition according to claim 6, wherein: In S3, the process of building a classifier based on ResNet-50 to distinguish target categories includes: Adding an optical flow branch: When building a classifier based on ResNet-50 for motion feature extraction, an optical flow branch is added to the input layer of ResNet-50. The Farneback dense optical flow algorithm is used to calculate the target's motion information. This algorithm, based on the assumptions of constant brightness and spatial consistency of the image, calculates the displacement of pixels in adjacent frames to obtain the target's motion speed and direction. Feature fusion and classification: Classification is performed by combining RGB image features with optical flow features. Features extracted from the RGB image are fused with motion features obtained from the optical flow branch to form a more comprehensive and rich feature vector. These fused features are input into a classifier built on ResNet-50. The classifier further extracts high-level features through multi-layer convolution and pooling operations, and finally uses a fully connected layer for classification prediction.

8. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S4, target tracking specifically includes the following steps: S4.

1. Trajectory prediction optimization: Introducing adaptive noise covariance matrix: In the Kalman filtering process, an adaptive noise covariance matrix is ​​introduced to optimize trajectory prediction; Adaptive adjustment parameters: used to adjust the coefficients of the process noise covariance matrix and the observation noise covariance matrix and It will adaptively adjust according to the target's movement speed, allowing the Kalman filter to more flexibly adapt to rapid changes in the target and adjust the predicted trajectory in time; Predicting target position: Based on the Kalman filter principle and the adjusted noise covariance matrix, the target's state estimate at the previous moment is used to predict the target's position and state at the current moment. S4.

2. Data association improvements: Calculate the cost matrix: In the Hungarian matching stage, the cost matrix is ​​calculated by combining appearance features and motion information. The calculation formula is: in, Represents the elements in the cost matrix, representing the The prediction target and The association cost between the detection targets; Calculate the distance metric: Represents the intersection-over-union distance, which is used to measure the degree of overlap between the predicted target box and the actual detected target box; is the appearance feature cosine distance, which is obtained by calculating the cosine similarity between the appearance feature vectors of the target; Adjust weights: It is an adjustable weight, with a value range between 0 and 1, and is set according to the specific application scenario and target characteristics; Matching goal: Based on the calculated cost matrix, use the Hungarian matching algorithm to find the optimal match.

9. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S5, the data storage process includes the following steps: S5.

1. Organize test result data: Collect data: summarize various data generated by target detection, classification, and tracking; Data verification: Conduct preliminary verification of the collected data to check its completeness and accuracy; S5.

2. Data format conversion: Select storage format: Choose an appropriate storage format based on the cloud database requirements and data characteristics; Data encoding: Encode the converted data to ensure the security and stability of the data during storage and transmission; S5.

3. Store in cloud database: Connect to the database: Use the corresponding database connection tools and interfaces to establish a connection with the cloud database; Insert data: Insert the sorted and converted data into the data table of the cloud database.

10. The method for detecting aerial targets based on image recognition according to claim 1, wherein: In S6, the alarm and feedback process includes the following steps: S6.

1. Evaluation of test results: Extract key information: Extract key information from the results of target detection, classification and tracking; Compare with preset rules: Compare the extracted key information with the preset alarm rules; S6.

2. Triggering of the hierarchical alarm mechanism: Determine threat level: Determine threat level based on target type and confidence level; Trigger alarm signal: trigger corresponding alarm signal according to the determined threat level; S6.

3. Feedback target information: Organize feedback data: Organize the detailed information of the target into a format that is easy for the recipient to understand and use; Send feedback information: Send the sorted target information to relevant personnel or systems through the API.

Images