Aerial target identification tracking method and system based on multiband electromagnetic fusion perception technology

Abstract

The invention relates to the technical field of target identification, in particular to an aerial target identification tracking method and system based on a multi-band electromagnetic fusion sensing technology. According to the invention, through the cooperative work of the infrared high-definition camera, the event type camera and the multispectral camera, the advantages of each sensor are fully played, and the recognition blind spots of a single sensor in a complex environment are effectively overcome, so that the accuracy and reliability of target recognition are greatly improved, and meanwhile, the accuracy and reliability of target recognition are greatly improved by introducing a laser ranging technology. The distance information between the target and the pod can be acquired in real time, so that the system accurately tracks the motion state of the target and timely adjusts the tracking strategy, and the real identity of the target can be deeply discriminated and recognized through the multiband data acquired by the multispectral camera in combination with the spectral characteristics of the known target in the data storage module. In addition, the method has the capability of coping with various environments such as low-illumination and high-dynamic scenes, and is suitable for wide application fields.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and more specifically, to an air target recognition and tracking method and system based on multi-band electromagnetic fusion perception technology. Background Art

[0002] With the development of unmanned aerial vehicle technology, target recognition and tracking have been increasingly widely used in military, security, traffic monitoring and other fields. Traditional target recognition methods mainly rely on a single sensor, such as an optical camera or an infrared camera. However, the recognition accuracy and robustness of a single sensor are limited in complex environments. For example, the performance of optical devices deteriorates under low light conditions, while infrared devices may be interfered with under daytime weather conditions. Therefore, researchers have begun to explore methods of fusing multiple sensors to enhance the performance of target recognition.

[0003] A system based on a multi-light-source fusion perception pod usually uses multiple sensors (such as an infrared high-definition camera, an event camera, and a multi-spectral camera) for image data fusion, thereby effectively improving the accuracy and speed of target recognition. Through laser ranging technology, the system can obtain the target position in real time and improve the tracking accuracy.

[0004] However, the existing technical solutions have deficiencies in multi-source information fusion and target judgment. Therefore, there is an urgent need for an air target recognition and tracking method and system based on multi-band electromagnetic fusion perception technology to achieve more efficient and accurate target recognition and tracking. Summary of the Invention

[0005] The purpose of the present invention is to provide an air target recognition and tracking method and system based on multi-band electromagnetic fusion perception technology to solve the problem of deficiencies in multi-source information fusion and target judgment in the existing technical solutions as mentioned in the above background art.

[0006] To achieve the above purpose, the present invention provides an air target recognition and tracking method based on multi-band electromagnetic fusion perception technology, including the following steps:

[0007] S1. Configure the perception pod: Configure a perception pod equipped with an infrared high-definition camera, an event camera, and a multi-spectral camera, and install a laser ranging device;

[0008] S2. Collect environmental images: Start the infrared high-definition camera to collect air environmental images and obtain real-time infrared image data;

[0009] S3. Capture target motion information: Start the event camera to quickly capture target motion information in the high-dynamic air scene;

[0010] S4. Multi-band acquisition and feature extraction: Use a multi-spectral camera to perform multi-band acquisition of images, and at the same time use the feature extraction module to deeply analyze and extract features of the target to distinguish the authenticity of the target;

[0011] S5. Image data fusion: Use the image data fusion module to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image;

[0012] S6. Image analysis and target recognition: Use the image analysis module to analyze the fused image and the multi-spectral image using image processing algorithms, extract key features, and identify the type of the target;

[0013] S7. Real-time update of target position and output of tracking data: According to the laser ranging data obtained by the laser ranging device, update the target position in real time for tracking, and use the data transmission module to output the tracking data and position status.

[0014] As a further improvement of this technical solution, the specific operation method for configuring the perception pod in step S1 is as follows:

[0015] S11. Equipment preparation: Prepare an infrared high-definition camera, an event camera, and a multi-spectral camera, and at the same time ensure that the power supplies and data interfaces of all cameras are normal;

[0016] S12. Installation and configuration: Fix the infrared high-definition camera, the event camera, and the multi-spectral camera on the perception pod, confirm that their viewing angles can cover the same target area, and install a laser ranging device on the pod to ensure the synchronous operation of the laser ranging device and the camera;

[0017] S13. System integration: Connect all devices and modules, including the data storage module, the feature extraction module, the image data fusion module, the image analysis module, the data transmission module, and the laser ranging device, to ensure normal data communication between the devices and modules.

[0018] As a further improvement of this technical solution, the specific operation method for environmental image acquisition in step S2 is as follows:

[0019] S21. Infrared image acquisition: Start the infrared high-definition camera, select a suitable operation mode, and at the same time set the image resolution and frame rate to ensure obtaining the best-quality infrared image under the current environmental lighting conditions;

[0020] S22. Data recording: Continuously collect infrared image data and store it in the data storage module for subsequent processing and analysis.

[0021] As a further improvement of this technical solution, the specific operation method for capturing target motion information in step S3 is as follows:

[0022] S31. Event-based camera configuration: Activate the event-based camera, set the sensing sensitivity and triggering conditions, and quickly respond to target movement;

[0023] S32. Target tracking: Start capturing movements in the high-dynamic scene. Once a movement is detected, immediately record the timestamp and relevant image data;

[0024] S33. Data storage: Store the captured event image data and relevant movement information in the data storage module, ensuring consistency with the timestamp of the infrared image data.

[0025] As a further improvement of this technical solution, the specific operation method of multi-band acquisition and feature extraction in step S4 is as follows:

[0026] S41. Multi-spectral image acquisition: Activate the multi-spectral camera, perform multi-band image acquisition on the target area, and set the band range to cover the required spectral features;

[0027] S42. Feature extraction: Use the feature extraction module to analyze the multi-spectral data, extract the spectral features of the target, and perform preliminary target discrimination;

[0028] S43. Authenticity discrimination: Compare the extracted features with the known target feature database to confirm the authenticity of the target or eliminate camouflage, where the known target feature database is stored in the data storage module.

[0029] As a further improvement of this technical solution, the specific operation method of image data fusion in step S5 is as follows:

[0030] S51. Image fusion module configuration: Activate the image data fusion module, and set the fusion algorithm to process the real-time data of the infrared high-definition camera and the event-based camera;

[0031] S52. Data fusion: Align the infrared image and the event image in time series, and use the fusion algorithm to generate a composite image;

[0032] S53. Output the composite image: Use the data storage module to store the fused composite image for subsequent use by the image analysis module.

[0033] As a further improvement of this technical solution, the specific operation method of image analysis and target recognition in step S6 is as follows:

[0034] S61. Selection of image processing algorithm: Select a suitable image processing algorithm in the image analysis module;

[0035] S62. Image analysis: Input the fused image and the multi-spectral image into the image analysis module, and apply the selected image processing algorithm for processing;

[0036] S63, Feature Extraction and Target Recognition: Extract key features from the analyzed image and determine the type of the target.

[0037] As a further improvement of this technical solution, the specific operation method for updating the target position in real time and outputting tracking data in step S7 is as follows:

[0038] S71, Activation of Laser Ranging Device: Activate the laser ranging device, regularly measure the distance between the target and the pod, and obtain the coordinate position of the target relative to the pod;

[0039] S72, Target Tracking Update: Based on the acquired laser ranging data and image information, update the target position in real time and generate the target motion trajectory;

[0040] S73, Data Output: Output the tracking data and position status through the data transmission module, provide real-time feedback to the user or the visualization system, and record them in the data storage module.

[0041] The present invention also provides an air target recognition and tracking system based on multi-band electromagnetic fusion perception technology for implementing the above-mentioned air target recognition and tracking method based on multi-band electromagnetic fusion perception technology, including: an unmanned aerial vehicle (UAV), a perception pod, and a ground control terminal. The perception pod is installed on the UAV, and the ground control terminal is used to control the flight of the UAV. The perception pod is internally integrated with an infrared high-definition camera, an event camera, and a multi-spectral camera. The UAV is internally integrated with a data storage module, a feature extraction module, an image data fusion module, an image analysis module, a data transmission module, and a laser ranging device;

[0042] The data storage module is used to store the data collected by the infrared high-definition camera, the event camera, the multi-spectral camera, and the laser ranging device, and is also used to store the data processed by the feature extraction module, the image data fusion module, the image analysis module, and the data transmission module;

[0043] The feature extraction module is used to deeply analyze and extract features from the multi-band images collected by the spectral camera;

[0044] The image data fusion module is used to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image;

[0045] The image analysis module is used to analyze the fused image and the multi-spectral image using image processing algorithms, extract key features, and identify the type of the target;

[0046] The data transmission module is used to output the tracking data and position status to the ground control terminal;

[0047] The laser ranging device is used to perform laser ranging on a target and transmit the ranging data to a data storage module for storage.

[0048] As a further improvement of this technical solution, the output ends of the infrared high-definition camera, event camera, multispectral camera, and laser ranging device are all electrically connected to the input end of the data storage module. The feature extraction module, image data fusion module, image analysis module, and data transmission module are all bidirectionally electrically connected to the data storage module. The data transmission module and the ground control terminal maintain real-time data intercommunication through the Internet.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] 1. In the present invention, through the collaborative work of the infrared high-definition camera, event camera, and multispectral camera, the advantages of each sensor are fully utilized, effectively overcoming the recognition blind spots of a single sensor in complex environments, thereby greatly improving the accuracy and reliability of target recognition.

[0051] 2. In the present invention, by introducing laser ranging technology, the distance information between the target and the pod can be obtained in real time, enabling the system to accurately track the motion state of the target and timely adjust the tracking strategy.

[0052] 3. In the present invention, through the multi-band data obtained by the multispectral camera and combined with the spectral characteristics of known targets in the data storage module, it helps to deeply discriminate and identify the true identity of the target, significantly reducing the false alarm rate.

[0053] 4. In the present invention, by setting multiple cameras to work collaboratively and using laser ranging technology for assistance, the device has the ability to cope with various environments such as low light and high dynamic scenes, and is applicable to a wide range of application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the overall steps of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0055] Figure 2 It is a schematic diagram of the first step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0056] Figure 3 It is a schematic diagram of the second step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0057] Figure 4 It is a schematic diagram of the third step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0058] Figure 5 Schematic diagram of the fourth step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0059] Figure 6 Schematic diagram of the fifth step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0060] Figure 7 Schematic diagram of the sixth step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0061] Figure 8 Schematic diagram of the seventh step of the air target recognition and tracking method based on multi-band electromagnetic fusion perception technology of the present invention.

[0062] Figure 9 Block diagram of the air target recognition and tracking system based on multi-band electromagnetic fusion perception technology of the present invention. Specific implementation manners

[0063] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0064] In a specific embodiment, as Figure 1 shown, the present invention provides an air target recognition and tracking method based on multi-band electromagnetic fusion perception technology, including the following steps:

[0065] The first step, as Figure 2 shown, configure the sensing pod: Configure a sensing pod equipped with an infrared high-definition camera, an event camera, and a multispectral camera, and install a laser ranging device.

[0066] 1. Equipment preparation.

[0067] Equipment list confirmation: Confirm that the prepared equipment includes: Infrared high-definition camera - with high-resolution and imaging capabilities in low-light environments; Event camera - capable of quickly capturing images in high-dynamic scenes; Multispectral camera - with the ability to collect images in multiple bands, covering visible light, near-infrared, and other bands.

[0068] Power supply check: Use a multimeter or voltage tester to check the power supply of each camera and laser rangefinder, ensure that the voltage range meets the device specifications (for example, an infrared high-definition camera requires DC 12V ± 10%), check the battery status (if rechargeable batteries are used), ensure sufficient charging, and avoid interruption during on-site use.

[0069] 2. Installation and configuration.

[0070] Device fixation: Fix the infrared high-definition camera, event camera, and multispectral camera on the preset mounting seats of the pod respectively, ensure that the angles of the cameras are adjustable, and at the same time confirm that the viewing angles of the infrared camera and the multispectral camera should be able to cover the same target area. A spirit level and angle gauge can be used for adjustment.

[0071] Installation of laser rangefinder: Install the laser rangefinder at an appropriate position on the pod, ensure that its output end points to the target area, and has good visual overlap with other cameras. At the same time, the laser rangefinder should be fixed to the pod base to avoid interference caused by vibration.

[0072] Ensuring synchronous operation: Configure the trigger parameters of each camera in the ground control terminal to ensure that the laser rangefinder can perform accurate measurements when collecting images. At the same time, set the trigger conditions so that the laser rangefinder starts simultaneously during each image acquisition. Use the formula T = L / V (where T is the ranging time, L is the target distance, and V is the speed of light) to ensure accurate distance information is calculated.

[0073] 3. System integration.

[0074] Connection of devices and modules: Connect each device and the control unit through interfaces (such as CAN bus or Ethernet). At the same time, for the data storage module (such as SSD or HDD), ensure connection through a high-speed data transfer interface to meet the requirements of real-time data storage. The feature extraction module, image data fusion module, image analysis module, and laser rangefinder should all be correctly configured.

[0075] Function confirmation: After completion of the connection, run the self-check function of each module to ensure that all interfaces between the modules work properly. For example, perform a performance test on the data transfer module through the control software to ensure that the data latency is within the specified range (such as less than 50ms).

[0076] Data communication test: Use a protocol analyzer to test the data traffic, confirm that the data communication between each device and module is normal, monitor the latency and signal loss conditions. At the same time, if data loss is found, check the interface connection and optimize the system design, and improve the data transfer efficiency through compression algorithms (such as JPEG compression).

[0077] Second step, such as Figure 3As shown, environmental image acquisition: Start the infrared high-definition camera to acquire aerial environmental images and obtain real-time infrared image data.

[0078] 1. Infrared image acquisition.

[0079] Start the infrared high-definition camera: In the control system, start the infrared high-definition camera through the software interface or physical switch. Ensure that all connections are normal and the camera is in standby mode. Execute the self-check program to check whether the basic functions of the camera are normal, such as image output, focusing, and image sensor status.

[0080] Select the operation mode: Select the appropriate operation mode according to the current lighting conditions. Night vision mode - used in extremely low-light environments to enhance image contrast. Automatic gain control mode - automatically adjusts the gain of the camera under changing lighting conditions to obtain clear images. At the same time, ensure that the selected mode is displayed and applied on the control interface.

[0081] Set the image resolution and frame rate: Set the image resolution (e.g., 1920x1080, i.e., full high definition) and frame rate (e.g., 30fps or 60fps) based on the following considerations:

[0082] Higher resolutions can capture more details but will consume more storage space and processing power.

[0083] The choice of frame rate needs to be based on the dynamic characteristics of the target. If the target moves quickly, a higher frame rate should be selected.

[0084] To reduce storage requirements, the high definition (HD) mode can also be used, with a resolution set to 1280x720.

[0085] Environmental light assessment: Before starting the camera, environmental light inspection can be carried out. Use a light meter to measure the environmental light intensity (unit: lx) to ensure that it meets the best working conditions of the infrared camera. For example, if the environmental light intensity is below 10 lx, the low-light enhancement function should be enabled.

[0086] Monitor the image quality: Start the image preview function to monitor the images output by the camera in real time. If the images are blurry or have obvious noise, adjust them through the automatic focusing mechanism or manual focus adjustment. Confirm the clarity of the infrared images. A good signal-to-noise ratio (SNR) is an important indicator of high-quality images. The calculation formula is:

[0087]

[0088] where P signal is the signal power, and P noise is the noise power.

[0089] 2. Data recording.

[0090] Continuously collect infrared image data: Start the data collection program and set the timed collection or continuous collection mode. In continuous mode, the camera will continuously collect images at the set frame rate. Use a queue or buffer mechanism to ensure that image data is not lost and can handle multiple frames of images. Record the timestamp of each frame of image so that the time information of each frame of data can be associated during subsequent analysis.

[0091] Storage method selection: Store the collected infrared image data in the data storage module in real time. You can choose to use a solid-state drive (SSD) to improve the write speed and data reading efficiency. Store it in a suitable data format (such as JPEG, TIFF), considering the balance between image quality and file size.

[0092] Regularly check the storage status: Regularly check the available space of the data storage module, display the current storage capacity in real time through the control interface, and set a limit to prevent the storage from being full. Set a data overwrite or circular storage policy, that is, when the storage space is insufficient, the oldest data can be overwritten (such as stored in a FIFO manner).

[0093] Data integrity verification: After the data is written, perform an integrity check to ensure that each frame of image data is correctly written to the storage device. You can check by calculating the checksum of the image, such as the MD5 hash value. Regularly back up the data to ensure that important data will not be lost due to equipment failure.

[0094] The third step, as Figure 4 shown, capture target motion information: Start the event-based camera to quickly capture the target motion information in the high-dynamic scene in the air.

[0095] 1. Event-based camera configuration.

[0096] Start the event-based camera: Ensure that the power supply of the event-based camera is normal, connect it to the control system, and start the device through the control interface or physical switch. Execute the self-check program to ensure that the camera can work properly, and check the status of the image sensor and the health of the system.

[0097] Set the induction sensitivity: Set the induction sensitivity of the camera according to the pseudo-force of the environment. Generally, the sensitivity can be divided into three levels: low, medium, and high:

[0098] Low sensitivity is suitable for low-speed targets or in good lighting conditions;

[0099] Medium sensitivity is suitable for general dynamic environments;

[0100] High sensitivity is suitable for fast-moving targets or complex backgrounds.

[0101] Images can be collected at different sensitivities through testing, and the sensitivity most suitable for the current environment can be selected.

[0102] Configure trigger conditions: Set trigger conditions according to specific requirements, such as motion detection, light change, image contrast, etc. For example, set the motion detection trigger condition to trigger recording when the speed of an object in a certain area exceeds a set threshold (assumed to be v, for example, 5 m / s).

[0103] Test the device response: After configuration, verify the working status of the camera by simulating motion conditions to ensure its fast response ability. During the test, monitor whether the model can respond within milliseconds.

[0104] 2. Target tracking.

[0105] Start capturing motion: Start the motion detection program and set the backend processing time window (for example, 200 milliseconds) to ensure that actions can still be captured when the target passes quickly. Set the system to monitor the detection area in continuous mode and capture the image stream.

[0106] Motion detection and data recording: Use image processing algorithms (such as optical flow method, background difference method) to detect motion in real time. Once the target appears, immediately trigger data recording, record the motion trajectory and speed of the target, using the formula: where d is the displacement of the target within a specific time interval, and t is the time interval.

[0107] Record the timestamp and related image data: Whenever motion is confirmed, immediately record the current timestamp (in the format of UTC time or system time) and the corresponding image data. Use a formatting function to automatically record the time and uniformly store all timestamps and image data for subsequent synchronization processing and analysis.

[0108] 3. Data storage.

[0109] Store event image data: Store the captured image data together with the timestamp in the data storage module to ensure data integrity. Use standard formats such as JPEG or PNG to reduce storage space requirements and maintain image quality. Establish a data directory structure to organize files based on timestamps.

[0110] Ensure timestamp consistency: During storage, ensure that the data collected by the event camera is consistent with the timestamp of the infrared image. In the data storage module, a synchronization mechanism is usually used. In each storage operation, attach the corresponding timestamp to map the two image data streams. Achieve time alignment of data by comparing timestamps.

[0111] Backup and Inspection: Regularly backup the stored data to ensure that important data will not be lost due to equipment failures. Use a regular backup schedule (such as daily, weekly) to prevent data loss. Use a data integrity check mechanism, such as CRC (Cyclic Redundancy Check), to verify files after storage.

[0112] Step 4, as Figure 5 shown, Multi - band Acquisition and Feature Extraction: Use a multispectral camera to perform multi - band acquisition of images. At the same time, use a feature extraction module to conduct in - depth analysis and feature extraction of the target to distinguish the authenticity of the target.

[0113] 1. Multispectral Image Acquisition.

[0114] Start the multispectral camera: Ensure that the multispectral camera is correctly installed and connected to the control system. Check the power supply, connection cables, and communication interfaces of the device to ensure normal operation. Start the camera through the system software interface, perform a self - test to confirm that the device is fault - free, and prepare to enter the working state.

[0115] Set the band range: According to the target characteristics and their reflected spectral characteristics, set the required band range. Multispectral images usually cover multiple narrow spectral bands, such as red, green, blue, and near - infrared bands.

[0116] An example of setting the band range can be:

[0117] Green band: 500 - 550nm

[0118] Red band: 620 - 670nm

[0119] Near - infrared band: 750 - 900nm

[0120] The target characteristics can be determined through relevant literature or previous studies to ensure coverage of the main reflection characteristics.

[0121] Collect image data: Within the set band range, start the image acquisition program to synchronously acquire multi - band images of the target area. Ensure that the acquisition of each band image is completed within a fixed time delay (such as 50 milliseconds) to reduce the impact of motion blur and time deviation on the data.

[0122] Data Saving and Format: After the data of each band is generated, store it using an appropriate data format (such as TIFF or HDF5) to maintain high - quality and accurate data. The directory structure can be classified and stored by date and band for subsequent processing and retrieval.

[0123] 2. Feature Extraction.

[0124] Using the feature extraction module: Start the feature extraction module and load the previously collected hyperspectral image data. Determine the feature extraction method to be used, such as principal component analysis (PCA), support vector machine (SVM), or deep learning algorithms, etc., to extract important features from the hyperspectral data.

[0125] Analyze the hyperspectral data: Standardize the images of each band to reduce the influence of noise, and calculate the mean and standard deviation of the images of each band. Isolate the target area through image segmentation techniques (such as threshold segmentation, region growing, etc.) and obtain the main features. The feature vector F can be designed as: F = [f1, f2,..., f n , where f n is the feature extracted from the nth band, such as reflectance, spectral feature value, etc.

[0126] Preliminary target discrimination: Conduct preliminary target discrimination based on the extracted features, such as classification or recognition, and use a machine learning model for evaluation. The evaluation can be based on clustering analysis of the features and similarity metrics (such as cosine similarity or Euclidean distance) for a preliminary judgment.

[0127] 3. Authenticity discrimination.

[0128] Compare with the feature database: Prepare a known target feature database to ensure that the database contains the spectral features of various targets and their metadata. This data can come from a genetic feature library, literature, or past experimental data. Compare the extracted target feature F with the features in the database through a feature matching algorithm (such as KNN, SVM, etc.).

[0129] Calculate the similarity: Use a suitable similarity metric formula to evaluate the similarity between the extracted features and the database features. Common methods include:

[0130]

[0131] where represents the dot product of vectors, and represents the norm of the vector. This formula is used to calculate the cosine similarity.

[0132] Confirm the authenticity of the target or exclude camouflage: If the similarity exceeds the set threshold (such as 0.8), the target can be considered valid; otherwise, mark it as camouflaged or uncertain. During the discrimination process, record the discrimination results and related parameters of each target, including the similarity score, to provide a basis for subsequent analysis.

[0133] Data storage and recording: Store the judgment results and related feature data in the data storage module to ensure good management for subsequent query and analysis.

[0134] Step Five. For example Figure 6As shown in the figure, image data fusion: Use the image data fusion module to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image.

[0135] 1. Image fusion module configuration.

[0136] Start the image data fusion module: Ensure that the image fusion module is correctly installed and connected to the real-time data streams of the infrared high-definition camera and the event camera. Start the image fusion module through the system interface and perform self-checks to confirm the effectiveness of the processing link, including the quality detection of the input signal and the device status check.

[0137] Set the fusion algorithm: Select a suitable image fusion algorithm according to the application requirements. Common methods include:

[0138] Weighted average method - Assign different weights to different source images;

[0139] Multi-scale fusion method - Such as wavelet transform, which can process the low-frequency and high-frequency components of the image separately;

[0140] Deep learning-based fusion method - Use neural networks to learn features for fusion.

[0141] Determine the parameters of each algorithm, such as weight settings (w1 and w2), and require that:

[0142] w1 + w2 = 1, 0 ≤ w1, w2 ≤ 1.

[0143] At the same time, configure the adaptability of the algorithm under specific conditions, such as noise suppression and tolerance to light changes.

[0144] Initialize the module settings: Determine and initialize parameters such as the resolution and frame rate of the input source. At the same time, set the buffer size of the data stream to handle data latency and ensure real-time performance.

[0145] 2. Data fusion.

[0146] Temporal alignment: Before data fusion, perform timestamp alignment of the images. Obtain the timestamps of the infrared images and event images, compare the acquisition times of each frame of data, and find the best match. At the same time, perform corresponding image interpolation or compensation according to the time deviation to ensure the accurate docking of multi-source data at the same time point.

[0147] Image preprocessing: Perform necessary preprocessing on the infrared images and event images, including image enhancement, noise removal, and contrast adjustment, to optimize the fusion effect. This can be achieved through filters or histogram equalization algorithms.

[0148] Data fusion: Apply the selected fusion algorithm to the aligned infrared image I IR and event image Ievent Perform fusion to generate the composite image I fused . For example, the fusion formula using the weighted average method:

[0149] I fused (x,y) = w1·I IR (x,y) + I event (x,y)

[0150] where (x,y) represents the pixel position in the image, and w1 and w2 are weights.

[0151] Fusion image optimization: During the fusion process, post-process the generated composite image, such as edge enhancement, contrast adjustment, etc., to improve the recognizability of the target.

[0152] 3. Output the composite image.

[0153] Store the fused composite image: Through the data storage module, store the generated composite image I fused to the specified location, and set the storage format (such as JPEG, TIFF, or RAW, etc.). The saved file structure can be classified by time and type for subsequent processing.

[0154] Record metadata: During the storage process, record relevant metadata, such as the image timestamp, basic information of the source image (such as resolution, exposure, etc.), and store it additionally in JSON or XML format for retrieval.

[0155] Make it available for the subsequent image analysis module: Ensure that the stored composite image can be accessed by the subsequent image analysis module in a timely manner, and set up an interface or API for use in subsequent target recognition, tracking, and analysis processes.

[0156] Sixth step, as Figure 7 shown, image analysis and target recognition: Use the image analysis module to analyze the fused image and the multi-spectral image using image processing algorithms, extract key features, and identify the type of the target.

[0157] 1. Selection of image processing algorithms.

[0158] Analyze the target recognition requirements: According to the specific application scenario, clarify the accuracy, real-time performance, and computing resource requirements for target recognition. For example, military surveillance may pay more attention to real-time performance, while environmental monitoring may pay more attention to accuracy.

[0159] Select appropriate image processing algorithms: According to the target requirements, select suitable image processing algorithms. Commonly used algorithms include:

[0160] Traditional methods: Edge detection (such as Canny algorithm, Sobel operator); Image segmentation (such as thresholding method, K-means clustering); Feature extraction (such as HOG, SIFT, SURF).

[0161] Machine learning-based methods: Support Vector Machine (SVM); Random Forest.

[0162] Deep learning methods: Convolutional Neural Network (CNN), such as YOLO, Faster R-CNN, etc.

[0163] Consider the computational complexity and implementation difficulty of the algorithm to ensure that the selected algorithm can run efficiently on the available hardware resources.

[0164] Set algorithm parameters: According to the selected algorithm, adjust its hyperparameters to optimize performance. For example, set the kernel function of SVM, the learning rate, number of epochs, and batch size of the deep learning model, etc.

[0165] 2. Image analysis.

[0166] Prepare input data: Load the composite image I fused and the multispectral image I multi into the image analysis module. Ensure that the data format matches the module requirements. Common formats include standard image formats (such as PNG, JPEG) or specific multi-dimensional array formats (such as Numpy arrays).

[0167] Apply the selected image processing algorithm: Input the loaded images sequentially into the selected processing algorithm for preprocessing (such as normalization, noise reduction, etc.). For example:

[0168] I processed = f(I fused , I multi )

[0169] where f represents the processing algorithm.

[0170] Execute processing steps: Image enhancement - Use methods such as histogram equalization and contrast stretching to enhance the image quality to improve the effect of subsequent feature extraction. Image segmentation: Perform image segmentation according to the algorithm and threshold to separate the target object from the background and obtain a binary image.

[0171] Analysis result evaluation: Evaluate the processed image results to ensure that no important information is lost, and at the same time visualize the preprocessing and segmentation effects.

[0172] 3. Feature extraction and target recognition.

[0173] Key feature extraction: Using the selected feature extraction method, extract key features from the analyzed image, such as shape features, color distribution, texture features, etc. For example, the following feature extraction methods can be used:

[0174] Histogram of Oriented Gradients (HOG):

[0175] where G(i) is the gradient calculated at the pixel point (x, y), and Φ(i) is the direction.

[0176] Object recognition: According to the extracted features, input them into an object recognition model (such as logistic regression, SVM, neural network, etc.) for object classification. Each object type c can be represented as:

[0177]

[0178] where F represents the extracted feature vector.

[0179] Decision threshold setting: Set the decision threshold for recognition. Only when the recognition probability exceeds this threshold is it determined as the target. For example, if the recognition probability P(c|F>T_{thresh}), then the target is judged as valid.

[0180] Output recognition result: Record and output the recognized result (including object type, location information, and confidence level) for use by subsequent processing modules (such as the object tracking module). At the same time, visualize the result for analysis and verification.

[0181] Step 7, as Figure 8 shown, update the target position in real time and output tracking data: According to the laser ranging data obtained by the laser ranging device, update the target position in real time, perform tracking, and use the data transmission module to output the tracking data and position status.

[0182] 1. Enable the laser ranging device.

[0183] Start the laser ranging device: Ensure that the laser ranging device is properly connected to the pod system and has been calibrated, including checking the status of the laser emission module and the receiving module. Start the ranging process to make the laser ranging device enter the working state and be ready for regular measurements.

[0184] Regularly measure the distance between the target and the pod: Set the measurement time interval (for example, measure once per second). Use the laser ranging formula to obtain the actual distance d between the target and the pod:

[0185]

[0186] where c is the speed of light (about 299792458 m / s), and T is the round-trip time of the laser signal.

[0187] Obtain the coordinate position of the target relative to the pod: Based on the distance value d obtained by laser ranging and the coordinate system of the pod (set as (x t , y t , z t )) and combined with the pitch and yaw angles of the target, calculate the relative coordinate position of the target (x g , y g , z g ):

[0188] x g = x t + d·cos(pitch)·sin(yaw)

[0189] y g = y t + d·cos(pitch)·cos(yaw)

[0190] z g = z t + d·sin(pitch).

[0191] And save the result in the memory of the suspension system for use in subsequent steps.

[0192] 2. Target tracking update.

[0193] Based on the obtained laser ranging data and image information: Compare the target position (x g , y g , z g ) provided by the laser ranging data with the target position data recognized in the image processing module to ensure the synchronization and consistency of the information sources.

[0194] Update the target position in real time: Use algorithms such as Kalman Filter to smooth and predictively update the target position to cope with noise and measurement errors.

[0195] Generate the target motion trajectory: Dynamically record the target position (x g , y g , z g ) after each update to generate the target motion trajectory. The motion trajectory can be described by a mathematical model and conforms to the functional form:

[0196] Trajectory(t) = (x(t), y(t), z(t))

[0197] where t is the time parameter, and the recorded trajectory is used to analyze the motion pattern and behavior of the target.

[0198] 3. Data output.

[0199] Output tracking data and position status through the data transmission module: Output the latest target position data (x g , y g , z g ) and the motion trajectory through the data transmission interface, and use real-time transmission protocols such as TCP / IP or UDP for data sending.

[0200] Provide real-time feedback to the user or visualization system: Transmit the tracking data to the user interface or visualization system, and display the target motion status in real time through the graphical user interface (GUI), supporting real-time monitoring and decision-making support.

[0201] Record in the data storage module: Store the obtained tracking data and target status information (including timestamp, position coordinates, speed, motion direction, etc.) in the data storage module for subsequent data analysis and backtracking. At the same time, format the output structure to ensure that the data is easy to query and use.

[0202] As Figure 9 shown, based on the above embodiment, the present invention also provides an air target recognition and tracking system based on multi-band electromagnetic fusion perception technology for implementing the above-mentioned air target recognition and tracking method based on multi-band electromagnetic fusion perception technology, including an unmanned aerial vehicle (UAV), a sensing pod, and a ground control terminal.

[0203] Control the flight of the UAV through the ground control terminal. At the same time, the sensing pod is installed on the UAV, and an infrared high-definition camera, an event camera, and a multispectral camera are integrated inside the sensing pod. At the same time, a data storage module, a feature extraction module, an image data fusion module, an image analysis module, a data transmission module, and a laser rangefinder are also integrated inside the UAV.

[0204] During use, first collect environmental image data through the infrared high-definition camera inside the sensing pod to obtain real-time infrared image data, and quickly capture the target motion information in the high-dynamic aerial scene through the event camera. At the same time, use the multispectral camera to perform multi-band image acquisition. Then, use the feature extraction module inside the UAV to deeply analyze and extract features of the target to distinguish the authenticity of the target. Next, use the image data fusion module to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image to enhance the saliency of the target. Finally, use the image analysis module to analyze the fused image and the multispectral image using image processing algorithms to extract key features, identify the type of the target, and at the same time, according to the laser ranging data obtained by the laser rangefinder, update the target position in real time, and use the data transmission module to transmit the processed and analyzed information to the ground control terminal, so as to ensure that the ground control terminal can better combine the information to control the flight of the UAV.

[0205] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and the descriptions in the specification are only preferred examples of the present invention, and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. An air target recognition and tracking method based on multi-band electromagnetic fusion perception technology, characterized in that It includes the following steps: S1. Configure the perception pod: Configure a perception pod equipped with an infrared high-definition camera, an event camera, and a multispectral camera, and install a laser ranging device; S2. Collect environmental images: Start the infrared high-definition camera to collect aerial environmental images and obtain real-time infrared image data; S3. Capture target motion information: Start the event camera to quickly capture target motion information in the high-dynamic aerial scene; S4. Multiband acquisition and feature extraction: Use the multispectral camera to perform multiband acquisition on the images, and at the same time use the feature extraction module to deeply analyze and extract features of the target to distinguish the authenticity of the target; S5. Image data fusion: Use the image data fusion module to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image; S6. Image analysis and target recognition: Use the image analysis module to analyze the fused image and the multispectral image using image processing algorithms, extract key features, and identify the type of the target; S7. Real-time update the target position and output tracking data: According to the laser ranging data obtained by the laser ranging device, real-time update the target position, perform tracking, and use the data transmission module to output the tracking data and position status.

2. The method for identifying and tracking aerial targets based on multi-band electromagnetic fusion perception technology according to claim 1, wherein The specific operation method for configuring the perception pod in step S1 is as follows: S11. Equipment preparation: Prepare an infrared high-definition camera, an event camera, and a multispectral camera, and at the same time ensure that the power supply and data interfaces of all cameras are normal; S12. Installation and configuration: Fix the infrared high-definition camera, the event camera, and the multispectral camera on the perception pod, confirm that their viewing angles can cover the same target area, and install a laser ranging device on the pod to ensure the synchronous operation of the laser ranging device and the camera; S13. System integration: Connect all devices and modules, including the data storage module, the feature extraction module, the image data fusion module, the image analysis module, the data transmission module, and the laser ranging device, and ensure normal data communication between the devices and modules.

3. The method for identifying and tracking an aerial target based on the multi-band electromagnetic fusion perception technology according to claim 1, wherein, The specific operation method for environmental image collection in step S2 is as follows: S21. Infrared image collection: Start the infrared high-definition camera, select a suitable operation mode, and at the same time set the image resolution and frame rate to ensure obtaining the best-quality infrared image under the current environmental lighting conditions; S22. Data recording: Continuously collect infrared image data and store it in the data storage module for subsequent processing and analysis.

4. The method for identifying and tracking aerial targets based on multi-band electromagnetic fusion perception technology according to claim 1, characterized in that, The specific operation method for capturing target motion information in step S3 is as follows: S31. Event camera configuration: Start the event camera, set the induction sensitivity and trigger conditions, and quickly respond to target motion; S32. Target tracking: Start capturing motion in the high-dynamic scene. Once motion is detected, immediately record the timestamp and related image data; S33. Data storage: Store the captured event image data and related motion information in the data storage module to ensure consistency with the data timestamp of the infrared images.

5. The method for identifying and tracking aerial targets based on multi-band electromagnetic fusion perception technology according to claim 1, characterized in that The specific operation method for multiband acquisition and feature extraction in step S4 is as follows: S41. Multispectral image acquisition: Start the multispectral camera, acquire multi-band images of the target area, and set the band range to cover the required spectral features; S42. Feature extraction: Use the feature extraction module to analyze the multispectral data, extract the spectral features of the target, and perform preliminary target discrimination; S43. Authenticity discrimination: Compare the extracted features with the known target feature database to confirm the authenticity of the target or exclude camouflage, where the known target feature database is stored in the data storage module.

6. The method for identifying and tracking airborne targets based on multi-band electromagnetic fusion perception technology according to claim 5, wherein The specific operation method of image data fusion in step S5 is as follows: S51. Image fusion module configuration: Start the image data fusion module and set the fusion algorithm to process the real-time data of the infrared high-definition camera and the event camera; S52. Data fusion: Align the infrared image and the event image in time series, and use the fusion algorithm to generate a composite image; S53. Output the composite image: Use the data storage module to store the fused composite image for subsequent use by the image analysis module.

7. The air target recognition and tracking method based on multi-band electromagnetic fusion perception technology according to claim 1, wherein, The specific operation method of image analysis and target recognition in step S6 is as follows: S61. Selection of image processing algorithm: Select a suitable image processing algorithm in the image analysis module; S62. Image analysis: Input the fused image and the multispectral image into the image analysis module and process them using the selected image processing algorithm; S63. Feature extraction and target recognition: Extract key features from the analyzed image and determine the type of the target.

8. The method for identifying and tracking aerial targets based on multi-band electromagnetic fusion perception technology according to claim 7, characterized in that, The specific operation method of real-time updating the target position and outputting tracking data in step S7 is as follows: S71. Enable the laser ranging device: Start the laser ranging device, regularly measure the distance between the target and the pod, and obtain the coordinate position of the target relative to the pod; S72. Target tracking update: Based on the acquired laser ranging data and image information, update the target position in real time and generate the target motion trajectory; S73. Data output: Output the tracking data and position status through the data transmission module, provide real-time feedback to the user or the visualization system, and record them in the data storage module.

9. An air target recognition and tracking system based on multi-band electromagnetic fusion sensing technology is used to implement the air target recognition and tracking method based on multi-band electromagnetic fusion sensing technology according to any one of claims 1-8, characterized in that Including: An unmanned aerial vehicle, a sensing pod, and a ground control terminal. The sensing pod is installed on the unmanned aerial vehicle. The ground control terminal is used to control the flight of the unmanned aerial vehicle. An infrared high-definition camera, an event camera, and a multispectral camera are integrated inside the sensing pod. A data storage module, a feature extraction module, an image data fusion module, an image analysis module, a data transmission module, and a laser ranging device are integrated inside the unmanned aerial vehicle; The data storage module is used to store the data collected by the infrared high-definition camera, the event camera, the multispectral camera, and the laser ranging device, and is also used to store the data processed by the feature extraction module, the image data fusion module, the image analysis module, and the data transmission module; The feature extraction module is used to deeply analyze and extract features from the multi-band images collected by the spectral camera; The image data fusion module is used to fuse the image data collected by the infrared high-definition camera and the event camera to generate a composite image; The image analysis module is used to analyze the fused image and the multi-spectral image by using image processing algorithms, extract key features, and identify the type of the target; The data transmission module is used to output tracking data and position status to the ground control terminal; The laser ranging device is used to perform laser ranging on the target and transmit the ranging data to the data storage module for storage.

10. The air target recognition and tracking system based on multi-band electromagnetic fusion perception technology according to claim 9, characterized in that, The output ends of the infrared high-definition camera, the event-type camera, the multi-spectral camera, and the laser ranging device are all electrically connected to the input end of the data storage module. The feature extraction module, the image data fusion module, the image analysis module, and the data transmission module are all bidirectionally electrically connected to the data storage module. The data transmission module and the ground control terminal maintain real-time data intercommunication through the Internet.

Images