Multi-spectral fusion target detection and tracking system and method for UAV-mounted optoelectronic turret

Abstract

The present invention relates to the technical field of target tracking of unmanned aerial vehicle optoelectronic turret, and in particular to a multi-spectral fusion target detection and tracking system and method for an unmanned aerial vehicle airborne optoelectronic turret. The system includes a visible light optical subsystem, an infrared optical subsystem, and a tracker; the visible light optical subsystem includes a visible light camera and a visible light filter wheel; the infrared optical subsystem includes an infrared camera and an infrared filter wheel; the visible light filter wheel and the infrared filter wheel both include a filter wheel, a filter, and a driving mechanism; the filter includes an all-pass band filter and N-1 filters of specific bands; when the unmanned aerial vehicle is flying in the air, the filters are automatically switched in turn, and the spectral fusion information is used to detect and confirm the target, and then the filter position is locked and the relevant tracking method is used for stable tracking, so as to improve the stability and accuracy of the tracking, so that the unmanned aerial vehicle optoelectronic turret has the ability of gaze and squint fusion reconnaissance, and significantly improves the efficiency of multi-spectral target detection and recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) optoelectronic turret target tracking, and in particular to a multi-spectral fusion target detection and tracking system and method for an unmanned aerial vehicle (UAV) airborne optoelectronic turret. Background Art

[0002] In drone aerial photography and monitoring missions, the stability and accuracy of target tracking are crucial. Existing technologies usually rely on a single sensor, such as a visible light camera, but under complex lighting and occlusion conditions, target detection performance will be significantly reduced. Traditional multispectral and hyperspectral imaging can only perform vertical downward imaging, that is, it can only detect targets directly below the drone, with low reconnaissance efficiency and inability to track targets.

[0003] The multi-spectral fusion target detection and tracking technology of the UAV-mounted optoelectronic turret is an important branch in the field of UAV reconnaissance. With the development of UAV technology, UAVs are increasingly used in civilian fields such as military reconnaissance, border monitoring, search and rescue, and mineral exploration. In these applications, target detection and tracking technology is one of the key technologies, especially for the detection and tracking of camouflaged targets in complex environments.

[0004] Traditional multispectral / hyperspectral reconnaissance technology can usually only perform vertical downward push-scan imaging, which limits the reconnaissance range and efficiency. With the development of science and technology, thermal imagers, lasers, etc. are integrated into the optoelectronic turret, which has the ability to perform tasks around the clock. However, these systems usually have significantly reduced target detection performance under complex lighting conditions. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multi-spectral fusion target detection and tracking system and method for an unmanned aerial vehicle airborne optoelectronic turret.

[0006] The first object of the present invention is to provide a multi-spectral fusion target detection and tracking system for an unmanned aerial vehicle airborne optoelectronic turret, comprising: a visible light optical subsystem, an infrared optical subsystem, and a tracker;

[0007] The visible light optical subsystem includes a visible light camera and a visible light filter wheel;

[0008] The infrared optical subsystem includes an infrared camera and an infrared filter wheel;

[0009] Both the visible light filter wheel and the infrared filter wheel include a filter wheel, a filter and a driving mechanism; N through holes are evenly provided along the circumference of the filter wheel, and the N filters are correspondingly arranged at the N through holes;

[0010] The filters include an all-pass band filter and N-1 specific band filters. The all-pass band filter is used to measure the ambient illumination, and the specific band filters can realize the detection of multiple bands by combining them.

[0011] The driving mechanism is connected to the filter wheel and is used to drive the filter wheel to rotate;

[0012] When the drone is flying in the air, the ambient illumination is measured first. Then the visible light filter wheel and the infrared filter wheel are rotated to switch to filters of different bands, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively. The similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively. M VI and M IR ;like M VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked at this time; the pixel coordinates of the camouflaged target are confirmed, and the visible light target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the target template is updated, and the infrared target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the tracker tracks the target and calculates the target position.

[0013] Preferably, the visible light camera and the infrared camera are exposed synchronously, and the target light beam passes through a filter on the visible light filter wheel and the infrared filter wheel respectively at each exposure time; the frame rate of the visible light camera and the infrared camera is 60~100fps.

[0014] Preferably, the number of through holes N on the filter wheel is ≥5.

[0015] The second object of the present invention is to provide a method for detecting and tracking a target using a multi-spectral fusion target detection system of an airborne photoelectric turret of a UAV, which specifically includes the following steps:

[0016] S1. Calibrate different types of camouflaged targets on the ground and calibrate the response characteristics of different camouflaged targets in different bands;

[0017] S2. When the UAV is flying in the air, the visible light filter wheel and the infrared filter wheel are adjusted to the full-pass band filter position, and the visible light camera and the infrared camera are adjusted to the maximum field of view to measure the ambient illumination;

[0018] S3. Rotate the visible light filter wheel and the infrared filter wheel to switch to filters of different bands, respectively, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively;

[0019] S4. Compare the similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve through spectral matching M VI and M IR ;like M VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked;

[0020] S5. After confirming the pixel coordinates of the camouflaged target, the visible light target spectrum data is placed in the memory of the airborne optoelectronic turret tracker; the target template is updated, and the infrared target spectrum data is placed in the memory of the airborne optoelectronic turret tracker;

[0021] S6. The tracker starts to execute the relevant tracking algorithm to track the target and calculate the target position.

[0022] Preferably, step S1 specifically includes the following sub-steps:

[0023] S101. Place the camouflage net indoors;

[0024] S102. With 1 lux as the step length, gradually adjust the indoor illuminance from 0 lux to m lux, and obtain m groups of different illuminances in the visible light band. The spectral response function of the camouflage net is obtained by combining N-1 specific visible light bands and the spectral response function to obtain m×(N-1) groups of visible light response curves;

[0025] S103. With 1 lux as the step length, gradually adjust the indoor illuminance from 0 lux to m lux, and obtain m groups of infrared bands under different illuminances. The spectral response function of the camouflage net is obtained by combining N-1 infrared specific bands and spectral response functions to obtain m×(N-1) groups of infrared response curves;

[0026] The visible light and infrared spectral response functions are as follows:

[0027]

[0028]

[0029] In the formula, Indicates illumination, is the spectral response value of the visible light sensor in the band λ, is the spectral response value of the infrared sensor in the band λ;

[0030] S104. Store the visible light response curve and the infrared response curve in the storage medium of the drone onboard computer.

[0031] Preferably, when the visible light filter wheel and the infrared filter wheel are rotating, the automatic brightness adjustment function of the visible light camera and the infrared camera is turned off;

[0032] The visible light filter wheel and the infrared filter wheel report the current filter positions in real time.

[0033] Preferably, the similarity between the visible light spectrum curve and the infrared spectrum curve and the laboratory measurement curve is M VI and M IR The expressions are as follows:

[0034]

[0035]

[0036] In the formula, Represents the visible spectrum curve of the target, represents the infrared spectrum curve of the target, and Slab(λ) represents the laboratory measurement curve;

[0037] The visible spectrum curve of the target and the infrared spectrum curve of the target are:

[0038]

[0039]

[0040] Where mean represents the average value of the spectral reflectance of the target area.

[0041] Preferably, the target position is calculated in step S6 by using a correlation matching method or an optical flow method.

[0042] Preferably, the camouflaged target includes four types, namely jungle, grassland, ocean or desert.

[0043] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0044] Expanding the reconnaissance range: The present invention can not only conduct vertical reconnaissance, but also oblique reconnaissance, and reconnaissance of any pitch angle area, greatly expanding the reconnaissance range;

[0045] Improve reconnaissance efficiency: By adopting synchronous high-speed multi-spectral rotor technology, the UAV optoelectronic turret has the ability of gaze and squint fusion reconnaissance, which significantly improves the efficiency of multi-spectral target detection and recognition;

[0046] Improve imaging frame rate: Use high frame rate visible light cameras and infrared cameras (e.g. 60fps to 100fps) to increase the imaging frame rate, making target tracking smoother and more accurate;

[0047] Improve tracking stability: Since the target has been spectrally filtered and calibrated in the visible light band and the mid-wave infrared band, the target characteristics of the tracker are not affected by environmental factors such as lighting and weather, which greatly improves the tracking stability;

[0048] Reduce the workload of operators: The present invention has a high degree of automation, which significantly reduces the workload of UAV operators and improves operating efficiency;

[0049] Enhanced versatility: The present invention is not only suitable for military reconnaissance, but also for border monitoring, search and rescue, mineral exploration and other civilian fields, and has a wide range of application prospects;

[0050] Improved intelligence level: The UAV optoelectronic payload system of the present invention can perform more diverse tasks and improves the intelligence level;

[0051] Improve anti-interference capabilities: The high-speed and intelligent information processing of UAV systems and the integration of sensor systems have increased the difficulty of anti-reconnaissance and interference, and improved the survivability and combat effectiveness of UAVs.

[0052] In summary, the present invention adopts the synchronous high-speed multi-spectral rotor technology to enable the UAV optoelectronic turret to have the gaze-squint fusion reconnaissance capability, significantly improving the efficiency of multi-spectral target detection and recognition. The application of this technology is not only limited to military reconnaissance, but also applicable to civilian fields such as border monitoring, search and rescue, and mineral exploration. The implementation of the present invention will promote the development of optoelectronic payload technology of UAVs, especially in terms of stealth performance, long flight time, multi-purpose and high intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention provides a flowchart of a multi-spectral fusion target detection and tracking method for an unmanned aerial vehicle airborne optoelectronic turret according to an embodiment of the present invention.

[0054] Figure 2 It is a schematic diagram of the structure of a visible light filter wheel and an infrared filter wheel.

[0055] Reference numerals:

[0056] 11. Filter wheel;

[0057] 12. Filter. DETAILED DESCRIPTION

[0058] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0060] The present invention provides a multi-spectral fusion target detection and tracking system for an unmanned aerial vehicle airborne optoelectronic turret, comprising: a visible light optical subsystem, an infrared optical subsystem, and a tracker;

[0061] The visible light optical subsystem includes a visible light camera and a visible light filter wheel;

[0062] The infrared optical subsystem includes an infrared camera and an infrared filter wheel;

[0063] Both the visible light filter wheel and the infrared filter wheel include a filter wheel, N filters and a driving mechanism; N through holes are evenly opened along the circumference of the filter wheel, and the N filters are correspondingly arranged at the N through holes, where N≥5;

[0064] Using an external trigger circuit, the visible light camera and the infrared camera are exposed synchronously, and the target light beam passes through a filter on the visible light filter wheel and the infrared filter wheel respectively during each exposure time;

[0065] The filters include an all-pass band filter and N-1 specific band filters. The all-pass band filter is used to measure the ambient illumination, and the specific band filters can realize the detection of multiple bands by combining them.

[0066] The driving mechanism is connected to the filter wheel and is used to drive the filter wheel to rotate; specifically, the driving mechanism is a stepping motor;

[0067] When the UAV is flying in the air, the visible light filter wheel and the infrared filter wheel are adjusted to the full-band filter position, and the visible light camera and the infrared camera are adjusted to the maximum field of view to measure the ambient illumination; then the visible light filter wheel and the infrared filter wheel are rotated to switch to filters of different bands, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively;

[0068] Through spectral matching technology, the similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively. MVI and M IR ;like M VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked at this time; the pixel coordinates of the camouflaged target are confirmed, and the visible light target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the target template is updated, and then the infrared target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret;

[0069] The tracker starts to execute the relevant tracking algorithm to track the target and calculate the target position.

[0070] Specifically, the visible light filter wheel and the infrared filter wheel report the current filter position in real time. When the visible light filter wheel and the infrared filter wheel are rotating, the automatic brightness adjustment function of the visible light camera and the infrared camera is turned off. The reason is as follows: during the rotation of the filter, the filter wheel with the filter installed will completely block the light path. If the automatic brightness adjustment is turned on, when the filter is rotated into place, the automatic dimming program will take a long time to adjust the brightness to the appropriate value; therefore, according to the trigger signal, the automatic brightness adjustment function of the visible light camera and the infrared camera is turned off during the rotation of the filter.

[0071] Specifically, I VI(λ) is the image data captured by visible light at wavelength λ, I IR(λ) is the image data captured by infrared at wavelength λ, then the visible spectrum curve and infrared spectrum curve of the target are:

[0072]

[0073]

[0074] In the formula, mean represents the average value of the spectral reflectance of the target area;

[0075] Through spectral matching technology, the relationship between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively; when the ambient illumination is When comparing the visible spectrum curves of the target And infrared spectrum curve The laboratory test curve S lab(λ) The similarity of is as follows:

[0076] The similarity of visible light spectrum is:

[0077]

[0078] The infrared spectrum similarity is:

[0079]

[0080] like M VI and M IR If both are greater than the threshold value, and the threshold value is 0.8, it is determined that the two spectra are similar and there may be a camouflage net in the area.

[0081] Specifically, after confirming the pixel coordinates (x, y) of the camouflaged target, the visible light target spectrum data I VI(λ) Put it into the memory of the tracker of the airborne optoelectronic turret, and the variable name is ; Update the target template and convert the infrared target spectrum data I IR(λ) Put it into the memory of the tracker of the airborne optoelectronic turret, and the variable name is .

[0082] Specifically, the target position is calculated by using correlation matching, optical flow method and other methods.

[0083] Figure 2 An optional visible light filter wheel and infrared filter wheel structure are shown, and the filters 12 are evenly arranged along the circumference of the filter wheel 11. According to actual needs, the number of filters 12 is not limited to Figure 2 Given the number, when the diameter of the filter wheel 11 is increased, the number of filters 12 can be further expanded.

[0084] The present invention also provides a method for detecting and tracking a target using a multi-spectral fusion target on an airborne photoelectric turret of a UAV, which uses the above system for tracking and specifically includes the following steps:

[0085] S1. Ground calibration: Calibrate different types of camouflaged targets on the ground, and calibrate the response characteristics of different camouflaged targets in different bands;

[0086] There are four types of camouflaged targets, namely jungle, grassland, ocean and desert. Taking the identification of jungle camouflaged targets as an example, the specific sub-steps include the following:

[0087] S101. Place the jungle camouflage net in the laboratory;

[0088] S102. Adjust the illumination in the laboratory , with 1 lux as the step length, gradually adjust from 0 lux to 5000 lux, and obtain 5000 sets of different illumination in the visible light band The spectral response function of the jungle camouflage net is obtained by combining N-1 specific visible light bands and spectral response functions to obtain 5000×(N-1) groups of visible light response curves;

[0089] S103. Adjust the illumination in the laboratory , with 1 lux as the step length, gradually adjust from 0 lux to 5000 lux, and obtain 5000 sets of infrared bands under different illuminations The spectral response function of the jungle camouflage net is obtained by combining N-1 infrared specific bands and spectral response functions to obtain 5000×(N-1) groups of infrared response curves;

[0090] The visible light and infrared spectral response functions are as follows:

[0091]

[0092]

[0093] In the formula, Indicates illumination, is the spectral response value of the visible light sensor in the band λ, is the spectral response value of the infrared sensor in the band λ;

[0094] S104. Store the visible light response curve and the infrared response curve in the storage medium of the drone onboard computer.

[0095] When the types of camouflage targets are grassland, ocean and desert, the calibration steps are the same.

[0096] S2. When the UAV is flying in the air, the visible light filter wheel and the infrared filter wheel are adjusted to the full-band filter position, and the visible light camera and the infrared camera are adjusted to the maximum field of view to measure the ambient illumination.

[0097] S3. Rotate the visible light filter wheel and the infrared filter wheel to switch to filters of different bands, respectively, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively;

[0098] When the visible light filter wheel and the infrared filter wheel are rotating, the automatic brightness adjustment function of the visible light camera and the infrared camera is turned off;

[0099] The visible light filter wheel and the infrared filter wheel report the current filter position in real time.

[0100] S4. Compare the similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve through spectral matching M VI and M IR ;likeM VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked;

[0101] Specifically, I VI(λ) is the image data captured by visible light at wavelength λ, I IR(λ) is the image data captured by infrared at wavelength λ, then the visible spectrum curve of the target and the infrared spectrum curve of the target are respectively:

[0102]

[0103]

[0104] In the formula, mean represents the average value of the spectral reflectance of the target area;

[0105] Through spectral matching technology, the relationship between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively; when the ambient illumination is When comparing the visible spectrum curves of the target And infrared spectrum curve The similarity with the laboratory measurement curve Slab(λ) is expressed as follows:

[0106] The similarity of visible light spectrum is:

[0107]

[0108] The infrared spectrum similarity is:

[0109]

[0110] In the formula, Represents the visible spectrum curve of the target, represents the infrared spectrum curve of the target, and Slab(λ) represents the laboratory measurement curve;

[0111] like M VI and M IR If both are greater than the threshold, the two spectra are judged to be similar and there may be a camouflage net in the area.

[0112] S5. After confirming the pixel coordinates of the camouflaged target, the visible light target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the target template is updated, and then the infrared target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret.

[0113] S6. The tracker starts to execute the relevant tracking algorithm to track the target and calculate the target position;

[0114] Specifically, the relevant tracking algorithm includes the following steps:

[0115] S601. Target area initialization: Determine the initial area of ​​the target through the target detection algorithm, and use the initial area of ​​the target as the starting point of the relevant tracking algorithm;

[0116] S602. Template matching and correlation calculation: Use the initial area of ​​the target as a template and search for the target by calculating the correlation between the template and each possible position in the image; the correlation is calculated by the following formula:

[0117] ;

[0118] in, R ( x , y ) is the current image position ( x , y ), T ( i , j ) is the pixel value of the template image, I ( x + i , y + j ) is the pixel value of the target image, μT and μI are the means of the template and target regions, respectively;

[0119] S603. Peak detection and target positioning: In the correlation graph, the new position of the target is determined by finding the local maximum; if the peak value is higher than the preset threshold, the target is considered to be successfully tracked at this position, and the target model is updated according to the tracking result.

[0120] Specifically, the target position is calculated by using correlation matching, optical flow method and other methods.

[0121] In this step, the tracker begins to track the target using the correlation tracking method. Since the target has been spectrally filtered and calibrated in the visible light band and the medium-wave infrared band, the target characteristics of the tracker are not affected by light, weather, etc., which greatly improves the stability of tracking.

[0122] Example 1

[0123] This embodiment provides a method for detecting and tracking a target with multi-spectral fusion on an airborne optoelectronic turret of a UAV, and uses a multi-spectral fusion onboard optoelectronic turret of a UAV for tracking; the system includes: a visible light optical subsystem, an infrared optical subsystem, and a tracker;

[0124] The visible light optical subsystem includes a visible light camera and a visible light filter wheel;

[0125] The infrared optical subsystem includes an infrared camera and an infrared filter wheel;

[0126] The visible light filter wheel and the infrared filter wheel both include a filter wheel, a filter and a driving mechanism; five through holes are evenly opened along the circumference of the filter wheel, and five filters are correspondingly arranged at the five through holes;

[0127] The filters include 1 all-pass band filter and 4 filters of specific bands;

[0128] The driving mechanism is connected to the filter wheel and is used to drive the filter wheel to rotate;

[0129] The wavelengths of the four specific bands of filters on the visible light filter wheel are as follows: Band 1 is 450nm~485nm; Band 2 is 485~495nm; Band 3 is 495~570nm; Band 4 is 570~750nm;

[0130] The wavelengths of the filters of the four specific wavelength bands on the infrared filter wheel are as follows: the first wavelength band is 3.8~4.0μm; the second wavelength band is 4.0~4.2μm; the third wavelength band is 4.2~4.4μm; the fourth wavelength band is 4.4~4.7μm;

[0131] Each camera system is equipped with four-band filters and one all-pass filter to cover all 16 band combinations and measure the ambient illumination in a timely manner; the filter wheel is controlled by a stepper motor to ensure accurate band switching.

[0132] Four types of camouflage targets are specified: jungle, grassland, ocean and desert:

[0133] When identifying targets camouflaged in jungles, a combination of visible light band 1 + infrared band 1 and a combination of visible light band 2 + infrared band 2 are used;

[0134] When identifying grassland camouflaged targets, a combination of visible light band 2 + infrared band 3 and a combination of visible light band 3 + infrared band 4 are used;

[0135] When identifying desert camouflaged targets, a combination of visible light band 3 + infrared band 1 and a combination of visible light band 4 + infrared band 2 are used;

[0136] When identifying camouflaged targets at sea, a combination of visible light band 4 + infrared band 3 and a combination of visible light band 1 + infrared band 4 are used.

[0137] Since only a combination of two bands is needed to identify camouflaged targets, taking the identification of jungle camouflaged targets as an example, if the original frame rate of the visible light camera is 60fps, then the visible light filter wheel only needs to be rotated to two positions to alternately provide 30fps visible light first band images and 30fps visible light second band images; if the original frame rate of the infrared camera is 60fps, then the infrared filter wheel only needs to be rotated to two positions to alternately provide 30fps infrared first band images and 30fps infrared second band images.

[0138] The method specifically comprises the following steps:

[0139] S1. Ground calibration: Calibrate different types of camouflaged targets on the ground, and calibrate the response characteristics of different camouflaged targets in different bands; taking the identification of jungle camouflaged targets as an example, it specifically includes the following sub-steps:

[0140] S101. Place the jungle camouflage net in the laboratory;

[0141] S102. Adjust the illumination in the laboratory , with 1 lux as the step length, gradually adjust from 0 lux to 5000 lux, and obtain 5000 sets of different illumination in the visible light band The spectral response function of the jungle camouflage net is obtained by combining N-1 specific visible light bands and spectral response functions to obtain 5000×(N-1) groups of visible light response curves;

[0142] S103. Adjust the illumination in the laboratory , with 1 lux as the step length, gradually adjust from 0 lux to 5000 lux, and obtain 5000 sets of infrared bands under different illuminations The spectral response function of the jungle camouflage net is obtained by combining N-1 infrared specific bands and spectral response functions to obtain 5000×(N-1) groups of infrared response curves;

[0143] The visible light and infrared spectral response functions are as follows:

[0144]

[0145]

[0146] In the formula, Indicates illumination, is the spectral response value of the visible light sensor in the band λ, is the spectral response value of the infrared sensor in the band λ;

[0147] S104. Store the visible light response curve and the infrared response curve in the storage medium of the drone onboard computer.

[0148] S2. When the UAV is flying in the air, the visible light filter wheel and the infrared filter wheel are adjusted to the full-band filter position, and the visible light camera and the infrared camera are adjusted to the maximum field of view to measure the ambient illumination.

[0149] S3. Rotate the visible light filter wheel and the infrared filter wheel to switch to filters of different bands respectively, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively; when the visible light filter wheel and the infrared filter wheel are rotating, turn off the automatic brightness adjustment function of the visible light camera and the infrared camera; the visible light filter wheel and the infrared filter wheel report the current filter position in real time.

[0150] S4. Compare the similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve through spectral matching M VI and M IR ;set up I VI(λ) is the image data captured by visible light at wavelength λ, I IR(λ) is the image data captured by infrared at wavelength λ, then the visible spectrum curve and infrared spectrum curve of the target are:

[0151]

[0152]

[0153] In the formula, mean represents the average value of the spectral reflectance of the target area;

[0154] Through spectral matching technology, the relationship between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively; when the ambient illumination is When comparing the visible spectrum curves of the target And infrared spectrum curve The similarity with the laboratory measurement curve Slab(λ) is expressed as follows:

[0155] The similarity of visible light spectrum is:

[0156]

[0157] The infrared spectrum similarity is:

[0158]

[0159] like M VI and M IR If both are greater than the threshold, the two spectra are judged to be similar and there may be a camouflage net in the area.

[0160] S5. After confirming the pixel coordinates of the camouflaged target, the visible light target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the target template is updated, and then the infrared target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret.

[0161] S6. The tracker starts to execute the relevant tracking algorithm to track the target and calculate the target position by using methods such as correlation matching and optical flow method. The relevant tracking algorithm specifically includes the following steps:

[0162] S601. Target area initialization: Determine the initial area of ​​the target through the target detection algorithm, and use the initial area of ​​the target as the starting point of the relevant tracking algorithm;

[0163] S602. Template matching and correlation calculation: Use the initial area of ​​the target as a template and search for the target by calculating the correlation between the template and each possible position in the image; the correlation is calculated by the following formula:

[0164] ;

[0165] in, R ( x , y ) is the current image position ( x , y ), T ( i , j ) is the pixel value of the template image, I ( x + i , y + j ) is the pixel value of the target image, μT and μI are the means of the template and target regions, respectively;

[0166] S603. Peak detection and target positioning: In the correlation graph, the new position of the target is determined by finding the local maximum; if the peak value is higher than the preset threshold, the target is considered to be successfully tracked at this position, and the target model is updated according to the tracking result.

[0167] The present invention aims to improve the detection and tracking capabilities of UAVs on camouflaged targets on the ground in the air, especially the detection and tracking of camouflaged targets in complex environments. The key technical points include the following aspects:

[0168] Use high frame rate visible light camera (such as 60fps to 100fps) and high frame rate infrared camera (such as 60fps to 100fps medium wave camera), add rotatable filters in their respective optical systems, a total of 4 bands, to achieve multispectral imaging. Use external trigger circuit to synchronously control the exposure of visible light camera and infrared camera, as well as the rotation of filter wheel in place, to ensure the synchronization and accuracy of imaging. Visible light and infrared filter wheels can feedback the current filter position, improving the response speed and flexibility of the system. Calibrate different types of camouflage targets on the ground, and establish a database of response characteristics of different targets in different bands to optimize the target recognition algorithm. When the UAV is flying in the air, the filters are automatically switched in turn, and the spectral fusion information is used to detect and confirm the target, and then the filter position is locked and the relevant tracking method is used for stable tracking, which improves the stability and accuracy of tracking.

[0169] Through these technical points, the present invention not only improves the drone's reconnaissance capability against camouflaged targets, but also reduces the operator's workload, which has important military and civilian value. In addition, the challenges facing optoelectronic payloads include the development of multi-band, multi-spectral, large array, high sensitivity, high frame rate, and the improvement of information processing capabilities to quickly analyze and process massive multi-dimensional data.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An unmanned aerial vehicle airborne optoelectronic turret multi-spectral fusion target detection and tracking system, characterized in that: include: Visible light optical subsystem, infrared optical subsystem, tracker; The visible light optical subsystem includes a visible light camera and a visible light filter wheel; The infrared optical subsystem includes an infrared camera and an infrared filter wheel; Both the visible light filter wheel and the infrared filter wheel include a filter wheel, a filter and a driving mechanism; N through holes are evenly provided along the circumference of the filter wheel, and the N filters are correspondingly arranged at the N through holes; The filters include an all-pass band filter and N-1 specific band filters. The all-pass band filter is used to measure the ambient illumination, and the specific band filters can realize the detection of multiple bands by combining them. The driving mechanism is connected to the filter wheel and is used to drive the filter wheel to rotate; When the drone is flying in the air, the ambient illumination is measured first. Then the visible light filter wheel and the infrared filter wheel are rotated to switch to filters of different bands, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively. The similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve is compared respectively. M VI and M IR ;like M VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked at this time; the pixel coordinates of the camouflaged target are confirmed, and the visible light target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the target template is updated, and the infrared target spectrum data is placed in the memory of the tracker of the airborne optoelectronic turret; the tracker tracks the target and calculates the target position.

2. The UAV airborne optoelectronic turret multi-spectral fusion target detection and tracking system according to claim 1 is characterized by: The visible light camera and the infrared camera are exposed synchronously, and the target light beam passes through a filter on the visible light filter wheel and the infrared filter wheel respectively during each exposure time; the frame rate of the visible light camera and the infrared camera is 60-100fps.

3. The UAV airborne optoelectronic turret multi-spectral fusion target detection and tracking system according to claim 1 is characterized by: The number of through holes N on the filter wheel is ≥5.

4. A method for detecting and tracking a target using a multi-spectral fusion target detection system of an unmanned aerial vehicle (UAV) mounted optoelectronic turret, wherein the method uses the multi-spectral fusion target detection and tracking system of an unmanned aerial vehicle (UAV) mounted optoelectronic turret according to any one of claims 1 to 3 for tracking, characterized in that: The specific steps include: S1. Calibrate different types of camouflaged targets on the ground and calibrate the response characteristics of different camouflaged targets in different bands; S2. When the UAV is flying in the air, the visible light filter wheel and the infrared filter wheel are adjusted to the full-band filter position, and the visible light camera and the infrared camera are adjusted to the maximum field of view to measure the ambient illumination; S3. Rotate the visible light filter wheel and the infrared filter wheel to switch to filters of different bands, respectively, and the visible light camera and the infrared camera capture image data in the corresponding bands respectively; S4. Compare the similarity between the target's visible light spectrum curve and infrared spectrum curve and the laboratory measurement curve through spectral matching M VI and M IR ;like M VI and M IR If both are greater than the threshold, it is determined that there may be a camouflage net in the area, and the visible light filter wheel and the infrared filter wheel are locked; S5. After confirming the pixel coordinates of the camouflaged target, the visible light target spectrum data is placed in the memory of the airborne optoelectronic turret tracker; the target template is updated, and the infrared target spectrum data is placed in the memory of the airborne optoelectronic turret tracker; S6. The tracker starts to execute the relevant tracking algorithm to track the target and calculate the target position.

5. The method for detecting and tracking a target using a multi-spectral fusion optoelectronic turret mounted on a UAV according to claim 4, characterized in that: The step S1 specifically includes the following sub-steps: S101. Place the camouflage net indoors; S102. With 1 lux as the step length, gradually adjust the indoor illuminance from 0 lux to m lux, and obtain m groups of different illuminances in the visible light band. The spectral response function of the camouflage net is obtained by combining N-1 specific visible light bands and the spectral response function to obtain m×(N-1) groups of visible light response curves; S103. With 1 lux as the step length, gradually adjust the indoor illuminance from 0 lux to m lux, and obtain m groups of infrared bands under different illuminances. The spectral response function of the camouflage net is obtained by combining N-1 infrared specific bands and spectral response functions to obtain m×(N-1) groups of infrared response curves; The visible light and infrared spectral response functions are as follows: In the formula, Indicates illumination, is the spectral response value of the visible light sensor in the band λ, is the spectral response value of the infrared sensor in the band λ; S104. Store the visible light response curve and the infrared response curve in the storage medium of the drone onboard computer.

6. The method for detecting and tracking a target using a multi-spectral fusion optoelectronic turret mounted on a UAV according to claim 4, characterized in that: When the visible light filter wheel and the infrared filter wheel are rotating, turning off the automatic brightness adjustment function of the visible light camera and the infrared camera; The visible light filter wheel and the infrared filter wheel report the current filter positions in real time.

7. The method for detecting and tracking a target using a multi-spectral fusion optoelectronic turret mounted on a UAV according to claim 4, characterized in that: Similarity between visible light spectrum curve and infrared spectrum curve and laboratory measurement curve M VI and M IR The expressions are as follows: In the formula, Represents the visible spectrum curve of the target, represents the infrared spectrum curve of the target, and Slab(λ) represents the laboratory measurement curve; The visible spectrum curve of the target and the infrared spectrum curve of the target are: Where mean represents the average value of the spectral reflectance of the target area.

8. The method for detecting and tracking a target using a multi-spectral fusion optoelectronic turret mounted on a UAV according to claim 4, characterized in that: The target position is calculated in step S6 by using a correlation matching method or an optical flow method.

9. The method for detecting and tracking a target using a multi-spectral fusion optoelectronic turret mounted on a UAV according to claim 4, characterized in that: The camouflaged targets include four types, namely jungle, grassland, ocean or desert.

Images