Image processing method for jointly detecting low, slow and small targets in the air by radar and photoelectric sensor

Abstract

The present application provides an image processing method for jointly monitoring low, slow and small targets by radar and photoelectric sensor, comprising: using a radar system to scan and detect a specified area, and inputting information of suspected low, slow and small targets into an image processing system; a photoelectric sensor system receiving information and obtaining images of suspected low, slow and small targets, and the image processing system performing matching between a set of information received from the radar system and the image data of the photoelectric sensor to generate a first 3D display image; the image processing system selecting a predetermined range of areas around the suspected low, slow and small targets, generating an image reprocessing area, and performing image enhancement calculation and image de-shake calculation on the image in the processing area to generate a second 3D display image; and the image processor outputting information of real low, slow and small targets and storing the information into a database.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of air target detection, and particularly relates to an image processing method for jointly detecting an air low, slow and small target by a radar and an optoelectronic sensor. BACKGROUND

[0002] With the explosive growth of unmanned aerial vehicles, a large number of low-altitude small slow aircrafts are brought by consumer-level small rotor unmanned aerial vehicles, which causes problems in supervision. Unmanned aerial vehicles have disturbed civil aviation airports, resulting in flight suspension accidents, and have brought major safety hazards to social life. At the same time, civil and military airports are long-term threatened by birds, and means for monitoring and investigating birds in the airport are needed. Therefore, the investigation, tracking and monitoring of low-altitude small slow (low, slow and small) targets are urgent needs at present.

[0003] The radar and optoelectronic sensor joint tracking system is an effective tool for detecting low, slow and small targets in the air, especially small unmanned aerial vehicles. The radar can efficiently monitor air targets in long distance, all-weather and all-time, but has the shortcomings of low imaging resolution and difficulty in imaging identification to determine the target properties. The optoelectronic tracking system has high imaging resolution and can perform 24-hour uninterrupted target imaging identification and tracking through a visible light system or an infrared optoelectronic system in high-visibility weather conditions. The optoelectronic tracking system also has high angle tracking accuracy. However, the distance accuracy of the optoelectronic tracking system is a major defect. If a 3D image function is used to display the field of view of the optoelectronic tracking system in real time, due to the large error of the distance data output by the optoelectronic device, the 3D map picture has serious image blur and distortion. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide an image processing method for jointly detecting an air low, slow and small target by a radar and an optoelectronic sensor, which uses a three-dimensional image to monitor the low, slow and small target in the air, and uses image enhancement technology and picture de-shake calculation to obtain a better low, slow and small target image.

[0005] In order to achieve the above purpose, the technical solution adopted by the present application is:

[0006] The image processing method for jointly detecting an air low, slow and small target by a radar and an optoelectronic sensor comprises the following steps:

[0007] Step 1: using a radar system to scan and detect a specified area, and inputting the flight data of a suspected low, slow and small target detected into an image processing system;

[0008] Step 2: The optoelectronic sensor system receives the flight data of the suspected low and slow small target outputted by the image processing system, the gimbal in the optoelectronic sensor system guides the optoelectronic sensor to point to the position of the flight data of the suspected low and slow small target by rotating, and acquires the image of the suspected low and slow small target, and inputs the image data into the image processing system.

[0009] Step 3: The image processing system performs matching on the image data of the optoelectronic sensor and the set of flight data information of the suspected low and slow small target received from the radar system, to generate a first 3D display image, and the image display device displays the first 3D display image.

[0010] Step 4:

[0011] If it is determined that the suspected low and slow small target in the first 3D display image is a real low and slow small target, go to step 8;

[0012] If the target definition in the first 3D display image does not reach the recognition threshold, perform step 5;

[0013] Step 5: The image processing system selects a predetermined range of surrounding area for the suspected low and slow small target, generates an image reprocessing area, and performs image enhancement calculation and image shake elimination calculation on the image reprocessing area; to generate a second 3D display image, and the image display device displays the second 3D display image;

[0014] Step 6:

[0015] If it is determined that the suspected low and slow small target in the second 3D display image is a real low and slow small target, go to step 8;

[0016] If the target definition does not reach the recognition threshold, return to step 5;

[0017] Step 7: Count the number of times that steps 5 and 6 are repeated in the loop, and if the number of times that steps 5 and 6 are repeated is greater than a predetermined number, go to step 8;

[0018] Step 8: The image processor outputs the image information and flight data information of the real low and slow small target, and stores the image information and flight data of the real low and slow small target after identification.

[0019] The flight data in step 1 includes azimuth, pitch and distance information.

[0020] The step 2 includes the following sub-steps:

[0021] Step 2.1, the photoelectric sensor system receives the azimuth, elevation and distance information of the suspected low, slow and small target output by the image processing system, guides the photoelectric sensor to the position of the azimuth, elevation and distance of the target, and obtains a spatial image of the position;

[0022] Step 2.2, the photoelectric sensor inputs the obtained spatial image to the image processing system, and converts the spatial image into digital image data by the image processing system.

[0023] The photoelectric sensor includes a visible light image sensor, a low-light image sensor, an ultraviolet image sensor and an infrared image sensor. The sensor is a camera, which is arranged on a gimbal and controlled by an automatic controller to realize image shooting in each direction at any time.

[0024] The camera can shoot images in each direction at any time, and a gimbal and an automatic controller are required.

[0025] The step 3 includes the following sub-steps:

[0026] Step 3.1, the image processing system performs matching on the set of azimuth, elevation and distance information of the suspected low, slow and small target received from the radar system and the digital image data of the photoelectric sensor;

[0027] Step 3.2, the image processing system stores the set of azimuth, elevation and distance information to a position data storage unit of a storage connected to the image processor, stores the digital image data to an image data storage unit of the storage connected to the image processor, and stores the matching result to an index data storage unit of the storage connected to the image processor;

[0028] Step 3.3, the image processing system generates a first 3D display image according to the received display instruction, extracts the azimuth, elevation and distance information and the matched digital image data from the storage.

[0029] The step 4 includes the following sub-steps:

[0030] Step 4.1, the first 3D display image displayed by the image display includes an image area containing the suspected low, slow and small target, a surrounding environment of the image area, azimuth, elevation and distance matched with the suspected low, slow and small target, and other flight data of the suspected low, slow and small target;

[0031] Step 4.2, the image recognition performed on the 3D display image includes determining an identification threshold for distinguishing the other flight data of the real low, slow and small target by using a deep learning algorithm.

[0032] Step 4.3, the other flight data of the suspected low slow small target is all greater than the identification threshold, it is determined to be a real low slow small target, the other flight data of the suspected low slow small target is less than the identification threshold, it is determined to be a non-low slow small target; the data part of the suspected low slow small target is greater than the identification threshold, it is determined that the target clarity does not reach the identification threshold.

[0033] The step 4.2 further comprises the following sub-steps:

[0034] Step 4.21, extracting the other flight data of the low slow small target, the other flight data comprising: the moving speed, moving direction, acceleration, angular velocity, angular acceleration of the target; the other flight data of the low slow small target is obtained from the radar system or calculated based on the flight data;

[0035] Step 4.22, dividing the low slow small targets into two groups according to a predetermined proportion, wherein the other flight data of one group of low slow small targets is used as a training group, and the other group of low slow small targets is used as a control group to perform deep learning training;

[0036] Step 4.23, obtaining the optimal other flight data value through deep learning training as the identification threshold for determining the other flight data of the real low slow small target.

[0037] When the flight data accumulated in the memory connected to the image processing system exceeds a predetermined value, step 4.22 is repeatedly executed.

[0038] The image enhancement calculation in step 5 comprises performing a binary operation on the pixels of the image; the image shake elimination calculation comprises using a digital filtering operation.

[0039] The step 6 comprises the following sub-steps:

[0040] Step 6.1, when the target clarity of the suspected low slow small target in the second 3D display image does not reach the identification threshold, the second 3D display image is renamed as the first 3D display image;

[0041] Step 6.2, the renamed first 3D display image is transferred to step 5.

[0042] The beneficial effects of the present application are:

[0043] The present application combines the detection signals of the radar system and the photoelectric sensor system to identify low slow small drones, combines the advantages of the radar system and the photoelectric sensor system, uses image enhancement and image shake elimination technology to process the images of the suspected low slow small target area, and improves the accuracy of image recognition. The method has good adaptability and high recognition probability. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a general flow chart of the image processing method of the present invention for jointly monitoring low, slow and small targets by radar and photoelectric sensor;

[0045] Figure 2 is a system block diagram of the present invention for jointly monitoring low, slow and small targets by radar and photoelectric sensor. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings.

[0047] ATTACHMENT Figure 2 is a system block diagram of the present invention for jointly monitoring low, slow and small targets by radar and photoelectric sensor. The system can be composed of existing radar and photoelectric sensor systems in network, or can be a special device composed of radar and photoelectric sensor. The device includes radar system, photoelectric sensor system, display device and image processing system, and can display three-dimensional graphic image on the display device. The radar system and photoelectric sensor system are arranged on independent rotating platforms, which can rotate 360 degrees omnidirectionally and have pitching motion capability.

[0048] EMBODIMENT

[0049] REFERENCE Figure 1 is a flow chart of the image processing method of the present invention for jointly monitoring low, slow and small targets by radar and photoelectric sensor. The present invention is mainly used for monitoring low, slow and small flight targets such as civilian unmanned aerial vehicles in large and super large cities. The monitoring and response of such civilian equipment are different from military targets, and more importantly, through monitoring, legal evidence of illegal flight is obtained for effective and strict implementation of urban space management. This management mode is different from the high interception rate and countermeasure required for monitoring military low, slow and small flight targets.

[0050] The present invention proposes an image processing system and method for jointly monitoring low, slow and small targets by radar and photoelectric sensor, which includes radar system, photoelectric sensor system, display device and image processing system, and the image processing system can provide three-dimensional graphic image on the display device. Secondly, the system is preferably installed on a motor vehicle, and the power supply for the radar system, photoelectric sensor system, display device and image processing system is configured on the motor vehicle. Such a vehicle can be moved to the direction where low, slow and small flight targets need to be monitored at any time as needed. The image processing method includes the following steps:

[0051] Step 1, using the radar system to scan and detect the specified area, and inputting the flight data of the suspected low, slow and small target into the image processing system, the flight data including azimuth, pitch and distance information,

[0052] Step 2, the photoelectric sensor system receives the azimuth, elevation and distance information of the suspected low, slow and small target output by the image processing system, the photoelectric sensor system guides the photoelectric sensor to the position of the above-mentioned azimuth, elevation and distance, acquires the image of the suspected low, slow and small target, and inputs the image data into the image processing system;

[0053] The radar system has a large detection area for monitoring flight targets and a fast reaction speed. First, the radar system is used to detect a predetermined airspace with high efficiency. Only when the radar system discovers a suspected low, slow and small target is the photoelectric sensor system enabled. This can reduce the consumption of working hours and improve the efficiency of accurately acquiring images of low, slow and small targets. The present application uses a method of first using a radar system to scan the area of a flight target to be detected, and then calling a photoelectric sensor system to directly acquire an image of a suspected low, slow and small target when a suspected low, slow and small target is detected.

[0054] Step 2 includes: the photoelectric sensor system receives the azimuth, elevation and distance information of the suspected low, slow and small target output by the image processing system, guides the photoelectric sensor to the position of the above-mentioned azimuth, elevation and distance, and acquires the spatial image of the position; the acquired spatial image is input into the image processing system, and is converted into digital image data by the image processing system.

[0055] Step 3, the image processing system performs matching on the set of azimuth, elevation and distance information of the suspected low, slow and small target received from the radar system and the image data of the photoelectric sensor to generate a first 3D display image, and the image display device displays the first 3D display image; the display image can be observed by the administrator to find the region of the suspected low, slow and small target and the topographic map of the region. The radar system, the photoelectric sensor system, the display device and the image processing system can display a three-dimensional graphic image on the display device.

[0056] Step 3 includes a sub-step:

[0057] Step 3.1, the image processing system performs matching on the set of azimuth, elevation and distance information of the suspected low, slow and small target received from the radar system and the digital image data of the photoelectric sensor;

[0058] Step 3.2, the image processing system stores the set of azimuth, elevation and distance information to a position data storage unit of a storage connected to the image processor; the image processing system stores the digital image data to an image data storage unit of a storage connected to the image processor; and the image processing system stores the matching result to an index data storage unit of a storage connected to the image processor;

[0059] Step 3.3, the image processing system generates a first 3D display image from the azimuth, elevation and distance information and the matched digital image data extracted from the memory according to the received display instruction.

[0060] Step 4, if it is determined that the suspected low slow small target in the first 3D display image is a real low slow small target, go to step 8; if the target definition in the first 3D display image does not reach the recognition threshold, perform step 5;

[0061] The step 4 includes the following sub-steps:

[0062] Step 4.1, the first 3D display image displayed by the image display includes: an image area containing the suspected low slow small target obtained by the image sensor, the surrounding environment of the image area, the azimuth, elevation and distance matched with the suspected low slow small target, and other flight data of the suspected low slow small target;

[0063] Step 4.2, the image recognition performed on the 3D display image includes: using a deep learning algorithm to determine the recognition threshold of the other flight data of the real low slow small target;

[0064] Step 4.3, if all the other flight data of the suspected low slow small target are greater than the recognition threshold, it is determined as a real low slow small target; if the other flight data of the suspected low slow small target are less than the recognition threshold, it is determined as a non-low slow small target; if part of the other flight data of the suspected low slow small target are greater than the recognition threshold, it is determined that the target definition does not reach the recognition threshold.

[0065] The other flight data of the low slow small target includes: the moving speed, moving direction, acceleration, angular speed and angular acceleration of the target; the other flight data of the suspected low slow small target is obtained from the radar system or calculated based on the flight data. These flight data can be directly measured by the radar system, for example: the moving speed and moving direction of the target. It can also be calculated by the image processing system according to the basic flight data output by the radar system, for example: other data such as acceleration, angular speed and angular acceleration are calculated from the moving speed and moving direction of the target.

[0066] The process of using a deep learning algorithm to determine the recognition threshold of the other flight data of the real low slow small target in step 4.2 includes the following sub-steps:

[0067] Step 4.21, extract the other flight data of the low slow small target, which includes: the moving speed, moving direction, acceleration, angular speed and angular acceleration of the target; the other flight data of the low slow small target is obtained from the radar system or calculated based on the flight data;

[0068] Step 4.22, the low slow small target is divided into two groups according to a predetermined ratio, one group of low slow small targets is used as a training group, and the other group of low slow small targets is used as a control group to perform deep learning training;

[0069] Step 4.23, the optimal other flight data value is obtained by deep learning training as a recognition threshold for determining the real low slow small target.

[0070] The image processing method provided by the application can be repeatedly executed when the accumulated flight data in the memory connected to the image processing system exceeds a predetermined value. The deep learning training method is repeatedly used to improve the accuracy of the recognition threshold.

[0071] Step 5, the image processing system selects a predetermined range of regions around the suspected low slow small target to generate an image reprocessing region, and performs image enhancement calculation and image deblurring calculation on the image reprocessing region; to generate a second 3D display image, and display the second 3D display image by the image display device;

[0072] The image enhancement calculation includes performing a binary operation on the pixels of the image;

[0073] The image deblurring calculation includes using a digital filtering operation, such as Kalman filter, the process is: setting the target state transition equation and the target state measurement equation; obtaining the angle and distance information set of the suspected low slow small target from the radar and the photoelectric device, and sending the information set to the image processing system; the image processing system extracts the distance measurement information of the target from the k previous information sets, and initializes the target state value and the state covariance based on the measurement information; combining the target state transition equation and the target state measurement equation, calculating the target state prediction value and the state prediction covariance based on the initial state value; obtaining the initial state covariance matrix, and calculating the state prediction covariance matrix: obtaining the measurement prediction value to calculate the measurement innovation, calculating the gain based on the state prediction covariance matrix and the measurement prediction covariance matrix, and using the gain to calculate the state filtering value and the covariance filtering value;

[0074] Step 6, if the suspected low slow small target of the second 3D display image is determined to be a real low slow small target, go to step 8; if the target definition does not reach the recognition threshold, return to step 5; the step 6 includes the following substeps:

[0075] Step 6.1, if the target definition of the suspected low slow small target of the second 3D display image does not reach the recognition threshold, the second 3D display image is renamed as the first 3D display image;

[0076] Step 6.2, the renamed first 3D display image is transferred to step 5.

[0077] Step 7, the loop is executed for a count of the number of times step 5 and step 6 are repeated, and if the number of times step 5 to step 6 are repeated is greater than a predetermined number, then step 8 is executed;

[0078] Step 8, the image processor outputs the image information and azimuth, elevation and range information of the real low, slow and small target, and stores the image information and flight data of the real low, slow and small target after being marked.

[0079] Finally, it should be noted that the above embodiments are merely intended to illustrate the technical solutions of the embodiments of the present application and not to limit the same, and although the embodiments of the present application have been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that the technical solutions of the embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method for jointly detecting low, slow and small targets in the air by radar and photoelectric sensors, characterized in that, The method comprises the following steps: Step 1: scanning a designated area using a radar system, and inputting flight data of a suspected low, slow and small target detected by the radar system into an image processing system; Step 2: receiving the flight data of the suspected low, slow and small target output by the image processing system, and guiding a pan-tilt head in the photoelectric sensor system to point the photoelectric sensor to a position of the flight data by rotating, so as to obtain an image of the suspected low, slow and small target and input the image into the image processing system; Step 3: performing matching between a set of the flight data of the suspected low, slow and small target received by the radar system and the image of the photoelectric sensor by the image processing system, so as to generate a first 3D display image, and displaying the first 3D display image by an image display device; Step 4: if it is determined that the suspected low, slow and small target in the first 3D display image is a real low, slow and small target, proceeding to Step 8; if the resolution of the target in the first 3D display image does not reach a recognition threshold, proceeding to Step 5; Step 5: selecting a predetermined range of areas around the suspected low, slow and small target by the image processing system, generating an image reprocessing area, and performing image enhancement calculation and image shake elimination calculation on the image reprocessing area by the image processing system, so as to generate a second 3D display image, and displaying the second 3D display image by the image display device; Step 6: if it is determined that the suspected low, slow and small target in the second 3D display image is a real low, slow and small target, proceeding to Step 8; if the resolution of the target does not reach the recognition threshold, returning to Step 5; Step 7: counting the number of times of repeating Steps 5 and 6, and proceeding to Step 8 if the number of times of repeating Steps 5 and 6 is greater than a predetermined number; Step 8: outputting image information and flight data information of the real low, slow and small target by the image processing system, and storing the image information and the flight data of the real low, slow and small target after marking the image information and the flight data.

2. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 1, characterized in that, The flight data in Step 1 comprises azimuth, pitch and distance information.

3. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 2, characterized in that, Step 2 comprises the following sub-steps: Step 2.1: receiving the azimuth, pitch and distance information of the suspected low, slow and small target output by the image processing system by the photoelectric sensor system, guiding the photoelectric sensor to point to a position of the azimuth, pitch and distance information, and obtaining a spatial image of the position; Step 2.2: inputting the spatial image obtained by the photoelectric sensor into the image processing system, and converting the spatial image into digital image data by the image processing system.

4. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 3, characterized in that, The photoelectric sensor comprises a visible light image sensor, a low-light image sensor, an ultraviolet image sensor and an infrared image sensor.

5. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 4, characterized in that, Step 3 comprises the following sub-steps: Step 3.1: performing matching between a set of the azimuth, pitch and distance information of the suspected low, slow and small target received by the radar system and the digital image data of the photoelectric sensor by the image processing system; Step 3.2: storing the set of the azimuth, pitch and distance information into a position data storage unit of a storage connected to the image processing system by the image processing system; Step 3.3: storing the digital image data into an image data storage unit of the storage connected to the image processing system by the image processing system. The image processing system stores the matching result to an index data storage unit of a memory connected to the image processor; Step 3.3, the image processing system extracts the azimuth, elevation and distance information from the memory and generates a first 3D display image according to the received display instruction and the matched digital image data.

6. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 2, characterized in that, The step 4 includes the following sub-steps: Step 4.1, the first 3D display image displayed by the image display includes: the image area containing the suspected low slow small target obtained by the image sensor, the surrounding environment of the image area, the azimuth, elevation and distance matched with the suspected low slow small target, and other flight data of the suspected low slow small target; Step 4.2, performing image recognition on the 3D display image includes: using a deep learning algorithm to determine an identification threshold for distinguishing the other flight data of the real low slow small target; Step 4.3, if all the other flight data of the suspected low slow small target are greater than the identification threshold, it is determined as a real low slow small target, if the other flight data of the suspected low slow small target are less than the identification threshold, it is determined as a non-low slow small target; if part of the other flight data of the suspected low slow small target are greater than the identification threshold, it is determined that the target clarity does not reach the identification threshold.

7. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 6, characterized in that, The step 4.2 further includes the following sub-steps: Step 4.21, extracting the other flight data of the low slow small target, the other flight data including: the moving speed, moving direction, acceleration, angular velocity and angular acceleration of the target; the other flight data of the low slow small target is obtained from a radar system or calculated based on flight data; Step 4.22, dividing the low slow small targets into two groups according to a predetermined proportion, wherein the other flight data of one group of low slow small targets is used as a training group, and the other flight data of another group of low slow small targets is used as a control group to perform deep learning training; Step 4.23, obtaining the optimal other flight data value through deep learning training as the identification threshold for distinguishing the other flight data of the real low slow small target.

8. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 7, characterized in that, When the cumulative flight data in the memory connected to the image processing system exceeds a predetermined value, step 4.22 is repeatedly performed.

9. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 1, characterized in that, The image enhancement calculation in step 5 includes performing a binary operation on the pixels of the image; and the image shake calculation includes using a digital filtering operation.

10. The image processing method for detecting low, slow and small targets in the air by radar and photoelectric sensor combination according to claim 1, characterized in that, The step 6 includes the following sub-steps: Step 6.1, if the target clarity of the suspected low slow small target in the second 3D display image does not reach the identification threshold, the second 3D display image is renamed as the first 3D display image; Step 6.2, the renamed first 3D display image is transferred to step 5.

Images