A low-illumination sensitive unit-based low-light night vision target recognition device

Abstract

The application discloses a low-illumination sensitive unit-based low-light night vision target recognition device, and relates to the field of unmanned planes.The device comprises a low-illumination sensitive unit, an embedded computing module, a flight control module and a carbon fiber plate.The low-illumination sensitive unit, the embedded computing module and the flight control module are connected to the carbon fiber plate according to the design position.The low-illumination sensitive unit, the embedded computing module and the flight control module are installed in the head cabin to the cabin inside the fixed-wing unmanned plane body.The low-illumination sensitive unit and the embedded computing module are electrically connected, and the low-illumination sensitive unit and the embedded computing module are electrically connected with the flight control module.The low-illumination sensitive unit can be used at night and under normal illumination, different weights are adopted for identification according to different time, the target recognition probability is improved, the line-of-sight angle information of the target is sent to the flight control module, and the unmanned plane is realized to fly with the target.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a low-light night vision target recognition device based on a low-light sensitive unit. Background Technology

[0002] With the development of intelligent drone technology, drones, which are characterized by low cost, simple structure, and remote control capability, are widely used in fields such as aerial photography, disaster reconnaissance, pesticide spraying, and traffic monitoring.

[0003] In complex scenarios such as disaster reconnaissance and traffic inspection at night, drones need to have a certain ability to identify targets at night. Existing drones can be operated by personnel for target identification or achieve autonomous guidance using a positioning system combined with visual sensors during the day. However, under starlight conditions at night, the failure of visual sensors often causes existing drones to be unable to locate and identify targets correctly, or even to take off normally. In other words, existing drones do not have the ability to perform complex tasks at night.

[0004] Therefore, those skilled in the art need to address the lack of nighttime positioning and target recognition capabilities in existing drone technologies. The universal low-light night vision target recognition device for drones based on a low-light sensitive unit proposed in this invention has enormous development potential and a wide range of application scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a low-light night vision target recognition device based on a low-light sensitive unit, which solves the problem that existing drones do not have the ability to locate and recognize targets at night, making it difficult to perform complex tasks such as disaster reconnaissance and traffic monitoring at night.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] This invention discloses a low-light night vision target recognition device based on a low-light sensitive unit, comprising a low-light sensitive unit, an embedded computing module, a flight control module, and a carbon fiber plate. The low-light sensitive unit, the embedded computing module, and the flight control module are connected to the carbon fiber plate according to their designed positions. The low-light sensitive unit, the embedded computing module, and the flight control module are respectively installed in the head compartment to the fuselage of a fixed-wing UAV. The embedded computing module is connected to the carbon fiber plate via copper pillars and is located above the low-light sensitive unit. The pitot tube is mounted on a bracket, and the bracket is fixed between the embedded computing module and the carbon fiber plate by four copper pillars. The low-light sensitive unit and the embedded computing module are electrically connected, and both the low-light sensitive unit and the embedded computing module are electrically connected to the flight control module.

[0008] Preferably, the low-light sensitive unit and the embedded computing module are connected via a USB 3.0 interface, the embedded computing module is connected to the flight control module via a UART interface, and the low-light sensitive unit and the flight control module are connected via a trigger interface.

[0009] Preferably, the low-light sensitive unit is a night vision image information acquisition device, including a low-light night vision camera and a camera lens. The camera lens is connected to the low-light night vision camera through an optical interface and is responsible for acquiring image information at night or under normal lighting conditions.

[0010] Preferably, the embedded computing module is an algorithm running device, which is responsible for driving the low-light sensitive unit to acquire images, aligning the satellite time obtained by the flight control module with the image acquisition sequence number, running the electronic stabilization algorithm and the target recognition algorithm, calculating the target line-of-sight angle information and sending it to the flight control module.

[0011] Preferably, the flight control module is an autopilot device for unmanned aerial vehicles (UAVs), responsible for acquiring satellite time and aircraft attitude data, controlling the UAV to fly automatically, and receiving the target line-of-sight angle and accompanying target flight sent by the embedded computing module. The flight control module includes a data processing main chip, an inertial measurement unit (IMU), and a barometer. Both the IMU and the barometer are external sensors, and both are connected to the data processing chip via a serial port.

[0012] Preferably, the calculation formula for the electronic stability enhancement algorithm assumes the aircraft pitch angle is... The roll angle is yaw angle is Then the rotation matrix for:

[0013]

[0014] Let the camera intrinsic parameter matrix be K, then the homography matrix H is:

[0015] .

[0016] Preferably, the target recognition algorithm has built-in target recognition weights for normal lighting conditions and target recognition weights for low-light night vision conditions, and switches between the two different weights according to whether the working scene is a low-light condition. The target recognition weights for low-light night vision conditions are obtained by training the low-light sensitive unit on a self-collected dataset.

[0017] Preferably, the formula for calculating the target viewing angle is as follows: Let the coordinates of the target center in the image be f, the camera focal length be f, and the pixel size be u. Then the horizontal azimuth angle α and the elevation angle β are respectively:

[0018] .

[0019] Preferably, the low-light sensitive unit and the flight control module are located on the same horizontal plane.

[0020] A method for operating a low-light night vision target recognition device based on a low-light sensitive unit as described above includes the following steps:

[0021] S1: A thread is created in the embedded computing module to drive the low-light sensitive unit to acquire images, and at the same time the embedded computing module sends a trigger command to the flight control module;

[0022] S2: After receiving the trigger command, the flight control module reads the satellite time and the aircraft attitude data and sends them to the embedded computing module. At the same time, the flight control module sends a trigger signal to the low-light sensitive unit through the trigger line. After receiving the trigger command, the low-light sensitive unit performs image acquisition and stores the image acquisition into the memory of the embedded computing module through the USB 3.0 line. The embedded computing module stores the acquired image and the received satellite time and aircraft attitude data into a structure to complete time alignment.

[0023] S3: After another thread of the embedded computing module detects that the image acquisition is successful, it starts the electronic stabilization and target recognition thread, and determines whether the working scene is a low-light condition based on the image grayscale and system time. Different weight files are selected, and the target recognition software outputs the position of the target in the image, the target category information, and the recognition confidence.

[0024] S4: When the target recognition confidence level is greater than a specified threshold, the line-of-sight angle of the target's position in the image is calculated and sent to the flight control module via a serial port. The flight control module controls the UAV to fly toward the target based on the line-of-sight angle input.

[0025] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0026] This invention discloses a low-light night vision target recognition device based on a low-light sensitive unit. The device can be used both at night and under normal lighting conditions. It not only has target indication functionality but also target information classification functionality. It employs different weights for recognition based on different times to improve the target recognition probability. Furthermore, it can send the target's line-of-sight angle information to the flight control module, enabling UAV-assisted flight. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings.

[0028] Figure 1 This is a perspective view of a low-light night vision target recognition device based on a low-light sensitive unit according to the present invention.

[0029] Figure 2 This is a front view of a low-light night vision target recognition device based on a low-light sensitive unit according to the present invention;

[0030] Figure 3 This is a top view of a low-light night vision target recognition device based on a low-light sensitive unit according to the present invention;

[0031] Figure 4 This is a side view of a low-light night vision target recognition device based on a low-light sensitive unit according to the present invention.

[0032] Explanation of reference numerals in the attached diagram: 1. Low-light sensitive unit; 2. Embedded computing module; 3. Flight control module; 4. Carbon fiber plate; 5. USB 3.0 interface; 6. UART interface; 7. Trigger interface; 8. Pitot tube; 9. Copper column; 10. Bracket. Detailed Implementation

[0033] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] like Figure 1-4 As shown, a low-light night vision target recognition device based on a low-light sensitive unit includes a low-light sensitive unit 1, an embedded computing module 2, a flight control module 3, and a carbon fiber plate 4. The low-light sensitive unit 1, the embedded computing module 2, and the flight control module 3 are connected to the carbon fiber plate 4 according to their designed positions. The low-light sensitive unit 1, the embedded computing module 2, and the flight control module 3 are installed in the head compartment to the cabin of a fixed-wing UAV. The embedded computing module 2 is connected to the carbon fiber plate 4 via copper pillars 9 and is located above the low-light sensitive unit 1. The pitot tube 8 is mounted on a bracket 10, and the bracket 10 is fixed between the embedded computing module 2 and the carbon fiber plate 4 by four copper pillars 9. The low-light sensitive unit 1 and the embedded computing module 2 are electrically connected, and both the low-light sensitive unit 1 and the embedded computing module 2 are electrically connected to the flight control module 3.

[0035] The low-light sensitive unit 1 and the embedded computing module 2 are connected via a USB 3.0 interface 5. The embedded computing module 2 is connected to the flight control module 3 via a UART interface 6. The low-light sensitive unit 1 and the flight control module 3 are connected via a trigger interface 7.

[0036] The low-light sensitive unit 1 is a night vision image information acquisition device, including a low-light night vision camera and a camera lens. The camera lens is connected to the low-light night vision camera through an optical interface and is responsible for acquiring image information at night or under normal lighting conditions.

[0037] Specifically, the sensor of the low-light night vision camera is model GSENSE2020, which features low noise, high quantum efficiency, and large pixel size.

[0038] Specifically, the working process of the low-light night vision camera is as follows: the low-light sensitive unit 1 receives the working trigger signal sent by the flight control module 3 through the trigger interface 7 and starts the image acquisition work. The camera lens receives the light reflected by the external object. The light is refracted by the lens and converged onto the low-light sensor of the low-light night vision camera. The low-light sensor converts the light signal into an electrical signal. The image is processed by the internal image processing circuit of the camera. Finally, the image is stored in the memory of the embedded computing module 2 through the USB 3.0 interface 5, completing one image information acquisition.

[0039] The embedded computing module 2 is an algorithm running device. It is responsible for driving the low-light sensitive unit 1 to acquire images, aligning the satellite time obtained by the flight control module 3 with the image acquisition sequence number, running the electronic stabilization algorithm and the target recognition algorithm, calculating the target line-of-sight angle information and sending it to the flight control module 3.

[0040] Specifically, the embedded computing module has an NVIDIA Carmel ARM v8.2 6-core 64-bit CPU and a 384-core NVIDIA Volta GPU. The embedded computing module has USB 3.0 and UART interfaces for image information processing algorithm operation and data transmission.

[0041] The flight control module 3 is an autopilot device for the UAV, responsible for acquiring satellite time and aircraft attitude data, controlling the UAV to fly automatically, and receiving the target line-of-sight angle and accompanying target flight sent by the embedded computing module. The flight control module 3 includes a data processing main chip, an inertial measurement unit (IMU), and a barometer. Both the IMU and the barometer are external sensors and are connected to the data processing chip via a serial port. The IMU and the barometer send the collected information to the data processing chip for processing, which is used for automatic flight, automatic take-off and landing functions according to the flight path.

[0042] The electronic stabilization algorithm is an image stabilization method that combines the aircraft attitude information measured by the inertial measurement unit of the flight control module 3. When the UAV is flying in the air, it is often disturbed by the airflow, which causes the aircraft to shake and vibrate. This directly leads to jitter between the image sequences collected by the low-light sensitive unit 1, affecting the target recognition function. In order to reduce the computational complexity of the electronic stabilization algorithm, the embedded computing module 2 uses the aircraft attitude data obtained from the inertial measurement unit of the flight control module 3 to perform motion estimation, filter compensation, and then obtains the compensated image through affine transformation.

[0043] The calculation formula for the electronic stability enhancement algorithm assumes the aircraft pitch angle is... The roll angle is yaw angle is Then the rotation matrix for:

[0044]

[0045] Since drones operate at long distances, the inter-frame variations caused by translation are negligible. Therefore, the inter-frame variations in drone aerial photography of this specific scene can be considered to be primarily caused by rotation. If the camera intrinsic matrix is ​​K, then the homography matrix H is:

[0046] .

[0047] The target recognition algorithm has built-in target recognition weights for normal lighting conditions and target recognition weights for low-light night vision conditions, and switches between the two different weights depending on whether the working scene is a low-light condition. The target recognition weights for low-light night vision conditions are obtained by training the low-light sensitive unit 1 through self-collected dataset.

[0048] Specifically, the target recognition algorithm uses two methods to determine whether the working scene is under low-light conditions: Method 1 compares the average gray value of the image collected by the low-light sensitive unit 1 with the average gray value of the template image. When the average gray value of the image collected by the low-light sensitive unit 1 is lower than the average gray value of the template image, the low-light night vision condition target recognition weight is used. Method 2 reads the satellite time information corresponding to the time-aligned image and converts the satellite time to Beijing time. If the time is displayed as nighttime, the low-light night vision condition target recognition weight is used.

[0049] The formula for calculating the target line-of-sight angle is as follows: Let the coordinates of the target center in the image be x, y, f be the camera focal length, and u be the pixel size. Then the horizontal azimuth angle α and the elevation angle β are respectively:

[0050] .

[0051] The low-light sensitive unit 1 and the flight control module 3 are located on the same horizontal plane, which makes it convenient for the aircraft attitude data collected by the flight control module 3 to be directly used for electronic stabilization.

[0052] A method for operating a low-light night vision target recognition device based on a low-light sensitive unit as described above includes the following steps:

[0053] S1: A thread is created in the embedded computing module 2 to drive the low-light sensitive unit 1 to acquire images, and at the same time the embedded computing module 2 sends a trigger command to the flight control module 3;

[0054] S2: After receiving the trigger command, the flight control module 3 reads the satellite time and the aircraft attitude data and sends them to the embedded computing module 2. At the same time, the flight control module 3 sends a trigger signal to the low-light sensitive unit 1 through the trigger line. After receiving the trigger command, the low-light sensitive unit 1 performs image acquisition and stores the image acquisition into the memory of the embedded computing module 2 through the USB 3.0 line. The embedded computing module 2 stores the acquired image and the received satellite time and aircraft attitude data into a structure to complete time alignment.

[0055] S3: After another thread of the embedded computing module 2 detects that the image acquisition is successful, it starts the electronic stabilization and target recognition thread, and judges whether the working scene is a low-light condition based on the image grayscale and system time. Different weight files are selected, and the target recognition software outputs the position of the target in the image, the target category information, and the recognition confidence.

[0056] S4: When the target recognition confidence level is greater than a specified threshold, the line-of-sight angle of the target's position in the image is calculated and sent to the flight control module 3 via a serial port. The flight control module 3 then controls the UAV to fly towards the target based on the line-of-sight angle input. Specifically, the specified threshold is a target recognition confidence level greater than 70%.

[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0058] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A low-light night vision target recognition device based on a low-light sensitive unit, characterized in that: The device includes a low-light sensitive unit (1), an embedded computing module (2), a flight control module (3), and a carbon fiber plate (4). The low-light sensitive unit (1), the embedded computing module (2), and the flight control module (3) are connected to the carbon fiber plate (4) according to their designed positions. The low-light sensitive unit (1), the embedded computing module (2), and the flight control module (3) are installed in the head compartment to the cabin of the fixed-wing UAV. The embedded computing module (2) is connected to the carbon fiber plate (4) through copper pillars (9) and is located above the low-light sensitive unit (1). The pitot tube (8) is installed on a bracket (10). The bracket (10) is fixed between the embedded computing module (2) and the carbon fiber plate (4) through four copper pillars (9). The low-light sensitive unit (1) and the embedded computing module (2) are electrically connected. The low-light sensitive unit (1) and the embedded computing module (2) are both electrically connected to the flight control module (3). The low-light sensitive unit (1) is a night vision image information acquisition device, including a low-light night vision camera and a camera lens. The camera lens is connected to the low-light night vision camera through an optical interface and is responsible for acquiring image information at night or under normal light conditions. The embedded computing module (2) is an algorithm running device, responsible for driving the low-light sensitive unit (1) to acquire images, aligning the satellite time obtained by the flight control module (3) with the image acquisition sequence number, running the electronic stabilization algorithm and the target recognition algorithm, calculating the target line-of-sight angle information and sending it to the flight control module (3). Among them, the electronic stabilization algorithm is a stabilization method that combines the aircraft attitude information measured by the inertial measurement unit of the flight control module (3). The embedded computing module (2) uses the aircraft attitude data obtained from the inertial measurement unit of the flight control module (3) to perform motion estimation, filter compensation, and then obtain the compensated image through affine transformation. The target recognition algorithm has built-in target recognition weights for normal lighting conditions and target recognition weights for low light night vision conditions. The two different weights are switched according to whether the working scene is a low light condition. The target recognition weights for low light night vision conditions are obtained by training the low light sensitive unit (1) on the self-collected dataset.

2. The low-light night vision target recognition device based on a low-light sensitive unit according to claim 1, characterized in that: The low-light sensitive unit (1) and the embedded computing module (2) are connected via a USB 3.0 interface (5). The embedded computing module (2) and the flight control module (3) are connected via a UART interface (6). The low-light sensitive unit (1) and the flight control module (3) are connected via a trigger interface (7).

3. The low-light night vision target recognition device based on a low-illuminance sensitive unit according to claim 1, characterized in that: The flight control module (3) is an autopilot device for UAVs, responsible for acquiring satellite time and aircraft attitude data, controlling the UAV to fly automatically, and receiving the target line of sight angle and accompanying target flight sent by the embedded computing module. The flight control module (3) includes a data processing main chip, an inertial measurement unit and a barometer. The inertial measurement unit and the barometer are both external sensors. The inertial measurement unit and the barometer are both connected to the data processing main chip through a serial port.

4. A low-light night vision target recognition device based on a low-illuminance sensitive unit according to claim 1, characterized in that: The calculation formula for the electronic stability enhancement algorithm assumes the aircraft pitch angle is... The roll angle is yaw angle is Then the rotation matrix for: Let the camera intrinsic parameter matrix be K, then the homography matrix H is: 。 5. A low-light night vision target recognition device based on a low-illuminance sensitive unit according to claim 1, characterized in that: The formula for calculating the target line-of-sight angle is as follows: Let the coordinates of the target center in the image be (x, y), f be the camera focal length, and u be the pixel size. Then the horizontal azimuth angle α and the elevation angle β are respectively: 。 6. A low-light night vision target recognition device based on a low-illuminance sensitive unit according to claim 1, characterized in that: The low-light sensitive unit (1) and the flight control module (3) are located on the same horizontal plane.

7. A method for operating the low-light night vision target recognition device based on a low-light sensitive unit as described in any one of claims 1-6, characterized in that, Includes the following steps: S1: A thread is opened in the embedded computing module (2) to drive the low-light sensitive unit (1) to acquire images, and at the same time the embedded computing module (2) sends a trigger command to the flight control module (3); S2: After receiving the trigger command, the flight control module (3) reads the satellite time and the aircraft pose data and sends them to the embedded computing module (2). At the same time, the flight control module (3) sends a trigger signal to the low-light sensitive unit (1) through the trigger line. After receiving the trigger command, the low-light sensitive unit (1) performs image acquisition and stores the image acquisition into the memory of the embedded computing module (2) through the USB 3.0 line. The embedded computing module (2) stores the acquired image and the received satellite time and aircraft pose data into a structure to complete time alignment. S3: After another thread of the embedded computing module (2) detects that the image acquisition is successful, it starts the electronic stabilization and target recognition thread, and judges whether the working scene is a low illumination condition based on the image grayscale and system time. Different weight files are selected, and the target recognition software outputs the position of the target in the image, the target category information, and the recognition confidence. S4: When the target recognition confidence is greater than the specified threshold, the line-of-sight angle of the target position in the image is calculated and sent to the flight control module (3) via serial port. The flight control module (3) controls the UAV to fly toward the target based on the line-of-sight angle input.

Images