Multi-target detection and tracking system and method based on compound eye event camera and turntable cooperation

Abstract

This invention provides a multi-target detection and tracking system and method that coordinates compound eye event cameras and turntables, solving the problems of high bandwidth and computational load, wasted resources, or unsuitability for large field-of-view multi-target detection and tracking in existing technologies. The system includes a central control unit, a defense unit, M bases, N two-dimensional turntables, M sets of compound eye event cameras, and N high-speed cameras. The central control unit is communicatively connected to the defense unit for receiving strike signals, the compound eye event cameras for acquiring event streams, the two-dimensional turntables for receiving rotation signals, and the high-speed cameras for acquiring target image data. It receives event streams acquired by the compound eye event cameras, detects targets, calculates the three-dimensional coordinates of targets in the world coordinate system, sends rotation signals to the two-dimensional turntables and tracks targets, receives target image data acquired by the high-speed cameras, detects and identifies enemy targets, calculates the three-dimensional coordinates of enemy targets in the world coordinate system, and sends strike signals to the defense unit.

Description

Technical Field

[0001] This invention relates to the detection and tracking of multiple targets, specifically to a multi-target detection and tracking system and method that coordinates a compound eye event camera and a turntable. Background Technology

[0002] Multi-object detection and tracking is an important technology in the field of computer vision. Its main purpose is to detect and track the position and trajectory of multiple targets in a video. Multi-object detection and tracking technology has been widely used in many fields.

[0003] Currently, most multi-target detection and tracking methods employ one or more traditional cameras for observation, while simultaneously using one or more other traditional cameras for tracking. However, a single traditional camera will produce motion blur when capturing high-speed targets, leading to detection and tracking failures. While using multiple traditional cameras can alleviate the motion blur problem caused by high-speed target movement to some extent, increasing the number of traditional cameras leads to increased bandwidth, computational load, and computation time. Moreover, even when there is no target movement, traditional cameras will continue to capture images, resulting in a waste of computational and storage resources.

[0004] Event cameras, also known as dynamic vision sensors, differ from traditional cameras in that they output asynchronous event streams based on the brightness changes of individual pixels, rather than images. Therefore, event cameras offer advantages such as high dynamic range and high temporal resolution, and also provide good detection and tracking performance even when targets are moving at high speeds. However, currently, a single event camera is typically used for single-target detection and tracking, which is unsuitable for multi-target detection and tracking with a large field of view. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problems of existing target detection and tracking systems and methods, such as large bandwidth, large computational load, and waste of computing and storage resources, or unsuitability for multi-target detection and tracking with a large field of view, and to provide a multi-target detection and tracking system and method that coordinates a compound eye event camera and a turntable.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multi-target detection and tracking system that combines a compound eye event camera with a turntable is characterized by:

[0008] It includes a central control unit, a defense unit, M bases, N two-dimensional turntables, and M sets of compound eye event cameras respectively installed on the M bases, and N high-speed cameras respectively installed on the N two-dimensional turntables; the compound eye event cameras and high-speed cameras are all facing the target monitoring area; where M and N are both integers greater than 1;

[0009] The central control unit is communicatively connected to the defense unit, the compound eye event camera, the two-dimensional turntable, and the high-speed camera. The central control unit is used to receive the event stream collected by the compound eye event camera, detect the target and calculate the three-dimensional coordinates of the target in the world coordinate system, send a rotation signal to the two-dimensional turntable based on the three-dimensional coordinates of the target in the world coordinate system and track the target, receive the target image data collected by the high-speed camera, detect and identify the enemy target and calculate the three-dimensional coordinates of the enemy target in the world coordinate system, and send an attack signal to the defense unit.

[0010] The defense unit is used to receive attack signals, attack enemy targets, and detect the attack results;

[0011] The compound eye event camera is used to collect the event stream triggered by the target and transmit it to the central control unit;

[0012] The two-dimensional turntable is used to receive rotation signals sent by the central control unit (3) according to the three-dimensional coordinates of the target in the world coordinate system, and to adjust the azimuth and pitch angles.

[0013] The high-speed camera is used to acquire target image data and transmit it to the central control unit.

[0014] Furthermore, the M bases are evenly distributed in the ground area corresponding to the target monitoring area;

[0015] The N two-dimensional turntables are evenly arranged in the ground area corresponding to the target monitoring area, which can ensure the maximum monitoring range and the minimum overlap of monitoring areas.

[0016] Furthermore, N is 24;

[0017] The compound eye event camera consists of two groups, with each group consisting of 25 event cameras;

[0018] The base is a hemispherical structure, and 25 event cameras are mounted on the base in a biomimetic compound eye style, achieving a large field of view monitoring through a small-volume mounting structure.

[0019] Meanwhile, a multi-target detection and tracking method using a compound eye event camera and a turntable in collaboration is also provided. This method employs the aforementioned multi-target detection and tracking system using a compound eye event camera and a turntable, and its unique feature lies in the following steps:

[0020] Step 1: Calibrate the M groups of compound eye event cameras, N two-dimensional turntables, and N high-speed cameras, and obtain the intrinsic and extrinsic parameters of the compound eye event cameras and high-speed cameras, as well as the azimuth and pitch information of the two-dimensional turntables.

[0021] Step 2: Use the M-group compound eye event cameras to collect the event stream triggered by the target. The central control unit receives the event stream collected by the compound eye event cameras for that time period every T1 time and performs noise reduction processing.

[0022] Step 3: The central control unit detects whether there are targets in the noise-reduced event stream. If there are targets, the three-dimensional coordinates of each target in the world coordinate system are calculated based on the intrinsic and extrinsic parameters of the compound eye event camera obtained in Step 1; otherwise, return to Step 2.

[0023] Step 4: Based on the three-dimensional coordinates of each target in the world coordinate system obtained in Step 3, determine the high-speed cameras corresponding to the monitoring areas of each target; based on the azimuth and elevation angle information of the two-dimensional turntable obtained in Step 1, schedule the two-dimensional turntables corresponding to each high-speed camera and the two-dimensional turntables corresponding to one of the idle high-speed cameras adjacent to each high-speed camera, so that each target is in the center field of view of two of the high-speed cameras and the target image data is tracked and recorded; if P targets appear in the same high-speed camera, then use 2P-2 of the adjacent idle high-speed cameras to track and record the remaining P-1 targets; where P is an integer greater than 1;

[0024] Step 5: The central control unit detects and identifies each target based on the target image data obtained in Step 4. If it is an enemy target, it calculates the three-dimensional coordinates of the enemy target in the world coordinate system based on the intrinsic and extrinsic parameters of the high-speed camera corresponding to the enemy target obtained in Step 1 and the azimuth and elevation angle information of the two-dimensional turntable after scheduling in Step 4, and executes Step 6. If it is not an enemy target, it stops the high-speed camera from tracking by controlling the two-dimensional turntable corresponding to it.

[0025] Step 6: The defense unit strikes the enemy target in the world coordinate system based on the three-dimensional coordinates of the enemy target obtained in Step 5. After a successful strike, the high-speed camera stops tracking by controlling the two-dimensional turntable corresponding to the enemy target, thus completing the detection and tracking of multiple targets.

[0026] Furthermore, step 6 specifically includes:

[0027] The defense unit strikes the enemy target based on the three-dimensional coordinates of the enemy target in the world coordinate system obtained in step 5, and detects the strike result. If the enemy target loses its threat after the strike, the strike is successful, and the high-speed camera stops tracking by controlling the two-dimensional turntable corresponding to the enemy target; otherwise, the strike fails, and the high-speed camera continues to track until the strike is successful by controlling the two-dimensional turntable corresponding to the enemy target, thus completing the detection and tracking of multiple targets.

[0028] Furthermore, the noise reduction process in step 2 specifically involves:

[0029] First, calculate the neighborhood event count matrix Q; where the size of Q is the same as the size of the spatial neighborhood, and the elements of Q are the number of events generated by each pixel in the newly arrived event spatial neighborhood;

[0030] The random noise judgment value R is then calculated using the following formula:

[0031] R = ||F*Q||1

[0032] Where F*Q represents the Hadamard product of the random noise filter F and the matrix Q, and ||||1 represents the norm of 1;

[0033] If R is greater than the threshold, the event is retained; otherwise, the event is filtered out.

[0034] Furthermore, Q is a 5×5 matrix;

[0035] The beneficial effects of this invention are:

[0036] This invention provides a multi-target detection and tracking system and method that coordinates a compound-eye event camera and a turntable, solving the problems of slow computation, high bandwidth, and wasted computing and storage resources inherent in traditional cameras. This invention uses a compound-eye event camera to observe targets, while simultaneously employing multiple high-speed cameras for detection and tracking. This allows for the acquisition of high-resolution target details. Each high-speed camera focuses only on its own monitoring area, dynamically coordinating with other high-speed cameras to observe targets in adjacent areas only when needed. This results in more accurate detection results. Furthermore, when multiple targets are present in the air, the stability and accuracy of the entire detection and tracking system are enhanced, enabling the detection and tracking of multiple targets with a large field of view. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the multi-target detection and tracking system of the present invention, which combines a compound eye event camera with a turntable.

[0038] Figure 2 This is a schematic diagram of the compound eye event camera and its base in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the structure of the two-dimensional turntable and high-speed camera in an embodiment of the present invention;

[0040] Figure 4 This is a flowchart of the multi-target detection and tracking method of the present invention, which uses a compound eye event camera and a turntable in collaboration.

[0041] Explanation of reference numerals in the attached figures:

[0042] 1-Event camera, 2-Base, 3-Central control unit, 4-Defense unit, 5-2D turntable, 6-High-speed camera. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figures 1-3 As shown, a multi-target detection and tracking system in collaboration with compound eye event cameras and turntables includes a central control unit 3, a defense unit 4, two bases 2, 24 two-dimensional turntables 5, two sets of compound eye event cameras, and 24 high-speed cameras 6 respectively mounted on the 24 two-dimensional turntables 5. Specifically, the bases 2 have a hemispherical structure, and the two bases 2 are evenly distributed in the ground area corresponding to the target monitoring area. The 24 two-dimensional turntables 5 are evenly distributed in the ground area corresponding to the target monitoring area. Each set of compound eye event cameras consists of 25 event cameras 1, which are mounted on the bases 2 in a biomimetic compound eye style. All 25 event cameras 1 and 24 high-speed cameras 6 face the target monitoring area.

[0045] The central control unit 3 is communicatively connected to the defense unit 4, event camera 1, 2D turntable 5, and high-speed camera 6. The central control unit 3 receives event streams collected by event camera 1, detects targets, calculates the target's 3D coordinates in the world coordinate system, sends rotation signals to the 2D turntable 5 based on the target's 3D coordinates, tracks the target, receives target image data collected by high-speed camera 6, detects and identifies enemy targets, calculates the enemy target's 3D coordinates in the world coordinate system, and sends attack signals to the defense unit 4. The defense unit 4 receives attack signals, attacks enemy targets, and detects the attack results. Event camera 1 collects event streams triggered by targets and transmits them to the central control unit 3. The 2D turntable 5 receives rotation signals sent by the central control unit 3 based on the target's 3D coordinates in the world coordinate system and adjusts the azimuth and elevation angles. High-speed camera 6 collects target image data and transmits it to the central control unit 3.

[0046] like Figure 4 As shown, the present invention also provides a multi-target detection and tracking method in collaboration between a compound eye event camera and a turntable, comprising the following steps:

[0047] Step 1: Calibrate each event camera 1, 2D turntable 5, and high-speed camera 6, and obtain the intrinsic and extrinsic parameters of each event camera 1 and high-speed camera 6, as well as the azimuth and pitch angle information of each 2D turntable 5.

[0048] Step 2: Use all event cameras 1 to collect the event stream triggered by the target. The central control unit 3 receives the event stream collected by all event cameras 1 for that time period every T1 time and performs noise reduction processing.

[0049] A denoising algorithm is used to denoise the event stream, focusing only on events indicating target movement. Based on existing events, it is determined whether new events are noisy events. The specific denoising steps are as follows:

[0050] First, calculate the neighborhood event count matrix Q; where Q is the same size as the spatial neighborhood, a 5×5 matrix, and the elements of Q are the number of events generated by each pixel in the newly arrived event spatial neighborhood;

[0051] The random noise judgment value R is then calculated using the following formula:

[0052] R = ||F*Q||1

[0053] Where F*Q represents the Hadamard product of the random noise filter F and the matrix Q. ||||1 represents the norm of 1;

[0054] If R is greater than the threshold, the event is retained; otherwise, the event is filtered out.

[0055] Step 3: The central control unit 3 detects whether there is a target in the event stream after noise reduction. If there is a target, it calculates the three-dimensional coordinates of each target in the world coordinate system based on the intrinsic and extrinsic parameters of each event camera 1 obtained in Step 1; otherwise, it returns to Step 2.

[0056] Step 4: Based on the three-dimensional coordinates of each target in the world coordinate system obtained in Step 3, determine the high-speed cameras 6 corresponding to the monitoring areas of each target; based on the azimuth and elevation angle information of the two-dimensional turntable 5 obtained in Step 1, schedule the two-dimensional turntable 5 corresponding to each determined high-speed camera 6 and the two-dimensional turntable 5 corresponding to one of the idle high-speed cameras 6 adjacent to each high-speed camera 6, so that each target is in the center field of view of two of the high-speed cameras 6 and the target image data is tracked and recorded; if P targets appear in the same high-speed camera 6, then use 2P-2 of the adjacent idle high-speed cameras 6 to track and record the remaining P-1 targets; where P is an integer greater than 1.

[0057] Step 5: The central control unit 3 detects and identifies each target based on the target image data obtained in step 4. If it is an enemy target, it calculates the three-dimensional coordinates of the enemy target in the world coordinate system based on the internal and external parameters of the high-speed camera 6 corresponding to the enemy target obtained in step 1 and the azimuth and elevation angle information of the two-dimensional turntable 5 after scheduling in step 4, and executes step 6. If it is not an enemy target, it stops the high-speed camera 6 from tracking by controlling the two-dimensional turntable 5 corresponding to it.

[0058] Step 6: Defense unit 4 strikes the enemy target in the world coordinate system based on the three-dimensional coordinates of the enemy target obtained in step 5, and detects the strike result. If the enemy target loses its threat after the strike, the strike is successful, and the high-speed camera 6 stops tracking by controlling the two-dimensional turntable 5 corresponding to the enemy target; otherwise, the strike fails, and the high-speed camera 6 continues to track until the strike is successful by controlling the two-dimensional turntable 5 corresponding to the enemy target, thus completing the detection and tracking of multiple targets.

[0059] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-target detection and tracking system that coordinates a compound eye event camera and a turntable, characterized in that: It includes a central control unit (3), a defense unit (4), M bases (2), N two-dimensional turntables (5), and M sets of compound eye event cameras installed on the M bases (2) and N high-speed cameras (6) installed on the N two-dimensional turntables (5); the compound eye event cameras and the high-speed cameras (6) are all facing the target monitoring area; where M and N are both integers greater than 1; The central control unit (3) is connected to the defense unit (4), compound eye event camera, two-dimensional turntable (5), and high-speed camera (6) respectively. The central control unit (3) is used to receive the event stream collected by the compound eye event camera and detect the target and calculate the three-dimensional coordinates of the target in the world coordinate system, send a rotation signal to the two-dimensional turntable (5) according to the three-dimensional coordinates of the target in the world coordinate system and track the target, receive the target image data collected by the high-speed camera (6) and detect and identify the enemy target and calculate the three-dimensional coordinates of the enemy target in the world coordinate system, and send an attack signal to the defense unit (4). The defense unit (4) is used to receive strike signals, strike enemy targets, and detect strike results; The compound eye event camera is used to collect the event stream triggered by the target and transmit it to the central control unit (3); The two-dimensional turntable (5) is used to receive the rotation signal sent by the central control unit (3) according to the three-dimensional coordinates of the target in the world coordinate system, and to adjust the azimuth and pitch angles. The high-speed camera (6) is used to acquire target image data and transmit it to the central control unit (3).

2. The multi-target detection and tracking system based on the compound eye event camera and turntable in collaboration according to claim 1, characterized in that: The M bases (2) are evenly arranged in the ground area corresponding to the target monitoring area; The N two-dimensional turntables (5) are evenly arranged in the ground area corresponding to the target monitoring area.

3. The multi-target detection and tracking system based on the compound eye event camera and turntable in collaboration according to claim 2, characterized in that: The value of N is 24; The compound eye event camera consists of two groups, each group consisting of 25 event cameras (1); The base (2) is a hemispherical structure, and 25 event cameras (1) are mounted on the base (2) in a bionic compound eye style.

4. A multi-target detection and tracking method using a compound eye event camera and a turntable in collaboration, employing the multi-target detection and tracking system using a compound eye event camera and a turntable as described in any one of claims 1 to 3, characterized in that, Includes the following steps: Step 1: Calibrate the M groups of compound eye event cameras, N two-dimensional turntables (5) and N high-speed cameras (6), and obtain the internal and external parameters of the compound eye event cameras and high-speed cameras (6), as well as the azimuth and pitch information of the two-dimensional turntables (5); Step 2: Use the M-group compound eye event camera to collect the event stream triggered by the target. The central control unit (3) receives the event stream collected by the compound eye event camera for that time period every T1 time and performs noise reduction processing. Step 3: The central control unit (3) detects whether there is a target in the event stream after noise reduction. If there is a target, it calculates the three-dimensional coordinates of each target in the world coordinate system based on the internal and external parameters of the compound eye event camera obtained in Step 1; otherwise, it returns to Step 2. Step 4: Based on the three-dimensional coordinates of each target in the world coordinate system obtained in Step 3, determine the high-speed cameras (6) corresponding to the monitoring areas of each target; based on the azimuth and pitch information of the two-dimensional turntable (5) obtained in Step 1, schedule the two-dimensional turntable (5) corresponding to each high-speed camera (6) and the two-dimensional turntable (5) corresponding to one of the idle high-speed cameras (6) adjacent to each high-speed camera (6) respectively, so that each target is in the center field of view of two of the high-speed cameras (6) and the target image data is tracked and recorded; if P targets appear in the same high-speed camera (6), then 2P-2 of the adjacent idle high-speed cameras (6) are used to track the remaining P-1 targets and record the target image data; where P is an integer greater than 1; Step 5: The central control unit (3) detects and identifies each target based on the target image data obtained in step 4. If it is an enemy target, it calculates the three-dimensional coordinates of the enemy target in the world coordinate system based on the internal and external parameters of the high-speed camera (6) corresponding to the enemy target obtained in step 1 and the azimuth and pitch information of the two-dimensional turntable (5) after scheduling in step 4, and executes step 6. If it is not an enemy target, it stops the high-speed camera (6) from tracking by controlling the two-dimensional turntable (5) corresponding to it. Step 6: The defense unit (4) strikes the enemy target in the world coordinate system according to the three-dimensional coordinates of the enemy target obtained in step 5. After the strike is successful, the high-speed camera (6) stops tracking by controlling the two-dimensional turntable (5) corresponding to the enemy target, thus completing the detection and tracking of multiple targets.

5. The multi-target detection and tracking method using a compound eye event camera and a turntable in collaboration according to claim 4, characterized in that, Step 6 specifically involves: The defense unit (4) strikes the enemy target in the world coordinate system according to the three-dimensional coordinates of the enemy target obtained in step 5, and detects the strike result. If the enemy target loses its threat after the strike, the strike is successful. The high-speed camera (6) stops tracking by controlling the two-dimensional turntable (5) corresponding to the enemy target. Otherwise, the strike fails. The high-speed camera (6) continues to track until the strike is successful by controlling the two-dimensional turntable (5) corresponding to the enemy target, thus completing the detection and tracking of multiple targets.

6. The multi-target detection and tracking method using a compound eye event camera and a turntable in collaboration according to claim 5, characterized in that, The noise reduction process in step 2 is as follows: First, calculate the neighborhood event count matrix Q; where the size of Q is the same as the size of the spatial neighborhood, and the elements of Q are the number of events generated by each pixel in the newly arrived event spatial neighborhood; The random noise judgment value R is then calculated using the following formula: R = ||F*Q|| Where F*Q represents the Hadamard product of the random noise filter F and the matrix Q, and ||||1 represents the norm of 1; If R is greater than the threshold, the event is retained; otherwise, the event is filtered out.

7. The multi-target detection and tracking method using a compound eye event camera and a turntable in collaboration according to claim 6, characterized in that: Q is a 5×5 matrix;

Images