Photoelectric pod target locking system based on multi-target detection tracking

Abstract

The invention relates to the technical field of airborne photoelectricity, in particular to a photoelectric pod target locking system based on multi-target detection and tracking. Comprising a multi-target detection module, a multi-target tracking module and a target locking control module. The multi-target detection module is used for detecting the positions of a plurality of targets in real time from images and video streams obtained by the photoelectric pod and outputting target position data; the multi-target tracking module is used for distributing an identity for each target according to the target position data, performing target tracking and outputting target motion trail data; the target locking control module is used for acquiring target motion trail data and pod attitude data, and adjusting the direction of the photoelectric pod in real time according to the target motion trail data and the pod attitude data; through the above mode, a plurality of targets can be tracked at the same time under complex conditions, and each target can be accurately locked.

Description

Technical Field

[0001] The present invention relates to the field of airborne optoelectronic technology, and particularly to an optoelectronic pod target locking system based on multi-target detection and tracking. Background Art

[0002] An optoelectronic pod is a key sensor on an aviation platform. The optoelectronic pod usually integrates various sensors such as a high-definition camera and an infrared sensor, and performs target recognition, tracking, and positioning through the image data and video stream obtained by the sensors. With the wide application of platforms such as unmanned aerial vehicles, aircraft, and satellites, the demand for optoelectronic pods has gradually increased.

[0003] However, the current optoelectronic pod system has many limitations in the application of multi-target scenarios. Especially in a dynamic and complex environment, there are still problems with the multi-target detection and tracking capabilities of the optoelectronic pod: Most traditional optoelectronic pods rely on manually selecting targets in the image and rely on traditional single-target image tracking algorithms to achieve target locking. These algorithms can achieve certain effects in the case of a single target and a relatively simple environment. However, in actual application scenarios, there are often multiple task targets, and operators need to make quick selections for locking. In such scenarios, traditional single-target tracking algorithms are inadequate. For example, in an urban traffic monitoring scenario, the single-target tracking algorithm may incorrectly lock or frequently switch the lock because it cannot quickly select the target area or distinguish multiple similar vehicle targets, resulting in a decline in the continuity and accuracy of target tracking.

[0004] In summary, it is very necessary to propose an optoelectronic pod target locking system that can simultaneously track multiple targets in complex situations and maintain precise locking of each target. Summary of the Invention

[0005] The purpose of the present invention is to provide an optoelectronic pod target locking system based on multi-target detection and tracking, which can simultaneously track multiple targets in complex situations and maintain precise locking of each target.

[0006] To achieve the above purpose, an optoelectronic pod target locking system based on multi-target detection and tracking adopted by the present invention includes a multi-target detection module, a multi-target tracking module, and a target locking control module. The multi-target tracking module is connected to the multi-target detection module, and the target locking control module is connected to the multi-target tracking module;

[0007] The multi-target detection module is used to: in the images and video streams obtained from the optoelectronic pod, detect the positions of multiple targets in real time and output target position data;

[0008] The multi-target tracking module is used to assign an identity to each target according to the target position data, track the target, and output the target motion trajectory data;

[0009] The target locking control module is used to obtain the target motion trajectory data and the pod attitude data, and adjust the direction of the optoelectronic pod in real time according to the target motion trajectory and the pod attitude.

[0010] Among them, the multi-target detection module includes a multi-target recognition unit, a target box generation unit, and a data processing unit. The target box generation unit is respectively connected to the multi-target recognition unit and the multi-target tracking module, and the data processing unit is connected to the target box generation unit.

[0011] Among them, the multi-target recognition unit is used to recognize various types of targets in images and video streams; the target types include vehicles, people, and buildings;

[0012] The data processing unit is used to process data for various types of targets in images and video streams;

[0013] The target box generation unit is used to generate a bounding box for each detected target, and output the target category and confidence data.

[0014] Among them, the data processing unit includes an optimization subunit, a multi-scale processing subunit, and a feature extraction subunit. The optimization subunit, the multi-scale processing subunit, and the feature extraction subunit are respectively connected to the target box generation unit.

[0015] Among them, the optimization subunit is used to perform non-maximum suppression on various types of targets in images and video streams;

[0016] The multi-scale processing subunit is used to perform multi-scale detection processing on various types of targets in images and video streams;

[0017] The feature extraction subunit is used to extract features for various types of targets in images and video streams.

[0018] Among them, the multi-target tracking module includes a target identity matching unit, a trajectory prediction unit, and a target loss association unit. The target identity matching unit is respectively connected to the multi-target detection module and the trajectory prediction unit, the trajectory prediction unit is connected to the target locking control module, and the target loss association unit is respectively connected to the target identity matching unit, the trajectory prediction unit, and the target locking control module.

[0019] Among them, the target identity matching unit is used to assign a unique identity to each target through appearance features and motion information;

[0020] The trajectory prediction unit is configured to predict the movement trajectory of the target based on the historical movement information of the target;

[0021] The lost target association unit is configured to re-associate the identity of the lost target for the appearance feature data and the movement trajectory data.

[0022] Wherein, the target locking control module includes a direction adjustment unit, a target template generation unit, and a control instruction generation unit. The direction adjustment unit is respectively connected to the multi-target tracking module and the target template generation unit, and the control instruction generation unit is connected to the target template generation unit.

[0023] Wherein, the direction adjustment unit is configured to adjust the direction of the optoelectronic pod in real time according to the target movement trajectory and the pod attitude;

[0024] The target template generation unit is configured to segment the area where the target is located and extract appearance features to generate a target template;

[0025] The control instruction generation unit is configured to generate real-time control instructions by combining the target state, the UAV flight parameters, and the pod adjustment requirements.

[0026] An optoelectronic pod target locking system based on multi-target detection and tracking according to the present invention. The multi-target detection module is configured to detect the positions of multiple targets in real time from the images and video streams obtained by the optoelectronic pod and output target position data; the multi-target tracking module is configured to assign identities to each target according to the target position data and perform target tracking, and output target movement trajectory data; the target locking control module is configured to obtain the target movement trajectory data and the pod attitude data, and adjust the direction of the optoelectronic pod in real time according to the target movement trajectory and the pod attitude; it can track multiple targets simultaneously in complex situations and maintain precise locking on each target. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Figure 1 It is a schematic structural diagram of the optoelectronic pod target locking system based on multi-target detection and tracking of the present invention.

[0029] Figure 2 It is a schematic diagram of the startup of the optoelectronic pod target locking system based on multi-target detection and tracking of the present invention and the UAV deployment.

[0030] 100 - Multi - target detection module, 101 - Multi - target recognition unit, 102 - Target box generation unit, 103 - Optimization sub - unit, 104 - Multi - scale processing sub - unit, 105 - Feature extraction sub - unit, 200 - Multi - target tracking module, 201 - Target identity matching unit, 202 - Trajectory prediction unit, 203 - Target loss association unit, 300 - Target locking control module, 301 - Direction adjustment unit, 302 - Target template generation unit, 303 - Control instruction generation unit. Detailed implementation manner

[0031] Please refer to Figure 1 and Figure 2 wherein Figure 1 is a schematic structural diagram of an optoelectronic pod target locking system based on multi - target detection and tracking, Figure 2 is a schematic diagram of the startup of an optoelectronic pod target locking system based on multi - target detection and tracking and the deployment of an unmanned aerial vehicle.

[0032] The present invention provides an optoelectronic pod target locking system based on multi - target detection and tracking, including a multi - target detection module 100, a multi - target tracking module 200, and a target locking control module 300. The multi - target tracking module 200 is connected to the multi - target detection module 100, and the target locking control module 300 is connected to the multi - target tracking module 200;

[0033] The multi - target detection module 100 is used to detect the positions of multiple targets in real - time from the images and video streams obtained by the optoelectronic pod, and output target position data;

[0034] The multi - target tracking module 200 is used to assign identities to each target according to the target position data, and perform target tracking, and output target motion trajectory data;

[0035] The target locking control module 300 is used to obtain the target motion trajectory data and the pod attitude data, and adjust the direction of the optoelectronic pod in real - time according to the target motion trajectory and the pod attitude.

[0036] In this embodiment, the multi - target detection module 100 detects the positions of multiple targets in real - time from the images and video streams obtained by the optoelectronic pod, and outputs target position data; the multi - target tracking module 200 assigns identities to each target according to the target position data, and performs target tracking, and outputs target motion trajectory data; the target locking control module 300 obtains the target motion trajectory data and the pod attitude data, and adjusts the direction of the optoelectronic pod in real - time according to the target motion trajectory and the pod attitude; it can track multiple targets simultaneously in complex situations and maintain precise locking on each target.

[0037] Further, the multi-object detection module 100 includes a multi-object recognition unit 101, an object bounding box generation unit 102, and a data processing unit. The object bounding box generation unit 102 is respectively connected to the multi-object recognition unit 101 and the multi-object tracking module 200, and the data processing unit is connected to the object bounding box generation unit 102.

[0038] Further, the multi-object recognition unit 101 is configured to recognize multiple types of objects in images and video streams; the object types include vehicles, people, and buildings.

[0039] The data processing unit is configured to perform data processing on multiple types of objects in images and video streams.

[0040] The object bounding box generation unit 102 is configured to generate a bounding box for each detected object and output object category and confidence data.

[0041] In this embodiment, based on a deep learning algorithm, using image processing techniques such as convolutional neural networks (CNNs), the positions of multiple objects are detected in real time from the images or video streams obtained by the optoelectronic pod. The multi-object recognition unit 101 recognizes multiple types of objects in images and video streams; the object types include categories such as people, vehicles, ships, and airplanes. The data processing unit performs data processing on multiple types of objects in images and video streams. The object bounding box generation unit 102 generates a bounding box for each detected object and outputs object category and confidence data.

[0042] Further, the data processing unit includes an optimization subunit 103, a multi-scale processing subunit 104, and a feature extraction subunit 105. The optimization subunit 103, the multi-scale processing subunit 104, and the feature extraction subunit 105 are respectively connected to the object bounding box generation unit 102.

[0043] Further, the optimization subunit 103 is configured to perform non-maximum suppression on multiple types of objects in images and video streams.

[0044] The multi-scale processing subunit 104 is configured to perform multi-scale detection processing on multiple types of objects in images and video streams.

[0045] The feature extraction subunit 105 is configured to perform feature extraction on multiple types of objects in images and video streams.

[0046] In this embodiment, in order to improve the detection accuracy and speed, multi-scale processing and feature extraction technologies are adopted to ensure the efficient detection of targets at different distances and resolutions. Among them, for various types of targets in images and video streams, the optimization subunit 103 is used for non-maximum suppression, and the multi-scale processing subunit 104 is used for multi-scale detection processing; the feature extraction subunit 105 is used for feature extraction.

[0047] Further, the multi-target tracking module 200 includes a target identity matching unit 201, a trajectory prediction unit 202, and a target loss association unit 203. The target identity matching unit 201 is respectively connected to the multi-target detection module 100 and the trajectory prediction unit 202. The trajectory prediction unit 202 is connected to the target locking control module 300. The target loss association unit 203 is respectively connected to the target identity matching unit 201, the trajectory prediction unit 202, and the target locking control module 300.

[0048] Further, the target identity matching unit 201 is configured to assign a unique identity to each target through appearance features and motion information.

[0049] The trajectory prediction unit 202 is configured to predict the motion trajectory of the target based on the historical motion information of the target.

[0050] The target loss association unit 203 is configured to re-associate the identity of the lost target for the appearance feature data and the motion trajectory data.

[0051] In this embodiment, the target identity matching unit 201 assigns a unique identity to each target through appearance features and motion information. Among them, by using the matching mechanism between the detection box and the tracker, through appearance feature matching and motion trajectory analysis, a unique identity is assigned to each target. Through the comprehensive analysis of deep learning embedding features and motion trajectories, target identity confusion is avoided, and the coherence of the tracking process is ensured. The trajectory prediction unit 202 uses post-processing methods such as model prediction and Kalman filtering based on the historical motion information of the target to predict the future motion trajectory of the target and enhance the tracking coherence. The target loss association unit 203 can re-associate the target identity through the target re-identification technology when the target is lost due to occlusion or fast movement.

[0052] Further, the target locking control module 300 includes a direction adjustment unit 301, a target template generation unit 302, and a control instruction generation unit 303. The direction adjustment unit 301 is respectively connected to the multi-target tracking module 200 and the target template generation unit 302. The control instruction generation unit 303 is connected to the target template generation unit 302.

[0053] Further, the direction adjustment unit 301 is configured to adjust the direction of the optoelectronic pod in real time according to the target motion trajectory and the pod attitude;

[0054] The target template generation unit 302 is configured to segment the area where the target is located and extract appearance features to generate a target template;

[0055] The control instruction generation unit 303 is configured to generate real-time control instructions by combining the target state, the UAV flight parameters, and the pod adjustment requirements.

[0056] In this embodiment, the direction adjustment unit 301 adjusts the direction of the optoelectronic pod in real time according to the target motion trajectory and the pod attitude; the target template generation unit 302 segments the area where the target is located and extracts appearance features to generate a target template; the control instruction generation unit 303 generates real-time control instructions by combining the target state, the UAV flight parameters, and the pod adjustment requirements.

[0057] As Figure 2 shown, task startup and UAV deployment:

[0058] The UAV carries the optoelectronic pod and flies along a preset path.

[0059] The optoelectronic pod starts to collect real-time video streams and transmits the data to the signal base station.

[0060] Target detection:

[0061] Real-time image processing: The multi-target detection module 100 performs real-time processing on the video stream collected by the optoelectronic pod to identify and locate the target.

[0062] Detection result output: Generate a bounding box and a class label for each target, and attach a confidence score, and transmit them to the ground control center.

[0063] Target selection and segmentation:

[0064] The operator observes the video stream through the ground control center interface and manually selects the target of interest.

[0065] The system applies an image segmentation algorithm to accurately segment the selected target and generates an appearance template of the target.

[0066] Multi-target tracking:

[0067] Tracking initialization: Based on the target detection result, activate the MOT model to assign a unique ID to each target.

[0068] Trajectory update: Real-time update the motion trajectory of the target and predict its future position.

[0069] Target locking and instruction generation:

[0070] Locking process: Through target template matching and motion prediction, the direction of the optoelectronic pod is adjusted in real time to lock the target at the center of the field of view.

[0071] Instruction generation and transmission: According to the locking state, control instructions are generated and transmitted to the UAV platform through the signal base station.

[0072] Real-time feedback and optimization:

[0073] The UAV feeds back the pod state, target tracking state and environmental data to the ground control center in real time.

[0074] The above-disclosed are only one or more preferred embodiments of the present invention. Of course, the scope of rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. An optoelectronic pod target locking system based on multi-target detection and tracking, characterized in that: It includes a multi-target detection module, a multi-target tracking module and a target locking control module, wherein the multi-target tracking module is connected to the multi-target detection module, and the target locking control module is connected to the multi-target tracking module; The multi-target detection module is used to detect the positions of multiple targets in real time from the images and video streams acquired by the optoelectronic pod and output target position data; The multi-target tracking module is used to assign an identity to each target according to the target position data, and to track the target and output the target motion trajectory data; The target locking control module is used to obtain target motion trajectory data and pod attitude data, and adjust the direction of the optoelectronic pod in real time according to the target motion trajectory and the pod attitude.

2. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 1, characterized in that: The multi-target detection module includes a multi-target recognition unit, a target frame generation unit and a data processing unit. The target frame generation unit is connected to the multi-target recognition unit and the multi-target tracking module respectively, and the data processing unit is connected to the target frame generation unit.

3. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 2, characterized in that: The multi-target recognition unit is used to recognize multiple types of targets in images and video streams; the target types include vehicles, people, and buildings; The data processing unit is used to perform data processing on various types of targets in images and video streams; The target frame generation unit is used to generate a bounding box for each detected target and output target category and confidence data.

4. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 3, characterized in that: The data processing unit includes an optimization subunit, a multi-scale processing subunit and a feature extraction subunit, and the optimization subunit, the multi-scale processing subunit and the feature extraction subunit are respectively connected to the target frame generation unit.

5. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 4, characterized in that: The optimization subunit is used to perform non-maximum suppression on various types of targets in images and video streams; The multi-scale processing subunit is used to perform multi-scale detection processing on various types of targets in images and video streams; The feature extraction subunit is used to extract features of various types of targets in images and video streams.

6. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 1, characterized in that: The multi-target tracking module includes a target identity matching unit, a trajectory prediction unit and a target loss association unit. The target identity matching unit is respectively connected to the multi-target detection module and the trajectory prediction unit, the trajectory prediction unit is connected to the target locking control module, and the target loss association unit is respectively connected to the target identity matching unit, the trajectory prediction unit and the target locking control module.

7. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 6, characterized in that: The target identity matching unit is used to assign a unique identity to each target through appearance features and motion information; The trajectory prediction unit is used to predict the movement trajectory of the target based on the historical movement information of the target; The target loss association unit is used to re-associate the identity of the lost target based on the appearance feature data and the motion trajectory data.

8. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 1, characterized in that: The target locking control module includes a direction adjustment unit, a target template generation unit and a control instruction generation unit. The direction adjustment unit is connected to the multi-target tracking module and the target template generation unit respectively, and the control instruction generation unit is connected to the target template generation unit.

9. The optoelectronic pod target locking system based on multi-target detection and tracking as claimed in claim 8, characterized in that: The direction adjustment unit is used to adjust the direction of the optoelectronic pod in real time according to the target motion trajectory and the pod posture; The target template generation unit is used to segment the target area, extract appearance features and generate a target template; The control instruction generation unit is used to generate real-time control instructions in combination with target status, UAV flight parameters and pod adjustment requirements.

Images