Target detection and tracking method and device based on multiscale field of view photoelectric cooperation, electronic equipment and storage medium

Abstract

This invention provides a target detection and tracking method, device, electronic device, and storage medium based on multi-scale field-of-view optoelectronic coordination, relating to the field of target detection and tracking technology. It constructs a multi-scale field-of-view coordination mechanism that integrates coarse observation from a panoramic lens with fine observation from an optoelectronic lens. Leveraging the wide coverage of the panoramic lens, it performs large-scale UAV target search and priority ranking. A coordinate mapping model enables precise cross-field-of-view guidance, achieving accurate transmission of UAV target position information. The telephoto advantage of the optoelectronic lens is utilized for detail capture and continuous tracking, significantly reducing the handover delay between the panoramic and optoelectronic lenses, thus preventing target loss. This method fundamentally solves the problems of large blind spots, missing target details, and insufficient dynamic tracking capabilities inherent in single-field-of-view lenses in low-altitude security scenarios, significantly improving the detection efficiency and recognition accuracy of UAV targets.

Description

Technical Field

[0001] This invention relates to the field of target detection and tracking technology, and in particular to a target detection and tracking method, device, electronic device and storage medium based on multi-scale field-of-view optoelectronic coordination. Background Technology

[0002] Currently, target detection and tracking technology has become a core supporting technology in the field of low-altitude security, widely used in key scenarios such as airport airspace protection, security of important venues, low-altitude control, and restricted area alert. One of its core objectives is to achieve accurate detection and continuous tracking of illegally intruding drones. With the rapid popularization of drone technology, the requirements for detection systems in terms of wide coverage, high precision, real-time performance, and anti-interference capabilities are becoming increasingly stringent. From a technological evolution perspective, existing drone detection and tracking solutions have gone through a development stage from those based on single-field-of-view lenses to those based on multi-field-of-view lens fusion.

[0003] Optical detection solutions based on a single field-of-view lens cannot simultaneously achieve wide coverage and high precision, making it difficult to meet the detection needs of drones in complex low-altitude environments. For example, if a panoramic field-of-view lens is used to pursue a large-scale, blind-spot-free low-altitude coverage, it inevitably sacrifices local imaging resolution, resulting in the loss of details of small, distant drone targets, making accurate identification and model determination impossible. If a high-magnification photoelectric lens is used to ensure fine detection, the small field of view will create numerous monitoring blind spots, easily missing high-speed maneuvering drone targets. Moreover, due to the small size, high speed, and flexible flight paths of drones, and the complex factors in the low-altitude environment such as cloud cover, light and shadow interference, and building obstruction, the fixed parameter control strategy of a single field-of-view lens is difficult to adapt to the dynamic motion characteristics of drones and the complex low-altitude environment, exhibiting poor adaptability and failing to achieve efficient tracking of multiple drones simultaneously.

[0004] Optical detection schemes based on multi-field-of-view lens fusion mainly combine panoramic lenses and electro-optical lenses. The panoramic lens first continuously scans the low-altitude area. Upon detecting a suspected UAV target, the approximate area is output to the electro-optical lens through manual annotation or simple coordinate calculation. The operator then manually controls the electro-optical lens to rotate to the target area, or achieves coarse alignment through simple automatic control, before switching to manual tracking mode. However, this scheme cannot solve problems such as target handover delays and tracking loss due to high-speed UAV maneuvers. It is particularly inadequate for detecting small, low-speed, and low-observable UAV targets, and still cannot meet the requirements for automated, high-precision UAV detection. Summary of the Invention

[0005] This invention provides a target detection and tracking method, device, electronic device, and storage medium based on multi-scale field-of-view optoelectronic coordination, in order to overcome the deficiencies existing in related technologies.

[0006] This invention provides a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination, comprising: The system acquires panoramic images captured by a panoramic lens, identifies target information of each UAV target in the panoramic images, and determines the tracking order of each UAV target based on the target information of each UAV target. According to the tracking order of each UAV target, each UAV target is tracked, and for the current tracked target, based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, the pixel position of the current tracked target under the panoramic lens is converted into the pixel position under the photoelectric lens. Based on the pixel position of the current tracking target under the photoelectric lens, the control parameters of the photoelectric lens are adjusted to align the photoelectric lens with the current tracking target, and the current tracking target is tracked based on the photoelectric image acquired by the photoelectric lens.

[0007] According to a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided by the present invention, the step of tracking the current target based on the optoelectronic image acquired by the optoelectronic lens includes: Calculate the motion parameters of the currently tracked target in the photoelectric image and the image quality parameters of the photoelectric image, and obtain the environmental parameters of the photoelectric lens; Based on the motion parameters, image quality parameters, and environmental parameters, with the joint optimization objectives of maximizing target tracking stability, optimizing imaging quality, and minimizing tracking power consumption, a reinforcement learning algorithm is used to adaptively adjust the control parameters of the photoelectric lens, and continuously track the current target based on the photoelectric lens.

[0008] According to a multi-scale field-of-view optoelectronic cooperative target detection and tracking method provided by the present invention, the step of determining the tracking order of each UAV target based on the target information of each UAV target further includes: determining the tracking level of each UAV target based on the target information of each UAV target; The coordinate mapping model based on the panoramic field-of-view coordinate system of the panoramic lens and the optical field-of-view coordinate system of the photoelectric lens converts the pixel position of the currently tracked target under the panoramic lens into the pixel position under the photoelectric lens, including: If the tracking level of the current tracked target is the target tracking level, then based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, the pixel position of the current tracked target in the panoramic image coordinate system is converted into the pixel position in the photoelectric image coordinate system.

[0009] According to the present invention, a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination is provided, wherein the target information further includes a target threat level, and each UAV target includes multiple designated targets with the same tracking level; the tracking of the current target based on the optoelectronic image acquired by the optoelectronic lens further includes: Based on the photoelectric image, predict the tracking difficulty coefficient of each of the specified targets; If the tracking level of each specified target is the target tracking level, and the distance between each specified target is greater than the preset distance, then the joint optimization target is to prioritize tracking the specified target with the highest target threat level and the highest tracking difficulty coefficient, and to maximize the tracking benefit, and to plan the time-sharing collaborative tracking sequence of the photoelectric lens for each specified target. According to the time-division collaborative tracking sequence, the photoelectric lens is controlled to perform time-division collaborative tracking of each of the designated targets.

[0010] According to a multi-scale field-of-view photoelectric cooperative target detection and tracking method provided by the present invention, the step of tracking the current target based on the photoelectric image acquired by the photoelectric lens further includes: Based on the photoelectric images, predict the flight trajectories of each of the specified targets; If the tracking level of each specified target is the target tracking level, the distance between each specified target is less than or equal to the preset distance, and the error between the flight trajectories of each specified target is less than the preset error, then the control parameters of the photoelectric lens are adjusted so that each specified target simultaneously enters the field of view of the photoelectric lens, and the specified targets are synchronously tracked based on the photoelectric lens.

[0011] According to the present invention, a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination is provided, the method further comprising: If the tracking level of the current target is a non-target tracking level, then the current target is tracked based on the panoramic image.

[0012] According to the present invention, a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination is provided, wherein the control parameters include gimbal attitude and lens focal length; the step of adjusting the control parameters of the optoelectronic lens based on the pixel position of the currently tracked target under the optoelectronic lens to align the optoelectronic lens with the currently tracked target includes: Based on the pixel position of the currently tracked target under the photoelectric lens, calculate the deviation between the currently tracked target and the field of view center of the photoelectric lens; Based on the deviation, the gimbal of the photoelectric lens is controlled to move until the currently tracked target is located in the center area of ​​the field of view of the photoelectric lens.

[0013] The present invention also provides a target detection and tracking device based on multi-scale field-of-view optoelectronic coordination, comprising: The target detection module is used to acquire panoramic images captured by the panoramic lens, identify the target information of each UAV target in the panoramic image, and determine the tracking order of each UAV target based on the target information of each UAV target. The coordinate mapping module is used to track each UAV target according to the tracking order of each UAV target, and for the current tracked target, based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, convert the pixel position of the current tracked target under the panoramic lens into the pixel position under the photoelectric lens. The target tracking module is used to adjust the control parameters of the photoelectric lens based on the pixel position of the current tracking target under the photoelectric lens, so that the photoelectric lens is aligned with the current tracking target, and to track the current tracking target based on the photoelectric image acquired by the photoelectric lens.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination as described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination as described above.

[0017] This invention provides a target detection and tracking method, device, electronic device, and storage medium based on multi-scale field-of-view optoelectronic coordination. By constructing a multi-scale field-of-view coordination mechanism that integrates coarse observation from panoramic lenses to fine observation from optoelectronic lenses, it leverages the wide coverage of panoramic lenses for large-scale UAV target search and priority ranking. A coordinate mapping model enables precise cross-field-of-view guidance, ensuring accurate transmission of UAV target position information. The telephoto advantage of optoelectronic lenses allows for detailed capture and continuous tracking, significantly reducing the handover delay between panoramic and optoelectronic lenses and preventing target loss. This method fundamentally solves the problems of large blind spots, missing target details, and insufficient dynamic tracking capabilities inherent in single-field-of-view lenses in low-altitude security scenarios. It significantly improves the detection efficiency and recognition accuracy of UAV targets, achieving seamless integration of wide-area low-altitude coverage and high-precision focused tracking. By performing fully automated detection and tracking of UAV targets, this method significantly reduces the workload of operators, minimizes the need for manual intervention, and enhances the applicability of the target detection and tracking system in unattended, all-weather security scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the target detection and tracking structure in the multi-scale field-of-view optoelectronic cooperative target detection and tracking method provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the target detection and tracking device based on multi-scale field-of-view optoelectronic coordination provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The target, optical detection solutions based on a single field-of-view lens, mainly include wide-coverage detection technology for UAVs based on panoramic lenses and precise tracking technology for UAVs based on high-magnification photoelectric lenses.

[0025] Wide-coverage drone detection technology based on panoramic lenses leverages the large field of view of panoramic lenses to achieve full-coverage scanning of specific low-altitude areas. This allows for rapid detection of drone targets within a given area, enabling large-scale monitoring without the need for field-of-view switching. This technology solves the problem of blind spots in single high-magnification photoelectric lenses and is suitable for detecting the wide distribution of low-altitude drone targets.

[0026] High-magnification photoelectric lens-based UAV precision tracking technology aims to achieve refined detection and continuous tracking of detected UAV targets, providing core technological support for low-altitude security and airport airspace protection. This technology integrates a high-magnification photoelectric lens, a precision gimbal, a high-frame-rate photoelectric detector, and an autofocus / exposure control module. It achieves rapid alignment and focus adjustment of the UAV target through optical zoom and gimbal rotation. In practice, this technology first obtains the initial position information of the UAV target through radar guidance or manual detection. Then, it locks onto the UAV target using image recognition algorithms. The gimbal's closed-loop control is driven by the pixel position deviation of the UAV target, while the focus of the photoelectric lens adaptively adjusts according to changes in the size of the UAV target to ensure the target is clearly centered in the lens's field of view. This technology can obtain high-definition detailed images of UAV targets, enabling aircraft identification, precise trajectory capture, and flight status assessment. It is suitable for threat assessment and guidance scenarios involving UAV targets.

[0027] To address the performance bottleneck of single-field-of-view lenses, existing technologies have developed optical detection solutions based on multi-field-of-view lens fusion, but these technologies are not yet mature and have not yet formed a true collaborative mechanism.

[0028] Although existing detection and tracking technologies using single-field-of-view lenses and optical detection schemes using multi-field-of-view lens fusion can meet some basic monitoring needs, they still have many shortcomings and deficiencies in practical applications, as follows: First, the inherent contradiction between field of view and accuracy is irreconcilable: this is the core bottleneck of existing drone detection solutions based on single-field-of-view lenses. For solutions based on panoramic lenses, whether fisheye panoramas or multi-camera stitched panoramas, the expansion of the field of view inevitably comes at the cost of sacrificing the imaging resolution of local areas. Fisheye lenses suffer from severe edge distortion, and the resolution of edge areas further decreases after correction; while multi-camera stitched panoramas can improve overall resolution, the loss of detail in the stitched areas is still unavoidable. Because drone targets are generally small, fly at low altitudes, and are highly maneuverable, they can only be presented as a set of 3-5 pixels in a mid-to-long-range panoramic image beyond 500 meters. It is impossible to extract effective feature information such as wings and fuselages, making it easy to misidentify birds, kites, balloons, and other interfering objects as drones, and also making it impossible to distinguish the type of drone target, resulting in a high false alarm rate and low identification accuracy.

[0029] For solutions based on high-magnification photoelectric lenses, the focal length and field of view are strictly inversely proportional. To clearly capture the details of UAV targets, the focal length needs to be set to a long focal length, such as 200mm or more. At this point, the field of view may shrink to less than 0.5°, only able to monitor a very small area of ​​tens of square meters, unable to achieve large-scale low-altitude coverage. In large-scale low-altitude control scenarios such as airport airspace and the vicinity of large venues, scanning is required through a preset trajectory, which can easily lead to the omission of high-speed maneuvering UAV targets due to scanning gaps, forming monitoring blind spots. This inherent contradiction between wide coverage and high precision makes existing single-field-of-view solutions unable to meet the high-end requirements of UAV target detection that simultaneously demands no blind spots and fine-grained identification and tracking. Moreover, this technology relies on externally inputted UAV target positions and cannot autonomously complete wide-area detection of UAV targets, making it difficult to cope with scenarios where multiple UAV targets intrude simultaneously.

[0030] Second, the lack of a collaborative mechanism leads to response delays and tracking loss: Existing UAV detection solutions based on multi-field-of-view lens combinations lack a deep collaborative mechanism between panoramic and electro-optical lenses, resulting in significant response delays during UAV target handover, which easily leads to the loss of highly maneuverable UAV targets. Specifically: First, the accuracy of coordinate transmission is insufficient. The coordinate systems of the panoramic and electro-optical lenses are independent, and existing solutions only perform coordinate conversions through simple geometric relationships. This results in significant errors in the UAV target position information received by the electro-optical lens, requiring additional manual adjustments to align with the target. Second, there is a delay in the response of control commands. In existing solutions, the target detection results from the panoramic lens need to be relayed through a host computer before being sent to the control module of the electro-optical lens. The data transmission and processing process has a delay of more than 500ms. For high-speed UAV targets, the actual displacement within this delay exceeds 8 meters, far exceeding the field of view of the electro-optical lens, directly leading to tracking loss.

[0031] Based on this, this embodiment of the invention provides a target detection and tracking method based on multi-scale field-of-view optoelectronic coordination, such as... Figure 1 As shown, the method includes: S1, acquire panoramic images captured by panoramic lenses, identify target information of each UAV target in the panoramic images, and determine the tracking order of each UAV target based on the target information of each UAV target; S2, each UAV target is tracked according to the tracking order of each UAV target, and for the current tracking target, the pixel position of the current tracking target under the panoramic lens is converted into the pixel position under the photoelectric lens based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the photoelectric lens field of view coordinate system. S3, based on the pixel position of the current tracking target under the photoelectric lens, adjust the control parameters of the photoelectric lens to align the photoelectric lens with the current tracking target, and track the current tracking target based on the photoelectric image acquired by the photoelectric lens.

[0032] Specifically, the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided in this embodiment of the invention uses a target detection and tracking device based on multi-scale field-of-view optoelectronic coordination as its execution entity. This device can be configured within a target detection and tracking system. This target detection and tracking system can have standardized interfaces, enabling easy integration into existing low-altitude security systems, providing a universal solution for different types of UAV target detection needs, and expanding the application scope and lifecycle of the technology.

[0033] First, step S1 is executed to acquire the panoramic image captured by the panoramic lens. A panoramic lens is an imaging device with a large field of view, designed to provide wide-coverage low-altitude field-of-view monitoring capabilities, enabling the detection of targets within the monitoring area without blind spots. The scanning frequency of the panoramic lens can be set as needed, for example, to 10 frames per second.

[0034] A panoramic lens can be a panoramic camera module stitched together from multiple cameras, or it can be a fisheye panoramic lens. Without deviating from the core principle of multi-scale field-of-view collaboration, panoramic lenses can be flexibly selected or their core parameters adjusted based on the spatial conditions, cost budget, and resolution requirements of the application scenario.

[0035] When a panoramic lens is composed of multiple cameras stitched together, the cameras used can be ordinary cameras, high-resolution industrial cameras, or long-focal-length cameras. The number of cameras in a panoramic lens can be set as needed, for example, up to 18. Each camera can be an 8-megapixel, 1 / 1.8-inch CMOS industrial camera. The cameras in the panoramic lens are arranged in a 3×6 array, and each camera has a focal length of 75mm.

[0036] High-resolution industrial cameras are suitable for applications requiring high detail in panoramic images, such as long-range drone detection and airspace protection at large airports. By increasing the image resolution of a single camera, the ability to render the features of long-range drone targets can be enhanced. Combined with the existing focal length parameters, the coverage range can be stably guaranteed, adapting to the needs of high-definition detection.

[0037] Long-focal-length cameras are suitable for applications involving ultra-long-range low-altitude protection, such as low-altitude control along borders. Combined with high-resolution CMOS, the observation distance of a single camera can be further extended. By optimizing the array layout, wide coverage can be ensured, which can meet the needs of early detection of long-range UAV targets.

[0038] Fisheye panoramic lenses are suitable for applications involving short-to-medium distance detection in confined spaces, such as the perimeter of venues and low-altitude protection at ports and docks. Their core advantages lie in their simple structure, low deployment cost, and compact size. Through mature distortion correction algorithms, they can ensure the integrity of low-altitude field of view coverage and meet basic detection needs that do not require high panoramic image clarity.

[0039] A panoramic image is image data containing complete field-of-view information captured by a panoramic lens. If a panoramic lens is composed of multiple cameras stitched together, the panoramic image can be obtained by stitching together single-camera images acquired by each camera at a preset scanning frame rate over a large area of ​​the monitored region using an image stitching algorithm.

[0040] Here, the image stitching algorithm used can be a SIFT feature matching-based algorithm. First, distortion correction is performed on the individual camera images acquired by each camera. Then, SIFT feature points are extracted from the individual camera images, and feature point matching is performed using a FLANN matcher. The RANSAC algorithm is used to remove mismatched points. The homography matrix between each individual camera image is calculated, and the projection transformation of the individual camera images is achieved using the homography matrix. Finally, a weighted fusion algorithm is used to eliminate stitching gaps between the individual camera images, resulting in a high-resolution panoramic image. The resolution of this panoramic image can be obtained by superimposing the resolutions of the individual camera images.

[0041] After obtaining the panoramic image, it can be preprocessed. For example, a Gaussian filter with a 5×5 convolution kernel can be used to remove noise from the panoramic image. An adaptive histogram equalization (CLAHE) algorithm can be used to enhance the contrast of the panoramic image. For low-altitude fog and haze environments, a dark channel prior algorithm can be added to remove fog, thereby improving the detection effect of UAV targets under complex weather conditions.

[0042] Subsequently, target recognition is performed on the panoramic image to obtain target information for each drone target within the panoramic image. A drone target refers to an unauthorized intrusion or unmanned aerial vehicle appearing within the monitoring area. Target information refers to the feature data of the drone target in the panoramic image, including but not limited to the drone target's identifier, location, target type, and motion parameters. The location of the drone target can include bounding box information and the center point pixel coordinates; the bounding box information can be represented by the world coordinates of the diagonal vertices of the bounding box.

[0043] Target types can include multi-rotor drones, fixed-wing drones, and racing drones. This target type can be determined through drone target identification and threat level assessment. Target identification can be achieved using drone identification algorithms. For detection scenarios with complex backgrounds, identification models with stronger global feature capture capabilities can be selected to better handle interference from complex backgrounds on drone target features, improving the stability and accuracy of target identification. Threat level assessment can be achieved using threat level assessment algorithms. For complex security scenarios with ambiguous threat definitions, assessment methods with stronger fuzzy feature processing capabilities can be selected to more accurately handle the ambiguity of target features, improving the rationality and reliability of threat level assessment.

[0044] Motion parameters refer to data indicators that reflect the dynamic flight characteristics of UAV targets. They are used to measure the maneuverability of UAV targets and can include flight speed, flight acceleration, flight direction, flight altitude, and trajectory of UAV targets.

[0045] When performing target recognition on panoramic images, image recognition algorithms can be used first to detect the identifiers, positions, and detection confidence of each UAV target in the panoramic image. Then, the positions of each UAV target in adjacent frames of panoramic images are associated with the identifiers of each UAV target. The flight speed of each UAV target is calculated by the position difference between each UAV target in adjacent frames of panoramic images. The flight direction is determined by the trend of the change in the pixel coordinates of the center point of each UAV target in adjacent frames of panoramic images, with a focus on determining whether it is heading towards core areas such as runways and terminals.

[0046] Here, the image recognition algorithm used can be an improved YOLOv10 target detection and tracking algorithm, a high-precision detection and tracking algorithm, or a lightweight detection and tracking algorithm, which can be selected according to the differentiated requirements of real-time performance and accuracy in the application scenario. The improved YOLOv10 target detection and tracking algorithm can also introduce a retina-like mechanism to enhance the ability to extract motion parameters of UAV targets at different scales and reduce tracking drift caused by rapid maneuvering of UAV targets and cloud obstruction.

[0047] Understandably, high-precision detection and tracking algorithms are suitable for applications with stringent requirements for detection accuracy, such as long-range, low-observable UAV detection and military-grade UAV identification. This algorithm enhances the resolution accuracy of UAV targets by strengthening feature extraction capabilities, effectively distinguishing UAV targets from interfering objects in low signal-to-noise ratio environments, thus meeting the demands of high-precision identification.

[0048] Lightweight detection and tracking algorithms are suitable for lightweight deployment scenarios such as portable devices and embedded systems, including portable drone detection devices and embedded low-altitude surveillance systems. Through model simplification and parameter optimization, this algorithm significantly improves detection speed and reduces computational consumption, enabling rapid response to high-speed maneuvering drone targets and adapting to resource-constrained deployment scenarios.

[0049] Tracking order refers to the sequence in which each UAV target is tracked.

[0050] In this step, the panoramic lens is first controlled to continuously scan and acquire images of the monitoring area at a preset frame rate, obtaining high-resolution panoramic images in real time. Then, a pre-set target detection algorithm is used to process each frame of the panoramic image to identify the presence of drone targets. When one or more drone targets are detected within the monitoring area, the pixel position of each drone target is extracted, and the motion parameters of the drone targets are calculated by combining the pixel position changes of the drone targets in adjacent frames of the panoramic image. Based on the target information of each drone target, the tracking order of each drone target can be determined according to a preset evaluation logic. The tracking order of each drone target can be obtained by first calculating the priority score of each drone target based on its target information and then sorting them in descending order of priority score.

[0051] The priority score can be calculated as follows: Priority Score = α × Type Weight + β × Speed ​​Weight + γ × Direction Weight, where α, β, and γ are weight coefficients, which can be set to α = 0.5, β = 0.3, and γ = 0.2, respectively. In the type weight, large fixed-wing drones and multi-rotor drones carrying payloads can be assigned a value of 1.0, while small recreational multi-rotor drones can be assigned a value of 0.6. In the speed weight, a value of 1.0 is assigned when the speed is greater than 15 m / s, 0.6 is assigned when the speed is between 5 and 15 m / s, and 0.2 is assigned when the speed is less than 5 m / s. In the direction weight, a value of 1.0 is assigned when moving towards the core area, and 0.4 is assigned when moving towards the edge area.

[0052] For example, drone targets closer to the core protected area, flying faster, or identified as high-threat types are tracked earlier; low-threat drone targets farther from the core protected area or hovering are tracked later. This generates an ordered list containing all drone targets. The drone targets in this ordered list are sorted according to their tracking order.

[0053] This step fully leverages the wide field of view of panoramic lenses to achieve broad coverage detection in low-altitude areas. Through intelligent recognition and prioritization of panoramic images, it can quickly filter out the most threatening drone targets in complex scenarios with multiple drone intrusions. This solves the pain point of traditional narrow field-of-view electro-optical lenses, which struggle to quickly detect drone targets or easily lose drone targets outside the field of view, providing reliable guidance data for subsequent accurate tracking.

[0054] Then, step S2 is executed to track each UAV target according to the tracking order. During the tracking process, an electro-optical lens is introduced. An electro-optical lens is an imaging device with high-magnification optical zoom and gimbal rotation capabilities, used to acquire electro-optical images of the UAV target. The initial focal length of the electro-optical lens can be 10mm, and the tracking frame rate is 30 frames per second.

[0055] The photoelectric image refers to a high-resolution image captured by a photoelectric lens in telephoto narrow field-of-view mode.

[0056] The control parameters of this optoelectronic lens are adjustable. These parameters can include the focal length of the optoelectronic lens, the scanning frequency of the panoramic lens, the gimbal attitude, the gimbal rotation speed, the tracking frame rate, and exposure parameters. The gimbal attitude can include the horizontal rotation angle θ and the vertical pitch angle of the gimbal. The focal length of the optical lens can be adjusted from 10 to 300 mm, the horizontal rotation angle of the gimbal can be adjusted from 0 to 360°, and the vertical pitch angle can be adjusted from -45° to 90°. Exposure parameters can include exposure time, aperture size, and ISO.

[0057] The currently tracked target refers to the UAV target being tracked in an ordered list. For the currently tracked target, a coordinate mapping model is used to convert the pixel position of the currently tracked target under the panoramic lens to the pixel position under the photoelectric lens, so as to unify the pixel positions in the panoramic image and the photoelectric image and ensure the consistency of the UAV target's position.

[0058] In this step, the panoramic field-of-view coordinate system is the three-dimensional field-of-view coordinate system of the panoramic lens, used to describe the three-dimensional position and orientation of the target relative to the panoramic lens. The photoelectric field-of-view coordinate system is the three-dimensional field-of-view coordinate system of the photoelectric lens, used to describe the three-dimensional position and orientation of the target relative to the photoelectric lens. The pixel position under the panoramic lens refers to the two-dimensional position in the panoramic image acquired by the panoramic lens, and the pixel position under the photoelectric lens refers to the two-dimensional position in the photoelectric image acquired by the photoelectric lens.

[0059] A coordinate mapping model is a pre-defined mathematical model used to describe the spatial positional relationship and imaging mapping relationship between a panoramic lens and an optical lens. It can eliminate parallax caused by differences in installation position and viewing angle between the panoramic and optical lenses. The coordinate mapping model can be established using a first rotation matrix. and the first translation vector Represented as: ; Wherein, the first rotation matrix Describe the attitude transformation from the panoramic field of view coordinate system to the optical television field of view coordinate system. Describe the offset of the origin position between the panoramic field of view coordinate system and the optical television field of view coordinate system. (x1, y1, z1) is the position of point p1 in the panoramic field of view coordinate system, and (x2, y2, z2) is the position of point p2 in the panoramic field of view coordinate system.

[0060] The coordinate mapping model can be obtained by jointly calibrating the panoramic lens and the photoelectric lens, or by using a deep learning image registration and mapping scheme or a LiDAR-assisted coordinate mapping scheme. The appropriate model can be flexibly selected based on the complexity of lens installation, calibration difficulty, and accuracy requirements.

[0061] In the process of obtaining the coordinate mapping model through joint calibration of the panoramic lens and the photoelectric lens, taking the panoramic lens as an example obtained by stitching together 18 cameras, the panoramic lens and the photoelectric lens are installed and fixed to ensure that the monitoring areas of the panoramic lens and the photoelectric lens have an overlap coverage of ≥30%, thus ensuring the continuity of UAV target handover. After starting the target detection and tracking system, the panoramic lens and the photoelectric lens are jointly calibrated.

[0062] The joint calibration steps include: ① Intrinsic parameter calibration: This can be achieved using Zhang's calibration method. First, select a 20mm×20mm checkerboard calibration board and place it at different distance nodes within the monitoring area. Collect multiple sets of images of the calibration board under different poses of each camera and photoelectric lens in the panoramic lens. Then, use Zhang's calibration method with the OpenCV open-source library to calibrate and obtain the intrinsic parameter matrix of each camera. Distortion coefficient and the intrinsic parameter matrix of the photoelectric lens and distortion coefficient The focal length of each camera on the x-axis Focal length on the y-axis All values ​​are calibrated to pixel values ​​corresponding to a 75mm focal length, and the principal point coordinates u0 and v0 are close to the image center; The intrinsic parameter matrices of each camera can be represented using a pinhole camera model, i.e.: ; in, , This represents the pixel position of the principal point in the image.

[0063] ②Extrinsic parameter calibration among cameras in a panoramic lens: Images of the overlapping area of ​​the checkerboard calibration board in the field of view of 18 cameras in the panoramic lens are acquired. The corner pixel coordinates of the checkerboard calibration board are extracted, and the camera located at the center of the 18-camera array is selected as the reference camera. The extrinsic parameters of each camera, i.e., the second rotation matrix of each camera relative to the reference camera, are calculated by the least squares method. Second translation vector Complete the coordinate system of each camera in the panoramic lens to obtain a unified panoramic coordinate system O1-X1Y1Z1; Camera extrinsic parameters are used to describe the camera's position and orientation in the world coordinate system, and are expressed through a second rotation matrix. Second translation vector Indicates a point in three-dimensional space. to image pixels The projection relationship formula is as follows: Projection relationship formula: Where s is the scale factor, The second rotation matrix is ​​3×3. It is the second translation vector of 3×1.

[0064] ③ External parameter calibration of panoramic and electro-optical lenses: Place the checkerboard calibration board in the overlapping area of ​​the fields of view of the panoramic and electro-optical lenses, and simultaneously acquire the pixel positions of the checkerboard calibration board in the panoramic and electro-optical images. Extract the position of the UAV target in the panoramic coordinate system and the position in the electro-optical coordinate system, and solve the first rotation matrix between the panoramic and electro-optical lenses through an iterative optimization algorithm. and the first translation vector Then, a coordinate mapping model is established.

[0065] After establishing the coordinate mapping model, it can be stored in the storage unit, providing a foundation for the accurate conversion of the UAV target's position in the future.

[0066] Deep learning-based image registration and mapping solutions can be applied to dynamic deployment scenarios with complex or non-rigidly fixed lens mounting postures, such as low-altitude surveillance on mobile platforms, where traditional calibration methods struggle to guarantee accuracy. By leveraging the feature correspondence between panoramic images and UAV targets in photoelectric images through deep learning, end-to-end mapping of UAV target coordinates is achieved without complex calibration processes, adapting to flexible deployment requirements.

[0067] LiDAR-assisted coordinate mapping solutions can be applied to scenarios requiring high-precision target localization, such as long-range UAV positioning and accurate threat range assessment. By leveraging the high-precision ranging capabilities of LiDAR and establishing a coordinate mapping model, the target alignment accuracy and distance measurement accuracy of UAVs can be further improved, meeting the demands of high-precision positioning.

[0068] In step S2, the pixel position of the currently tracked target in the panoramic image is first obtained, and then transformed into the panoramic field-of-view coordinate system. To achieve field-of-view linkage between the panoramic lens and the photoelectric lens, a pre-obtained coordinate mapping model is called from the storage unit. This model is used to transform the position of the currently tracked target in the panoramic field-of-view coordinate system into the photoelectric field-of-view coordinate system. Furthermore, the position of the currently tracked target in the photoelectric field-of-view coordinate system is transformed into the pixel position under the photoelectric lens, realizing the data flow from panoramic detection to photoelectric aiming.

[0069] In this step, a pre-established coordinate mapping model enables field-of-view fusion and data alignment between heterogeneous panoramic and optoelectronic lenses. The coordinate transformation error is controlled within 0.1°, and the target handover time is reduced to less than 100ms. This overcomes the handover failure problem caused by the independence of the panoramic and optoelectronic systems and large coordinate transformation errors in existing technologies. Precise coordinate transformation guides the optoelectronic lens to quickly point to the location of the UAV target, significantly shortening the response time for UAV target handover and ensuring that high-speed moving UAV targets are not lost during lens switching.

[0070] Finally, step S3 is executed. Aligning the photoelectric lens with the currently tracked target means adjusting the gimbal attitude of the photoelectric lens to place the currently tracked target in the center area of ​​the photoelectric lens's field of view. The center area of ​​the field of view can be a neighborhood of 2 or 5 pixels from the center of the photoelectric lens's field of view.

[0071] By utilizing the pixel position of the currently tracked target under the photoelectric lens, the control parameters of the photoelectric lens can be adjusted to align the lens with the target. For example, the pixel position of the target can be used to generate control commands to drive the gimbal of the photoelectric lens to rotate rapidly, aligning the optical axis center of the lens with the target and bringing it into the lens's field of view. The rotation speed of the gimbal can be adaptively adjusted according to the target's flight speed. For instance, if the target's flight speed is >10 m / s, the gimbal rotation speed can be set to 40° / s; if the target's flight speed is ≤10 m / s, the gimbal rotation speed can be set to 20° / s.

[0072] The horizontal rotation angle θ and the vertical pitch angle of the gimbal It can be calculated using the following formula: , ; in,( This represents the current position of the tracked target in the optical television field coordinate system.

[0073] Simultaneously, based on the estimated distance between the current tracking target and the photoelectric lens, and combined with the size of the current tracking target in the photoelectric image, the target value of the focal length of the photoelectric lens is calculated and the focal length of the photoelectric lens is adjusted to the target value, so as to achieve adaptive adjustment of the focal length of the photoelectric lens.

[0074] Once the photoelectric lens captures the target, it immediately switches to a visual tracking mode based on the photoelectric image. In this mode, the photoelectric image can be analyzed in real time, and image tracking algorithms, such as correlation filtering or feature matching, can be used to continuously lock onto the target. The gimbal attitude is adjusted in a closed loop based on the pixel position deviation of the target in the photoelectric image to ensure that the target remains centered in the photoelectric lens's view, thereby obtaining a clear and detailed image of the target for further evidence collection or identification.

[0075] For image tracking algorithms, when dealing with highly nonlinear motion scenarios such as aerobatic maneuvers and evasive maneuvers of UAV targets, tracking algorithms adapted to nonlinear motion can be selected. These algorithms utilize multi-state simulation mechanisms to better adapt to complex maneuver trajectories, improving the tracking stability of highly maneuverable UAV targets and meeting the detection requirements of complex maneuver scenarios. For scenarios with limited computing power, such as portable devices, lightweight tracking algorithms can be selected to achieve quasi-adaptive adjustment of control parameters. This significantly reduces computing power consumption while ensuring basic control response performance, adapting to the needs of lightweight embedded deployments.

[0076] This step utilizes the high-magnification zoom capability of the photoelectric lens to achieve high-precision identification of small, distant drone targets. Through automatic adjustment of the control parameters of the photoelectric lens and visual closed-loop tracking, smooth locking of the drone target can be achieved under the guidance of the panoramic image. This solves the problem that although panoramic lenses have a large field of view, their low resolution makes it impossible to see details such as the drone model and payload. It truly achieves an organic combination of visibility and clarity.

[0077] The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided in this invention constructs a multi-scale field-of-view coordination mechanism, from coarse observation by a panoramic lens to fine observation by an optoelectronic lens. It leverages the wide coverage of the panoramic lens for large-scale UAV target search and priority ranking, utilizes a coordinate mapping model for precise cross-field-of-view guidance, and achieves accurate transmission of UAV target position information. Furthermore, it utilizes the telephoto advantage of the optoelectronic lens for detail capture and continuous tracking, significantly reducing the handover delay between the panoramic and optoelectronic lenses and preventing target loss. This method fundamentally solves the problems of large blind spots, missing target details, and insufficient dynamic tracking capabilities inherent in single-field-of-view lenses in low-altitude security scenarios. It significantly improves the detection efficiency and recognition accuracy of UAV targets, achieving seamless integration of large-scale low-altitude coverage and high-precision focused tracking. By performing fully automated detection and tracking of UAV targets, this method significantly reduces the workload of operators, minimizes the need for manual intervention, and enhances the applicability of the target detection and tracking system in unattended, all-weather security scenarios.

[0078] Existing control strategies are mostly based on fixed parameters, which results in poor system robustness and weak anti-interference ability. Regarding adaptation to the motion state of drone targets, the existing electro-optical lenses have a fixed tracking frame rate. When a drone target suddenly accelerates or dives from a hovering state, the fixed tracking frame rate cannot capture the change in the drone target's position in time, and cannot quickly adjust the focal length to adapt to the change in the size of the drone target, causing the drone target to go out of the field of view or the image to be blurry. Regarding adaptation to ambient lighting, the exposure parameters of the panoramic lens and the electro-optical lens are independent of each other. When there are strong direct light, backlight, low light at dawn or dusk, the exposure parameters cannot be adjusted in a coordinated manner, resulting in the panoramic lens being unable to effectively detect drone targets, or the electro-optical lens being unable to capture clear images. Regarding adaptation to complex backgrounds, the parameters of the existing target detection algorithm are fixed. When there are complex backgrounds such as swaying leaves, water reflections, or flocks of birds flying in low-altitude scenes, it is easy to misjudge the interference objects as drone targets, resulting in a high false alarm rate, or the background noise may mask the characteristics of small drones, leading to missed detections. The existing solutions cannot dynamically adjust the threshold and feature extraction strategy of the detection algorithm according to the complexity of the background, further reducing the reliability of the system in complex low-altitude environments. Especially for low-speed, small unmanned aerial vehicle (UAV) targets, existing solutions lack specific low signal-to-noise ratio feature enhancement mechanisms, resulting in severely insufficient detection capabilities.

[0079] Based on this, and building upon the above embodiments, the multi-scale field-of-view optoelectronic cooperative target detection and tracking method, wherein tracking the current target based on the optoelectronic image acquired by the optoelectronic lens includes: Calculate the motion parameters of the currently tracked target in the photoelectric image and the image quality parameters of the photoelectric image, and obtain the environmental parameters of the photoelectric lens; Based on the motion parameters, image quality parameters, and environmental parameters, with the joint optimization objectives of maximizing target tracking stability, optimizing imaging quality, and minimizing tracking power consumption, a reinforcement learning algorithm is used to adaptively adjust the control parameters of the photoelectric lens, and continuously track the current target based on the photoelectric lens.

[0080] Specifically, in this embodiment, an adaptive control strategy based on reinforcement learning algorithm is introduced in step S3 to cope with the complex and ever-changing low-altitude detection environment.

[0081] Since the motion parameters of the currently tracked target include parameters such as the target's flight speed, flight acceleration, flight direction, and flight altitude, when calculating the motion parameters of the currently tracked target in the photoelectric image, the target information of the currently tracked target can be identified by real-time analysis of the photoelectric images continuously acquired by the photoelectric lens through image recognition algorithms.

[0082] Image quality parameters refer to data indicators that evaluate the clarity and suitability of photoelectric images captured by photoelectric lenses for algorithm recognition. These mainly include image sharpness, contrast, and signal-to-noise ratio. By evaluating the quality of each frame of the photoelectric image and using sharpness evaluation algorithms, such as the Laplacian gradient function, the image quality parameters of the photoelectric image can be calculated.

[0083] Environmental parameters refer to the physical conditions of the external environment in which the photoelectric lens is located. These are mainly acquired in real time by sensors integrated into the photoelectric lens or gimbal, or quantified by analyzing the brightness distribution of the histogram of the photoelectric image. Environmental parameters include ambient light intensity, atmospheric visibility, and cloud thickness.

[0084] Subsequently, with the joint optimization objectives of maximizing target tracking stability, optimizing imaging quality, and minimizing tracking power consumption, the motion parameters, image quality parameters, and environmental parameters of the UAV target are used as state inputs, and the control parameters of the electro-optical lens are used as action outputs. The tracking success rate of the UAV target is then calculated. Image sharpness rating With system power consumption The weighted value is used as the reward function. Through offline training and online iterative optimization, the agent can autonomously learn the optimal control strategy under different environmental parameters, achieving dynamic adaptive adjustment of the control parameters. For example, when a drone target is detected to be changing direction rapidly or diving, the agent can autonomously increase the gimbal rotation response speed and optimize the tracking frame rate to avoid losing the drone target; when encountering cloud cover or sudden changes in lighting, it can simultaneously adjust the exposure parameters of the panoramic lens and the photoelectric lens to ensure stable drone target imaging quality.

[0085] The reward function can be expressed as: ; Among them, ω1, ω2, and ω3 are all weighting coefficients, which can be set to ω1=0.5, ω2=0.3, and ω3=0.2 respectively, to ensure that the system power consumption is reduced as much as possible while ensuring the tracking effect and imaging quality.

[0086] The formula for an agent to predict a state can be expressed as: in, The state prediction value at time k. Here is the state transition matrix. This is the state estimate at time k-1. To control the input matrix, This is the control input at time k-1. Let be the prediction error covariance matrix at time k. Let be the estimation error covariance matrix at time k-1. Let be the process noise covariance matrix.

[0087] The formula for an agent to update its state can be expressed as: in, The Kalman gain at time k, For the observation matrix, Observation noise covariance matrix, Let k be the observation value at time k. The state estimate at time k. Let k be the estimation error covariance matrix. It is an identity matrix.

[0088] Understandably, successful tracking can be indicated by the detection confidence of the currently tracked target in the photoelectric image being greater than or equal to a confidence threshold. This confidence threshold can be set as needed, for example, it can be set to 0.5.

[0089] Based on the agent's decision-making output using reinforcement learning, the control parameters of the photoelectric lens are dynamically and adaptively adjusted through linkage control. The core adjustment logic is optimized autonomously by the agent, eliminating the need for manual preset of fixed thresholds. The control effect in typical UAV detection scenarios is as follows: ① UAV maneuvering state adaptation scenario: When a UAV is detected to be accelerating rapidly (acceleration a > 2m / s²), 2 When the drone is diving or changing direction, the intelligent agent autonomously increases the tracking frame rate of the photoelectric lens and the rotation speed of the gimbal, while optimizing the focus adjustment to prevent the target from leaving the field of view; when the drone is hovering (|a|≤0.5m / s 2When the light intensity drops sharply (average brightness V < 80) or encounters fog or haze (visibility < 2km), the agent simultaneously increases the exposure time (maximum not exceeding 1 / 30 second to avoid image blurring) and aperture value to improve sensitivity. Simultaneously, it activates image denoising and dehazing enhancement algorithms, controlling the panoramic lens to adjust exposure parameters synchronously to ensure stable drone detection performance in panoramic images. When strong direct sunlight occurs (average brightness V > 250), the agent quickly reduces the exposure time and aperture value, activating the lens shading compensation mechanism to prevent overexposure and loss of drone features. Target occlusion scenarios: When the drone is detected to be partially obscured by clouds, leaves, or buildings (improving the target occlusion rate of YOLOv10 output to >30%), the agent appropriately increases the focal length, narrows the tracking field of view, and focuses on the unobstructed area of ​​the drone (such as the wings and tail). Simultaneously, it increases the tracking frame rate and combines Kalman filtering to predict the drone's trajectory, enhancing the ability to quickly re-identify the drone after the occlusion is removed. For example, when the target is a high-speed racing drone with a speed of 15 m / s or higher, the reinforcement learning agent can adjust the control strategy within 0.05 seconds, increasing the tracking frame rate to 60 frames / second and the gimbal rotation speed to 50° / s, ensuring the racing drone remains stably centered in the field of view. When nighttime light intensity drops sharply, the agent simultaneously adjusts the exposure parameters of both lenses to maintain the photoelectric image sharpness above 85 points and the panoramic image drone detection confidence level at no less than 0.4.

[0090] This reinforcement learning control technology significantly improves the adaptability to complex low-altitude scenarios and the flexible movement of UAV targets, and its tracking stability and robustness are significantly better than traditional control methods.

[0091] In this embodiment of the invention, through multi-dimensional parameter perception, the motion parameters of the currently tracked target, the image quality parameters of the photoelectric image, and the environmental parameters of the photoelectric lens can be comprehensively grasped. This changes the limitation of traditional photoelectric tracking, which relies solely on passive adjustment based on positional deviation. It provides rich and accurate data input for subsequent intelligent decision-making, enabling the target detection and tracking system to perceive environmental changes and the UAV's target intent. This method achieves a cognitive intelligent upgrade of photoelectric tracking by introducing reinforcement learning algorithms. Unlike traditional control methods based on fixed thresholds or PID rules, this method can handle complex high-dimensional and nonlinear scenarios. By incorporating stability, imaging quality, and power consumption into a unified optimization framework, a dynamic balance can be found between not losing the UAV target, high-quality photoelectric images, and energy efficiency. Especially when facing complex tactical maneuvers performed by the UAV or severe weather interference, this method exhibits strong robustness and adaptability, significantly reducing the tracking loss rate and improving the practical effectiveness of the target detection and tracking system. This solution significantly solves the problems of unstable tracking, blurred images, or excessive energy consumption caused by rigid control strategies in existing optoelectronic systems when facing highly maneuverable targets or complex lighting environments. It ensures continuous and high-definition locking of UAV targets under various extreme conditions, enabling the target detection and tracking system to maintain stable tracking performance in complex dynamic scenarios and improving its adaptability to various maneuverable targets.

[0092] Based on the above embodiments, determining the tracking order of each UAV target based on the target information of each UAV target further includes: determining the tracking level of each UAV target based on the target information of each UAV target; The coordinate mapping model based on the panoramic field-of-view coordinate system of the panoramic lens and the optical field-of-view coordinate system of the photoelectric lens converts the pixel position of the currently tracked target under the panoramic lens into the pixel position under the photoelectric lens, including: If the tracking level of the current tracked target is the target tracking level, then based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, the pixel position of the current tracked target in the panoramic image coordinate system is converted into the pixel position in the photoelectric image coordinate system.

[0093] Specifically, by utilizing the target information of each UAV target, the tracking level of each UAV target can also be determined. The tracking level of each UAV target can be obtained by grouping them according to their priority scores. For example, UAV targets with a priority score ≥ 0.8 are classified as first priority, UAV targets with a priority score between 0.4 and 0.8 are classified as second priority, and UAV targets with a priority score < 0.4 are classified as third priority. First priority can be a high threat level, second priority can be a medium threat level, and third priority can be a low threat level. First priority can correspond to large fixed-wing UAV targets, multi-rotor UAV targets carrying suspicious payloads, and UAV targets approaching the core area at high speed, etc.; second priority can correspond to small recreational multi-rotor UAVs, etc.; and third priority can correspond to UAVs that have deviated from the core area, etc.

[0094] Furthermore, in converting the pixel position of the currently tracked target under the panoramic lens to the pixel position under the photoelectric lens, the conversion can be performed only when the tracking level of the currently tracked target is at the target tracking level. That is, if the tracking level of the currently tracked target is at the non-target tracking level, no conversion is needed, and the panoramic lens can be used to continuously track the currently tracked target. The target tracking level can be set as needed, for example, it can be the first priority or the second priority, and the non-target tracking level can be the third priority.

[0095] In this embodiment of the invention, by introducing an intelligent hierarchical screening mechanism, limited photoelectric tracking resources can be concentrated on UAV targets at the target tracking level, effectively avoiding meaningless high-precision tracking of birds, kites, or low-threat UAV targets. This solves the problem of system resources being abused in complex scenarios with multiple UAV targets, leading to missed detection of key targets, and improves the system's task processing efficiency and security targeting.

[0096] Based on the above embodiments, the target information further includes a target threat level, and each of the UAV targets includes multiple designated targets with the same tracking level; the tracking of the current target based on the photoelectric image acquired by the photoelectric lens further includes: Based on the photoelectric image, predict the tracking difficulty coefficient of each of the specified targets; If the tracking level of each specified target is the target tracking level, and the distance between each specified target is greater than the preset distance, then the joint optimization target is to prioritize tracking the specified target with the highest target threat level and the highest tracking difficulty coefficient, and to maximize the tracking benefit, and to plan the time-sharing collaborative tracking sequence of the photoelectric lens for each specified target. According to the time-division collaborative tracking sequence, the photoelectric lens is controlled to perform time-division collaborative tracking of each of the designated targets.

[0097] Specifically, the target threat level refers to the threat level obtained through threat assessment of UAV targets, with all designated targets having the same tracking level. The tracking difficulty coefficient is a quantitative indicator reflecting the ease with which each designated target can be stably tracked, typically related to the target's maneuverability and the probability of environmental obstruction. The time-sharing cooperative tracking sequence refers to a time-scheduled plan for rapid switching of electro-optical lenses among multiple spatially dispersed UAV targets.

[0098] When a panoramic camera simultaneously detects multiple designated targets belonging to the target tracking level, the target detection and tracking system first uses a Kalman filter algorithm to predict the flight trajectory and position range of each designated target within a preset time period in the future. It then calculates the tracking difficulty coefficient by combining the target's flight speed, occlusion probability, target size, and maneuvering characteristics. For example, targets undergoing rapid dives or irregular maneuvers are considered to have a higher tracking difficulty coefficient.

[0099] The system determines the spatial distance between each designated target. If the distance between all designated targets is greater than a preset distance, it means that the electro-optical lens cannot simultaneously cover all designated targets within a single field of view. In this case, the target detection and tracking system will activate a time-sharing cooperative scheduling strategy. Using a greedy algorithm, it prioritizes tracking designated targets with high threat levels and high tracking difficulty, and prioritizes targets with the highest tracking benefits as joint optimization objectives. Specifically, it assigns higher benefit weights to UAV targets with high threat levels and high tracking difficulty. This generates a time-sharing cooperative tracking sequence, prioritizing the tracking of designated targets that are fast-flying, maneuverable, and heading towards the core area.

[0100] For example, when two high-threat designated targets, A and B, are far apart, and the flight speed and maneuvering frequency of designated target A are higher than those of designated target B, the electro-optical lens is prioritized for short-term, high-frequency tracking of the most threatening and maneuverable designated target A. Then, the tracking is quickly switched to designated target B, and after completing one round of evidence collection, it is rapidly switched back to designated target A, ensuring that the switching time between different targets is kept extremely short. During time-sharing coordinated scheduling, the switching time between two adjacent designated targets is less than 150ms, and the tracking duration for each designated target is 0.5 seconds or continuously acquiring 3 frames of electro-optical images. The preset distance can be set as needed, for example, to 100 meters.

[0101] Understandably, during the target switching process, the agent predicts the position and motion parameters of the next target in advance and determines the control parameters of the electro-optic lens. Then, through the control parameters of the electro-optic lens, it controls the photoelectric lens to quickly switch to the next target.

[0102] In this embodiment of the invention, a time-sharing scheduling strategy based on maximizing benefits breaks through the limitation of traditional single photoelectric lenses that can only track one target at a time. When facing saturation attack scenarios involving multiple dispersed drone targets, the system can intelligently allocate observation time windows based on the threat and difficulty level of the drone targets, achieving near-simultaneous monitoring of multiple dispersed drone targets by the photoelectric lens, maximizing hardware efficiency, and ensuring that high-risk targets are not lost.

[0103] Existing detection and tracking technologies can only passively receive the position information of drone targets in panoramic lenses and cannot predict the flight trajectory of drone targets. When drone targets undergo rapid changes in motion such as changing direction or accelerating, photoelectric lenses cannot adjust their tracking strategies in advance, further increasing the probability of tracking loss.

[0104] Based on the above embodiments, the step of tracking the current target based on the photoelectric image acquired by the photoelectric lens further includes: Based on the photoelectric images, predict the flight trajectories of each of the specified targets; If the tracking level of each specified target is the target tracking level, the distance between each specified target is less than or equal to the preset distance, and the error between the flight trajectories of each specified target is less than the preset error, then the control parameters of the photoelectric lens are adjusted so that each specified target simultaneously enters the field of view of the photoelectric lens, and the specified targets are synchronously tracked based on the photoelectric lens.

[0105] Specifically, a flight trajectory refers to the line connecting the spatial position changes of a designated target over a period of time, which can represent the path changes of a designated target performing agile maneuvers such as hovering, diving, and changing direction. Simultaneous tracking refers to locking onto and monitoring multiple designated targets simultaneously within the same field of view of an electro-optical lens.

[0106] When tracking multiple designated targets, the target detection and tracking system continuously analyzes the movement trends of each target. When the distance between multiple designated targets at the target tracking level is less than or equal to a preset distance, and the error between the flight trajectories of each target is less than a preset error (i.e., the flight trajectories of the targets are highly similar), the system abandons the time-sharing strategy and dynamically adjusts the control parameters of the electro-optical lens. Specifically, it sends control commands to reduce the focal length of the electro-optical lens to expand the field of view until the calculated field of view can simultaneously accommodate all the designated targets. Subsequently, in the wide field of view mode of the electro-optical lens, a multi-target tracking algorithm is used to synchronously track and update the status of all designated targets within the electro-optical image. When a new UAV target appears, the system promptly determines the target information and tracking order of the UAV target.

[0107] In this embodiment of the invention, an adaptive field-of-view adjustment scheme is provided for special scenarios such as UAV formation flight or close-range cluster intrusion. By expanding the field of view to achieve synchronous tracking, the gimbal wear and image shake caused by frequent mechanical rotation of the photoelectric lens between close targets are avoided, ensuring continuous control over the overall situation of the enemy formation and improving the stability of tracking and the integrity of intelligence.

[0108] Based on the above embodiments, the method further includes: If the tracking level of the current target is a non-target tracking level, then the current target is tracked based on the panoramic image.

[0109] Specifically, when the target detection and tracking system classifies the currently tracked target as a non-target tracking level, it will not send control commands to the photoelectric lens. In this case, it can continuously track the target within the panoramic field of view using only the panoramic image captured by the panoramic lens. The target detection and tracking system can also mark the currently tracked target with a specific low-threat identifier, such as a yellow box, on the panoramic monitoring screen and continuously update its location information until it flies out of the monitoring area or its threat level changes.

[0110] In this embodiment of the invention, a tiered resource release mechanism is constructed. For a large number of low-threat targets, only the panoramic lens is used for escort surveillance, without occupying the electro-optical lens. This ensures that the electro-optical lens can always be in standby mode or focused on tracking high-threat targets, achieving optimal allocation of system computing and hardware resources.

[0111] Based on the above embodiments, the control parameters include gimbal attitude and lens focal length; adjusting the control parameters of the photoelectric lens based on the pixel position of the current tracked target under the photoelectric lens to align the photoelectric lens with the current tracked target includes: Based on the pixel position of the currently tracked target under the photoelectric lens, calculate the deviation between the currently tracked target and the field of view center of the photoelectric lens; Based on the deviation, the gimbal of the photoelectric lens is controlled to move until the currently tracked target is located in the center area of ​​the field of view of the photoelectric lens.

[0112] Specifically, the target detection and tracking system first adaptively calculates the initial focal length of the electro-optical lens based on the target distance and size information detected by the panoramic field of view. Simultaneously, it sends a control signal containing a focal length adjustment command to the electro-optical lens, driving the lens zoom motor to achieve the preset focal length. This process ensures that the electro-optical lens is at the optimal magnification level the moment it intervenes in tracking, avoiding the problems of insufficient target visibility due to too short a focal length or difficulty in searching due to too long a focal length.

[0113] Ideally, when the photoelectric lens is aligned with the currently tracked target, the target should coincide with the optical axis of the lens. Deviation refers to the difference in the horizontal and vertical directions between the pixel position of the target under the lens and the center of the field of view. This deviation reflects the angular error between the current orientation of the lens and the actual orientation of the target.

[0114] The calculated deviation is used to generate control commands, which are then used to control the gimbal of the photoelectric lens to move and change its attitude until the currently tracked target is located in the center of the field of view of the photoelectric lens, thus completing the precise alignment between the photoelectric lens and the currently tracked target.

[0115] Based on the above embodiments, when the UAV target flies away from the monitoring area, or the tracking level is reduced to the non-target tracking level for more than 10 seconds, or the operator issues a tracking end command, the photoelectric lens is controlled to stop tracking and adjust to the initial working state, the panoramic lens continues to perform large-scale low-altitude monitoring, and the target detection and tracking system returns to the initial detection mode to wait for the detection and tracking of the next UAV target.

[0116] Based on the above embodiments, existing multi-lens UAV detection solutions are mostly simple superpositions of panoramic and optoelectronic systems, lacking a unified hardware integration architecture and software management platform. This results in high system complexity, poor stability, and high operation and maintenance costs. At the hardware level, the two systems use independent power modules, data transmission modules, and control modules, leading to bulky equipment and complex wiring. This makes them inflexible in space-constrained scenarios such as airport runway perimeters and hilltop surveillance points. Furthermore, incompatibility issues exist between panoramic and optoelectronic systems from different manufacturers, making system integration difficult and prone to hardware conflicts. At the software level, the control interfaces of the two systems are independent, requiring operators to monitor the operating status of both systems simultaneously and manually perform a series of operations such as UAV target detection, coordinate transfer, and lens switching. This not only increases the workload of operators but also increases the risk of target loss due to human error. In addition, system maintenance requires separate calibration and repair of both systems, resulting in a complex maintenance process and maintenance costs 2-3 times higher than a single system. This low integration design makes existing multi-lens solutions unsuitable for unattended, 24 / 7 continuous UAV detection scenarios, limiting their application in low-altitude security.

[0117] Based on this, such as Figure 2 As shown in the embodiment of the present invention, the target detection and tracking method based on multi-scale field of view optoelectronic coordination is provided in the hardware layer, which integrates a panoramic lens power supply module and an optoelectronic lens power supply module, respectively used to power the panoramic lens and the optoelectronic lens.

[0118] At the device layer, there are global lenses and photoelectric lenses, which are used to acquire panoramic images and photoelectric images, respectively.

[0119] At the business layer, this includes data processing processes such as target detection and tracking, coordinate mapping model calibration, and tracking level assessment, as well as linkage control processes such as focal length adjustment of the optoelectronic lens and gimbal attitude tracking of the optoelectronic lens.

[0120] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a target detection and tracking device based on multi-scale field-of-view optoelectronic coordination, comprising: The target detection module 31 is used to acquire panoramic images captured by the panoramic lens, identify the target information of each UAV target in the panoramic image, and determine the tracking order of each UAV target based on the target information of each UAV target. The coordinate mapping module 32 is used to track each of the UAV targets according to the tracking order of each UAV target, and for the current tracked target, based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, convert the pixel position of the current tracked target under the panoramic lens into the pixel position under the photoelectric lens. The target tracking module 33 is used to adjust the control parameters of the photoelectric lens based on the pixel position of the current tracking target under the photoelectric lens, so that the photoelectric lens is aligned with the current tracking target, and to track the current tracking target based on the photoelectric image acquired by the photoelectric lens.

[0121] Specifically, the functions of each module in the multi-scale field-of-view optoelectronic cooperative target detection and tracking device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-mentioned method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0122] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided in the above embodiments.

[0123] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided in the above embodiments.

[0125] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and no specific limitation is made herein.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection and tracking method based on multi-scale field-of-view optoelectronic coordination, characterized in that, include: The system acquires panoramic images captured by a panoramic lens, identifies target information of each UAV target in the panoramic images, and determines the tracking order of each UAV target based on the target information of each UAV target. According to the tracking order of each UAV target, each UAV target is tracked, and for the current tracked target, based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, the pixel position of the current tracked target under the panoramic lens is converted into the pixel position under the photoelectric lens. Based on the pixel position of the current tracking target under the photoelectric lens, the control parameters of the photoelectric lens are adjusted to align the photoelectric lens with the current tracking target, and the current tracking target is tracked based on the photoelectric image acquired by the photoelectric lens.

2. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to claim 1, characterized in that, The tracking of the current target based on the photoelectric image acquired by the photoelectric lens includes: Calculate the motion parameters of the currently tracked target in the photoelectric image and the image quality parameters of the photoelectric image, and obtain the environmental parameters of the photoelectric lens; Based on the motion parameters, image quality parameters, and environmental parameters, with the joint optimization objectives of maximizing target tracking stability, optimizing imaging quality, and minimizing tracking power consumption, a reinforcement learning algorithm is used to adaptively adjust the control parameters of the photoelectric lens, and continuously track the current target based on the photoelectric lens.

3. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to claim 1, characterized in that, The step of determining the tracking order of each UAV target based on the target information of each UAV target further includes: determining the tracking level of each UAV target based on the target information of each UAV target; The coordinate mapping model based on the panoramic field-of-view coordinate system of the panoramic lens and the optical field-of-view coordinate system of the photoelectric lens converts the pixel position of the currently tracked target under the panoramic lens into the pixel position under the photoelectric lens, including: If the tracking level of the current tracked target is the target tracking level, then based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, the pixel position of the current tracked target in the panoramic image coordinate system is converted into the pixel position in the photoelectric image coordinate system.

4. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to claim 3, characterized in that, The target information also includes the target threat level, and each of the UAV targets includes multiple designated targets with the same tracking level; the tracking of the current target based on the photoelectric image acquired by the photoelectric lens also includes: Based on the photoelectric image, predict the tracking difficulty coefficient of each of the specified targets; If the tracking level of each specified target is the target tracking level, and the distance between each specified target is greater than the preset distance, then the joint optimization target is to prioritize tracking the specified target with the highest target threat level and the highest tracking difficulty coefficient, and to maximize the tracking benefit, and to plan the time-sharing collaborative tracking sequence of the photoelectric lens for each specified target. According to the time-division collaborative tracking sequence, the photoelectric lens is controlled to perform time-division collaborative tracking of each of the designated targets.

5. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to claim 4, characterized in that, The tracking of the current target based on the photoelectric image acquired by the photoelectric lens further includes: Based on the photoelectric images, predict the flight trajectories of each of the specified targets; If the tracking level of each specified target is the target tracking level, the distance between each specified target is less than or equal to the preset distance, and the error between the flight trajectories of each specified target is less than the preset error, then the control parameters of the photoelectric lens are adjusted so that each specified target simultaneously enters the field of view of the photoelectric lens, and the specified targets are synchronously tracked based on the photoelectric lens.

6. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to claim 3, characterized in that, The method further includes: If the tracking level of the current target is a non-target tracking level, then the current target is tracked based on the panoramic image.

7. The target detection and tracking method based on multi-scale field-of-view optoelectronic coordination according to any one of claims 1-6, characterized in that, The control parameters include gimbal attitude and lens focal length; adjusting the control parameters of the photoelectric lens based on the pixel position of the current tracked target under the photoelectric lens to align the photoelectric lens with the current tracked target includes: Based on the pixel position of the currently tracked target under the photoelectric lens, calculate the deviation between the currently tracked target and the field of view center of the photoelectric lens; Based on the deviation, the gimbal of the photoelectric lens is controlled to move until the currently tracked target is located in the center area of ​​the field of view of the photoelectric lens.

8. A target detection and tracking device based on multi-scale field-of-view optoelectronic coordination, characterized in that, include: The target detection module is used to acquire panoramic images captured by the panoramic lens, identify the target information of each UAV target in the panoramic image, and determine the tracking order of each UAV target based on the target information of each UAV target. The coordinate mapping module is used to track each UAV target according to the tracking order of each UAV target, and for the current tracked target, based on the coordinate mapping model between the panoramic field of view coordinate system of the panoramic lens and the optical field of view coordinate system of the photoelectric lens, convert the pixel position of the current tracked target under the panoramic lens into the pixel position under the photoelectric lens. The target tracking module is used to adjust the control parameters of the photoelectric lens based on the pixel position of the current tracking target under the photoelectric lens, so that the photoelectric lens is aligned with the current tracking target, and to track the current tracking target based on the photoelectric image acquired by the photoelectric lens.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target detection and tracking method based on multi-scale field-of-view optoelectronic coordination as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target detection and tracking method based on multi-scale field of view optoelectronic coordination as described in any one of claims 1-7.

Images