A method and device for automatically guiding, identifying and tracking a target

By using YOLOv5s and DeepSORT algorithms for multi-object recognition and tracking on a small unmanned flight platform, combining the servo system of the photoelectric pod and laser rangefinder, the target locking and tracking problems of small unmanned flight platforms in complex backgrounds are solved. , achieving high precision and high efficiency target tracking strikes.

CN115047903BActive Publication Date: 2024-11-29BEIJING MECHANICAL EQUIP INST
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210490085.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-07
Publication Date
2024-11-29
Estimated Expiration
2042-05-07

AI Technical Summary

Technical Problem

The existing TV guidance technology is difficult to achieve autonomous intelligent detection and locking targets on small unmanned flight platforms, and in complex contexts, the target types are diverse and maneuverable, making it difficult to lock and track stably.

Method used

The YOLOv5s algorithm and DeepSORT algorithm are used for multi-object recognition and tracking, and the servo system of the photoelectric pod controls the camera to rotate with the target movement to achieve tracking of multiple targets. For single targets, the scale estimation algorithm and background suppression regularization algorithm are used for tracking, and the target distance is calculated through a laser rangefinder to achieve tracking and strike targets.

Benefits of technology

It realizes automatic guidance identification and target tracking of small unmanned flight platforms in complex contexts, improves target detection accuracy and tracking speed, and is suitable for multi-type targets and multi-objective locking tasks of the same type.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115047903B_ABST
    Figure CN115047903B_ABST
Patent Text Reader

Abstract

The present disclosure is about a method, device, electronic device and storage medium for automatically guiding, identifying and tracking targets. The present disclosure uses an automatic target detection and tracking guidance algorithm: for target detection, based on the YOLOv5 recognition model, a lightweight feature extraction basic network is constructed to improve the speed of target detection on the airborne end; for multi-target problems, the DeepSORT algorithm is combined to perform inter-frame multi-target association; for single-target stable tracking, based on the ECO correlation filter tracking algorithm, the scale feature extraction calculation is optimized, combined with the Kalman filter to predict the motion trajectory, narrow the search range, and improve the tracking speed; for the problem of target mobility and loss, a target occlusion and loss judgment strategy is designed, and the filter training is performed based on the identified target in the target detection stage to improve the tracking accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] TV guidance technology was first used in terminal guidance of missiles. The US White Star Eye 1 / 2, GBU-15(V)1 / B guided bombs, etc. all use TV guidance technology. In recent years, with the development of unmanned aerial platforms towards attack, miniaturization, and low cost, and the increasing maturity of image processing technology, countries around the world are competing to use TV guidance technology to develop remote guided strike or direct attack flight platforms.

[0003] This type of low-altitude small unmanned aerial platform generally has a wingspan of between 15cm and 30cm. Currently, most similar products are used to perform ground attack missions. Their guidance systems mainly include radar and television guidance. Radar-guided attack systems have been applied to disposable small attack drones very early on. Since they emit a large amount of electromagnetic waves, they are easily affected by electronic interference, and their combat capabilities are greatly limited. Television guidance can obtain rich target information, which is conducive to the search for targets in complex backgrounds. Applying television guidance technology to small unmanned aerial platforms has become a development trend.

[0004] The TV guidance uses the camera installed at the front of the flight platform as a sensor to detect the target, measure the angular coordinates of the target in space, lock and track the target, and then control the platform guidance and attack the target. According to its function, it is divided into two guidance modes: semi-automatic "man in the loop" and fully automatic "fire and forget". Semi-automatic guidance requires human control from the entire process from target locking to hitting. This method is inefficient and requires a large amount of manpower. Once the information transmission loop is interrupted, it will directly lead to system paralysis. Fully automatic guidance is that after the shooter aims and locks the target, the missile completes the task of target tracking and attacking by itself. This method is highly efficient, especially in the application of anti-tank missiles in a single-soldier combat environment, it can effectively protect the shooter's life safety. This guidance method has high requirements for the design of the target tracking algorithm.

[0005] At present, semi-automatic TV guidance technology is relatively mature, and is mainly used in large missiles to strike ground targets. Its platform has a long flight distance, a long guidance time, a long reserved judgment decision and correction time, and the guidance process can be manually zoomed in time to clearly observe the target. The commonly equipped seeker has high performance and is expensive. The target size is large (such as pedestrians, vehicles, buildings, etc.), the relative movement speed is slow, and it is easy to lock. The existing fully automatic TV guidance technology has simple application targets and scenes, and mostly uses target detection methods such as statistical pattern recognition, template matching, and support vector machine training classification. It has low intelligence, reliability, and sensitivity, and is not suitable for target locking under complex backgrounds.

[0006] However, for small flight platforms with limited payload, especially those with cost constraints, the existing TV guidance algorithms have the following shortcomings:

[0007] 1) Unable to detect and lock the target autonomously: Even with the existing fully automatic TV guidance technology, the initial target locking is mostly completed by the shooter aiming, and the commonly used intelligent detection methods are mostly traditional image processing methods such as background segmentation and moving target detection. Their detection accuracy is low and they cannot accurately identify the target type, let alone deal with the situation where there are multiple targets in the field of view;

[0008] 2) Target types are diverse and flexible, making it difficult to stably lock on targets: This type of small flying platform can be used to deal with aerial and ground targets within a small range (≤4km). Unlike vehicles or fixed buildings, aerial rotor drones and other typical "low, slow and small" targets, and ground pedestrians, they are flexible in movement and do not conform to the laws of linear motion. Simple linear trajectory prediction makes it difficult to stably lock on targets;

[0009] 3) The target is blurred and has large scale changes, making it difficult to continuously track the target: Considering the short time (2-3 seconds) reserved after the flight platform locks the target, the camera zoom response is not timely and the cost issue, a fixed-focus camera is used. This selection also makes it impossible to always capture clear targets during the guidance process, and the target scale changes greatly, making continuous tracking difficult.

[0010] In the prior art, there is a method for detecting and tracking multiple targets in an air-space background (publication number: CN107993245B), which is applied to air-space targets. Although multi-target detection and tracking is achieved, a simple binary segmentation method based on image color channels is used to detect targets, which has low reliability and is not suitable for target detection in complex ground environments. A multi-target detection and tracking method based on YOLOv3 multi-Bernoulli video (publication number: CN110084831) combines YOLOv3 and multi-Bernoulli filtering to achieve detection and tracking of multiple targets. The detection algorithm has a large amount of calculation and is not suitable for implementation on an airborne computing end. The tracking algorithm considers the correlation between detection targets more and is not suitable for continuous tracking of a single target with large scale changes without detection assistance.

[0011] Therefore, one or more methods are needed to solve the above problems.

[0012] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0013] The purpose of the present disclosure is to provide a method, device, electronic device and computer-readable storage medium for automatically guiding, identifying and tracking a target, thereby overcoming one or more problems caused by limitations and defects of related technologies at least to a certain extent.

[0014] According to one aspect of the present disclosure, a method for automatically guiding, identifying and tracking a target is provided, comprising:

[0015] A target search step, wherein the servo system of the optoelectronic pod of the flying platform controls the rotation of the camera of the flying platform based on a preset scanning control logic to achieve a full-range search for the target;

[0016] A multi-target detection step, after the camera of the flight platform finds the target, the multi-target is identified based on the YOLOv5s algorithm and the DeepSORT algorithm, and the camera of the flight platform is controlled to rotate with the target through the servo system of the optoelectronic pod of the flight platform to achieve tracking of the multi-target;

[0017] The single target tracking and striking step, after determining one of the multiple targets as the strike target, enables the servo system of the optoelectronic pod of the flying platform to control the camera to collect the image of the strike target, and tracks the strike target based on the scale estimation algorithm and the background suppression regularization algorithm, and calculates the distance between the flying platform and the strike target based on the laser rangefinder to track and strike the strike target.

[0018] In an exemplary embodiment of the present disclosure, the multi-target detection step of the method includes:

[0019] After the camera of the flying platform finds the target and identifies multiple targets, the camera of the flying platform is controlled by the servo system of the optoelectronic pod of the flying platform to rotate along with the movement of the target, thereby tracking the multiple targets;

[0020] Enables tracking filter training to complete the learning of target features.

[0021] In an exemplary embodiment of the present disclosure, the method further includes:

[0022] After the camera of the flight platform finds the target and identifies multiple targets, feature extraction is performed on the images of the multiple targets based on the YOLOv5s fast detection network;

[0023] Improving target attention for the features based on a channel attention module and a spatial attention module;

[0024] Multiple targets are tracked and identified based on the DeepSORT algorithm.

[0025] In an exemplary embodiment of the present disclosure, the single target tracking and striking step of the method further includes:

[0026] When tracking the strike target, the search area of ​​the tracking algorithm is predicted and adjusted based on the target scale change rate of the image of the strike target.

[0027] In an exemplary embodiment of the present disclosure, the method further includes:

[0028] When tracking the strike target, the target feature calculation size limit value is adjusted based on the distance between the flying platform and the strike target calculated by the laser rangefinder.

[0029] In an exemplary embodiment of the present disclosure, the method further includes:

[0030] Based on the background suppression regularization algorithm, according to the tracking response result of the image of the strike target, the regularization coefficient matrix of the background suppression regularization algorithm is adaptively adjusted to achieve background suppression of the strike target.

[0031] In an exemplary embodiment of the present disclosure, the method further includes:

[0032] Performing loss / occlusion determination on the strike target based on the peak sidelobe ratio calculated by the ECO tracking algorithm and the historical position and trajectory prediction of the strike target;

[0033] When it is determined to be lost, it changes to a global search for the target;

[0034] When it is determined to be blocked, the search for the strike target continues according to the historical position and trajectory prediction of the strike target.

[0035] In an exemplary embodiment of the present disclosure, the single target tracking and striking step of the method further includes:

[0036] After one of the multiple targets is determined as a strike target, a servo system of an optoelectronic pod of the flight platform controls a camera to collect an image of the strike target, and tracks the strike target based on a scale estimation algorithm and a background suppression regularization algorithm;

[0037] The distance between the flying platform and the strike target is calculated based on a laser rangefinder, and the three-dimensional coordinates of the strike target are calculated based on geodetic coordinates, so as to achieve tracking and striking of the strike target.

[0038] In one aspect of the present disclosure, there is provided a device for automatically guiding, identifying and tracking a target, comprising:

[0039] A target search module, used to enable the servo system of the optoelectronic pod of the flying platform to control the rotation of the camera of the flying platform based on a preset scanning control logic, so as to realize a full-range search for the target;

[0040] A multi-target detection module is used to identify multiple targets based on the YOLOv5s algorithm and the DeepSORT algorithm after the camera of the flight platform finds the target, and control the camera of the flight platform to rotate with the target through the servo system of the optoelectronic pod of the flight platform to achieve tracking of multiple targets;

[0041] The single target tracking and striking module is used to control the camera of the servo system of the optoelectronic pod of the flying platform to collect the image of the striking target after one of the multiple targets is determined as the striking target, and to track the striking target based on the scale estimation algorithm and the background suppression regularization algorithm, and to calculate the distance between the flying platform and the striking target based on the laser rangefinder to track and strike the striking target.

[0042] In one aspect of the present disclosure, there is provided an electronic device, comprising:

[0043] Processor; and

[0044] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of the above items.

[0045] In one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above items is implemented.

[0046] An exemplary embodiment of the present invention discloses a method for automatically guiding, identifying and tracking targets. The present invention uses an automatic target detection and tracking guidance algorithm: for target detection, based on the YOLOv5 recognition model, a lightweight feature extraction basic network is constructed to improve the target detection speed on the airborne side; for multi-target problems, the DeepSORT algorithm is combined to perform inter-frame multi-target association; for single-target stable tracking, based on the ECO correlation filter tracking algorithm, the scale feature extraction calculation is optimized, combined with the Kalman filter to predict the motion trajectory, narrow the search range, and improve the tracking speed; for the problem of target mobility and loss, a target occlusion and loss judgment strategy is designed, and filter training is performed based on the identified target in the target detection stage to improve tracking accuracy.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.

[0049] Figure 1A flowchart of a method for automatically guiding, identifying and tracking a target according to an exemplary embodiment of the present disclosure is shown;

[0050] Figure 2A-2B A flow chart of a target detection and tracking guidance algorithm of a method for automatically guiding and identifying a tracking target according to an exemplary embodiment of the present disclosure is shown;

[0051] Figure 3 A hardware system and functional diagram of a guidance algorithm for a method of automatically guiding, identifying and tracking a target according to an exemplary embodiment of the present disclosure are shown;

[0052] Figure 4 A schematic block diagram of a device for automatically guiding, identifying and tracking a target according to an exemplary embodiment of the present disclosure is shown;

[0053] Figure 5 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure; and

[0054] Figure 6 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0056] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0058] In this exemplary embodiment, a method for automatically guiding, identifying and tracking a target is first provided; Figure 1 As shown in, the method for automatically guiding, identifying and tracking a target may include the following steps:

[0059] The target search step S110 is to enable the servo system of the optoelectronic pod of the flying platform to control the rotation of the camera of the flying platform based on a preset scanning control logic to achieve a full range search for the target;

[0060] In a multi-target detection step S120, after the camera of the flight platform finds the target, the multi-targets are identified based on the YOLOv5s algorithm and the DeepSORT algorithm, and the camera of the flight platform is controlled to rotate with the target through the servo system of the optoelectronic pod of the flight platform to achieve tracking of the multi-targets;

[0061] In the single target tracking and striking step S130, after one of the multiple targets is determined as the strike target, the servo system of the optoelectronic pod of the flying platform controls the camera to collect the image of the strike target, and tracks the strike target based on the scale estimation algorithm and the background suppression regularization algorithm, and calculates the distance between the flying platform and the strike target based on the laser rangefinder to track and strike the strike target.

[0062] An exemplary embodiment of the present invention discloses a method for automatically guiding, identifying and tracking targets. The present invention uses an automatic target detection and tracking guidance algorithm: for target detection, based on the YOLOv5 recognition model, a lightweight feature extraction basic network is constructed to improve the target detection speed on the airborne side; for multi-target problems, the DeepSORT algorithm is combined to perform inter-frame multi-target association; for single-target stable tracking, based on the ECO correlation filter tracking algorithm, the scale feature extraction calculation is optimized, combined with the Kalman filter to predict the motion trajectory, narrow the search range, and improve the tracking speed; for the problem of target mobility and loss, a target occlusion and loss judgment strategy is designed, and filter training is performed based on the identified target in the target detection stage to improve tracking accuracy.

[0063] Next, a method for automatically guiding, identifying and tracking a target in this example embodiment will be further described.

[0064] In the embodiment of this example, a low-altitude fast flying platform is used as a carrier, with a flight speed of 30m / s-50m / s, a maximum flight distance of not less than 5km, and a maximum flight altitude of not less than 500m. A "visible light + laser ranging" pod is used as a detection sensor, and an automatic target detection and tracking guidance algorithm is designed: for target detection, a lightweight feature extraction basic network is constructed based on the YOLOv5 recognition model to improve the airborne target detection speed; for multi-target problems, the DeepSORT algorithm is combined to perform inter-frame multi-target association; for single-target stable tracking, the ECO correlation filter tracking algorithm is used to optimize the scale feature extraction calculation, combined with the Kalman filter to predict the motion trajectory, narrow the search range, and improve the tracking speed; for the problem of target maneuverability and loss, a target occlusion and loss judgment strategy is designed, and the filter training is performed based on the identified target in the target detection stage to improve the tracking accuracy.

[0065] In the embodiment of this example, the guidance algorithm based on target detection and tracking is used for a small unmanned aerial platform. After visually discovering a target, the initial stage is to detect the target's GPS position through a ground optoelectronic or radar system, guide the unmanned aerial platform to launch close to the target, and control the visible light camera to rotate through the servo system to intelligently identify and search for the target, thus entering the television guidance stage.

[0066] In this exemplary embodiment, if Figure 2A The target detection, tracking and guidance flow chart is shown in Figure 1. The process from the unmanned aerial platform approaching the target after launch to the target attack can be divided into three stages:

[0067] 1) Target search phase: This process starts the target detection function. The target has not yet entered the camera's field of view. The servo system controls the camera to rotate over a large range to search for the target according to the preset scanning control logic.

[0068] 2) Target discovery and stable detection stage: Once the target is detected, the multi-target detection and tracking function is turned on. At this time, the servo system rotates according to the concentrated position of the multiple targets to ensure that the target appears in the camera's field of view. At the same time, the training of the tracking filter is turned on to learn the target features in advance.

[0069] 3) Target locking and stable tracking stage: when the long-term single-target stable tracking function is turned on, the servo system controls the camera to lock the target according to the target movement, the laser rangefinder calculates the target distance, and calculates the target's three-dimensional coordinates in real time after a series of positioning conversions such as the optoelectronic pod, flight platform, and geodetic coordinates to track and strike the target.

[0070] Among them, the methods of switching to single-target stable tracking include: 1. Intelligent switching, detecting and tracking single targets or multiple targets according to preset strategies, judging and finding the optimal attack target, and the number of stable target detection frames reaches the threshold requirement; 2. Manual switching, the ground shooter manually selects the target according to the returned target detection image and switches to the target tracking stage.

[0071] In the target search step S110, the servo system of the optoelectronic pod of the flying platform may control the rotation of the camera of the flying platform based on a preset scanning control logic to achieve a full-range search of the target.

[0072] In the multi-target detection step S120, after the camera of the flying platform finds the target, the multi-targets can be identified based on the YOLOv5s algorithm and the DeepSORT algorithm, and the camera of the flying platform can be controlled by the servo system of the optoelectronic pod of the flying platform to rotate with the target to achieve tracking of the multi-targets.

[0073] In the embodiment of this example, the multi-target detection step of the method includes:

[0074] After the camera of the flying platform finds the target and identifies multiple targets, the camera of the flying platform is controlled by the servo system of the optoelectronic pod of the flying platform to rotate along with the movement of the target, thereby tracking the multiple targets;

[0075] Enables tracking filter training to complete the learning of target features.

[0076] In the embodiment of this example, the method further includes:

[0077] After the camera of the flight platform finds the target and identifies multiple targets, feature extraction is performed on the images of the multiple targets based on the YOLOv5s fast detection network;

[0078] Improving target attention for the features based on a channel attention module and a spatial attention module;

[0079] Multiple targets are tracked and identified based on the DeepSORT algorithm.

[0080] In the embodiment of this example, for multi-target detection, a lightweight detection model is designed mainly based on the limited performance of the onboard computing unit and the small target characteristics of the application scenario. The implementation approach is to use the YOLOv5s model as the basic framework, perform lightweight design on the feature extraction part, introduce the attention mechanism to optimize the accuracy of small target detection, and combine it with DeepSORT to achieve inter-frame same-target trajectory association, that is, multi-target tracking.

[0081] In the embodiment of this example, the lightweight design of the target extraction basic network includes: based on the end-to-end YOLOv5s fast detection network, the classic lightweight network MobileNet is optimized and designed to replace the feature extraction basic network, specifically including: for the problem of large convolution calculation amount, channel compression is performed by adding convolution before the depth-separable convolution operation, and the point-by-point convolution part with the highest proportion in the depth-separable convolution is used to improve the calculation speed; for the ReLU low-dimensional data collapse problem, the Mish activation function is used to replace part of the ReLU, and a smaller gradient flow is allowed at negative values ​​so that information can better penetrate the network, thereby improving the accuracy and generalization ability of the network; for the problem of no reuse of features, a residual structure is added to skip one or more weight layers so that the gradient can reach a shallower level unimpeded.

[0082] In the embodiment of this example, the design of introducing the attention mechanism in YOLOv5s includes: on the basis of replacing the feature extraction network of YOLOv5s with the MobileNet network improved by the above design, the attention mechanism is introduced by embedding the CBAM module for the situation of insufficient multi-scale detection and missed detection of small targets, which is used to let the network know which part to focus on, and accordingly realize the prominent performance of important features, while suppressing less prominent features. In this process, the feature map obtained by the network is given to the CBAM module as input. According to the channel attention and spatial attention modules divided by CBAM, the input feature map is first convolved and given to the channel attention module in the CBAM module, and then passed through the spatial attention module, and finally the feature adjustment is performed with the input feature map to obtain the output of the entire module. The spatial attention module of the first part will perform a global average pooling operation and a maximum pooling operation on the input feature map, and then obtain the output through two fully connected layers, and then obtain two feature sums, and finally obtain the scaling factor through the sigmoid activation function, and multiply the scaling factor with the input feature map to obtain the output of the spatial attention module. In the second part, the spatial attention module performs average pooling and maximum pooling operations on the output of the channel attention module, and then performs channel splicing on the outputs after the two operations. Then, through convolution and sigmoid activation functions, the scaling factor of the spatial attention module is obtained, and the scaling factor of the spatial attention module is multiplied by the output of the channel attention module to obtain the output of the spatial attention module. Finally, the output of the two module groups is added to the input of the CBAM module to obtain the new features of the entire CBAM output. Experimental comparison shows that the effect obtained by first passing through the channel attention module and then through the spatial attention module is better than the processing method of processing the input through the channel attention module and the spatial attention module separately, and the processing method of first passing through the spatial attention module and then through the channel attention module.

[0083] In the embodiment of this example, YOLOv5s+DeepSORT multi-target tracking includes: for multi-target tracking, the DeepSORT tracking algorithm is used. Compared with the multi-target online tracking algorithm SORT (Simple Online and Realtime Tracking), it combines the motion information and appearance information of the target as an association metric, which improves the tracking failure problem caused by the target disappearing and reappearing. DeepSORT uses the results of the detector to initialize the tracker. Each tracker will set a counter. After the Kalman filter, the counter accumulates. When the prediction result and the detection result are successfully matched, the counter is set to 0. If the tracker does not match the appropriate detection result within a period of time, the tracker is deleted. DeepSORT assigns a tracker to the newly appearing detection result in each frame. When the prediction results of the tracker for 3 consecutive frames can match the detection result, it is confirmed that a new trajectory has appeared, otherwise the tracker is deleted.

[0084] In the embodiment of this example, multi-target tracking is introduced to link unrelated target detections between frames, which is suitable for dealing with multi-type target and multi-target locking tasks of the same type, especially the recent popular drone swarm problem. The introduction of multi-target tracking improves the strike judgment ability of a single flight platform based on multi-target tracking, and also provides a capability foundation for multiple flight platforms to coordinate strikes against drone swarms.

[0085] In the single target tracking and striking step S130, after one of the multiple targets is determined as the strike target, the servo system of the optoelectronic pod of the flying platform controls the camera to collect an image of the strike target, and based on the ECO correlation filter tracking method, a scale estimation algorithm and a background suppression regularization algorithm are used for optimization to achieve stable tracking of the strike target, and the distance between the flying platform and the strike target is calculated based on a laser rangefinder to achieve tracking and striking of the strike target.

[0086] In the embodiment of this example, the single target tracking and striking step of the method further includes:

[0087] When tracking the strike target, the search area of ​​the tracking algorithm is predicted and adjusted based on the target scale change rate of the image of the strike target.

[0088] In the embodiment of this example, the method further includes:

[0089] When tracking the strike target, the target feature calculation size limit value is adjusted based on the distance between the flying platform and the strike target calculated by the laser rangefinder.

[0090] In the embodiment of this example, the scale estimation and calculation simplification design includes: under a fixed-focus camera, when the unmanned aerial platform approaches the target, the target scale changes greatly and the clarity decreases. In this scenario, it is necessary to expand the target search range during tracking, but the information richness is low and the calculation amount is increased when calculating and extracting features for large-size blurred images.

[0091] In the embodiment of this example, the search area is adjusted based on the prediction of the target scale change, and the target scale change rate λ(t) is introduced, that is, a continuous differentiable function r(t) of any order with respect to time t is used to describe the target scale. The target scale at time t0 is r(t0), then

[0092]

[0093] In the formula, is the partial derivative of the target scale function r(t) with respect to time t. In practical applications, the exact analytical form of the target scale function cannot be obtained. Considering the inertia of the target motion process, based on the target scales of the first four frames, the target scale of the next frame is estimated using a cubic polynomial method, and the target search range is adjusted accordingly.

[0094] In the embodiment of this example, the size limit of fuzzy target feature calculation includes that the target detection stage can accurately identify the minimum target imaging of about 50*25 pixels (the image pixel size is 1920*1080). In the process of locking and tracking the target, especially when the flying platform is less than 10 meters away from the target, the target scale increases by nearly 20 times. In this case, the feature extraction and analysis of each pixel continues, and the calculation speed is greatly reduced, which cannot meet the real-time calculation requirements. Considering that although the target imaging size increases in the process of approaching the target, the clarity of the photo taken with a fixed-focus camera is significantly reduced, and not much rich feature information is added. Therefore, the present invention reduces the image size of the search area before feature extraction, matching and tracking in view of the size enlargement process.

[0095] The specific adjustment logic is to set the maximum feature calculation threshold. When the search area is expanded and the convolution size box exceeds the threshold, the calculation image area is downsampled to the threshold size. That is, the tracking and matching calculations that exceed the threshold are all processed by the threshold size before feature analysis.

[0096] In the embodiment of this example, the method further includes:

[0097] Based on the background suppression regularization algorithm, according to the tracking response result of the image of the strike target, the regularization coefficient matrix of the background suppression regularization algorithm is adaptively adjusted to achieve background suppression of the strike target.

[0098] In the embodiment of this example, the background suppression regularization design includes that the backgrounds of the unmanned aerial platform's air and ground application scenarios are relatively complex. Although the ECO algorithm adopts the spatial regularization method, during the spatial regularization operation, the regularization coefficient matrix ω is a pre-set fixed matrix, and the regularization coefficient matrix ω cannot be adaptively adjusted according to the target background conditions. When there are similar targets in the background, the parameters of this area should be punished intensively, and at the same time, the regularization coefficient for the area without similar targets should be appropriately reduced.

[0099] In the embodiment of this example, the adaptive background suppression regularization method is: according to the filter response result of the tracker, the regularization coefficient matrix ω of the spatial regularization term is adaptively adjusted so that it can focus on punishing the area with similar background. Specifically, the adaptive background suppression regularization coefficient matrix ω is introduced LA ,ω LA The final regularization coefficient matrix can be obtained by linearly superimposing it with ω. Then the objective function E(f) of background suppression can be obtained:

[0100]

[0101] The background suppression regularization coefficient matrix ω LA The update is adaptively adjusted according to the response of the filter, ω LA for:

[0102]

[0103] In the formula, k is the regularization gain coefficient, is the filter response at time t.

[0104] In the embodiment of this example, the method further includes:

[0105] Performing loss / occlusion determination on the strike target based on the peak sidelobe ratio calculated by the ECO tracking algorithm and the historical position and trajectory prediction of the strike target;

[0106] When it is determined to be lost, it changes to a global search for the target;

[0107] When it is determined to be blocked, the search for the strike target continues according to the historical position and trajectory prediction of the strike target.

[0108] In the embodiment of this example, the unmanned aerial platform is fast, and its typical aerial drone targets are flexible and maneuverable, and can easily leave the camera's field of view. Its ground application scenarios are more complex, such as typical moving targets such as pedestrians and vehicles, which not only easily leave the field of view, but also have obvious occlusion. To address this issue, the peak side lobe ratio (PSLR) calculated by the ECO tracking algorithm is used to determine target occlusion and loss, and the tracking search strategy is adjusted based on the Kalman filter prediction of the target trajectory. The judgment and decision logic is as follows: Figure 2B As shown in the figure, when the PSLR calculated for a new frame of image is lower than the threshold, it indicates that the response related to the target template is weak and the target is not found. At this time, combined with the target historical position and trajectory prediction, when the target historical position is in the boundary area of ​​the field of view and the predicted trajectory is to leave the field of view, the target is judged to be lost. When the target historical position and trajectory prediction are both in the internal area of ​​the field of view, the target is judged to be occluded. The search range of the next frame is determined according to the judgment and trajectory prediction. When it is judged that the target is lost, the next frame will search for the target globally. When it is judged that the target is occluded, the search range will not be changed, and the search area position will be moved only according to the trajectory prediction of the target. If the target is not found for 30 consecutive frames, the tracking fails and the target detection search stage is re-entered.

[0109] In the embodiment of this example, the single target tracking and striking step of the method further includes:

[0110] After one of the multiple targets is determined as a strike target, the servo system of the optoelectronic pod of the flight platform controls the camera to collect an image of the strike target, and tracks the strike target based on an optimized ECO tracking algorithm;

[0111] The distance between the flying platform and the strike target is calculated based on a laser rangefinder, and the three-dimensional coordinates of the strike target are calculated based on geodetic coordinates, so as to achieve tracking and striking of the strike target.

[0112] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0113] In addition, in this exemplary embodiment, a device for automatically guiding, identifying and tracking a target is also provided.

[0114] In the embodiment of this example, the guidance algorithm is implemented by the optoelectronic pod and information processor of the flight platform. The hardware system composition and functions are as follows: Figure 3As shown. The optoelectronic pod includes a visible light camera, a laser rangefinder and a servo system. The visible light camera consists of a fixed-focus optical lens and an imaging movement. A low-cost camera is selected for visible light imaging of the target. The laser rangefinder is used to measure the slant distance of the target. The servo system uses a three-axis stabilized platform to expand the camera's target search range, control the camera tracking phase after locking the target, and output the angle information when tracking the target. The information processor is used to complete the parameter setting of the visible light measurement camera, image acquisition and storage, target detection and tracking, data acquisition and time control of the positioning and timing module, and control of the servo system.

[0115] Reference Figure 4 As shown, the device 400 for automatically guiding, identifying and tracking targets may include: a target search module 410, a multi-target detection module 420 and a single target tracking and striking module 430.

[0116] in:

[0117] A target search module 410, configured to enable a servo system of an optoelectronic pod of a flying platform to control the rotation of a camera of the flying platform based on a preset scanning control logic, so as to realize a full-range search for a target;

[0118] The multi-target detection module 420 is used to identify multiple targets based on the YOLOv5s algorithm and the DeepSORT algorithm after the camera of the flight platform finds the target, and control the camera of the flight platform to rotate with the target through the servo system of the optoelectronic pod of the flight platform to achieve tracking of multiple targets;

[0119] The single target tracking and striking module 430 is used to control the camera of the servo system of the optoelectronic pod of the flying platform to collect the image of the striking target after one of the multiple targets is determined as the striking target, and to track the striking target based on the scale estimation algorithm and the background suppression regularization algorithm, and to calculate the distance between the flying platform and the striking target based on the laser rangefinder to track and strike the striking target.

[0120] The specific details of each of the above-mentioned device modules for automatically guiding, identifying and tracking a target have been described in detail in a corresponding method for automatically guiding, identifying and tracking a target, and will not be repeated here.

[0121] It should be noted that, although several modules or units of an apparatus 400 for automatically guiding, identifying and tracking a target are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0122] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0123] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0124] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present invention is described in detail. The electronic device 500 shown in FIG5 is only an example and should not bring any limitation to the functions and application scope of the embodiment of the present invention.

[0125] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), and a display unit 540.

[0126] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 510 can perform the following steps: Figure 1 Steps S110 to S130 shown in FIG.

[0127] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0128] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5203, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0129] Bus 550 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0130] The electronic device 500 may also communicate with one or more external devices 570 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 550. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0131] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0132] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0133] refer to Figure 6As shown, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0134] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0137] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0138] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0139] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0140] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for automatically guiding, identifying and tracking a target, characterized in that: The method comprises: A target search step, wherein the servo system of the optoelectronic pod of the flying platform controls the rotation of the camera of the flying platform based on a preset scanning control logic to achieve a full-range search for the target; The multi-target detection step is as follows: after the camera of the flight platform finds the target, based on the YOLOv5s fast detection network, feature extraction is performed on the image of the multi-target; the target attention is improved based on the channel attention module and the spatial attention module; the multi-target is tracked and identified based on the DeepSORT algorithm, and the camera of the flight platform is controlled to rotate with the target movement through the servo system of the optoelectronic pod of the flight platform to achieve tracking of the multi-target; and tracking filter training is enabled to complete the learning of target features; The single target tracking and striking step, after determining one of the multiple targets as the strike target, enables the servo system of the optoelectronic pod of the flight platform to control the camera to collect the image of the strike target, and based on the ECO correlation filter tracking method, uses the scale estimation algorithm and the background suppression regularization algorithm for optimization to achieve stable tracking of the strike target, calculates the distance between the flight platform and the strike target based on the laser rangefinder, and calculates the three-dimensional coordinates of the strike target based on the geodetic coordinates to achieve tracking and striking of the strike target; wherein, based on the ECO correlation filter tracking method, uses the scale estimation algorithm and the background suppression regularization algorithm for optimization to achieve stable tracking of the strike target, including: Performing loss / occlusion determination on the strike target based on the peak sidelobe ratio calculated by the ECO tracking algorithm and the historical position and trajectory prediction of the strike target; When it is determined to be lost, it changes to a global search for the target; When it is determined to be blocked, the search for the strike target continues according to the historical position and trajectory prediction of the strike target.

2. The method according to claim 1, characterized in that The single target tracking and striking step of the method further includes: When tracking the strike target, the search area of ​​the tracking algorithm is predicted and adjusted based on the target scale change rate of the image of the strike target.

3. The method according to claim 1, characterized in that The method further comprises: When tracking the strike target, the target feature calculation size limit value is adjusted based on the distance between the flying platform and the strike target calculated by the laser rangefinder.

4. The method according to claim 1, characterized in that The method further comprises: The ECO tracking algorithm is optimized based on the background suppression regularization algorithm. According to the tracking response result of the image of the strike target, the regularization coefficient matrix of the background suppression regularization algorithm is adaptively adjusted to achieve background suppression of the strike target.

5. A device for automatically guiding, identifying and tracking a target, characterized in that: The device comprises: A target search module, used to enable the servo system of the optoelectronic pod of the flying platform to control the rotation of the camera of the flying platform based on a preset scanning control logic, so as to realize a full-range search for the target; A multi-target detection module is used to extract features from images of multiple targets based on a YOLOv5s fast detection network after the camera of the flight platform finds a target; improve target attention based on the channel attention module and the spatial attention module; track and identify multiple targets based on a DeepSORT algorithm, and control the camera of the flight platform to rotate with the target through the servo system of the optoelectronic pod of the flight platform to achieve tracking of multiple targets; enable tracking filter training to complete the learning of target features; The single target tracking and striking module is used to control the camera of the servo system of the optoelectronic pod of the flying platform to collect the image of the striking target after determining one of the multiple targets as the striking target, and to track the striking target based on the optimized ECO tracking algorithm, calculate the distance between the flying platform and the striking target based on the laser rangefinder, and calculate the three-dimensional coordinates of the striking target based on the geodetic coordinates, so as to track and strike the striking target; wherein, the tracking of the striking target based on the optimized ECO tracking algorithm includes: Performing loss / occlusion determination on the strike target based on the peak sidelobe ratio calculated by the ECO tracking algorithm and the historical position and trajectory prediction of the strike target; When it is determined to be lost, it changes to a global search for the target; When it is determined to be blocked, the search for the strike target continues according to the historical position and trajectory prediction of the strike target.

6. An electronic device, characterized in that: include Processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

Citation Information

Patent Citations

  • A method for multi-target detection and tracking in aerospace background

    CN107993245B

  • An unmanned aerial vehicle visual target tracking method based on scale adaptive kernel correlation filtering

    CN109816698A

  • Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning

    CN111932588A