Small aircraft detecting and tracking method, device and equipment and storage medium

By combining the YOLOv7-tiny object detection model and the kernel-related filtering algorithm, the problem of low detection and tracking accuracy of small aircraft is solved, and high-precision tracking in complex environments is achieved to ensure that the target can be continuously identified and tracked.

CN119006528AInactive Publication Date: 2024-11-22JIANGSU NORTH LAKE OPTOELECTRONICS CO LTD

Patent Information

Application Number
CN202411479673.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing civilian small aircraft detection and tracking methods based on machine vision algorithms have low detection and tracking accuracy and are not strong in environmental adaptability.

Method used

Using the YOLOv7-tiny object detection model combined with the kernel-related filtering algorithm, small aircraft are identified and tracked through real-time video processing, and re-detected when the target is lost to ensure continuous tracking. The Gaussian weighted denoising filter template is used to improve the Adaptability.

Benefits of technology

Improves the accuracy and accuracy of detection and tracking of small aircraft, ensuring that targets can be continuously identified and tracked in complex environments until they fly away from the field of view.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006528A_ABST
    Figure CN119006528A_ABST
Patent Text Reader

Abstract

The invention relates to a small aircraft detection and tracking method, device and equipment and a storage medium, and is applied to the technical field of artificial intelligence, and the method comprises the steps: obtaining a video shot in real time, the video comprises a small aircraft image, and the small aircraft is a to-be-detected target; inputting the video into a trained YOLOv7-tiny target detection model, identifying the target, and marking the identified target; selecting a to-be-tracked target from the identified targets, and tracking the to-be-tracked target by adopting a trained kernel correlation filtering algorithm; and judging whether the to-be-tracked target is tracked and lost or not, and if the to-be-tracked target is tracked and lost, detecting the to-be-tracked target again. The method has the technical effects that the small target detection precision is improved, the influence of the surrounding environment is reduced, and the environmental adaptability of the algorithm under the complex background is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a small aircraft detection and tracking method, device, equipment and storage medium. Background Art

[0002] The technical routes for effective supervision of small civil aircraft mainly include: radar detection, microwave irradiation, electromagnetic detection and photoelectric tracking; however, due to the cost limitations of radar, microwaves and electromagnetics, methods such as radar detection, microwave irradiation, electromagnetic detection and photoelectric tracking are not suitable for widespread promotion.

[0003] At present, based on the flight characteristics of small civilian aircraft within the visible range, people have designed and developed machine vision algorithms and adopted artificial intelligence algorithms to achieve target detection and tracking, thus solving the problems of expensive equipment in the technical route. However, the detection and tracking accuracy of the detection and tracking method based on machine vision algorithms is low. Summary of the invention

[0004] In order to help solve the problem of low detection and tracking accuracy of detection and tracking methods based on machine vision algorithms, the present application provides a small aircraft detection and tracking method, device, equipment and storage medium.

[0005] In a first aspect, the present application provides a small aircraft detection and tracking method, which adopts the following technical solution: the method comprises: Acquire a video shot in real time, wherein the video includes an image of a small aircraft, and the small aircraft is a target to be detected; Input the video into the trained YOLOv7-tiny target detection model, identify the target and mark the identified target; Selecting a target to be tracked from the identified targets, and tracking the target to be tracked using a trained kernel correlation filter algorithm; Determine whether the target to be tracked is lost, and if the target to be tracked is lost, re-detect the target to be tracked.

[0006] In a specific implementation scheme, the YOLOv7-tiny target detection model includes an MPConv module and an ACmix module, and the YOLOv7-tiny target detection model training method includes: Obtaining a training sample video shot in real time, and dividing the training set sample video into a plurality of training sample images; Calling the MPConv module to evenly divide the training sample image into four training sample sub-images; Performing a convolution operation on the four training sample sub-images and generating a convolution image feature map; Calling the ACmix module to process the convolution image feature map and output a plurality of high-dimensional feature maps; Concatenate all the high-dimensional feature maps and output a complete feature map; A target detection loss function is constructed. If the value of the target detection loss function meets a preset threshold, the training of the YOLOv7-tiny target detection model is completed.

[0007] In a specific implementation scheme, constructing the target detection loss function includes: The target detection loss function is constructed using SIoU, and the target detection loss function includes angle loss; The angle loss includes: ; in, represents the angle loss, Represents the height difference between the center point of the predicted box of the target and the center point of the real box of the target, Represents the distance between the center point of the predicted box of the target and the center point of the real box of the target.

[0008] In a specific implementation scheme, the training method of the kernel correlation filtering algorithm includes: Acquire a tracking sample video, wherein the tracking sample video includes a plurality of images, and each frame of the image includes a target area of ​​a target to be tracked; Constructing a ridge regression classifier and a target optimization function of the ridge regression classifier, and constructing a filtering template according to the target optimization function; Extracting the HoG features of the target area in each frame image and generating a HoG feature circulant matrix; Performing Fourier transform on the HoG feature circulant matrix and the filter template of the target area to obtain a response feature map of the target area; Calculating the peak value of the response characteristic map of the target area, determining the movement deviation of the target to be tracked according to the peak value of the response characteristic map of the target area, finding the position of the target to be tracked in each frame image and updating the filter template; The filter template is denoised using a Gaussian weighted algorithm to generate a target template.

[0009] In a specific implementation scheme, after the filtering template is denoised using the Gaussian weighted algorithm and a target template is generated, the method further includes: Extracting target areas of different scales of the target to be tracked, and respectively calculating the peak values ​​of the response characteristic graphs corresponding to the target areas of different scales; According to the peak values ​​of the response characteristic graphs corresponding to the target areas of different scales, a matching algorithm is used to determine the target matching template corresponding to the target area of ​​the target to be tracked.

[0010] In a specific implementation scheme, the determining whether the target to be tracked is lost, and if the target to be tracked is lost, re-detecting the target to be tracked includes: Adopting the APCE confidence combined with the image response change gradient discrimination algorithm to determine whether the target to be tracked is lost; If the judgment result meets the preset rule, the target to be tracked is not lost; If the discrimination result does not conform to the preset rule, the tracking of the target to be tracked is lost, and the target to be tracked is detected again.

[0011] In a specific implementation scheme, the APCE confidence determination algorithm includes: ; in, represents the degree of variation between the maximum and minimum responses in the target area, represents the maximum response value of the i-th frame image, represents the minimum response value of the i-th frame image, w represents the length of the target area detection window, h represents the width of the target area detection window, and n represents the number of frames of the image in the video; The response change gradient includes: ; in, represents the change gradient of the small aircraft image response, and They respectively represent the response changes of two consecutive image frames.

[0012] In a second aspect, the present application provides a small aircraft detection and tracking device, which adopts the following technical solution: the device comprises: A video acquisition module, used to acquire a video shot in real time, wherein the video includes an image of a small aircraft, and the small aircraft is a target to be detected; A target recognition module is used to input the video into a trained YOLOv7-tiny target detection model, identify the target and mark the identified target; A target tracking module is used to select a target to be tracked from the identified targets and track the target to be tracked using a trained kernel correlation filter algorithm; The tracking judgment module is used to judge whether the tracking of the target to be tracked is lost, and if the tracking of the target to be tracked is lost, re-detect the target to be tracked.

[0013] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the above-mentioned small aircraft detection and tracking methods.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned small aircraft detection and tracking methods.

[0015] In summary, this application has the following beneficial technical effects: The trained YOLOv7-tiny target detection model is used to detect small aircraft. Improvements are made to the trained YOLOv7-tiny target detection model to improve the accuracy of small target detection. In addition, in order to deal with the problem of tracking loss caused by rapid target movement, a re-detection and positioning mechanism is constructed to continue target tracking after re-detecting the target, thereby improving the accuracy of target detection and tracking to ensure that the target can be detected and identified, and can be continuously tracked until the target flies away from the target scene. In the training of the kernel correlation filter algorithm, Gaussian weighting method is used for denoising to reduce the influence of the surrounding environment of the filter template and improve the algorithm's environmental adaptability in complex backgrounds. In addition, when training the KCF algorithm, a multi-scale target response analysis algorithm is used to update the denoised target template to improve the target template's ability to adapt to changes in target scale, thereby further improving the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a small aircraft detection and tracking method in an embodiment of the present application; Figure 2 It is a schematic diagram of the separation and merging of the MPConv modules in the embodiment of the present application; Figure 3 is a schematic diagram of the angle loss in the loss function in the embodiment of the present application; Figure 4 It is a complete block diagram of the training process of the small aircraft tracking method in the embodiment of the present application; Figure 5 This is a complete flowchart of the small aircraft detection and tracking method in the embodiment of the present application; Figure 6 is a schematic diagram of a small aircraft detection and tracking device in an embodiment of the present application; Figure 7 It is a schematic diagram used to embody a computer device in an embodiment of the present application.

[0017] Reference numerals: 601, video acquisition module; 602, target recognition module; 603, target tracking module; 604, tracking judgment module. DETAILED DESCRIPTION

[0018] The following combination Figure 1-Figure 7 This application is described in further detail.

[0019] The embodiment of the present application discloses a small aircraft detection and tracking method, which is applied to a small aircraft detection and tracking system. The detection and tracking method can accurately detect and identify small aircraft in motion, and track the detected targets to achieve effective supervision of civil small aircraft.

[0020] The technical routes for effective supervision of small civil aircraft mainly include: radar detection, microwave irradiation, electromagnetic detection and photoelectric tracking, etc. However, due to the technical and cost limitations of radar, microwave and electromagnetic, for example, the cost of radar equipment is high, and the target reflection area of ​​small aircraft is too small, which is not conducive to radar detection; in addition, microwave and electromagnetic detection equipment are expensive and have special requirements for the use environment; therefore, radar detection, microwave irradiation, electromagnetic detection and photoelectric tracking are not suitable for general promotion.

[0021] At present, in view of the flight characteristics of small civilian aircraft within the visible range, people have designed and developed machine vision algorithms and artificial intelligence algorithms to achieve target detection and tracking, thereby solving the problems of expensive equipment in the technical route; however, the detection and tracking method based on machine vision algorithms has low detection and tracking accuracy, and the existing technical solutions are not very adaptable to the environment. In order to help improve the accuracy of target detection and tracking, and enhance the adaptability of the solution to the environment, this application provides a small aircraft detection and tracking method.

[0022] Reference Figure 1 , the method comprises the following steps: S10, obtaining a real-time video, where the video includes an image of a small aircraft, and the small aircraft is a target to be detected.

[0023] Specifically, real-time video data is obtained, and a segment of the captured video data contains several frames of image data, each of which contains a small aircraft; the purpose of the embodiment of the present application is to detect and identify small aircraft in the video data, and to track the detected small aircraft in each subsequent frame of the image to achieve the purpose of effective supervision; thus, the small aircraft in the image of the small aircraft in the video data is the target to be detected and identified by the detection and tracking method. Specifically, the user can use a spectral detector, such as a multi-spectral visual image sensor, to take real-time photos of a target scene containing a small aircraft, and the image or video data obtained after the shooting can be transmitted to the system through data transmission and other methods for further data processing.

[0024] S20, input the video into the trained YOLOv7-tiny target detection model, identify the target and mark the identified target.

[0025] Specifically, the system inputs the acquired video data into the trained YOLOv7-tiny target detection model, detects and identifies the target through the trained target detection model, and the identified target is the target to be detected mentioned above, which can also be understood as a small aircraft to be detected and identified. After detecting and identifying the small aircraft, the identified target is marked. The marking method can be selected according to the actual situation. There is no restriction here. In the embodiment of the present application, a more convenient way of adding a detection frame is selected. The YOLOv7-tiny target detection model is a lightweight version of the target detection algorithm YOLO. It is a real-time target detection algorithm based on deep learning. It realizes real-time and efficient target detection through lightweight model architecture, multi-scale prediction strategy, optimization strategy and other designs; on the basis of the existing YOLOv7-tiny target detection model, the model is improved and enhanced, and the MPConv module, ACmix attention mechanism and SIoU classification loss function are used to improve the accuracy of small target detection.

[0026] S30, selecting a target to be tracked from the identified targets, and using the trained kernel correlation filter algorithm to track the target to be tracked.

[0027] Specifically, the target to be tracked is selected from the identified targets. Specifically, it can be understood that there may be more than one target detected and identified by the target detection model. There may be multiple small aircraft at the same time. Then, a target to be tracked can be selected from all the identified targets according to user needs. Specifically, the user can manually select the target to be tracked. After the system receives the information selected by the user, it selects one of the targets to be tracked according to the information selected by the user. Subsequently, the target to be tracked is tracked and detected using the kernel correlation filter algorithm. The kernel correlation filter detection algorithm, also known as Kernelized Correlation Filters, also known as the KCF algorithm, is a kernel-based tracking algorithm for target tracking.

[0028] S40, determining whether the target to be tracked is lost, and if the target to be tracked is lost, re-detecting the target to be tracked.

[0029] Specifically, a small aircraft may lose target tracking during rapid movement. If target tracking is lost, the system will first determine whether the target is lost. If it is not lost, the system will maintain the status quo and continue tracking. If tracking is lost, the system will re-detect the target to be tracked and continue tracking after detecting the target. Specifically, when tracking is lost, the system can use the trained YOLOv7-tiny target detection model to re-detect the target and mark it with a frame after detecting the target. Finally, when the target to be tracked flies out of the field of view of the detection video, which can also be understood as when the aircraft completely flies away from the current target scene, the system stops detection and tracking and returns to image detection mode.

[0030] In the present application, a trained YOLOv7-tiny target detection model is used to detect small aircraft. Improvements are made to the trained YOLOv7-tiny target detection model to improve the accuracy of small target detection. In addition, in order to deal with the problem of tracking loss caused by rapid target movement, a re-detection and positioning mechanism is constructed to continue target tracking after re-detecting the target, thereby improving the accuracy of target detection and tracking to ensure that the target can be detected and identified, and can continue to be tracked until the target flies away from the target scene.

[0031] In one embodiment, the YOLOv7-tiny target detection model includes an MPConv module and an ACmix module, and the steps of the YOLOv7-tiny target detection model training method can be specifically performed as follows: First, obtain the training sample video shot in real time, and divide the training set sample video into several training sample images; specifically, a multispectral visual image sensor can be used to shoot the target scene containing a small aircraft in real time under different meteorological conditions such as day and night, and obtain the video under different environmental conditions, and obtain multispectral image information including infrared, white light and low light as training sample data, so as to improve the adaptability of the target detection model to different environments, and improve the accuracy of the target detection model in detecting and identifying targets under different environmental conditions; for the obtained video data, it is necessary to first divide the video data into several frames of image data, so as to facilitate the subsequent processing of the image data and identify the target in each frame of image data.

[0032] The backbone network of the YOLOv7-tiny target detection model includes the MPConv module, which can improve the network's feature extraction capability and computational efficiency through multi-path convolution. The lower branch of this module contains two convolution operations with different step sizes. In order to avoid the loss of small target feature extraction caused by convolution with a step size of 2, separation and cooperation operations can be performed to reasonably avoid it. After dividing the video data into image data, call the MPConv module to evenly divide each frame of the training sample image into four training sample sub-images. Each training sample sub-image is one-fourth the size of the original image. Figure 2 After that, the convolution operation is performed on the four training sample sub-graphs. Specifically, the four training sample sub-graphs are concatenated and merged through 1*1 convolution, and a convolution image feature map is generated. The step size of the convolution image feature map is 1.

[0033] The YOLOv7-tiny target detection model also includes an ACmix module with convolutional attention and self-attention. This module combines self-attention and convolution techniques. By integrating the advantages of these two techniques, the performance and efficiency of the model can be improved. After obtaining the convolution image feature map, the ACmix module is called to process the convolution image feature map. The specific processing methods include shifting, aggregating, and convolution processing of the image feature map, and finally output multiple high-dimensional feature maps. Finally, all high-dimensional feature maps are spliced ​​using the Concat operation, and the complete feature map is output. Specifically, the output complete feature map can take into account both global and local features. Among them, the Concat operation, that is, Concatenation, connection or series connection, is an operation that splices two or more data sequences (such as arrays, lists, tensors, etc.) along a specified axis into a new data sequence. This operation does not change the elements in the original data sequence, but combines them in a specified order and dimension.

[0034] When training the model, the target detection loss function is constructed to detect whether the model training meets the requirements. If the value of the target detection loss function meets the preset threshold, the YOLOv7-tiny target detection model training is completed. Specifically, the SIoU function is used to construct the target detection loss function. The SIoU (Scalable Intersection over Union) loss function is an improved IoU (Intersection over Union) loss function. It considers the influence of angle, distance and shape on the abnormal convergence of the loss function, improves the regression accuracy of the network, and improves the detection and recognition accuracy. The target detection loss function includes angle loss, refer to Figure 3 , the angle loss can be expressed as: ; in, represents the angle loss, Represents the height difference between the center point of the target's predicted box and the center point of the target's real box, Represents the distance between the center point of the predicted box of the target and the center point of the real box of the target.

[0035] In this application, the YOLOv7-tiny target detection model algorithm is used to detect the target in each frame of the video. The MPConv module, ACmix attention mechanism and classification loss function are added to the YOLOv7-tiny target detection model, and the SIoU algorithm is used to further improve the accuracy of small target detection.

[0036] In one embodiment, the steps of the training method of the kernel correlation filtering algorithm may be specifically performed as follows: First, obtain a tracking sample video, which includes several frames of images, each of which contains the target area of ​​the target to be tracked. Specifically, the tracking sample video can be a real-time training sample video acquired in advance. The target in the first frame of the image is detected by the YOLOv7-tiny target detection model. There may be multiple detected targets, so the user can manually select the target to be tracked and set it as the target to be tracked. The system obtains relevant information about the target to be tracked and selects the target to be tracked in the image. The detected target is tracked in each subsequent frame. After that, a ridge regression classifier and a target optimization function of the ridge regression classifier are constructed, and a filter template is constructed according to the target optimization function; specifically, the constructed ridge regression classifier can be expressed as: ; in, represents the ridge regression classifier, Represents a linear combination of training samples. Training samples can also be understood as several frames of images in the tracking sample video. The kernel function is represented in one-dimensional data, and the HoG feature circulant matrix of KCF is represented in two-dimensional image data coordinates. In order to improve the performance of the ridge regression classifier, the kernel function representation is introduced, and the ridge regression classifier can be represented as: ; in, represents the linear combination coefficient, represents the next frame test sample sequence, k represents the current frame training sample sequence, represents the ridge regression classifier, i represents the image number, and n represents the total number of image frames.

[0037] The purpose of training is to minimize the objective optimization function value of the ridge regression classifier. The constructed objective optimization function can be expressed as: ; By taking the derivative of the target optimization function and setting the derivative to 0, we can obtain: ; in, Represents the kernel function, which can be represented by k1. represents the introduced regularization coefficient, I represents the unit matrix, y represents the regression label vector, Represents the filter template.

[0038] After that, the HoG features of the target area in each frame of the image are extracted and the HoG feature circulant matrix is ​​generated. The HoG feature circulant matrix is ​​also the kernel function k. After obtaining the HoG feature circulant matrix, the circulant matrix is ​​Fourier diagonalized, and the following can be obtained after Fourier diagonalization: ; in, represents the kernel correlation matrix. After Fourier diagonalization, the target optimization function can be simplified as follows: ; in, Represents the Fourier transform form of the coefficients updated in the current frame, represents the Fourier transform form of the kernel correlation matrix, Represents the Fourier transform form of the coefficients obtained from the previous frame update.

[0039] After Fourier diagonalization using the circulant matrix, the HoG feature circulant matrix and the filter denoising template of the target area in the frame image starting from the second frame image are Fourier transformed to obtain the response feature map of the target area in the subsequent frame image, where the response feature map of the first frame image is the response feature map of the target obtained after recognition, and the response feature map of the image after the second frame is obtained by predicting the KCF algorithm. The calculation method can be expressed as: ; in, represents the Fourier transform form of the predicted response characteristic map, Represents the Fourier transform form of the regression prediction result, Represents matrix dot product.

[0040] Finally, the peak value of the response feature map of the target area is calculated based on the response feature map of the target area, the movement deviation of the target is determined according to the peak value of the response feature map of the target area, and the position of the target in the current frame image is found. The filter template is continuously updated during the training process. The update method can be expressed as: ; in, represents the learning factor, and Respectively represent the Fourier transform forms of the coefficients calculated for the current frame and the previous frame.

[0041] In the process of training the filter template, the Gaussian weighted algorithm is used to denoise the filter template and generate the denoised target template. The filter template of the kernel correlation filter algorithm (KCF algorithm) comes from the image block intercepted at the target position of the current frame image. The template size is consistent with the detection window size of the image block. Since there is non-target environmental noise in the template, in order to reduce the impact of the noise on the tracking performance, the Gaussian weighted method can be used to remove excessive redundant background information; in addition, in order to improve the adaptability in complex scenes and avoid the interference of the surrounding environment on the target template, the Gaussian weighted method can be used to make only the target exist in the template image, and the area around the target is filled with 0. Specifically, the interception function of the Gaussian weighted method can be expressed as: ; ; ; Among them, x and y represent the upper left corner coordinates of the external contour of the target to be tracked, h represents the width of the target marking box, Represents the coefficient of the Gaussian function, specifically the variance of the window data.

[0042] Considering that the distance between the target and the shooting point is different during the movement of the target, the scale of the target in the captured image will change. In order to solve the scale transformation problem during the target movement, the image information of the target area of ​​multiple scales can be extracted. After using the Gaussian weighted algorithm to denoise the filter template and generate the target template, the following steps can be performed: Target areas at different scales of the target to be tracked are extracted, and the peak values ​​of the response feature maps of the target areas at different scales are calculated respectively; according to the peak values ​​of the response feature maps of the target areas at different scales, the target matching template corresponding to the target area can be determined by using the existing matching algorithm.

[0043] In the present application, in the training of the kernel correlation filter algorithm, after the filter template is updated, the Gaussian weighting method is used to denoise the filter template, reduce the influence of the surrounding environment of the filter template, and improve the environmental adaptability of the algorithm in complex backgrounds. In addition, a multi-scale target response analysis algorithm is used to improve the adaptability to targets with scale changes, thereby improving the accuracy of target tracking.

[0044] In one embodiment, determining whether the target to be tracked is lost, if the target to be tracked is lost, the step of redetecting the target to be tracked may be specifically performed as follows: The APCE confidence combined with the response change gradient discrimination algorithm is used to determine whether the target to be tracked is lost; if the discrimination result meets the preset rules, the target to be tracked is not lost; if the discrimination result does not meet the preset rules, the target to be tracked is lost and the target to be tracked is redetected.

[0045] Target occlusion can be understood as a situation of tracking loss. In order to effectively deal with the problem of target occlusion, the average value of gradient changes in multiple frames is usually collected and compared with the set response change gradient threshold to determine whether an abnormal situation occurs or the target is occluded. If an abnormal situation occurs, template update training will not be performed.

[0046] On the basis of the original APCE confidence judgment method, the original APCE confidence judgment method is improved, and the response change gradient is introduced into the judgment standard. The constructed new confidence judgment method can be expressed as: ; in, represents the degree of variation between the maximum and minimum responses in the target area, represents the maximum response value of the i-th frame image, represents the minimum response value of the i-th frame image, w represents the length of the target area detection window, h represents the width of the target area detection window, and n represents the number of frames of the image in the video.

[0047] The response change gradient of the image can be expressed as: ; in, represents the gradient of the image response, and They respectively represent the response changes of two consecutive image frames.

[0048] According to the APCE confidence judgment and the image response change gradient judgment, the target is not tracked and lost. The method can be expressed as: ; The above judgment condition can be understood as: when the response change value Greater than threshold , the image response changes Whether greater than or less than the response gradient , it is determined that the target is not tracked and lost; when the response change value Less than threshold , the image response changes Whether greater than or less than the response gradient , then it is determined that the target tracking is lost. In general, the response change value With threshold There will be no equal situation, the response changes value With threshold If they are equal, it means that the target in the two frames has not moved. If it has not moved, no prediction is needed. The response change of the image With the response gradient Generally, they will not be equal. If they are equal, it means that the target in the images of the previous and next frames has not moved. If there is no movement, no prediction is required.

[0049] It should be noted that the complete KCF algorithm training process also includes the training of the target tracking loss judgment process. While judging whether the target is lost, the KCF filter template is continuously updated to improve the tracking accuracy. For details, refer to Figure 4, the multi-scale target detection step in the KCF algorithm training obtains the target matching template, and then the multiple detection and tracking state. The multiple detection and tracking state is mainly a two-step judgment process for the target. First, it is judged whether the detected and tracked target has flown out of the field of view. If the target has flown out of the current field of view, it returns to call the YOLOv7-tiny target detection model algorithm to detect the target in the image; if the target has not flown out of the field of view, it uses the APCE confidence combined with the image response change gradient to detect whether the target is lost. If it is determined that the target is not lost, the target matching template is updated and training is continued for subsequent frame images; if it is determined that the target is lost, the YOLOv7-tiny target detection model algorithm is used to re-detect the target in the image to determine whether the target is re-detected successfully. If the target detection is unsuccessful, the YOLOv7-tiny target detection model is called to detect the target. If the target detection is successful, the target matching template is updated, training is continued for subsequent frame images, and the tracked target in the subsequent frame images is marked with a frame; after updating the target matching template, the Gaussian weighted method is used to denoise the target matching template, and all frame images in the video are processed in a loop until the target flies away from the current field of view.

[0050] During re-detection training, if it is determined that the target tracking is lost, the update of the filter template will be stopped. In the original marking box area (x, y, w, h), the peak values ​​of the response feature map are recalculated with (x-2w, y, w, h), (x+2w, y, w, h), (x, y-2w, w, h), and (x, y+2w, w, h) as the four re-detection areas, and the corresponding windows of the maximum values ​​of these peak values ​​are used as the optimal position area for target tracking. If the maximum value of the peak value is still too small, it is determined that the re-detection positioning has failed, then the tracking mode is exited, the target detection and recognition module is called, and the target to be tracked is re-detected.

[0051] In the present application, taking into account that the target may be lost during rapid movement, a mechanism of multiple detection and tracking status is introduced. Before determining whether the target is lost, it is first determined whether the target has left the scene. After the determination is completed, tracking is performed or the target is re-detected based on the determination result. Multiple judgments are made to improve the accuracy of target detection and tracking.

[0052] Reference Figure 5, which is a block diagram of the complete process of small aircraft detection and tracking. The system first loads the trained YOLOv7-tiny target detection model, then reads the video stream to obtain the video data of the target scene shot in real time, uses the YOLOv7-tiny target detection model to identify the target and add a frame mark, then manually selects the tracking target. The user manually selects the target to be tracked, the system selects the target to be tracked, and uses the kernel correlation filter algorithm to track the target and add a frame mark. After that, multiple detections first determine whether the tracked target flies out of the field of view. If it flies out of the field of view, the YOLOv7-tiny target detection model is used again for target detection; if the tracked target does not fly out of the field of view, it is determined whether the target is lost. If it is not lost, the kernel correlation filter algorithm is used to continue tracking and adding a frame mark until the target flies out of the field of view; if the target tracking is lost, the re-detection mechanism is started. If the target is successfully re-detected, the KCF algorithm is used for tracking. If the target is not successfully re-detected, the detection algorithm is used to identify the detection and tracking target and add a frame mark until the small aircraft flies out of the field of view, and the next video data is detected and tracked.

[0053] Figure 1 FIG. 1 is a flow chart of a method for detecting and tracking a small aircraft in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0054] Based on the above method, an embodiment of the present application also discloses a small aircraft detection and tracking device.

[0055] Reference Figure 6 , the device includes the following modules: The video acquisition module 601 is used to acquire a real-time video, where the video includes an image of a small aircraft, and the small aircraft is a target to be detected; The target recognition module 602 is used to input the video into the trained YOLOv7-tiny target detection model, recognize the target and mark the recognized target; The target tracking module 603 is used to select a target to be tracked from the identified targets and track the target to be tracked using the trained kernel correlation filter algorithm; The tracking judgment module 604 is used to judge whether the tracking of the target to be tracked is lost. If the tracking of the target to be tracked is lost, the target to be tracked is re-detected.

[0056] In one embodiment, the target recognition module 602 is also used to obtain a training sample video shot in real time, divide the training set sample video into a number of training sample images; call the MPConv module to evenly divide the training sample image into four training sample sub-images; perform a convolution operation on the four training sample sub-images, and generate a convolution image feature map; call the ACmix module to process the convolution image feature map, and output multiple high-dimensional feature maps; splice all high-dimensional feature maps, and output a complete feature map; construct a target detection loss function, and if the value of the target detection loss function meets a preset threshold, the YOLOv7-tiny target detection model training is completed.

[0057] In one embodiment, the target recognition module 602 is further used to construct a target detection loss function using SIoU, where the target detection loss function includes angle loss; the angle loss includes: ; in, represents the angle loss, Represents the height difference between the center point of the target's predicted box and the center point of the target's real box, Represents the distance between the center point of the predicted box of the target and the center point of the real box of the target.

[0058] In one embodiment, the target tracking module 603 is also used to obtain a tracking sample video, which includes a plurality of images, each frame of which includes a target area of ​​the target to be tracked; construct a ridge regression classifier and a target optimization function of the ridge regression classifier, and construct a filter template based on the target optimization function; extract the HoG features of the target area in each frame of the image, and generate a HoG feature circulant matrix; perform Fourier transform on the HoG feature circulant matrix and the filter template of the target area to obtain a response feature map of the target area; calculate the peak value of the response feature map of the target area, determine the movement deviation of the target to be tracked based on the peak value of the response feature map of the target area, find the position of the target to be tracked in each frame of the image and update the filter template; use a Gaussian weighted algorithm to denoise the filter template and generate a target template.

[0059] In one embodiment, the target tracking module 603 is also used to extract target areas of different scales of the target to be tracked, and calculate the peak values ​​of the response feature maps corresponding to the target areas of different scales respectively; according to the peak values ​​of the response feature maps corresponding to the target areas of different scales, a matching algorithm is used to determine the target matching template corresponding to the target area of ​​the target to be tracked.

[0060] In one embodiment, the tracking judgment module 604 is also used to use the APCE confidence combined with the image response change gradient discrimination algorithm to determine whether the target to be tracked is lost; if the discrimination result meets the preset rules, the target to be tracked is not lost; if the discrimination result does not meet the preset rules, the target to be tracked is lost, and the target to be tracked is redetected.

[0061] In one embodiment, the tracking determination module 604, the APCE confidence determination algorithm includes: ; in, Indicates the degree of change between the maximum and minimum responses in the target area. represents the maximum response value of the i-th frame image, represents the minimum response value of the i-th frame image, w represents the length of the target area detection window, h represents the width of the target area detection window, and n represents the number of frames of the image in the video; The response gradients include: ; in, represents the gradient of the small aircraft image response, and They respectively represent the response changes of two consecutive image frames.

[0062] The small aircraft detection and tracking device provided in the embodiment of the present application can be applied to the small aircraft detection and tracking method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0063] It should be noted that: the small aircraft detection and tracking device provided in the embodiment of the present application only uses the division of the above-mentioned functional modules / functional units as an example when performing small aircraft detection and tracking. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the small aircraft detection and tracking device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the small aircraft detection and tracking method provided in the above method embodiment and the implementation method of the small aircraft detection and tracking device provided in this embodiment belong to the same concept. The specific implementation process of the small aircraft detection and tracking device provided in this embodiment is detailed in the above method embodiment, and will not be repeated here.

[0064] The embodiment of the present application also discloses a computer device.

[0065] Specifically, if Figure 7As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above chips.

[0066] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0067] The embodiment of the present application also discloses a computer-readable storage medium.

[0068] Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the process of the implementation of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0069] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A small aircraft detection and tracking method, characterized in that: The method comprises: Acquire a video shot in real time, wherein the video includes an image of a small aircraft, and the small aircraft is a target to be detected; Input the video into the trained YOLOv7-tiny target detection model, identify the target and mark the identified target; Selecting a target to be tracked from the identified targets, and tracking the target to be tracked using a trained kernel correlation filter algorithm; Determine whether the target to be tracked is lost, and if the target to be tracked is lost, re-detect the target to be tracked; The YOLOv7-tiny target detection model includes an MPConv module and an ACmix module, and the YOLOv7-tiny target detection model training method includes: Obtain a training sample video shot in real time, and divide the training set sample video into a number of training sample images; Calling the MPConv module to evenly divide the training sample image into four training sample sub-images; Performing a convolution operation on the four training sample sub-images and generating a convolution image feature map; Calling the ACmix module to process the convolution image feature map and output a plurality of high-dimensional feature maps; Concatenate all the high-dimensional feature maps and output a complete feature map; A target detection loss function is constructed. If the value of the target detection loss function meets a preset threshold, the training of the YOLOv7-tiny target detection model is completed.

2. The method according to claim 1, characterized in that: The construction of the target detection loss function includes: The target detection loss function is constructed using SIoU, and the target detection loss function includes angle loss; The angle loss includes: in, represents the angle loss, Represents the height difference between the center point of the predicted box of the target and the center point of the real box of the target, Represents the distance between the center point of the predicted box of the target and the center point of the real box of the target.

3. The method according to claim 1, characterized in that: The training method of the kernel correlation filtering algorithm includes: Acquire a tracking sample video, wherein the tracking sample video includes a plurality of images, and each frame of the image includes a target area of ​​a target to be tracked; Constructing a ridge regression classifier and a target optimization function of the ridge regression classifier, and constructing a filtering template according to the target optimization function; Extracting the HoG features of the target area in each frame image and generating a HoG feature circulant matrix; Performing Fourier transform on the HoG feature circulant matrix and the filter template of the target area to obtain a response feature map of the target area; Calculating the peak value of the response characteristic map of the target area, determining the movement deviation of the target to be tracked according to the peak value of the response characteristic map of the target area, finding the position of the target to be tracked in each frame image and updating the filter template; The filter template is denoised using a Gaussian weighted algorithm to generate a target template.

4. The method according to claim 3, characterized in that: After the filtering template is denoised by using the Gaussian weighted algorithm and a target template is generated, the method further includes: Extracting target areas of different scales of the target to be tracked, and respectively calculating the peak values ​​of the response characteristic graphs corresponding to the target areas of different scales; According to the peak values ​​of the response characteristic graphs corresponding to the target areas of different scales, a matching algorithm is used to determine the target matching template corresponding to the target area of ​​the target to be tracked.

5. The method according to claim 1, characterized in that: The determining whether the target to be tracked is lost, and if the target to be tracked is lost, re-detecting the target to be tracked comprises: Adopting the APCE confidence combined with the image response change gradient discrimination algorithm to determine whether the target to be tracked is lost; If the judgment result meets the preset rule, the target to be tracked is not lost; If the discrimination result does not conform to the preset rule, the tracking of the target to be tracked is lost, and the target to be tracked is detected again.

6. The method according to claim 5, characterized in that: The APCE confidence determination algorithm includes: in, represents the degree of variation between the maximum and minimum responses in the target area, represents the maximum response value of the i-th frame image, represents the minimum response value of the i-th frame image, w represents the length of the target area detection window, h represents the width of the target area detection window, and n represents the number of frames of the image in the video; The response change gradient includes: in, represents the change gradient of the small aircraft image response, and They respectively represent the response changes of two consecutive image frames.

7. A small aircraft detection and tracking device, used to implement the small aircraft detection and tracking method according to any one of claims 1 to 6, characterized in that: The device comprises: A video acquisition module (601) is used to acquire a video shot in real time, wherein the video includes an image of a small aircraft, and the small aircraft is a target to be detected; A target recognition module (602) is used to input the video into a trained YOLOv7-tiny target detection model, recognize the target and mark the recognized target; A target tracking module (603) is used to select a target to be tracked from the identified targets, and track the target to be tracked using a trained kernel correlation filter algorithm; The tracking judgment module (604) is used to judge whether the tracking of the target to be tracked is lost, and if the tracking of the target to be tracked is lost, re-detect the target to be tracked.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • An infrared small unmanned aerial vehicle target detection and tracking method under a complex background

    CN109816695A

  • Air target tracking method based on YOLOv3

    CN110706266A

  • Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning

    CN111932588A

  • Re-detection tracking algorithm fusing multi-feature scale self-adaption

    CN115205591A

  • Acoustic tracking and countering interception device, system and method for invading unmanned aerial vehicle

    CN115902786A

Cited By

  • Multi-spectral small target identification tracking method based on deep learning

    CN120471951A

  • Unmanned aerial vehicle single target tracking method, heterogeneous embedded system and device

    CN122115500A