Unmanned aerial vehicle tracking method, device and system

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle tracking method, device and system, and relates to the technical field of data processing, and the method comprises the steps: obtaining the corresponding image data of a to-be-tracked unmanned aerial vehicle; inputting the corresponding image data of the to-be-tracked unmanned aerial vehicle into a target detection model based on deep learning, performing target detection on the to-be-tracked unmanned aerial vehicle, and outputting a target detection result; the target detection model is obtained based on training of an unmanned aerial vehicle image data set; the unmanned aerial vehicle image data set comprises image data of at least two spectrums; the image data of the at least two spectrums comprises an RGB image collected under the condition of a good view and an infrared image collected under the condition of a poor view by using an unmanned aerial vehicle tracking system; and tracking the to-be-tracked unmanned aerial vehicle by adopting a tracking algorithm based on the target detection result and the corresponding image data of the to-be-tracked unmanned aerial vehicle. The problem that unmanned aerial vehicle tracking is not accurate enough in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and system for tracking an unmanned aerial vehicle. Background Art

[0002] Drones are becoming more and more popular due to their low cost, ease of operation, and ability to capture high-quality images. However, the rapid development of drones has brought many problems in certain sensitive areas. It is well known that drones can interfere with the operation of airports and other critical infrastructure, putting life and property at risk. At the same time, privacy and security issues have also emerged, as drones can be used for invasive surveillance and invade an individual's personal space. In order to avoid these problems, drone positioning and tracking technology is particularly important. Drones have the characteristics of being low, small, slow, and flying against complex backgrounds, so it is very costly and easy to fail to monitor drones manually.

[0003] There are four main methods of drone detection on the market, including GPS-based tracking, radar systems, acoustic tracking, and visual tracking. GPS-based tracking relies on satellite signals to determine the location of the drone. However, this method may not be effective if the drone does not use GPS or uses a different frequency. In addition, accessing or manipulating other users' GPS data may also be illegal.

[0004] Radar systems, on the other hand, can detect drones by measuring their position, velocity, acceleration, and orientation. However, they can be affected by interference from other sources of radio waves, and can be expensive and complex to install and maintain. Acoustic tracking is another method that uses microphones to detect the sound of drones. However, this method has a limited range and may not be able to distinguish between different drones of the same model, or between drones and other noise sources. Visual tracking, such as using cameras and image processing techniques, has proven to be highly accurate and cost-effective, however existing visual tracking methods perform poorly in low visibility conditions such as fog or darkness.

[0005] From the above, we can see that how to improve the accuracy of drone tracking remains to be solved. Summary of the invention

[0006] The embodiments of the present invention provide a drone tracking method, device, system, electronic device and storage medium to solve the problem of inaccurate drone tracking in the related art.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] According to one aspect of the present invention, a drone tracking method comprises acquiring corresponding image data of a drone to be tracked; inputting the corresponding image data of the drone to be tracked into a target detection model based on deep learning, performing target detection on the drone to be tracked, and outputting a target detection result; the target detection model is obtained by training based on a drone image data set; the drone image data set comprises image data of at least two spectra; the image data of at least two spectra comprises an RGB image collected by a drone tracking system under good field of view and an infrared image collected under poor field of view; based on the target detection result and the corresponding image data of the drone to be tracked, a tracking algorithm is used to track the drone to be tracked.

[0009] According to one aspect of the present invention, a drone tracking device includes a data set acquisition module for acquiring corresponding image data of a drone to be tracked; a drone detection module for inputting the corresponding image data of the drone to be tracked into a target detection model based on deep learning, performing target detection on the drone to be tracked, and outputting a target detection result; the target detection model is obtained by training based on a drone image data set; the drone image data set includes image data of at least two spectra; the image data of at least two spectra include RGB images collected by a drone tracking system under good field of view and infrared images collected under poor field of view; the drone tracking module is used to track the drone to be tracked using a tracking algorithm based on the target detection result and the corresponding image data of the drone to be tracked.

[0010] According to one aspect of the present invention, a UAV tracking system includes a control end, an acquisition end, a detection end and a tracking end, wherein the control end is used to confirm two field of view conditions: good field of view and poor field of view; the acquisition end is used to obtain corresponding image data of the UAV to be tracked, and according to the instruction of the control end, it is determined to collect RGB images in the case of good field of view and infrared images in the case of poor field of view; the detection end is used to input the corresponding image data of the UAV to be tracked into a target detection model based on deep learning, perform target detection on the UAV to be tracked, and output the target detection result; the tracking end is used to track the UAV to be tracked by using a tracking algorithm based on the target detection result output by the detection end and the corresponding image data of the UAV to be tracked acquired by the acquisition end.

[0011] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the drone tracking method as described above.

[0012] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the drone tracking method as described above.

[0013] According to one aspect of the present invention, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the drone tracking method as described above.

[0014] The beneficial effects brought by the technical solution provided by the present invention are:

[0015] In the above technical solution, the present invention solves the problem of inaccurate UAV tracking existing in the related art.

[0016] Specifically, the present invention first obtains a drone image data set and corresponding image data of the drone to be tracked, the drone image data set includes image data of at least two spectra, namely RGB and infrared, and then constructs a target detection model based on deep learning and connects it to a tracking algorithm. The target detection model is obtained by training based on the drone image data set, and then the corresponding image data of the drone to be tracked is calculated, and the target detection result is output. Finally, the tracking algorithm is used to perform tracking calculation based on the target detection result and the corresponding image data of the drone to be tracked to achieve tracking and positioning of the drone.

[0017] In the above process, the present invention identifies the UAV to be tracked by combining the infrared image and the RGB image. When the field of view is good, the RGB image can be used to well display the details of the UAV to be tracked. When the field of view is poor, the infrared image can be used to display the UAV to be tracked without being affected by the environment. The details of the RGB image and the stability of the infrared image complement each other, thereby improving the image quality of the UAV visual tracking and effectively improving the accuracy of the UAV tracking, thereby effectively solving the problem of inaccurate UAV tracking existing in the related art.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a schematic diagram of an implementation environment according to the present invention;

[0021] Figure 2 is a flow chart of a method for tracking a drone according to an exemplary embodiment;

[0022] Figure 3 is a flow chart of a method for tracking a drone according to an exemplary embodiment;

[0023] Figure 4 is a structural schematic diagram of a drone tracking system according to an exemplary embodiment;

[0024] Figure 5 yes Figure 4 The appearance diagram of the photoelectric load in the corresponding embodiment;

[0025] Figure 6 is a schematic diagram of the effect in an application scenario according to an exemplary embodiment;

[0026] Figure 7 is a block diagram of a drone tracking device according to an exemplary embodiment;

[0027] Figure 8 is a schematic structural diagram of an electronic device according to an exemplary embodiment;

[0028] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0029] The above-mentioned drawings have shown clear embodiments of the present invention, which will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0031] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0032] The following is an introduction and explanation of several terms involved in the present invention:

[0033] RGB image: a color image. A color image can be seen as a combination of three grayscale images. Each pixel in a color image is represented by three components: R (red), G (green), and B (blue), and contains rich detail information.

[0034] Infrared image: It is an image formed by the infrared imaging device collecting the radiation of the target in the infrared band. It does not require the help of external ambient light and emits infrared light for night vision imaging. It has a wide night vision range and is less affected by the environment. It is not affected by harsh environments such as rain, snow, wind and frost. The imaging is clear and accurate, and it can identify camouflage and resist interference.

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 The schematic diagram of an implementation environment involved in a drone tracking method is shown in FIG. The implementation environment includes a drone tracking system, and the drone tracking system includes a control terminal 110 , a collection terminal 130 , a detection terminal 150 , and a tracking terminal 170 .

[0037] Specifically, the control terminal 110 is used to confirm the visual field conditions, wherein the visual field conditions include good visual field and poor visual field, so as to send corresponding instructions to the acquisition terminal 130 based on different visual field conditions. The instruction transmission between the control terminal 110 and the acquisition terminal 130 is realized by a communication connection established by wired or wireless means, which is not limited here.

[0038] The acquisition terminal 130 is used to acquire images. According to the instructions sent by the control terminal 110, it is determined to acquire RGB images when the field of view is good, and to acquire infrared images when the field of view is poor. The acquisition terminal 130 can be any electronic device with image and text acquisition functions. For example, the electronic device can be a camera for acquiring images, which is not specifically limited here.

[0039] The acquisition end 130 and the detection end 150, and the detection end 150 and the tracking end 170 can establish a communication connection through wired or wireless means to achieve data transmission between the two. For example, the transmitted data can be an RGB image or an infrared image.

[0040] The detection end 150 can also be considered as a cloud, a cloud platform, a platform end, a service end, etc. The detection end 150 can be a server, a desktop computer, a laptop computer, etc., or a server cluster composed of multiple servers, or a cloud computing center composed of multiple servers, so as to better provide background services to the mass collection end 130. For example, the background service includes a drone detection service.

[0041] The tracking end 170 can also be considered as a cloud, a cloud platform, a platform end, a service end, etc. The tracking end 150 can be a server, a desktop computer, a laptop computer, etc., or a server cluster composed of multiple servers, or a cloud computing center composed of multiple servers, so as to better provide background services to the mass collection end 130 and the detection end 170. For example, the background service includes a drone tracking service.

[0042] Taking the tracking end 170 providing drone tracking service as an example, with the interaction between the control end 110 and the collection end 130, the interaction between the collection end 130 and the detection end 150, and the interaction between the detection end 150 and the tracking end 170, in an application scenario, the control end 110 sends the corresponding instruction to the collection end 130 after confirming the field of view, and the collection end 130 collects the corresponding image according to the received instruction. After the collection end 130 collects the image, it sends the image to the tracking end 150. Then, the detection end 150 can receive the image sent by the collection end 130, and then provide drone detection service based on the image. After the detection end 150 detects the drone, it sends the detection result to the tracking end 150. Then, the tracking end 150 can receive the detection result sent by the detection end 150, and then provide drone tracking service based on the detection result. Specifically, the detection results obtained by the detection end 150 are used to indicate whether there is a drone in the image and the detection area used to indicate the location of the drone in the image. Finally, the drone in the detection area is tracked in the tracking end 170 based on the detection area and image data to achieve tracking and positioning of the drone.

[0043] Of course, in other application scenarios, the detection end and the tracking end can also be integrated and deployed on the same electronic device, which does not constitute a specific limitation here.

[0044] See also Figure 2 The embodiment of the present invention provides a method for tracking a drone, which is applicable to an electronic device, which may be Figure 1 The detection end 150 and the tracking end 170 in the drone tracking system in the implementation environment are shown.

[0045] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0046] like Figure 2 As shown, the method may include the following steps:

[0047] Step 210: Acquire corresponding image data of the drone to be tracked.

[0048] The image data includes RGB images and infrared images.

[0049] The image data is obtained by shooting and collecting the environment where the target to be tracked is located through an image acquisition device. The image acquisition device can be an electronic device with an image acquisition function, such as a camera, a smart phone equipped with a camera, etc. The image acquisition device can be deployed around the environment where the target to be tracked is located.

[0050] It can be understood that the shooting can be a single shot or continuous shooting. Then, for the same target to be tracked, for continuous shooting, a video can be obtained, and the image can be any frame in the video. For multiple shots, multiple photos can be obtained, and the image can be any one of the multiple photos. In other words, the image in this embodiment can come from a dynamic image, such as multiple frames in a video, or multiple photos, and can also come from a static image, such as any frame in a video, or any one of multiple photos. Accordingly, the drone tracking in this embodiment is performed in frames.

[0051] In one possible implementation, an embodiment of the present invention uses an uncooled infrared focal plane detector in combination with an infrared lens to collect corresponding image data of the flight of the drone to be tracked at different distances, different positions in the field of view, and different backgrounds.

[0052] Step 230: Input the corresponding image data of the UAV to be tracked into a target detection model based on deep learning, perform target detection on the UAV to be tracked, and output the target detection result.

[0053] The target detection model is trained based on a drone image dataset. The drone image dataset includes at least two spectral image data; the at least two spectral image data include: RGB images collected by a drone tracking system under good field of view and infrared images collected under poor field of view.

[0054] In one possible implementation, the training process of the target detection model includes obtaining a drone image data set, the data set includes training samples, the training samples are image data that annotates the detection area of ​​the drone to be tracked and include at least two spectra, RGB and infrared; the current training sample in the drone image data set is input into the machine learning model for training to obtain a loss value; if the loss value meets the convergence condition, the target detection model is obtained; otherwise, the model parameters of the machine learning model are updated, and another training sample in the drone image data set is obtained and input into the machine learning model with the updated model parameters to continue training until the loss value meets the convergence condition.

[0055] In a possible implementation, the corresponding image data of the UAV to be tracked is annotated using the open source annotation software Labelme. During the annotation, the UAV to be tracked is selected with a rectangular box by moving the mouse, so that the rectangular box is as close as possible to the edge of the UAV to be tracked.

[0056] In a possible implementation, by inputting an image into a trained target detection model, first adjusting the image size, then taking a value for every other pixel of the image, compressing the width and height of the image, expanding the number of channels, and then through the processing of convolution-normalization-activation function, performing channel adjustment and feature extraction, inputting the feature layer into the feature pyramid, performing feature fusion, strengthening feature extraction, and finally decoupling and outputting the target detection result, the target detection result at least includes whether there is a drone to be tracked in the image and the detection area of ​​the drone to be tracked for indicating the position of the drone to be tracked in the image. Of course, in other embodiments, whether there is a drone to be tracked can also be determined by whether there is a target and the target category in the target detection result. For example, when there is a target and the target category is a drone to be tracked, it can be determined that there is a drone to be tracked, which does not constitute a specific limitation here.

[0057] The image features may be color features, shape features, texture features, edge features, etc., which are not specifically limited here.

[0058] Specifically, the extraction of color features can analyze the color presented by a specific pixel point in a variety of ways and extract its color feature components. For example, the color features of a specific area can be extracted by manually marking the area, and the area can be represented by the average values ​​of the three components of the area in a color space, or three color histograms can be established. The extraction of shape features must be based on the segmentation of objects or areas in the image. The algorithms for extracting shape features include SIFT and HOG. Texture features are a kind of visual features that reflect homogeneous phenomena in images. They reflect the surface structure organization and arrangement properties of the object surface with slow or periodic changes. They are expressed by the grayscale distribution of pixels and their surrounding spatial neighborhoods. The extraction of texture features includes feature extraction algorithms such as local binary patterns and grayscale co-occurrence matrices. Edge features are intended to represent edges or discontinuous areas with obvious changes in a digital image. An edge is a boundary line between different areas in an image. Usually an edge image is a binary image. Edge features can reflect areas with sharp changes in brightness. The extraction of edge features includes the Canny algorithm and the Sobel algorithm, which are not limited here.

[0059] Step 250 : Based on the target detection result and the corresponding image data of the UAV to be tracked, a tracking algorithm is used to track the UAV to be tracked.

[0060] In a possible implementation, a tracking algorithm is used to track the unmanned aerial vehicle to be tracked in the detection frame according to the detection frame and the image data, such as the tracking algorithm KCF, etc., which is not limited here.

[0061] Through the above process, the embodiment of the present invention identifies the drone to be tracked by combining the infrared image and the RGB image. When the field of view is good, the RGB image can be used to well display the details of the drone to be tracked. When the field of view is poor, the infrared image can be used to display the drone to be tracked without being affected by the environment. The details of the RGB image and the stability of the infrared image complement each other, thereby improving the image quality of the drone visual tracking and effectively improving the accuracy of the drone tracking, thereby effectively solving the problem of inaccurate drone tracking existing in the related art.

[0062] See also Figure 3 The embodiment of the present invention provides a method for tracking a drone, which is applicable to an electronic device, which may be Figure 1 The detection end 150 and the tracking end 170 in the implementation environment are shown.

[0063] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0064] like Figure 3As shown, the method may include the following steps:

[0065] Step S1, in response to a trigger operation of a control terminal in a UAV tracking system, a message indicating a visual field condition is obtained.

[0066] Among them, the visual field conditions include good visual field and poor visual field. Good visual field refers to sunny days, high visibility and other conditions, while poor visual field refers to rain and fog, darkness, strong light conditions, extreme weather and other conditions, which are not limited here.

[0067] For example, when a button indicating a good field of view is triggered in the control terminal, information indicating a good field of view is transmitted to the acquisition terminal.

[0068] When the button indicating poor visual field in the control terminal is triggered, the information indicating the poor visual field is transmitted to the acquisition terminal.

[0069] Step S2, when the message indicates that the field of view is good, the RGB image is obtained through the acquisition end in the drone tracking system; when the message indicates that the field of view is poor, the infrared image is obtained through the acquisition end in the drone tracking system.

[0070] Among them, when the field of view is good, the RGB image can well show the details of the UAV to be tracked, and when the field of view is poor, the infrared image is almost unaffected, maintaining stable image quality.

[0071] Step S3, performing image compression and channel adjustment on the current frame image in the corresponding image data of the UAV to be tracked to obtain an intermediate image.

[0072] In a possible implementation, the image is resized to 640*640, and then a value is taken for every other pixel in the image to compress the width and height of the image and expand the number of channels.

[0073] Step S4, performing feature extraction and feature fusion on the intermediate image to obtain a feature image.

[0074] In a possible implementation, channel adjustment and feature extraction are performed through a 5-layer convolution-normalization-activation function, and the feature layer is input into a feature pyramid to perform feature fusion and enhance feature extraction to obtain a feature image.

[0075] Step S5, positioning the drone to be tracked in the corresponding image data of the drone to be tracked according to the feature image to obtain a target detection result.

[0076] In a possible implementation, a detection end in a drone tracking system performs drone detection on a feature image to obtain a target detection result.

[0077] Step S6, based on the target detection result, determine the detection area of ​​the drone to be tracked in the current frame image, find the position area corresponding to the detection area of ​​the drone to be tracked in the subsequent frame image of the corresponding image data, and sample the pixels in the found position area to obtain multiple sampling data; each sampling data includes multiple sampling points.

[0078] Specifically, a detection area of ​​the drone to be tracked is detected in the current frame of the multi-frame image data, and a position corresponding to the detection area in the image data after the current frame in the multi-frame image data is used as a sampling point, and the sampling point is sampled to obtain multiple sampling data.

[0079] Step S7, performing correlation operations on the multiple sampling points in each sampling data and the multiple pixel points corresponding to the detection area of ​​the to-be-tracked UAV in the current frame image, to obtain the response values ​​of the multiple sampling points in each sampling data.

[0080] Specifically, the sampling data corresponding to each sampling point is correlated with the image data at the position corresponding to each sampling point in the detection area of ​​the unmanned aerial vehicle to be tracked to obtain the response value of each sampling point.

[0081] Step S8, determining the detection area of ​​the UAV to be tracked in the subsequent frame image according to the response values ​​of the multiple sampling points in each sampling data, so as to achieve tracking and positioning of the UAV to be tracked.

[0082] Specifically, the position of the UAV to be tracked in the current image data is obtained according to the response value. For example, the sampling point with the largest response value is the position of the UAV to be tracked, the sampling point with a response value that meets a set threshold is the position of the UAV to be tracked, etc., which is not limited here.

[0083] Through the above process, the embodiment of the present invention first obtains a drone image data set and corresponding image data of the drone to be tracked, the drone image data set includes image data of at least two spectra, RGB and infrared, obtains the drone image data by dual-spectrum imaging, and then constructs a target detection model based on deep learning, and docks with the tracking algorithm. The target detection model is obtained based on the training of the drone image data set, and then the corresponding image data of the drone to be tracked is calculated, and the target detection result is output. Finally, the tracking algorithm is used to perform tracking calculation based on the target detection result and the corresponding image data of the drone to be tracked to achieve tracking and positioning of the drone to be tracked, thereby effectively solving the problem of inaccurate drone tracking in the related art.

[0084] See also Figure 4 An embodiment of the present invention provides a drone tracking system, which is used to implement a drone tracking method. The components of the system are now described, but this does not constitute a specific limitation.

[0085] like Figure 4 As shown, the system may include the following components: a carrier aircraft, an optoelectronic payload, and ground debugging equipment. Figure 1 The detection end and tracking end in the photoelectric load correspond to Figure 1 The acquisition end in the ground debugging equipment corresponds to Figure 1 The aircraft carrier includes antenna, image transmission equipment, lithium battery, flight control computer and data link.

[0086] Specifically, Figure 5 As shown, the optoelectronic payload includes a stable platform 510, an optical sensor 530 and an electronic processing unit 550, wherein the stable platform 510 is used to provide structural support for the entire acquisition end and assist the optical sensor in image capture by moving, the optical sensor 530 is used to capture images, and the electronic processing unit 550 is used to transmit the images captured by the optical sensor and control the stable platform to move and assist the optical sensor in capturing images. The optoelectronic payload is used to acquire images, and according to the instructions of the ground debugging equipment, RGB images are captured when the field of view is good, and infrared images are captured when the field of view is poor; the carrier is used to track the drone according to the images acquired by the acquisition end; the ground debugging equipment is used to confirm the field of view, which includes good field of view and poor field of view.

[0087] In the above process, the embodiment of the present invention carries out targeted design according to actual needs, takes the improvement of reconnaissance performance as the core goal, is characterized by miniaturization, lightness and intelligence, has more comprehensive tactical functions and more advanced technical indicators, and realizes flexible collection of high-quality image data of the drone to be tracked through a small volume and integrated optoelectronic payload.

[0088] Among them, the antenna is used to connect to the network; the image transmission device is used to receive the corresponding image data of the UAV to be tracked obtained by the acquisition end; the lithium battery is used to power the entire tracking end; the flight control computer is used to track the UAV to be tracked using a tracking algorithm based on the target detection results output by the detection end based on the corresponding image data of the UAV to be tracked obtained by the acquisition end, and obtain the tracking results; the data link is used to store the tracking results obtained by the flight control computer.

[0089] In one possible implementation, the photoelectric payload can be used to transmit infrared and RGB spectral images. Then, based on the collected images, data annotation is first performed, and the deep learning model is trained to obtain the model. Different spectra are used for detection in good and bad fields of view, respectively, and then the detection frame is passed to the KCF algorithm for subsequent tracking.

[0090] In one possible implementation, the YOLOv5 target detection network is trained for drone target detection, and image datasets under two spectra (RGB / infrared) are established. An uncooled infrared focal plane detector is combined with an infrared lens to collect flight pictures of the drone to be tracked at different distances, different positions in the field of view, and different backgrounds. The collected pictures are annotated using the open source annotation software Labelme. When annotating, the target drone to be tracked is selected with a rectangular box by moving the mouse. The rectangular box should be as close to the edge of the target drone to be tracked as possible. The format of the dataset is VOC2007, and the annotation information will be saved in an XML suffix file with the same name as the picture.

[0091] Through the above process, the embodiment of the present invention obtains image data through the drone tracking system, then performs drone detection on the image in the image data to obtain a detection result, and finally tracks the drone to be tracked in the detection frame according to the detection frame and the image data, thereby realizing the tracking and positioning of the drone to be tracked, and complementing each other through the details of the RGB image and the stability of the infrared image, thereby improving the image quality of drone visual tracking, and effectively improving the accuracy of drone tracking, thereby effectively solving the problem of inaccurate drone tracking existing in related technologies.

[0092] Figure 6 The schematic diagram shows the effect of the embodiment of the present invention in the application scenario. It can be seen from the figure that the embodiment of the present invention selects the position of the UAV to be tracked at each moment in the picture, and marks the running trajectory of the UAV to be tracked, clearly showing the tracking route of the UAV to be tracked and the position of the UAV to be tracked at each moment and the specific parameters in the tracking process, which intuitively reflects the embodiment of the present invention.

[0093] The following is an embodiment of the device of the present invention, which can be used to perform the drone tracking method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the drone tracking method involved in the present invention.

[0094] See also Figure 7 An embodiment of the present invention provides a drone tracking device 700, including but not limited to: a data set acquisition module 710, a drone detection module 730 and a drone tracking module 750.

[0095] The data set acquisition module 710 is used to acquire corresponding image data of the drone to be tracked.

[0096] The drone detection module 730 is used to input the corresponding image data of the drone to be tracked into a target detection model based on deep learning, perform target detection on the drone to be tracked, and output the target detection result; the target detection model is obtained by training the drone image data set; the drone image data set includes at least two spectral image data; the at least two spectral image data include RGB images collected by the drone tracking system under good field of view and infrared images collected under poor field of view.

[0097] The drone tracking module 750 is used to track the drone to be tracked by using a tracking algorithm based on the target detection result and the corresponding image data of the drone to be tracked.

[0098] It should be noted that the drone tracking device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing drone tracking. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the drone tracking device will be divided into different functional modules to complete all or part of the functions described above.

[0099] In addition, the drone tracking device and the drone tracking method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0100] Figure 8 A schematic diagram of the structure of an electronic device according to an exemplary embodiment. The electronic device is suitable for Figure 1 The collecting end 150 and the tracking end 170 in the implementation environment are shown.

[0101] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 8 One or more components of exemplary electronic device 2000 are shown.

[0102] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 8 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0103] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0104] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Figure 1 The interaction between the collection end 130 and the tracking end 150 in the implementation environment is shown.

[0105] Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 8 As shown, this is not intended to be a specific limitation.

[0106] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0107] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0108] The application program 253 is a computer program that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 8 (not shown), each module may respectively include a computer program for the electronic device 2000. For example, the drone tracking device may be regarded as an application 253 deployed on the electronic device 2000.

[0109] The data 255 may be images, texts, etc. stored in a disk, or training sets, etc., stored in the memory 250 .

[0110] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the drone tracking method is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0111] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0112] See also Fig. 9In an embodiment of the present invention, an electronic device 4000 is provided, and the electronic device 400 may include: a drone tracking system, etc.

[0113] exist Fig. 9 In the embodiment, the electronic device 4000 includes at least one processor 4001, at least one communication bus 4002 and at least one memory 4003.

[0114] The processor 4001 and the memory 4003 are connected, such as through a communication bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0115] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0116] The communication bus 4002 may include a path to transmit information between the above components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0117] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0118] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0119] The computer-readable instructions are executed by one or more processors 4001 to implement the drone tracking method in the above-mentioned embodiments.

[0120] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the drone tracking method as described above.

[0121] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the drone tracking method as described above.

[0122] Compared with the related art, the present invention has the following beneficial effects:

[0123] 1. The present invention proposes a method for tracking drones. The present invention identifies the drone to be tracked by combining infrared images and RGB images. When the field of view is good, the RGB image can well display the details of the drone to be tracked. When the field of view is poor, the infrared image can display the drone to be tracked without being affected by the environment. The details of the RGB image and the stability of the infrared image complement each other, thereby improving the image quality of the drone visual tracking and effectively improving the accuracy of drone tracking, thereby effectively solving the problem of inaccurate drone tracking in related technologies.

[0124] 2. The present invention combines multi-spectrum with deep learning target detection and tracking algorithms. Deep learning target detection is used for detection to determine the detection area of ​​the drone to be tracked. The KCF algorithm is used for further tracking to complete positioning. The combined application of infrared / RGB multi-spectrum perfectly solves the problems caused by a single spectrum.

[0125] 3. The present invention combines the fastest existing detection and tracking algorithms with the stability of multi-spectral decision making. The present invention can achieve near real-time detection in a CPU-only environment without calling CUDA.

[0126] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0127] The above contents are only preferred exemplary embodiments of the present invention and are not intended to limit the implementation scheme of the present invention. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope required by the claims.

Claims

1. A drone tracking method, characterized in that: The method comprises: Obtain corresponding image data of the UAV to be tracked; Inputting corresponding image data of the UAV to be tracked into a target detection model based on deep learning, performing target detection on the UAV to be tracked, and outputting a target detection result; the target detection model is obtained by training based on a UAV image data set; the UAV image data set includes image data of at least two spectra; the image data of at least two spectra include RGB images collected by the UAV tracking system under good field of view and infrared images collected under poor field of view; Based on the target detection result and the corresponding image data of the UAV to be tracked, a tracking algorithm is adopted to track the UAV to be tracked.

2. The method according to claim 1, characterized in that The step of obtaining corresponding image data of the drone to be tracked includes: In response to a trigger operation of a control terminal in the drone tracking system, a message indicating a visual field condition is obtained; When the field of vision is good, the RGB image is obtained through the acquisition terminal in the UAV tracking system; or When the field of vision is poor, infrared images are obtained through the acquisition terminal in the UAV tracking system; Corresponding image data of the UAV to be tracked is obtained from several frames of RGB images and / or several frames of infrared images.

3. The method according to claim 1, characterized in that The step of calculating the corresponding image data of the drone to be tracked and outputting the target detection result includes: Performing image compression and channel adjustment on the current frame image in the corresponding image data of the unmanned aerial vehicle to be tracked to obtain an intermediate image; Performing feature extraction and feature fusion on the intermediate image to obtain a feature image; The drone to be tracked in the corresponding image data of the drone to be tracked is located according to the feature image to obtain a target detection result; the target detection result at least includes whether the drone to be tracked exists in the image data and a detection area of ​​the drone to be tracked.

4. The method according to claim 1, characterized in that The method of tracking the UAV to be tracked by using a tracking algorithm based on the target detection result and the corresponding image data of the UAV to be tracked includes: Determine a detection area of ​​the unmanned aerial vehicle to be tracked in the current frame image based on the target detection result; In subsequent frame images of the corresponding image data, a position area corresponding to the detection area of ​​the drone to be tracked is searched, and pixels in the searched position area are sampled to obtain a plurality of sampling data; each of the sampling data includes a plurality of sampling points; Correlation operations are performed on the plurality of sampling points in each of the sampling data with the plurality of pixel points corresponding to the detection area of ​​the to-be-tracked UAV in the current frame image to obtain response values ​​of the plurality of sampling points in each of the sampling data; The detection area of ​​the unmanned aerial vehicle to be tracked in the subsequent frame image is determined according to the response values ​​of the multiple sampling points in each of the sampling data, so as to realize the tracking and positioning of the unmanned aerial vehicle to be tracked.

5. The method according to any one of claims 1 to 4, characterized in that: The training process of the target detection model includes: Acquire a drone image dataset, wherein the drone image dataset includes training samples, wherein the training samples are image data of at least two spectra, RGB and infrared, with the detection area of ​​the drone to be tracked being annotated; Inputting a current training sample in the drone image data set into the machine learning model for training to obtain a loss value; If the loss value satisfies the convergence condition, the target detection model is obtained; Otherwise, the model parameters of the machine learning model are updated, and another training sample in the drone image data set is obtained to input the updated model parameters to continue training the machine learning model until the loss value meets the convergence condition.

6. A drone tracking device, characterized in that: The device comprises: A data set acquisition module is used to acquire corresponding image data of the UAV to be tracked; A drone detection module is used to input the corresponding image data of the drone to be tracked into a target detection model based on deep learning, perform target detection on the drone to be tracked, and output a target detection result; the target detection model is obtained by training based on a drone image data set; the drone image data set includes image data of at least two spectra; the image data of at least two spectra include RGB images collected by the drone tracking system under good field of view and infrared images collected under poor field of view; The drone tracking module is used to track the drone to be tracked by using a tracking algorithm based on the target detection result and the corresponding image data of the drone to be tracked.

7. A drone tracking system, characterized in that: The system includes a control end, a collection end, a detection end and a tracking end, wherein: The control terminal is used to confirm two visual field conditions: good visual field and poor visual field; The acquisition end is used to obtain the corresponding image data of the UAV to be tracked, and collects RGB images in the case of good vision and infrared images in the case of poor vision according to the instructions of the control end; The detection end is used to input the corresponding image data of the UAV to be tracked into a target detection model based on deep learning, perform target detection on the UAV to be tracked, and output the target detection result; The tracking end is used to track the UAV to be tracked by using a tracking algorithm based on the target detection result output by the detection end and the corresponding image data of the UAV to be tracked acquired by the acquisition end.

8. The system according to claim 7, characterized in that The control terminal is used to confirm two vision conditions: good vision and poor vision, including: When the button indicating a good field of view in the control terminal is triggered, a signal indicating a good field of view is transmitted to the acquisition terminal; When the button indicating poor visual field in the control end is triggered, a signal indicating poor visual field is transmitted to the acquisition end.

9. The system according to claim 7, characterized in that The acquisition end includes a stable platform, an optical sensor and an electronic processing unit, wherein: The stable platform is used to provide structural support for the entire acquisition end and assist the optical sensor in image capture by moving; The optical sensor is used to capture images; The electronic processing unit is used to transmit the image captured by the optical sensor and control the movement of the stabilizing platform to assist the optical sensor in shooting.

10. The system according to claim 7, characterized in that The tracking end includes an antenna, an image transmission device, a lithium battery, a flight control computer and a data link, wherein: The antenna is used to connect to the network; the image transmission device is used to receive the corresponding image data of the drone to be tracked acquired by the acquisition end; the lithium battery is used to power the entire tracking end; The flight control computer is used to track the UAV to be tracked using a tracking algorithm based on the target detection result output by the detection end for the corresponding image data of the UAV to be tracked acquired by the acquisition end, and obtain a tracking result; the data link is used to store the tracking result obtained by the flight control computer.

Images