A visible light unmanned aerial vehicle air situation positioning method, device, equipment and storage medium

Abstract

The application discloses a visible light unmanned aerial vehicle air space positioning method and device, equipment and a storage medium, and relates to the technical field of target detection. The method comprises the following steps: training a target detection model and then respectively deploying the target detection model to a ground monitoring device and at least two optical cameras; obtaining a first image by shooting a target air space through the ground monitoring device, and identifying an unmanned aerial vehicle from the first image by using the target detection model; obtaining a second image by shooting the target air space through each optical camera, and judging whether the target air space has the unmanned aerial vehicle or not by using the target detection model according to the second image; and if yes, identifying the position of the unmanned aerial vehicle according to the positions of the ground monitoring device and the optical cameras. By training and deploying the target detection model, the position of the unmanned aerial vehicle can be detected by at least two optical cameras in addition to the ground monitoring device, the cost of positioning the unmanned aerial vehicle can be greatly reduced, and the position of the unmanned aerial vehicle can be accurately identified.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a visible light UAV air situation positioning method, device, equipment and storage medium. Background Technology

[0002] With the rapid development of the low-altitude economy, how to regulate airspace has become a major challenge in maintaining public safety. Most existing drone identification and positioning technologies utilize the reflection characteristics of radio waves to determine the drone's location by receiving the radio signals emitted by the drone. This technology is called radio frequency scanning technology, but the hardware cost of such equipment is relatively high. Another method uses visible light recognition detection algorithms and deep learning to identify drones in the image, but this method can only detect the presence of drones, not pinpoint their location. Summary of the Invention

[0003] The main objective of this application is to propose a visible light UAV air situation positioning method, device, equipment, and storage medium to reduce the cost of identifying UAV locations and to accurately identify UAV locations.

[0004] To achieve the above objectives, one aspect of this application proposes a visible light UAV air situation positioning method, the method comprising the following steps:

[0005] The target detection model is trained and then deployed on ground monitoring equipment and at least two optical cameras respectively; wherein the ground monitoring equipment and each of the optical cameras are located in different positions;

[0006] The ground monitoring equipment captures a first image of the target airspace, and the target detection model identifies the drone based on the first image.

[0007] The optical cameras are used to capture a second image of the target airspace, and the target detection model is used to determine whether a drone exists in the target airspace based on the second image.

[0008] If so, the location of the UAV is determined based on the positions of the ground monitoring equipment and each of the optical cameras.

[0009] In some embodiments, training the target detection model includes the following steps:

[0010] Obtain drone image dataset;

[0011] The original YOLOv11 object detection model is improved to obtain the improved YOLOv11 object detection model, which is used as the object detection model.

[0012] A first loss function is constructed to optimize the accuracy of target category classification, and a second loss function is constructed to optimize the location of detection boxes, thereby obtaining the final loss function;

[0013] The improved YOLOv11 target detection model is trained based on the UAV image dataset, the first loss function, and the second loss function.

[0014] In some embodiments, acquiring the drone image dataset includes the following steps:

[0015] Acquire third-party images including birds and drones;

[0016] A fourth image, including one from a drone, was captured using non-dedicated camera equipment;

[0017] Obtain a fifth image that includes distractors and a random background;

[0018] Acquire the sixth image taken by the drone under different weather conditions, lighting conditions, detection distances, and drone models;

[0019] The third, fourth, fifth, and sixth images are blurred and noise is added to obtain an initial image dataset;

[0020] The drone image dataset is obtained by labeling detection boxes for drones in the initial image dataset.

[0021] In some embodiments, improving the original YOLOv11 object detection model to obtain an improved YOLOv11 object detection model as the object detection model includes the following steps:

[0022] Replace the C2PSA module in the backbone network of the original YOLOv11 target detection model with the C2LSKA module;

[0023] Replace the PSA module of the backbone network in the original YOLOv11 target detection model with the LSKA module;

[0024] The improved YOLOv11 target detection model's backbone network further includes a C3K2 structure, an SPPF structure, and a detection head structure.

[0025] In some embodiments, constructing a first loss function for optimizing target category classification accuracy and a second loss function for optimizing detection box positions, thereby constructing the final loss function, includes the following steps:

[0026] The first loss function is constructed as follows:

[0027] ;

[0028] in, This represents the first loss function. A coefficient representing the weights that balance positive and negative samples; This represents the predicted probability of the target detection model for the correct category; This represents the focusing parameter used to adjust the weights of easy and difficult samples;

[0029] The second loss function is constructed as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] in, This represents the second loss function; This represents the detection box predicted by the target detection model. This represents the actual detection bounding box. and These represent the width and height of the actual detection bounding box, respectively, with scale being the scaling factor. and These represent the x-axis coordinates of the center points of the predicted and ground truth bounding boxes, respectively. and These represent the y-axis coordinates of the center points of the predicted and actual detection boxes, respectively.

[0038] Therefore, the final loss function of the object detection model is:

[0039] ;

[0040] in, This represents the final loss function. , All are balancing factors.

[0041] In some embodiments, the step of using the target detection model to identify the drone based on the image includes the following steps:

[0042] The target detection model is used to annotate detection boxes for the UAV on the image;

[0043] Determine the confidence level that the detection frame contains a drone;

[0044] If the confidence level reaches a preset threshold, it is determined that a drone exists in the image.

[0045] In some embodiments, determining the location of the drone based on the positions of the ground monitoring equipment and each of the optical cameras includes the following steps:

[0046] Based on the position, azimuth angle, and elevation angle of the ground monitoring equipment and each of the optical cameras, corresponding rays are drawn out respectively;

[0047] The distances to each ray in space and the nearest point are calculated as the location of the UAV.

[0048] To achieve the above objectives, another aspect of this application proposes a visible light UAV air situation positioning device, the device comprising:

[0049] A model training and deployment unit is used to train an object detection model and then deploy the object detection model to a ground monitoring device and at least two optical cameras respectively; wherein the ground monitoring device and each of the optical cameras are located in different positions;

[0050] The preliminary identification unit is used to capture a first image of the target airspace through the ground monitoring equipment and use the target detection model to identify the UAV based on the first image.

[0051] The identification unit is used to capture a second image of the target airspace using each of the optical cameras and to use the target detection model to determine whether a drone exists in the target airspace based on the second image.

[0052] The UAV positioning unit is used to determine the location of the UAV based on the ground monitoring equipment and the positions of each of the optical cameras if the UAV is identified in the target airspace by using the target detection model based on the second image.

[0053] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0054] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0055] The embodiments of this application include at least the following beneficial effects:

[0056] This application trains a target detection model and deploys it on ground monitoring equipment and at least two optical cameras. The ground monitoring equipment and each optical camera are located in different positions. A first image is captured by the ground monitoring equipment of the target airspace, and the target detection model identifies the drone based on the first image. Second images are captured by each optical camera of the target airspace, and the target detection model determines whether a drone exists in the target airspace based on the second image. If so, the drone's location is determined based on the positions of the ground monitoring equipment and each optical camera. By training and deploying the target detection model, this application can detect the location of a drone using at least two optical cameras, in addition to the ground monitoring equipment, significantly reducing the cost of drone location and enabling accurate identification of the drone's location based on the ground monitoring equipment and each optical camera. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating a visible light UAV air situation positioning method provided in an embodiment of this application;

[0059] Figures 2(a) to 2(e) are examples of drones with different datasets provided in the embodiments of this application;

[0060] Figure 3 Example images of blurred images provided in the embodiments of this application;

[0061] Figure 4 This is an example image of an image with added noise provided in an embodiment of this application;

[0062] Figure 5 A schematic diagram of the original YOLOv11 target detection model structure provided for embodiments of this application;

[0063] Figure 6A schematic diagram of the improved YOLOv11 target detection model structure provided in the embodiments of this application;

[0064] Figure 7 A schematic diagram of the C3K2 structure provided in the embodiments of this application;

[0065] Figure 8 A schematic diagram of the SPPF structure provided in the embodiments of this application;

[0066] Figure 9 A schematic diagram of the C2LSKA structure provided in the embodiments of this application;

[0067] Figure 10 This is a schematic diagram of the detection head structure provided in an embodiment of this application;

[0068] Figure 11 A schematic diagram illustrating the annotation of rays in an image, provided as an embodiment of this application;

[0069] Figure 12(a) and Figure 12(b) are the front view and top view of the embodiment of this application, respectively, showing how to solve for the position of a target point using two rays.

[0070] Figures 13(a) and 13(b) are front and top views of the embodiment of this application, respectively, showing how to solve for the position of a target point using three rays.

[0071] Figure 14 This is a schematic diagram of the structure of a visible light UAV air situation positioning device provided in an embodiment of this application;

[0072] Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0074] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0075] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0077] This application provides a visible light UAV air situation positioning method, apparatus, device, and storage medium, relating to the field of target detection technology. The visible light UAV air situation positioning method, apparatus, device, and storage medium provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a visible light UAV air situation positioning method, but is not limited to the above forms.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] Reference Figure 1 This application provides a visible light UAV air situation positioning method, which may include, but is not limited to, steps S100 to S130, as follows:

[0080] S100: Train the target detection model and then deploy the target detection model on the ground monitoring equipment and at least two optical cameras respectively; wherein the ground monitoring equipment and each of the optical cameras are located in different positions;

[0081] S110: The ground monitoring equipment captures a first image of the target airspace and uses the target detection model to identify the UAV based on the first image;

[0082] S120: The optical cameras are used to capture a second image of the target airspace, and the target detection model is used to determine whether a drone exists in the target airspace based on the second image;

[0083] S130: If so, the position of the UAV is determined by the position identification of the ground monitoring equipment and each of the optical cameras.

[0084] Optionally, the training of the target detection model includes the following steps:

[0085] Obtain drone image dataset;

[0086] The original YOLOv11 object detection model is improved to obtain the improved YOLOv11 object detection model, which is used as the object detection model.

[0087] A first loss function is constructed to optimize the accuracy of target category classification, and a second loss function is constructed to optimize the location of detection boxes, thereby obtaining the final loss function;

[0088] The improved YOLOv11 target detection model is trained based on the UAV image dataset, the first loss function, and the second loss function.

[0089] Optionally, acquiring the drone image dataset includes the following steps:

[0090] Acquire third-party images including birds and drones;

[0091] A fourth image, including one from a drone, was captured using non-dedicated camera equipment;

[0092] Obtain a fifth image that includes distractors and a random background;

[0093] Acquire the sixth image taken by the drone under different weather conditions, lighting conditions, detection distances, and drone models;

[0094] The third, fourth, fifth, and sixth images are blurred and noise is added to obtain an initial image dataset;

[0095] The drone image dataset is obtained by labeling detection boxes for drones in the initial image dataset.

[0096] Optionally, the step of improving the original YOLOv11 object detection model to obtain an improved YOLOv11 object detection model as the object detection model includes the following steps:

[0097] Replace the C2PSA module in the backbone network of the original YOLOv11 target detection model with the C2LSKA module;

[0098] Replace the PSA module of the backbone network in the original YOLOv11 target detection model with the LSKA module;

[0099] The improved YOLOv11 target detection model's backbone network further includes a C3K2 structure, an SPPF structure, and a detection head structure.

[0100] Optionally, the construction of a first loss function for optimizing the accuracy of target category classification and a second loss function for optimizing the location of detection boxes, thereby constructing the final loss function, includes the following steps:

[0101] The first loss function is constructed as follows:

[0102] ;

[0103] in, This represents the first loss function. A coefficient representing the weights that balance positive and negative samples; This represents the predicted probability of the target detection model for the correct category; This represents the focusing parameter used to adjust the weights of easy and difficult samples;

[0104] The second loss function is constructed as follows:

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, This represents the second loss function; This represents the detection box predicted by the target detection model. This represents the actual detection bounding box. and These represent the width and height of the actual detection bounding box, respectively, with scale being the scaling factor. and These represent the x-axis coordinates of the center points of the predicted and ground truth bounding boxes, respectively. and These represent the y-axis coordinates of the center points of the predicted and actual detection boxes, respectively.

[0113] Therefore, the final loss function of the object detection model is:

[0114] ;

[0115] in, This represents the final loss function. , All are balancing factors.

[0116] Optionally, the step of using the target detection model to obtain the drone based on the image recognition includes the following steps:

[0117] The target detection model is used to annotate detection boxes for the UAV on the image;

[0118] Determine the confidence level that the detection frame contains a drone;

[0119] If the confidence level reaches a preset threshold, it is determined that a drone exists in the image.

[0120] Optionally, the step of identifying the location of the UAV based on the positions of the ground monitoring equipment and each of the optical cameras includes the following steps:

[0121] Based on the position, azimuth angle, and elevation angle of the ground monitoring equipment and each of the optical cameras, corresponding rays are drawn out respectively;

[0122] The distances to each ray in space and the nearest point are calculated as the location of the UAV.

[0123] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0124] This embodiment aims to provide a low-cost drone positioning solution that can locate drones using as few as two optical cameras. By combining deep learning algorithms with traditional algorithms, it can not only identify the drone model but also locate the drone's position.

[0125] Specifically, this embodiment may include the following technical solutions:

[0126] 1. Data collection.

[0127] First, we collected drone-related data based on open-source datasets online, including Drone vs Birds, drone tracking datasets, Cranfield-Synthetic-Drone-Detection, and DroneDataset. The Drone vs Birds dataset, in particular, encompasses a wealth of drone and environmental data, as well as some bird data. Drones at a distance often resemble birds in appearance. Therefore, this dataset can help researchers study the problem of distinguishing between drones and birds at long distances. The drone tracking datasets are primarily used to study drone tracking and localization techniques, but they also include drone images captured by various consumer cameras (smartphones, GoPros, etc.), which perfectly meets the ground-view requirements of this project. The Cranfield-Synthetic-Drone-Detection dataset is a fully synthetic dataset of drones generated using structured domain randomization. It contains multiple datasets generated using different styles: drones and birds only, general noise, realistic noise, random backgrounds, etc. The dataset also simulates drone size data at different distances (20m, 40m, 80m, 160m, 320m). The Drone Dataset is mainly prepared for academic and competition purposes, but it also contains about 1,400 drone images and label files. Most of the images are from image search tools from Google and Yandex, obtained through image (web) crawling, while other images are cropped from drone videos downloaded from YouTube.

[0128] However, this method can only cover some drone models. Therefore, this embodiment also manually collected drone-related materials, including visible light images of drones under different weather conditions, lighting conditions, detection distances, and drone models. An example of the dataset is shown in Figure 2. Figures 2(a) to 2(e) respectively show examples of the Drone-vs-Bird dataset, the dronetracking datasets dataset, the Cranfield-Synthetic-Drone-Detection dataset, the Drone Dataset dataset, and manually collected drone images.

[0129] 2. Data augmentation and data annotation.

[0130] In actual operation, due to the influence of lighting on the movement of the drone and the data acquisition equipment, the real-time images captured may exhibit motion blur and noise. Furthermore, the dataset is generally clear. Therefore, this embodiment employs random addition of blur and noise for data augmentation. The image in Figure 2(e) after blurring is shown below. Figure 3 As shown in Figure 2(e), after adding noise, it is as follows: Figure 4 As shown.

[0131] Using X-anylabeling software, the dataset images were labeled using the YOLO object detection format. The labeled data was then divided into training, validation, and test sets in an 8:1:1 ratio, and the model was trained and tested based on this dataset.

[0132] 3. Model improvement.

[0133] The model in this embodiment is an improvement on the YOLOv11 algorithm. The original YOLOv11 object detection model structure is as follows: Figure 5 As shown.

[0134] This embodiment improves the C2PSA module by adopting the C2LSKA module, which has lower computational cost but higher performance. The PSA module is modified to become the LSKA module. The improved YOLOv11 target detection model structure is as follows: Figure 6 As shown.

[0135] The input is first processed through the first and second 3×3 convolutional blocks. Each 3×3 convolutional block consists of a regular convolution with a 3×3 kernel, a stride of 2, and padding of 1, a batch normalization layer, and a SiLU activation function. Then, it is processed by the first C3K2 module, whose network structure is as follows: Figure 7 As shown.

[0136] The input to the C3K2 module is first processed by a 3×3 convolutional block (as above), then undergoes channel splitting. One portion is directly input to the stitching layer, while the other portion is both directly input to the stitching layer and processed through two C3K modules. The result of the first C3K module also participates in the stitching operation. Finally, the stitched result is processed by another 3×3 convolutional block to obtain the final output. The C2K module structure is as follows: Figure 7 The demonstration also showed that the input was processed by two separate 3×3 convolutional blocks. The output of the left 3×3 convolutional block was processed through two bottleneck modules before being input to the concatenation layer. The output of the right 3×3 convolutional block was directly input to the concatenation layer. The concatenated output was then processed by a second 3×3 convolutional block to obtain the final output. The structure of the bottleneck layer is as follows: Figure 7 As shown in the left part, the input is first processed through two layers of 3×3 convolutional blocks, and then the output is obtained after being added together by jump connections.

[0137] The output of the first C3K2 module is processed by the third 3×3 convolutional block and then input to the second C3K2 module, then to the fourth 3×3 convolutional block, then processed by the third C3K2 module, and finally input to the SPPF layer. The structure of the SPPF layer is as follows: Figure 8 As shown.

[0138] The input to SPPF is first processed through a 1×1 convolutional block, which contains a 1×1 regular convolution, a batch normalization layer, and a SiLU activation function layer. The output of the 1×1 convolutional block is directly input to the concatenation layer and also passes through three max pooling layers with a pooling kernel size of 5×5. The output of each pooling layer is input to the concatenation layer, and the output of the concatenation layer passes through another 1×1 convolutional block to obtain the final output.

[0139] The output of the SPPF layer is processed by the C2LSKA layer, and then passed through the first upsampling layer. The structure of the C2LSKA module is as follows: Figure 9 As shown.

[0140] The input to C2LSKA is processed by a 1×1 regular convolution and a batch normalization. The result is then fed into the concatenation layer via a jump connection and processed through N LSKA layers (N = 2). The output of the second LSKA layer is also fed into the concatenation layer, and the output of the concatenation layer is processed by a 1×1 regular convolution and a batch normalization to obtain the final output. The LSKA structure is taken from the paper "Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN," and its structure is as follows: Figure 9 As shown in the right part, the process first undergoes a first depthwise separable convolution with a kernel size of 1×3 to extract horizontal features. Then, it undergoes a second depthwise separable convolution with a kernel size of 3×1 to extract vertical features. Next, it undergoes a first depthwise separable dilated convolution with a dilation rate of 2 and a kernel size of 1×3. Then, it undergoes a second depthwise separable dilated convolution with the same dilation rate of 2 and a kernel size of 3×1. Finally, it undergoes a 1×1 regular convolutional layer. The result is then combined with the output using a Hadma product to obtain the final output.

[0141] The output of the first upsampling layer is concatenated with the output of the second C3K2 module. After concatenation, it is processed by the fourth C3K2 module and then input to the second upsampling layer. This processed output is then concatenated with the output of the first C3K2 module and input to the fifth C3K2 layer. The output of the fifth C3K2 layer is directly input to the first detection head and simultaneously processed by the fifth 3×3 convolutional block. After processing, it is concatenated with the output of the fourth C3K2 module and input to the sixth C3K2 module. The output of the sixth C3K2 module is input to the second detection head and simultaneously input to the sixth 3×3 convolutional block. The output of the sixth 3×3 convolutional block is concatenated with the output of the C2LSKA layer and then processed by the seventh C3K2 module. Finally, it is input to the third detection head. The structure of the detection head is as follows: Figure 10 As shown.

[0142] The input to the detection head is processed on both the left and right sides. The left side first undergoes two 3×3 convolutions with stride and padding of 1, followed by a 1×1 convolution with stride of 1 and padding of 0 to obtain the final output. The right side first undergoes a depthwise separable convolution with stride and padding of 1 and kernel of 3×3, followed by a regular convolution with stride and padding of 1 and kernel of 1×1. After processing, the input is fed into a second depthwise separable convolution with stride and padding of 1 and kernel of 3×3, followed by a second regular convolution with stride and padding of 1 and kernel of 1×1, and finally a 1×1 convolution with stride of 1 and padding of 0 to obtain the final output.

[0143] 4. Model training and deployment.

[0144] The model training primarily involves two loss functions: Focal Loss, responsible for the accuracy of the target class classification, and Focal Loss. Focal Loss addresses class imbalance, performing particularly well in object detection. It adjusts the weights of difficult and easy samples, making the model focus more on hard-to-classify samples and suppress easy samples. Its loss function is:

[0145] ;

[0146] Among them The coefficient that balances the weights of positive and negative samples is set to 0.2 here. The probability of the model predicting the correct class is defined as:

[0147] ;

[0148] To adjust the focusing parameter for the weights of easy and difficult samples, we set it to 0.5.

[0149] Another loss function is the one responsible for optimizing the location of the detection boxes. This embodiment uses the Shape-IoU loss function. Existing detection box regression methods typically consider the geometric relationship between the ground truth (GT) box and the predicted box, calculating the loss based on the relative position and shape of the detection boxes. However, they neglect the influence of the inherent properties of the detection boxes themselves, such as shape and scale, on the detection box regression. To address the shortcomings of existing research, Shape-IoU proposes a detection box regression method that focuses on the shape and scale of the detection boxes themselves. The Shape-IoU loss function expression includes:

[0150] ;

[0151] ;

[0152] ;

[0153] ;

[0154] ;

[0155] ;

[0156] in This represents the detection box predicted by the model. This represents the actual detection bounding box. and represents the width and height of the actual detection bounding box, respectively, with scale being a scaling factor related to the size of the target in the dataset. and These represent the x-axis coordinates of the center points of the predicted and ground truth bounding boxes, respectively. and Represent the y-axis coordinates of the center points of the predicted and ground truth bounding boxes, respectively. The definition is as follows:

[0157] ;

[0158] Therefore, the final loss function is:

[0159] ;

[0160] in , These are all balancing factors, with values ​​of 1.0 and 7.5. During model training, the model weights with the smallest loss error are saved, converted from PyTorch format to ONNX format, and then converted to TensorRT format for deployment on the camera chip.

[0161] The present application will now be described with more specific implementation details.

[0162] First, the trained target detection model is deployed onto the chip of the ground monitoring equipment. During operation, the real-time image from the ground monitoring equipment is input into the trained target detection model. When the trained target detection model detects a suspected unidentified flying object, the zoom function is activated. When the zoom level is increased and the target confidence (i.e., the probability that the algorithm considers it to be a drone) is greater than 0.5, the terminal platform mobilizes nearby optical cameras to conduct a coordinated search. After receiving the linkage signal, the nearby optical cameras move their gimbals to point towards the target airspace. When some of these optical cameras (at least two) also detect the drone, the air situation localization algorithm is activated.

[0163] The principle of air situation positioning is as follows: First, based on the GPS coordinates of the ground monitoring equipment, and combined with the azimuth and elevation angles of its pan-tilt unit, several rays are drawn out, such as... Figure 11 As shown.

[0164] Then, the ray sets are paired and combined without repetition. For example, the three rays A, B, and C can be combined as AB, AC, and BC. Then, the point in space closest to the sum of distances to two rays is found. At this point, the principle of the common perpendicular is introduced. For example, the principle for finding the point closest to the sum of distances to rays A and B is as follows: First, find the point E on ray A that is closest to ray B. Then, find the point F on ray B that is closest to ray A. The coordinate expression of point E is as follows:

[0165] ;

[0166] in Let AB represent vectors, A represent point A, B represent point B, v be the direction vector of AE, and w be the direction vector of BF. Dot product, This represents the cross product. The formula for calculating point F is:

[0167] ;

[0168] The coordinates of point P, which is closest to both rays, are:

[0169] ;

[0170] Where x, y, and z represent the spatial coordinates x, y, and z of the point, respectively. When only two rays exist, the coordinates of point P are the position of the UAV, as shown in Figures 12(a) and 12(b). Figures 12(a) and 12(b) are the front view and top view, respectively, of solving the target point position using two rays.

[0171] When three usable rays exist, solve for point P on the common perpendicular of each ray by combining them pairwise. AB P AC ,P BC Then, the distances from the three rays and the coordinates of the nearest point P are:

[0172] ;

[0173] That is, respectively for P AB P AC ,P BC The mean of x, y, and z is calculated to obtain the final target, as shown in Figure 13. Figure 13(a) and Figure 13(b) are the front view and top view of the target point position obtained by using three rays, respectively.

[0174] Similarly, the same logic applies to solving for four or more rays. The method in this embodiment significantly reduces equipment costs, requiring only at least two ground-based cameras to locate the drone.

[0175] In summary, the beneficial effects of this embodiment include: air situation localization of unidentified flying objects can be completed using only two optical cameras, greatly reducing equipment costs; the target detection model has been improved, increasing its recognition accuracy and reducing computational load; the training loss function has been improved, increasing detection accuracy; and the use of a custom air situation localization algorithm has reduced the difficulty of solving the problem and increased the speed of unidentified flying object localization.

[0176] Reference Figure 14 This application also provides a visible light UAV air situation positioning device, which can realize the above-mentioned visible light UAV air situation positioning method. The device includes:

[0177] A model training and deployment unit is used to train an object detection model and then deploy the object detection model to a ground monitoring device and at least two optical cameras respectively; wherein the ground monitoring device and each of the optical cameras are located in different positions;

[0178] The preliminary identification unit is used to capture a first image of the target airspace through the ground monitoring equipment and use the target detection model to identify the UAV based on the first image.

[0179] The identification unit is used to capture a second image of the target airspace using each of the optical cameras and to use the target detection model to determine whether a drone exists in the target airspace based on the second image.

[0180] The UAV positioning unit is used to determine the location of the UAV based on the ground monitoring equipment and the positions of each of the optical cameras if the UAV is identified in the target airspace by using the target detection model based on the second image.

[0181] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0182] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method of this application. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0183] It is understood that the content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the methods of this application, and the beneficial effects achieved are the same as those achieved by the methods of this application.

[0184] Please see Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0185] The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0186] The memory 1502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501.

[0187] The input / output interface 1503 is used to implement information input and output;

[0188] The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0189] Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504);

[0190] The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0191] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of this application.

[0192] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0193] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0194] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0195] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0198] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0199] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0201] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A visible light UAV air situation positioning method, characterized in that, The method includes the following steps: The target detection model is trained and then deployed on ground monitoring equipment and at least two optical cameras respectively; wherein the ground monitoring equipment and each of the optical cameras are located in different positions; The ground monitoring equipment captures a first image of the target airspace, and the target detection model identifies the drone based on the first image. The optical cameras are used to capture a second image of the target airspace, and the target detection model is used to determine whether a drone exists in the target airspace based on the second image. If so, the location of the UAV is determined based on the positions of the ground monitoring equipment and each of the optical cameras. The training of the target detection model includes the following steps: Obtain drone image dataset; The original YOLOv11 object detection model is improved to obtain the improved YOLOv11 object detection model, which is used as the object detection model. A first loss function is constructed to optimize the accuracy of target category classification, and a second loss function is constructed to optimize the location of detection boxes, thereby obtaining the final loss function; The improved YOLOv11 target detection model is trained based on the UAV image dataset, the first loss function, and the second loss function. The construction of a first loss function for optimizing target category classification accuracy and a second loss function for optimizing detection box positions, thereby obtaining the final loss function, includes the following steps: The first loss function is constructed as follows: ； in, This represents the first loss function. A coefficient representing the weights that balance positive and negative samples; This represents the predicted probability of the target detection model for the correct category; This represents the focusing parameter used to adjust the weights of easy and difficult samples; The second loss function is constructed as follows: ； ； ； ； ； ； ； in, This represents the second loss function; This represents the detection box predicted by the target detection model. This represents the actual detection bounding box. and These represent the width and height of the actual detection bounding box, respectively, with scale being the scaling factor. and These represent the x-axis coordinates of the center points of the predicted and ground truth bounding boxes, respectively. and These represent the y-axis coordinates of the center points of the predicted and actual detection boxes, respectively. Therefore, the final loss function of the object detection model is: ； in, This represents the final loss function. , All are balancing factors.

2. The visible light UAV air situation positioning method according to claim 1, characterized in that, The process of acquiring the drone image dataset includes the following steps: Acquire third-party images including birds and drones; A fourth image, including one from a drone, was captured using non-dedicated camera equipment; Obtain a fifth image that includes distractors and a random background; Acquire the sixth image taken by the drone under different weather conditions, lighting conditions, detection distances, and drone models; The third, fourth, fifth, and sixth images are blurred and noise is added to obtain an initial image dataset; The drone image dataset is obtained by labeling detection boxes for drones in the initial image dataset.

3. The visible light UAV air situation positioning method according to claim 1, characterized in that, The improvement of the original YOLOv11 object detection model to obtain the improved YOLOv11 object detection model includes the following steps: Replace the C2PSA module in the backbone network of the original YOLOv11 target detection model with the C2LSKA module; Replace the PSA module of the backbone network in the original YOLOv11 target detection model with the LSKA module; The improved YOLOv11 target detection model's backbone network further includes a C3K2 structure, an SPPF structure, and a detection head structure.

4. The visible light UAV air situation positioning method according to claim 1, characterized in that, The steps of using the target detection model to identify the drone based on the image include the following steps: The target detection model is used to annotate detection boxes for the UAV on the image; Determine the confidence level that the detection frame contains a drone; If the confidence level reaches a preset threshold, it is determined that a drone exists in the image.

5. A visible light UAV air situation positioning method according to any one of claims 1 to 4, characterized in that, The process of determining the location of the drone based on the positions of the ground monitoring equipment and each of the optical cameras includes the following steps: Based on the position, azimuth angle, and elevation angle of the ground monitoring equipment and each of the optical cameras, corresponding rays are drawn out respectively; The distances to each ray in space and the nearest point are calculated as the location of the UAV.

6. A visible light UAV air situation positioning device, characterized in that, The device is applied to the visible light UAV air situation positioning method as described in claim 1, and the device comprises: A model training and deployment unit is used to train an object detection model and then deploy the object detection model to a ground monitoring device and at least two optical cameras respectively; wherein the ground monitoring device and each of the optical cameras are located in different positions; The preliminary identification unit is used to capture a first image of the target airspace through the ground monitoring equipment and use the target detection model to identify the UAV based on the first image. The identification unit is used to capture a second image of the target airspace using each of the optical cameras and to use the target detection model to determine whether a drone exists in the target airspace based on the second image. The UAV positioning unit is used to determine the location of the UAV based on the ground monitoring equipment and the positions of each of the optical cameras if the UAV is identified in the target airspace by using the target detection model based on the second image.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images