Unmanned aerial vehicle hidden target filtering method and device, electronic equipment and storage medium

Abstract

The present application belongs to the technical field of unmanned aerial vehicle monitoring, and particularly relates to a method and device for filtering hidden danger targets of unmanned aerial vehicles, an electronic device and a storage medium. The method comprises: obtaining real-time data of unmanned aerial vehicles and preprocessing the real-time data to obtain target data; performing target reasoning and scene segmentation reasoning on scene images in the target data respectively, and correspondingly outputting hidden danger targets containing label information, confidence and target detection frame information and scene region masks, and constructing the hidden danger targets and the scene region masks into a first data set; obtaining GPS coordinates and feature vectors of the hidden danger targets based on unmanned aerial vehicle GPS data, unmanned aerial vehicle three-axis attitude angles, unmanned aerial vehicle mounted camera parameters and target detection frame information of the hidden danger targets, and constructing the GPS coordinates and the feature vectors into a second data set; and filtering the hidden danger targets based on the first data set and the second data set. The present application aims to solve the problems of single filtering dimension, poor multi-model coordination and insufficient real-time performance of unmanned aerial vehicle targets in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of drone monitoring technology, specifically relating to a method, device, electronic equipment and storage medium for filtering potential targets from drones. It is applicable to the accurate identification and filtering of potential targets such as illegal floating objects and unauthorized personnel in scenarios such as water monitoring and regional security. Background Technology

[0002] With the widespread application of drone technology in environmental monitoring, security inspection and other fields, the vision equipment carried by drones can collect ground / water surface image data in real time. However, due to limitations in the imaging environment, such as changes in lighting, background interference and model accuracy, the original detection results often contain a large number of redundant targets, resulting in low efficiency of subsequent data processing and a high false alarm rate.

[0003] In the relevant prior art, Chinese patent document CN120455846A discloses a method, system, and computer-readable storage medium for accurately calculating the geographic coordinates of potential hazards based on UAV and artificial intelligence detection. The method includes: the UAV performing real-time identification and analysis of targets in video images and video frames; when the analysis shows that the current frame image contains a potential hazard, the control interface instructs the UAV to hover; based on the pixel coordinates of the potential hazard in the video image and the camera focal length, film width and height, and imaging size of the UAV gimbal, the relative adjustment angle of the gimbal is calculated to center the potential hazard; based on the obtained relative adjustment angle, the gimbal is adjusted to center the potential hazard in the gimbal image; after the adjustment is completed, the GPS data of the potential hazard is calculated.

[0004] However, in existing technologies, target filtering often relies on a single rule, such as a fixed confidence threshold or the result of a single model inference, which has the following drawbacks: 1) Single filtering dimension: Filtering is based solely on basic features such as confidence level and target size, without combining scene semantics. For example, targets with potential water hazards need to be associated with the water area, which may lead to missed or false detections. 2) Poor multi-model collaboration: If it is necessary to combine segmentation model, localization model and feature extraction model, each model runs independently without a unified scheduling mechanism, resulting in high inference latency; 3) Low resource utilization: The repeated loading of the inference engine by multiple models takes a long time for initialization, and the long time for multiple models to run inference makes it difficult to meet the low latency requirements of real-time monitoring of UAVs. 4) Poor scalability: Adding new filtering strategies, such as alarm area determination, to adapt to different monitoring scenarios requires modification of the core code, which cannot achieve flexible adaptation.

[0005] To address the aforementioned issues, there is an urgent need for a hazard target filtering solution that integrates multi-model collaborative reasoning, scenario-based rule filtering, and resource optimization to improve the accuracy and real-time performance of drone monitoring. Summary of the Invention

[0006] The present invention aims to provide a method for filtering potential targets of drones, so as to overcome the shortcomings of existing technologies in drone target filtering, such as single dimension, poor multi-model coordination, and insufficient real-time performance.

[0007] The present invention also discloses an apparatus equipped with a method for filtering potential targets caused by unmanned aerial vehicles (UAVs).

[0008] The detailed technical solution of this invention is as follows: A method for filtering potential targets from unmanned aerial vehicles (UAVs), the method comprising: S1. Acquire real-time data from the UAV and preprocess it to obtain target data; wherein, the real-time data from the UAV includes scene images collected by the camera mounted on the UAV, UAV GPS data, UAV three-axis attitude angles, and UAV camera parameters. S2. Perform target reasoning and scene segmentation reasoning on the scene images in the target data respectively, and output the hidden danger targets and scene region masks containing label information, confidence level and target detection box information respectively. Construct the label information, confidence level and target detection box information of the hidden danger targets and the scene region mask into the first dataset. S3. Based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters and target detection box information of the potential hazard target in the target data, obtain the GPS coordinates and feature vector of the potential hazard target, and construct them into a second dataset; S4. Filter the potential hazards based on the first dataset and the second dataset.

[0009] According to a preferred embodiment of the present invention, in step S2, the YOLOv8n model is used to perform target inference on the scene image in the target data, and outputs a potential hazard target containing label information, confidence level, and target detection box information; wherein, the label information contains the type of the potential hazard target, and the target detection box information contains the coordinates and size of the target detection box; All the obtained hidden danger targets are constructed into a hidden danger target set. , Indicates the first One potential hazard target, This indicates the number of potential hazards.

[0010] According to a preferred embodiment of the present invention, in step S2, the YOLOv5-seg model is used as the scene segmentation model to perform scene segmentation inference on the scene image in the target data, and a scene region mask M is output. The scene region mask M is a binary image, where 1 represents a scene region and 0 represents a non-scene region.

[0011] According to a preferred embodiment of the present invention, in step S3, the MobileNet network model is used as the feature extraction model to extract the feature vector of the potential hazard target, specifically including: Based on the coordinates of the target detection box of the potential hazard, the scene image containing the potential hazard is cropped to obtain a target cropped image; The target cropped image is input into the feature extraction model, which outputs an r-dimensional feature vector.

[0012] According to a preferred embodiment of the present invention, in step S4, filtering is performed on the potential hazard target, including basic filtering, wherein the basic filtering specifically includes: Based on the confidence level of the potential hazards, potential hazards below a preset confidence threshold are filtered out. Based on the size of the target detection box of the potential hazard, potential hazard targets below a preset pixel threshold are filtered out.

[0013] According to a preferred embodiment of the present invention, in step S4, filtering the potential hazard target further includes scene filtering, wherein the scene filtering specifically comprises: Read the target set of hidden dangers Internal hidden danger targets, set hidden danger targets The coordinates of the target detection box are ,in _x_coordinate of the top-left pixel of the target detection box, _y_coordinate of the top-left pixel of the target detection box, _width_coordinate of the target detection box, and _height_coordinate of the target detection box, all in pixels. The scene region mask M is a binary image and a W×H array, where, Represents pixels Belongs to a specific scene area. Represents pixels It does not belong to a specific scene area; Let the scaling factor be s, where s ≥ 1 and is an integer. The scene region mask M is scaled using nearest neighbor interpolation to obtain a scaled scene region mask M' with dimensions of... : , ; For potential hazards Target detection bounding box coordinates Scale the target detection box to obtain its coordinates. ; Invert the scaled scene region mask M', denoted as ; The Manhattan distance transform algorithm is used to obtain Distance map ,in Represents pixels The shortest distance to the nearest specific scene area, in pixels; From the distance map Extract scaled target detection box coordinates Corresponding area ; Computational area Minimum distance value in ; If the minimum distance value If the distance is less than or equal to a preset distance threshold, then the target is identified as a potential hazard. If a scene is adjacent to a specific area, it will be retained; otherwise, it will be determined as not adjacent and deleted.

[0014] According to a preferred embodiment of the present invention, in step S4, filtering the potential hazard targets further includes hazard determination, wherein the hazard determination includes alarm area matching, hazard type matching strategy, and duplicate target filtering, wherein: The alarm area matching method is to filter potential hazards located within the alarm area based on their GPS coordinates. The hazard type matching is based on the type matching of hazard targets to a preset hazard target type, and hazard targets that are not preset hazard target types are eliminated; The duplicate target filtering specifically includes: Set the current hidden danger target feature database as The target set of hidden dangers is ; Traversing the set of hidden danger targets According to the target of the hidden danger The target detection bounding box coordinates are used to crop the scene image acquired by the corresponding UAV to obtain the potential hazard target. The target cropped image; Feature extraction networks are used to extract features from cropped images of potential hazards, thus obtaining the target. The corresponding eigenvectors: ; In the formula: Represents the feature extraction function; Indicates the first One hidden danger target The corresponding feature vector; Database of hazard target characteristics Existing targets and hidden dangers The feature vector of a target with the same type of hidden danger is represented as: , Indicate potential hazards The number of similar hidden danger targets is calculated. eigenvectors Each of the feature vectors of its own type and all other hazard targets is compared with its feature vector. similarity ,by For example, the calculation formula is:

[0015] In the formula: The dot product operation between vectors is represented by ||||, and the modulo operation between vectors is represented by ||||. If the hidden danger target If the feature similarity of a target to one or more other similar hazard targets is greater than a similarity threshold, then that target or group of similar hazard targets is removed; otherwise, it is retained, and its feature vector is added to the hazard target feature database. middle.

[0016] In another aspect of the present invention, an apparatus is provided for implementing the method for filtering potential targets from unmanned aerial vehicles as described above, the apparatus comprising: The data acquisition module is used to acquire real-time data from the UAV and preprocess it to obtain target data; wherein, the real-time data of the UAV includes scene images collected by the camera mounted on the UAV, UAV GPS data, UAV three-axis attitude angles and UAV mounted camera parameters. The model inference module is used to perform target inference and scene segmentation inference on the scene images in the target data respectively, and outputs the hidden danger targets and scene region masks containing label information, confidence and target detection box information, and constructs the label information, confidence and target detection box information of the hidden danger targets and the scene region mask into the first dataset; The target calculation module is used to obtain the GPS coordinates and feature vectors of the potential hazard targets based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters and target detection box information of the potential hazard targets in the target data, and construct them into a second dataset; The target filtering module is used to perform filtering on the potential hazards based on the first dataset and the second dataset.

[0017] In another aspect of the invention, an electronic device is also provided, comprising: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the drone hazard target filtering method as described above.

[0018] In another aspect of the invention, a machine-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform the drone hazard target filtering method as described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) High filtering accuracy: By parallel reasoning of two types of models, target detection and scene segmentation, and by performing layered filtering of potential targets according to the filtering strategy, combined with scene semantics and GPS positioning, the false detection rate of potential targets in actual applications is reduced by more than 45%.

[0020] (2) Strong real-time performance: In terms of hardware, this invention adopts TensorRT and hardware device NPU engine acceleration, batch processing optimization, engine caching and other strategies to meet the real-time monitoring needs of UAVs.

[0021] (3) Good scalability: The filtering strategy adopted in this invention can be dynamically managed through configuration files. Adding a new scene only requires modifying the model and configuration parameters, without modifying the core code, which is easy to operate. Attached Figure Description

[0022] Figure 1 This is a flowchart of the UAV hazard target filtering method described in this invention.

[0023] Figure 2 This is a schematic diagram of the pipeline for multi-model collaborative reasoning in Embodiment 1 of the present invention.

[0024] Figure 3 This is a schematic diagram of the filtering strategy in Embodiment 1 of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] The object of the present invention is to overcome the defects of single target filtering dimension, poor multi-model collaboration, and insufficient real-time performance in the prior art for unmanned aerial vehicles (UAVs), and to provide a method for filtering potential hazard targets of UAVs based on multi-model reasoning. Through the design of multi-model collaborative reasoning and hierarchical rule filtering, accurate and efficient filtering of potential hazard targets is achieved.

[0030] The following further describes the method and device for filtering potential hazard targets of UAVs according to the present invention with specific embodiments.

[0031] Embodiment 1 Refer Figure 1 In this embodiment, a method for filtering potential hazard targets of UAVs is provided, and the method includes: S1. Obtain real-time UAV data and preprocess it to obtain target data.

[0032] In this embodiment, the real-time UAV data includes the scene images collected by the camera mounted on the UAV, UAV GPS data, the three-axis attitude angles of the UAV, and the parameters of the camera mounted on the UAV. Among them, the scene images collected by the camera mounted on the UAV are BGR format images, with a width of W, a height of H, and 3 channels; the UAV GPS data includes latitude, longitude, and relative height, denoted as lat, lng, and alt respectively; the three-axis attitude angles of the UAV include yaw angle, pitch angle, and roll angle, denoted as yaw, pitch, and roll respectively; the parameters of the camera mounted on the UAV include the width, height, and focal length of the sensor, denoted as w, h, and f respectively.

[0033] The preprocessing operations include data format verification, data caching, etc., which are all conventional operations and will not be elaborated here.

[0034] The preprocessed scene images, UAV GPS data, the three-axis attitude angles of the UAV, and the parameters of the camera mounted on the UAV will be used as target data.

[0035] S2. Respectively perform target reasoning and scene segmentation reasoning on the scene images in the target data, and correspondingly output potential hazard targets including label information, confidence level, and target detection box information, as well as a scene area mask. Construct the label information, confidence level, target detection box information of the potential hazard targets, and the scene area mask into a first data set.

[0036] Specifically, in this embodiment, a target detection model can be used to perform target reasoning on the scene images in the target data, and output potential hazard targets including label information, confidence level, and target detection box information. Among them, the label information includes the types of potential hazard targets; the target detection box information includes the coordinates, dimensions, etc. of the target detection box.

[0037] Preferably, the target detection model can be the YOLOv8n model.

[0038] Finally, all the obtained hidden danger targets are constructed into a hidden danger target set. , Indicates the first One potential hazard target, This indicates the number of potential hazards.

[0039] On the other hand, this embodiment can use a scene segmentation model to perform scene segmentation inference on the scene images in the target data and output a scene region mask M, i.e. a binary image, where 1 represents a scene region and 0 represents a non-scene region.

[0040] In practical applications, such as water monitoring, the YOLOv5-seg model can be used for scene segmentation.

[0041] Finally, the label information, confidence level, target detection box information, and scene region mask obtained above are used to construct the first dataset.

[0042] It should be understood that the hidden danger targets and scene area masks in this embodiment are obtained based on multi-model collaborative reasoning. Therefore, in each round of reasoning, the scene images input by the target detection model and the scene segmentation model are the same image data.

[0043] In practical implementation, collaborative inference in the pipeline is achieved by loading pre-trained models for object detection and scene segmentation and running the code. Furthermore, hardware-accelerated plugins (such as DeepStream) can be used to develop the inference pipeline to achieve parallel inference of multiple models, leveraging the hardware engine TensorRT and the neural network processing unit (NPU) to accelerate model inference. Specific parameters... Figure 2 As shown, this approach maximizes hardware resource utilization and reduces inference time.

[0044] S3. Based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters, and target detection box information of the potential hazard target in the target data, obtain the GPS coordinates and feature vector of the potential hazard target, and construct them into a second dataset.

[0045] The first step is to obtain the GPS coordinates of the potential hazard. This can be achieved using existing methods. For example, the method disclosed in the published patent document "CN120455846A - A method, system and computer-readable storage medium for accurately calculating the geographic coordinates of potential hazards based on UAV and artificial intelligence detection" can be used. This involves inputting the target detection box coordinate data from the target detection box information, as well as the preprocessed UAV GPS data, the UAV's three-axis attitude angles, and the UAV's mounted camera parameters. The output will then be the GPS coordinates of the potential hazard. .

[0046] Next, the feature vector of the potential hazard target is obtained. Based on the target detection box information obtained in step S2 and the scene image acquired by the UAV-mounted camera in step S1, the scene image is first cropped according to the target detection box coordinates to obtain the target cropped image. Then, the feature extraction model is used to input the target cropped image and output an r-dimensional feature vector.

[0047] The feature extraction model utilizes hardware-provided tools for TensorRT engine acceleration, avoiding redundant initialization, and allows for setting the batch size of the model to reduce inference latency. Preferably, the feature extraction model can employ the MobileNet network model.

[0048] Finally, the GPS coordinates and feature vectors of the potential hazards obtained above are used to construct the second dataset.

[0049] S4. Filter the potential hazards based on the first dataset and the second dataset.

[0050] In this embodiment, hazard targets are filtered according to a set filtering strategy. This filtering strategy is primarily layered filtering, including three layers: basic filtering, scenario filtering, and hazard assessment. Figure 3 As shown.

[0051] Specifically, the basic filtering aims to filter out invalid potential targets, eliminating targets that are not bound to identification items, have a confidence level lower than a set threshold, or have a target detection box size smaller than a set pixel threshold. Specifically: 1) Confidence filtering: Based on the confidence level of the potential hazard, filter out potential hazard targets that are below a preset confidence threshold; 2) Size filtering: Based on the size of the target detection box of the potential hazard, filter out potential hazard targets that are lower than the preset pixel threshold.

[0052] The scene filtering mainly involves determining and filtering targets that are only present in specific scenes based on their scene association. This is done by calculating the minimum distance between the potential hazard target and the scene region mask, thus eliminating targets without scene association. Specifically: if a detected potential hazard target is a scene-associated target, a distance map is calculated based on the scene region mask M. This distance map stores the pixel distances of each pixel in the scene image from the scene region (i.e., the specific scene region segmented by the pre-trained segmentation model in step S2). The minimum pixel distance stored in the distance map within the target detection box region corresponding to the potential hazard target is obtained. If this minimum pixel distance is less than a preset distance threshold, the corresponding potential hazard target is retained; otherwise, it is eliminated. If a detected potential hazard target is a non-scene-associated target, the same process is repeated: the minimum pixel distance stored in the distance map within the target detection box region corresponding to the potential hazard target is obtained. If this minimum pixel distance is less than a preset distance threshold, the corresponding potential hazard target is eliminated; otherwise, it is retained.

[0053] Furthermore, the specific implementation process of scene filtering in this embodiment is as follows: Set a set of potential hazards ; Read the target set of hidden dangers Hidden danger targets within, assuming hidden danger targets The coordinates of the target detection box are ,in _x_coordinate of the top-left pixel of the target detection box, _y_coordinate of the top-left pixel of the target detection box, _width_coordinate of the target detection box, and _height_coordinate of the target detection box, all in pixels. The scene region mask M is a binary image, assuming M is a W×H array, where... Represents pixels Belongs to a specific scene area. Represents pixels It does not belong to a specific scene area; Let the scaling factor be s, where s ≥ 1 and is an integer. The scene region mask M is scaled using nearest neighbor interpolation to obtain a scaled scene region mask M' with dimensions of... : , ; For potential hazards Target detection bounding box coordinates Scale the target detection box to obtain its coordinates. ; Invert the scaled scene region mask M', denoted as ; The Manhattan distance transform algorithm is used to obtain Distance map ,in Represents pixels The shortest distance to the nearest specific scene area, in pixels; From the distance map Extract scaled target detection box coordinates Corresponding area ; Computational area Minimum distance value in ; If the minimum distance value ,in If a preset distance threshold, such as 50 pixels, is set, then the target is identified as a potential hazard. If it is adjacent to a specific scene area, it is considered not adjacent.

[0054] It should be understood that, based on realistic logic, we assume the target of the potential hazard. If a scene should not be located near a specific area, it should be removed.

[0055] Repeat the above process until the entire set of potential hazards has been traversed. Of all the potential hazards, only those that conform to the relevant logic of the specific area scenario will be retained.

[0056] For example, assuming the drone inspection scenario is a water conservancy scenario, the water surface segmentation model in step S2 will obtain the location of the water surface area in the scene image. The distance map has the same size as the original image, assuming it is an array of 1920*1080. Then, each original storage content in the distance map array is the minimum pixel distance between the current pixel point in the scene image and the water surface area.

[0057] Since the target detection box is a region, this region corresponds to several distances. The minimum distance is found and compared with the set threshold. If the target is duckweed, it is known that it is associated with the water surface, so the minimum distance should be less than the threshold and it should be retained. If the target is a car, it is known that it is not associated with the water surface, so the minimum distance should be greater than the threshold and it should be removed.

[0058] The hazard determination mainly includes alarm area matching, hazard type matching strategy, and feature extraction for duplicate target filtering.

[0059] The alarm area matching process involves checking whether the target is located within the configured alarm area based on its calculated GPS coordinates. If it is, the target is retained; otherwise, it is removed.

[0060] Specifically, assuming the GPS coordinates of the alarm area are known, the alarm area matching process is as follows: The center pixel coordinates of the detected potential hazard target Convert to GPS coordinates and filter out invalid GPS data; Using relevant function libraries, such as matplotlib, the GPS coordinate data of the alarm area can be used to generate a closed polygon. The ray-mapping method is used to determine whether the potential hazard is within the closed polygonal area. If it is, it is retained; otherwise, it is discarded.

[0061] The hazard type matching involves matching the hazard target type with a preset hazard target type and eliminating hazard targets that do not belong to the preset hazard target type. For example, if the preset hazard target type is "personnel," and the type in the hazard target label information identified by target reasoning is not "personnel," then it is eliminated.

[0062] The feature extraction process filters duplicate targets. It extracts features from detected potential hazards, traverses all potential hazards, and calculates the similarity of the feature vectors of the currently detected potential hazards and those of the same type in the target feature vector library based on the feature vectors. Potential hazards with a similarity greater than a preset similarity threshold are considered duplicate potential hazards and are removed. Finally, the remaining potential hazards and their feature vectors maintain a target feature vector library.

[0063] The target feature vector library is initially empty. During the drone's flight, when a potential target is detected for the first time (assuming the first detected target is a car or a person), the feature vectors extracted from these two targets are directly added to the target feature vector library. When a car or a person is detected a second time, the similarity between the feature vectors of the second detected car and the first detected car is calculated. If the similarity is higher than a preset similarity threshold, the second detected car target is removed. Similarly, the similarity between the feature vectors of the second detected person and the first detected person is calculated. If the similarity is lower than a preset similarity threshold, the second detected "person" target is retained, and its feature vector is inserted into the target feature vector library, waiting for subsequent detections of "person" targets to be compared. This process continues to maintain a target feature vector library.

[0064] Specifically, the current database of potential hazard characteristics is set as follows: The set of potential hazards detected based on scene images collected by drones is .

[0065] First, traverse the target set of potential hazards. To obtain the first potential hazard target Assuming potential hazards The coordinates of the target detection box are ,in `top` is the x-coordinate of the top-left pixel of the target detection bounding box, `width` is the width of the target detection bounding box, and `height` is the height of the target detection bounding box; all units are pixels. Then, based on the target hazard... The target detection bounding box coordinates are cropped from the scene image captured by the corresponding UAV to obtain the potential hazard target. The target cropped image.

[0066] By analogy, all potential hazards can be identified. The corresponding target cropped image.

[0067] Using a feature extraction network, such as MobileNet, features are extracted from the obtained cropped target image, that is, the feature vector corresponding to each potential hazard target is obtained. The feature extraction process can be represented as: ; In the formula: Represents the feature extraction function; Indicates the first One hidden danger target The corresponding feature vector.

[0068] Perform feature vector similarity calculation: [This involves] analyzing the feature database of potential hazards. Existing targets and hidden dangers The feature vector of a target with the same type of hidden danger is represented as: , Indicate potential hazards The number of similar potential hazards. Calculate the potential hazard targets. eigenvectors Each of the feature vectors of its own type and all other hazard targets is compared with its feature vector. similarity ,by For example, the calculation formula is:

[0069] In the formula: The dot product operation between vectors is represented by ||||, and the modulo operation between vectors is represented by ||||.

[0070] Set a similarity threshold. It is used to determine whether a potential hazard is a duplicate target. The value range is [0,1], and the specific value can be adjusted according to the actual application scenario.

[0071] Filter duplicate potential hazards; if a potential hazard target... The feature similarity with one or more other potential hazards of the same category is greater than the similarity threshold. If the target is found to be a hazard, then remove the target or more of the same type; otherwise, retain the target and add its feature vector to the hazard target feature database. middle.

[0072] Based on the above process, duplicate potential hazards are filtered out. For the retained potential hazards, their feature vectors are added to the potential hazard feature database. The middle part is used for subsequent reasoning results.

[0073] Repeat the above process until the target set of potential hazards has been traversed. All potential hazards in the target.

[0074] Based on the above hierarchical filtering of potential drone targets, the final output is as follows: The filtered potential target results are formatted as JSON, including data such as labels, confidence scores, target detection box coordinates, and feature vectors, and then sent to the drone ground station or alarm platform via network communication.

[0075] The effectiveness of the method of the present invention will be verified by using specific application examples below.

[0076] The hardware environment for deployment is as follows: a drone and its NVIDIA Jetson Xavier NX module and gimbal camera, with an imaging resolution of 1920×1080, a film width and height of 7.53mm and 5.64mm respectively, and a focal length of 6.83mm.

[0077] The software environment deployed in this example is: Ubuntu 20.04, Python 3.8, TensorRT 8.5, and OpenCV 4.5.

[0078] The model configuration in this example is as follows: the object detection model uses YOLOv8n, with a total of 53 detection categories. It is converted to .engine format using TensorRT and the NPU in NVIDIA Jetson Xavier NX is used to accelerate model inference; the scene segmentation model is set to the water surface segmentation model, using YOLOv5-seg; the feature extraction model is MobileNetV3-small, and the batch size is set to 8, that is, eight targets can be processed at the same time in one inference.

[0079] The filtering strategy used in this example is configured as follows: basic filtering includes confidence filtering and scale filtering, where the threshold for confidence filtering is set to 0.6 and the threshold for scale filtering is set to 20 pixels; in scene filtering, the distance threshold for scene association checking is set to 50 pixels. The categories to be associated with the scene are floating objects, duckweed, algae, and aquatic plants. After these categories are identified, the distance obtained from scene association checking must be less than or equal to 50 pixels; otherwise, they need to be discarded. For categories other than those mentioned above, the distance obtained from scene association checking must be greater than 50 pixels. 50 pixels, otherwise it needs to be removed; in the hazard judgment, alarm area matching and feature vector duplicate target filtering are performed. The configured alarm GPS area includes 1523 points such as [35.23944900177172, 117.35001289454569], [35.23949053998757, 117.34888972240805], and [35.23976563447406, 117.34879401527432]. Feature vector similarity threshold Set it to 0.5.

[0080] Specifically, the steps include the following: Step S1: Receive data transmitted back from the UAV, in which the image resolution is 1920×1080, the UAV GPS data latitude is 35.196816, longitude is 117.3615345, altitude is 80.02m, and the yaw angle of the three-axis attitude angles is 156.69°, pitch angle is -44.9°, and roll angle is 0°; Step S2: The multi-model collaborative reasoning module performs inference: 1) The target detection outputs four targets: [{"label": "Ordinary Personnel", "confidence": 0.95, "bbox": [550, 200, 80, 150]}; {"label": "Ordinary Boats", "confidence": 0.80, "bbox": [450, 900, 150, 250]}; {"label": "Duckweed", "confidence": 0.75, "bbox": [560, 250, 50, 40]}; {"label": "Cars", "confidence": 0.99, "bbox": [100, 100, 120, 80]}]; where label represents the type, confidence is the confidence score, and bbox is the coordinates of the target detection box; 2) The water surface segmentation output mask is M, and the mask array data is [1,1,1,...0,0,0....,1,1,1].

[0081] Step S3: Calculate the feature vector of the above-mentioned hidden danger targets as [-0.0317523293197155, ..., -0.02607119083404541, ..., [-0.04774148389697075, ..., -0.031844913959503174].

[0082] Step S4: Layered rule filtering: 1) Basic filtering: All potential hazards have a confidence level greater than 0.6 and a target detection box size of 20 pixels; no rejection is allowed. 2) Scene filtering: The minimum pixel distance between duckweed and water is 70 pixels. Although it is a scene-related target, the minimum pixel distance of 70 is greater than the 50 pixel threshold, so it is removed; the minimum pixel distance between car and water is 38 pixels. Although it is less than the 50 pixel threshold, it is a non-scene-related target, so it is removed; the remaining targets are ordinary people and ordinary boats. 3) Hazard identification: Only alarm area matching strategy and feature extraction are used for duplicate target filtering strategy. The GPS coordinate data of ordinary personnel and ordinary vessels are [35.1975, 117.3650] and [35.1900, 117.3500], respectively. Among them, after judgment, ordinary personnel are located inside the alarm GPS area and their feature vectors have a similarity of less than the threshold of 0.5 with the feature vectors of the same type of hazard targets in the database, so they are retained; ordinary vessels are located outside the alarm GPS area and are removed.

[0083] Step S5: Finally, output one ordinary personnel hazard target, including its confidence level, detection box coordinates, and feature vector, and update the feature vector to the hazard target feature database. middle.

[0084] Step S6: Repeat steps S1-S5 until no more drone image data is input.

[0085] The above example utilized a tracking algorithm to perform interval inference of image data. The final average inference time per frame was 72ms, with the average model inference time being 45ms, the average rule filtering time being 27ms, and the average inference time per interval frame being 30ms.

[0086] The final example results show that the accuracy of identifying potential hazards is 92%; the processing time per frame is 72ms, which meets the requirement of less than 80ms; the initialization time of the model engine and inference pipeline is 25s for the first start, and there is no initialization time for subsequent frames that reuse the engine.

[0087] In summary, the UAV hazard target filtering method of this invention improves scene adaptability through multi-model collaborative reasoning, enhances accuracy through hierarchical rule filtering, and improves real-time performance through TensorRT and NPU acceleration and model caching. It is suitable for scenarios such as water monitoring and regional security. Practical application testing shows that the false detection rate of hazard targets is reduced by more than 45%, and the processing time per frame is less than 90 milliseconds, meeting the real-time monitoring requirements of UAVs.

[0088] Example 2 This embodiment provides an apparatus for implementing a method for filtering potential targets from unmanned aerial vehicles (UAVs), the apparatus comprising: The data acquisition module is used to acquire real-time data from the UAV and preprocess it to obtain target data; wherein, the real-time data of the UAV includes scene images collected by the camera mounted on the UAV, UAV GPS data, UAV three-axis attitude angles and UAV mounted camera parameters. The model inference module is used to perform target inference and scene segmentation inference on the scene images in the target data respectively, and outputs the hidden danger targets and scene region masks containing label information, confidence and target detection box information, and constructs the label information, confidence and target detection box information of the hidden danger targets and the scene region mask into the first dataset; The target calculation module is used to obtain the GPS coordinates and feature vectors of the potential hazard targets based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters and target detection box information of the potential hazard targets in the target data, and construct them into a second dataset; The target filtering module is used to perform filtering on the potential hazards based on the first dataset and the second dataset.

[0089] Example 3 This embodiment also provides an electronic device, including: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the drone hazard target filtering method as described above.

[0090] In this embodiment, the electronic device may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile computing device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable computing device, consumer electronic device, etc.

[0091] Example 4 This embodiment also provides a machine-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the drone hazard target filtering method described above.

[0092] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0093] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0094] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for filtering potential targets from unmanned aerial vehicles (UAVs), characterized in that, The method includes: S1. Acquire real-time data from the UAV and preprocess it to obtain target data; wherein, the real-time data from the UAV includes scene images collected by the camera mounted on the UAV, UAV GPS data, UAV three-axis attitude angles, and UAV camera parameters. S2. Perform target reasoning and scene segmentation reasoning on the scene images in the target data respectively, and output the hidden danger targets and scene region masks containing label information, confidence level and target detection box information respectively. Construct the label information, confidence level and target detection box information of the hidden danger targets and the scene region mask into the first dataset. S3. Based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters and target detection box information of the potential hazard target in the target data, obtain the GPS coordinates and feature vector of the potential hazard target, and construct them into a second dataset; S4. Filter the potential hazards based on the first dataset and the second dataset.

2. The method for filtering potential targets from unmanned aerial vehicles according to claim 1, characterized in that, In step S2, the YOLOv8n model is used to perform target inference on the scene images in the target data, and outputs the potential hazards containing label information, confidence level, and target detection box information; wherein, the label information contains the type of the potential hazard, and the target detection box information contains the coordinates and size of the target detection box; All the obtained hidden danger targets are constructed into a hidden danger target set. , Indicates the first One potential hazard target, This indicates the number of potential hazards.

3. The method for filtering potential targets from unmanned aerial vehicles according to claim 2, characterized in that, In step S2, the YOLOv5-seg model is used as the scene segmentation model to perform scene segmentation inference on the scene images in the target data, and outputs a scene region mask M. The scene region mask M is a binary image, where 1 represents a scene region and 0 represents a non-scene region.

4. The method for filtering potential targets from unmanned aerial vehicles according to claim 1, characterized in that, In step S3, the MobileNet network model is used as the feature extraction model to extract the feature vector of the potential hazard target, specifically including: Based on the coordinates of the target detection box of the potential hazard, the scene image containing the potential hazard is cropped to obtain a target cropped image; The target cropped image is input into the feature extraction model, which outputs an r-dimensional feature vector.

5. The method for filtering potential targets from unmanned aerial vehicles according to claim 1, characterized in that, In step S4, filtering is performed on the potential hazard targets, including basic filtering, which specifically includes: Based on the confidence level of the potential hazards, potential hazards below a preset confidence threshold are filtered out. Based on the size of the target detection box of the potential hazard, potential hazard targets below a preset pixel threshold are filtered out.

6. The method for filtering potential targets from unmanned aerial vehicles according to claim 3, characterized in that, In step S4, filtering is performed on the potential hazard target, including scene filtering, which specifically includes: Read the target set of hidden dangers Internal hidden danger targets, set hidden danger targets The coordinates of the target detection box are ,in _x_coordinate of the top-left pixel of the target detection box, _y_coordinate of the top-left pixel of the target detection box, _width_coordinate of the target detection box, and _height_coordinate of the target detection box, all in pixels. The scene region mask M is a binary image and a W×H array, where, Represents pixels Belongs to a specific scene area. Represents pixels It does not belong to a specific scene area; Let the scaling factor be s, where s ≥ 1 and is an integer. The scene region mask M is scaled using nearest neighbor interpolation to obtain a scaled scene region mask M' with dimensions of... : , ; For potential hazards Target detection bounding box coordinates Scale the target detection box to obtain its coordinates. ; Invert the scaled scene region mask M', denoted as ; The Manhattan distance transform algorithm is used to obtain Distance map ,in Represents pixels The shortest distance to the nearest specific scene area, in pixels; From the distance map Extract scaled target detection box coordinates Corresponding area ; Computational area Minimum distance value in ; If the minimum distance value If the distance is less than or equal to a preset distance threshold, then the target is identified as a potential hazard. If a scene is adjacent to a specific area, it will be retained; otherwise, it will be determined as not adjacent and deleted.

7. The method for filtering potential targets from unmanned aerial vehicles according to claim 6, characterized in that, In step S4, filtering the potential hazards further includes hazard determination, which comprises alarm area matching, hazard type matching strategy, and duplicate target filtering, wherein: The alarm area matching method is to filter potential hazards located within the alarm area based on their GPS coordinates. The hazard type matching is based on the type matching of hazard targets to a preset hazard target type, and hazard targets that are not preset hazard target types are eliminated; The duplicate target filtering specifically includes: Set the current hidden danger target feature database as The target set of hidden dangers is ; Traversing the set of hidden danger targets According to the target of the hidden danger The target detection bounding box coordinates are used to crop the scene image acquired by the corresponding UAV to obtain the potential hazard target. The target cropped image; Feature extraction networks are used to extract features from cropped images of potential hazards, thus obtaining the target. The corresponding eigenvectors: ； In the formula: Represents the feature extraction function; Indicates the first One hidden danger target The corresponding feature vector; Database of hazard target characteristics Existing targets and hidden dangers The feature vector of a target with the same type of hidden danger is represented as: , Indicate potential hazards The number of similar hidden danger targets is calculated. eigenvectors Each of the feature vectors of its own type of hidden danger targets similarity : In the formula: The dot product operation between vectors is represented by ||||, and the modulo operation between vectors is represented by ||||. If the hidden danger target If the feature similarity of a target to one or more other similar hazard targets is greater than a similarity threshold, then that target or group of similar hazard targets is removed; otherwise, it is retained, and its feature vector is added to the hazard target feature database. middle.

8. An apparatus for implementing the method for filtering potential targets from unmanned aerial vehicles as described in any one of claims 1 to 7, characterized in that, The device includes: The data acquisition module is used to acquire real-time data from the UAV and preprocess it to obtain target data; wherein, the real-time data of the UAV includes scene images collected by the camera mounted on the UAV, UAV GPS data, UAV three-axis attitude angles and UAV mounted camera parameters. The model inference module is used to perform target inference and scene segmentation inference on the scene images in the target data respectively, and outputs the hidden danger targets and scene region masks containing label information, confidence and target detection box information, and constructs the label information, confidence and target detection box information of the hidden danger targets and the scene region mask into the first dataset; The target calculation module is used to obtain the GPS coordinates and feature vectors of the potential hazard targets based on the UAV GPS data, UAV three-axis attitude angles, UAV mounted camera parameters and target detection box information of the potential hazard targets in the target data, and construct them into a second dataset; The target filtering module is used to perform filtering on the potential hazards based on the first dataset and the second dataset.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory that stores instructions, which, when executed by the at least one processor, cause the at least one processor to perform the drone hazard target filtering method as described in any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores executable instructions that, when executed, cause the machine to perform the drone hazard target filtering method as described in any one of claims 1 to 7.

Images