An unmanned aerial vehicle automatic patrol method and system based on AI vision

Abstract

The application provides an unmanned aerial vehicle automatic patrol method and system based on AI vision, relates to the technical field of data processing, and comprises the following steps: acquiring static environment prior data of a target patrol area, and constructing an initial environment cognition map; collecting multi-view visual observation data, and performing cognition consistency evaluation, and calculating a visual cognition uncertainty index corresponding to each observation position; based on the visual cognition uncertainty index, performing grid division on the target patrol area, generating a spatial cognition confidence map, screening out grid units with a cognition confidence value lower than an initial cognition confidence value to form a to-be-supplemented blind observation area set; based on the to-be-supplemented blind observation area set, constructing an active perception optimization target function; solving the active perception optimization target function, generating an updated patrol path; and controlling the target patrol unmanned aerial vehicle to execute the updated patrol path, and synchronously updating the spatial cognition confidence map and the initial environment cognition map in the execution process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an AI vision-based method and system for automatic patrol of drones. Background Technology

[0002] With the widespread application of drone technology in security patrols, environmental monitoring, and other fields, traditional automated patrol schemes based on fixed routes or simple rules are gradually revealing their limitations. These schemes typically rely on preset GPS coordinates for reciprocating flights, lacking a deep understanding of the visual characteristics of the surrounding environment and struggling to distinguish between "flying past without clear observation" and "flying past with full awareness." When encountering changes in lighting, seasonal shifts, or sudden obstructions, drones often fail to recognize the decreased reliability of their observation data, leading to a high rate of missed detections of concealed targets or camouflaged behavior. Furthermore, existing path replanning mechanisms are mostly based on immediate anomaly detection triggers, lacking effective accumulation and confidence modeling of historical observation information. This easily leads to repeated patrols or missed visits to key areas, making it difficult to achieve a dynamic balance between energy management and mission efficiency, and failing to meet the urgent need for autonomous cognition and decision-making capabilities in complex scenarios.

[0003] Therefore, an AI vision-based method and system for automatic drone patrol is provided. Summary of the Invention

[0004] To address the aforementioned technical problems of failing to detect concealed targets or camouflaged behavior in complex scenarios, resulting in repeated patrols or missed visits to key areas, the present invention aims to provide an AI vision-based drone automatic patrol method and system.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an AI vision-based method for automatic patrol of unmanned aerial vehicles, the method comprising: Acquire static environmental prior data of the target patrol area and construct an initial environmental cognition map, wherein the initial environmental cognition map includes: spatial topology, typical semantic distribution and statistical information of normal visual features; The target patrol drone is controlled to perform the first round of patrol mission based on a preset initial path, and collects multi-view visual observation data during the flight; The onboard AI vision module is used to evaluate the cognitive consistency of the multi-view visual observation data and calculate the visual cognitive uncertainty index corresponding to each observation position. Based on the aforementioned visual cognitive uncertainty index, the target patrol area is divided into grids to generate a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. Grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value are selected to form a set of observation areas to be filled in; Based on the set of observation areas to be filled, and combined with the constraints of the UAV's current position, remaining energy, and observation angle, an active perception optimization objective function is constructed; the active perception optimization objective function is solved to generate an updated patrol path; The target patrol drone is controlled to execute the updated patrol path, and the spatial cognition confidence map and the initial environmental cognition map are updated synchronously during the execution.

[0006] Furthermore, the calculation process of the visual cognitive uncertainty index includes: Extract deep semantic features from the current multi-view visual observation data and align their distribution with the normal visual features at the corresponding locations in the initial environmental cognition map; Calculate the KL divergence or JS divergence between semantic feature distributions as the first uncertainty component; Using a Bayesian neural network, the prediction entropy of the current observation result in semantic classification is calculated as the second uncertainty component. The first uncertainty component and the second uncertainty component are weighted and fused to obtain the visual cognitive uncertainty index.

[0007] Furthermore, the process of dividing the target patrol area into grids includes: Using the digital elevation model of the target patrol area as the base map, and combining the spatial distribution characteristics of the visual perception uncertainty index, the target patrol area is divided into grids to obtain multiple grid units. For each grid cell, the visual cognitive uncertainty index corresponding to all observation positions falling within that cell is statistically analyzed, and the initial cognitive confidence value of that grid cell is calculated based on a preset confidence transformation function. The spatial cognitive confidence map is composed of all grid cells and their corresponding initial cognitive confidence values.

[0008] Furthermore, the process of calculating the cognitive confidence value recorded in each grid cell at the current moment includes: Obtain visual cognitive uncertainty indices corresponding to all observation locations falling within the grid cell, and form a set of uncertainty indices. Based on a preset time decay factor, different weights are assigned to indicators at different historical moments in the uncertainty indicator set, wherein the weight of the indicator at the current moment is higher than that of the indicator at historical moments; a weighted average method is used to fuse the values ​​of each indicator in the uncertainty indicator set to obtain the fused uncertainty indicator of the grid cell. Based on the fusion uncertainty index, the cognitive confidence value of the grid cell at the current moment is obtained by calculating using a preset monotonically decreasing transformation function.

[0009] Furthermore, the process of selecting grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value to form a set of observation areas to be filled includes: Traverse each grid cell in the spatial cognitive confidence map; mark grid cells whose current cognitive confidence value is lower than the initial cognitive confidence value as grid cells to be filled in; aggregate the spatial range of all grid cells to be filled in to generate the set of observation areas to be filled in.

[0010] Furthermore, based on the set of observation areas to be filled, and combined with the current position of the UAV, remaining energy, and observation angle constraints, the process of constructing the active perception optimization objective function includes: For each grid cell in the set of observation areas to be filled, a cognitive cost sub-item is defined to quantify the expected increase in the cognitive confidence value of the grid cell after the UAV performs one effective observation above the grid cell. Based on the current location of the UAV and the location distribution of the grid cells to be filled, a path cost sub-item is constructed, wherein the path cost sub-item is positively correlated with the total length of the UAV's flight trajectory; Based on the remaining energy of the UAV and the preset energy consumption model, an energy cost sub-item is constructed, wherein the energy cost sub-item is used to constrain the solution space of the active perception optimization objective function to not exceed the executable range of the current remaining energy of the UAV; Based on the physical rotation range of the UAV's onboard gimbal and obstacle avoidance safety constraints, an observation angle constraint sub-item is constructed to limit the range of pitch and heading angle values ​​for candidate observation nodes. The cognitive cost sub-item, path cost sub-item, energy cost sub-item, and observation angle constraint sub-item are weighted and combined to generate the active perception optimization objective function.

[0011] Furthermore, the process of controlling the target patrol drone to execute the updated patrol path and simultaneously updating the spatial cognitive confidence map and the initial environmental cognitive map during execution includes: According to the order of the node sequence in the updated patrol path, the target patrol drone is controlled to fly to each observation node in sequence; At each observation node, the airborne AI vision module is controlled to collect current multi-view visual observation data, and to evaluate the cognitive consistency of the current multi-view visual observation data, and to calculate the updated visual cognitive uncertainty index corresponding to the current observation position. Based on the updated visual cognitive uncertainty index, locate the grid cell to which it belongs, and recalculate the cognitive confidence value of the grid cell at the current moment to complete the update of the corresponding grid cell in the spatial cognitive confidence map. The updated multi-view visual observation data and its corresponding deep semantic features are integrated into the normal visual feature statistics of the corresponding position in the initial environmental cognition map to achieve dynamic correction of the initial environmental cognition map. After completing the execution of all observation nodes in the updated patrol path, the updated spatial cognition confidence map and the initial environmental cognition map are output as the input basis for the next round of patrol tasks.

[0012] A second aspect of the present invention also provides an AI vision-based unmanned aerial vehicle (UAV) automatic patrol system, comprising: an environmental mapping module, a patrol execution module, a visual evaluation module, a confidence generation module, an area filtering module, a path planning module, and an update and optimization module; The environmental mapping module is used to acquire static environmental prior data of the target patrol area and construct an initial environmental cognitive map; The patrol execution module is used to control the target patrol drone to perform the first round of patrol missions based on a preset initial path or to perform updated patrol missions based on an updated patrol path, and to collect corresponding multi-view visual observation data during flight. The visual evaluation module is used to evaluate the cognitive consistency of the multi-view visual observation data through the airborne AI vision module and calculate the visual cognitive uncertainty index corresponding to each observation position. The confidence generation module divides the target patrol area into grids based on the visual cognitive uncertainty index and generates a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. The region filtering module filters out grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value, forming a set of observation regions to be filled in; The path planning module constructs an active perception optimization objective function based on the set of observation areas to be filled, combined with the current position of the UAV, remaining energy, and observation angle constraints; solves the active perception optimization objective function to generate an updated patrol path; The update and optimization module is used to update the spatial cognition confidence map and the initial environmental cognition map during the process of updating the patrol path.

[0013] Compared with existing technologies, the beneficial effects of this invention are as follows: By introducing a visual cognitive uncertainty index and a spatial cognitive confidence map, this application achieves a fundamental shift in the UAV patrol mode from "experience-driven" to "cognitive state-driven." This application enables UAVs to proactively identify their weaknesses in understanding the environment and dynamically generate updated patrol paths based on an active perception optimization objective function, prioritizing areas where cognitive uncertainty is minimized. This significantly improves the ability to detect concealed and abnormal targets with limited energy consumption. By establishing a gridded confidence update mechanism and a closed-loop feedback process, the logical completeness and self-consistency of the patrol mission are ensured, ineffective and repetitive patrols are avoided, and the overall system's intelligence level and mission execution robustness are improved. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a schematic diagram illustrating the steps of an AI vision-based drone automated patrol method.

[0016] Figure 2 This is a schematic diagram of a module for an AI vision-based drone automatic patrol system.

[0017] Figure 3 This is a flowchart for evaluating the uncertainty index of visual cognition.

[0018] Figure 4 This is a flowchart illustrating the process of selecting and processing observations to be filled in. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] like Figure 1 As shown, an AI vision-based drone automatic patrol method includes: Acquire static environmental prior data of the target patrol area and construct an initial environmental cognition map, wherein the initial environmental cognition map includes: spatial topology, typical semantic distribution and statistical information of normal visual features; The process of acquiring prior static environmental data of the target patrol area through drone pre-inspection, satellite remote sensing data, existing regional geographic information system (GIS) data, and supplementary data from on-site surveys includes, but is not limited to: data collection scope: full coverage of the target patrol area, taking into account regional boundaries, key points (such as intersections, building corners, densely vegetated areas), and concealed areas to ensure data integrity; data types (fitting the needs of subsequent visual observation and comparison): spatial geometric data: regional terrain elevation data, coordinates, dimensions, and positional relationships of static targets such as buildings / roads / vegetation; semantic data: category labels of static targets (such as roads, buildings, vegetation, fixed facilities, etc.) and distribution range data; visual feature data: visual data of static targets (RGB images, depth images) under multiple perspectives and normal lighting (such as sunny days and cloudy days), avoiding data interference from extreme weather (heavy fog, heavy rain).

[0022] Based on the collected spatial geometric data, a graph theory model is used to construct a topological network. Nodes are key static targets within the region (such as building vertices and road intersections), and edges are spatial relationships between targets (such as distance and connectivity). The topological network is simplified and optimized by removing redundant nodes (such as duplicate points with an adjacent distance of less than 0.5m), and the coordinate information of key nodes is marked to form a spatial topological structure that can be quickly called for subsequent UAV path planning and location positioning.

[0023] The collected semantic data is classified and labeled to clarify the semantic category system (such as roads, buildings, vegetation, and fixed facilities). A gridded statistical method is used to divide the target patrol area into grids of a preset size (consistent with the grid size of the subsequent spatial cognition confidence map). The semantic categories and proportions in each grid are recorded to form a typical semantic distribution matrix and clarify the normal semantic characteristics of each area.

[0024] The collected normal visual data is preprocessed (denoising, size normalization, illumination correction), and the visual feature vectors of each semantic target are extracted by a pre-trained deep CNN model. The visual feature vectors of the same semantic target are statistically analyzed, and the mean, variance and other statistical parameters of the feature vectors are calculated to form the normal visual feature statistical information of each semantic target. The feature variation range under different observation perspectives is recorded to provide a benchmark for subsequent observation feature distribution alignment and deviation calculation.

[0025] In this embodiment, spatial topology, typical semantic distribution, and normal visual feature statistics are linked and integrated to establish a mapping relationship among the three (e.g., a certain topological node corresponds to a certain semantic category, and is associated with the normal visual feature statistics of that category). The map is then lightweighted to adapt to the storage and retrieval requirements of the airborne AI vision module, forming an initial environmental cognition map that can be used for subsequent patrol comparison.

[0026] The target patrol drone is controlled to perform the first round of patrol mission based on a preset initial path, and collects multi-view visual observation data during the flight; The onboard AI vision module performs a cognitive consistency assessment on the multi-view visual observation data and calculates the visual cognitive uncertainty index corresponding to each observation position. The visual cognitive uncertainty index is used to quantify the degree of deviation and unreliability between the current observation result and the initial environmental cognitive map. Based on the aforementioned visual cognitive uncertainty index, the target patrol area is divided into grids to generate a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. Grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value are selected to form a set of observation areas to be filled in; Based on the set of observation areas to be filled, and combined with the constraints of the UAV's current position, remaining energy, and observation angle, an active perception optimization objective function is constructed; the active perception optimization objective function is solved to generate an updated patrol path; The target patrol drone is controlled to execute the updated patrol path, and the spatial cognition confidence map and the initial environmental cognition map are updated synchronously during the execution.

[0027] like Figure 3As shown, in this embodiment, the calculation process of the visual cognitive uncertainty index includes: It should be noted that the multi-view visual observation data (such as RGB images and depth images) collected during the drone flight are preprocessed to eliminate noise interference caused by flight jitter, lighting changes, and lens distortion, ensuring the accuracy of feature extraction. Deep semantic features of the multi-view visual observation data are extracted by pre-trained deep convolutional neural networks (CNNs, such as ResNet and ViT models). These features can accurately represent the core semantic information of the observation area (such as roads, buildings, vegetation, and abnormal obstacles), and maintain dimensional consistency with the normal visual features in the initial environmental cognition map.

[0028] The multi-view visual observation data is denoised (using Gaussian filtering), grayscale corrected, and size normalized to obtain standardized multi-view visual observation data, denoted as . .

[0029] Standardized multi-view visual observation data Input a pre-trained deep CNN model, compute it through forward propagation, and output the deep semantic features of the current multi-view visual observation data. ;in The feature dimension is determined by the model structure and is usually taken as 256, 512, etc.

[0030] Recall the normal visual features corresponding to the current observation location from the initial environmental cognition map. By normalizing features, deep semantic features are... and normal visual characteristics Scaling to the [0,1] interval and converting it to a probability distribution based on the Softmax function, so that the sum of each component is 1, achieves deep semantic features. With normal visual characteristics Aligning distributions with the same dimension, value range, and probability distribution ensures the rationality of subsequent divergence calculations.

[0031] It should be noted that the first uncertainty component is used to quantify the deviation between the semantic feature distribution of real-time observation data and the normal visual feature distribution. The larger the deviation, the greater the difference between the current observation area and the preset normal environment, and the higher the uncertainty. KL divergence (Kullback-Leibler Divergence) or JS divergence (Jensen-Shannon Divergence) is used for calculation. The two can be selected according to the actual scenario (KL divergence is more sensitive to distribution differences, while JS divergence has a more uniform value range and is suitable for subsequent weighted fusion).

[0032] Calculate the KL divergence between semantic feature distributions. The KL divergence is used to measure the probability distributions between two probability distributions. (i.e., the distribution of deep semantic features in the observed data) and The asymmetric difference between (normal visual feature distributions) is defined as the deep semantic feature distribution. Compared to the normal visual feature distribution The information gain is calculated using the following formula:

[0033] ;in Let KL divergence be a metric. For the aligned deep semantic features The probability of each component is used to convert deep semantic features into a probability distribution through the Softmax function; For the aligned normal visual features The probability of each component is used to transform the normal visual features into a probability distribution through the Softmax function; For feature dimensions.

[0034] To further explain, when and When the KL divergence approaches infinity, it is determined that there is a significant anomaly at the observation location.

[0035] It should be noted that JS divergence is a symmetric form of KL divergence, with a value range of [0,1], thus avoiding the problems of KL divergence being asymmetric and having no upper limit on the value.

[0036] The JS divergence between semantic feature distributions is calculated using the following formula: ;in ,Right now and The average distribution; the meanings of the remaining parameters are consistent with those of the KL divergence.

[0037] Let the JS divergence or KL divergence be the first uncertainty component, and denoted as . .

[0038] It should be noted that the second uncertainty component is used to quantify the unreliability of the current observation results in semantic classification. Even if the observed features deviate slightly from the normal features, if the AI ​​vision module is uncertain about the semantic classification result of the observed area (e.g., it cannot clearly determine whether it is normal vegetation or an abnormal obstacle), it should still be included in the uncertainty assessment. A Bayesian neural network (BNN) is used to calculate the prediction entropy; the higher the prediction entropy, the higher the uncertainty of the classification result.

[0039] In this embodiment, the process of calculating the predictive entropy of the current observation result in semantic classification using a Bayesian neural network includes: Aligned deep semantic features The input is a pre-trained Bayesian neural network (BNN), which, by introducing weight uncertainty, outputs the posterior probability distribution of the currently observed region belonging to various semantic categories. ,in For the first semantic tags, such as For roads, For buildings, etc.; Calculate prediction entropy for: ;in The total number of semantic categories is determined by the typical semantic distribution of the initial environmental cognition map, such as four categories: roads, buildings, vegetation, and abnormal targets. . For aligned deep semantic features Belongs to the Posterior probability of class semantics.

[0040] and the predicted entropy As the second uncertainty component, and denoted as .

[0041] To further explain, when all When they tend to be equal, the second uncertainty component Reaching the maximum value This indicates that the classification result is completely uncertain; when a certain When the value tends to 1 and the rest tend to 0, the second uncertainty component... A value approaching 0 indicates that the classification result is certain.

[0042] The first uncertainty component With the second uncertainty component Scale uniformly to the [0,1] range to eliminate differences in value range; For the scaled first uncertainty component With the second uncertainty component The visual cognitive uncertainty index is obtained by performing weighted fusion. ;in , These are the weighting coefficients.

[0043] like Figure 4 As shown, in this embodiment, the process of dividing the target patrol area into grids includes: Using the digital elevation model of the target patrol area as the base map, preprocessing is performed to eliminate terrain data errors, ensure that the gridded division matches the actual terrain, and match the spatial topology of the initial environmental awareness map: The original digital elevation model (DEM) data of the target patrol area (resolution determined according to patrol accuracy requirements, such as 1m×1m or 5m×5m) is retrieved for coordinate calibration to ensure consistency with the UAV's GPS positioning coordinates and the spatial topological coordinates of the initial environmental awareness map, thus avoiding positional deviations. A moving window filtering method (such as a 3×3 window) is used to eliminate noise points (such as isolated elevation anomalies) in the DEM data, resulting in a smoothed DEM base map. Based on the preset boundary coordinates of the target patrol area, the DEM base map is cropped to remove redundant terrain data outside the area, ensuring that the gridded area only covers the target patrol area.

[0044] Spatial distribution characteristics of statistical visual cognitive uncertainty index: Traverse all observation locations within the target patrol area and statistically analyze the visual cognitive uncertainty index at each location. Within the calculation area The spatial distribution density (number of observation points per unit area) and mean value are used to form a spatial distribution heat map of uncertainty indicators, which clarifies the spatial range of high uncertainty areas and low uncertainty areas.

[0045] The terrain complexity factor for determining the grid size is as follows: The terrain slope (in degrees) at each location is obtained based on smoothed digital elevation model data. A steeper slope indicates more complex terrain, resulting in a smaller terrain complexity factor (fineer grid). The formula is: The value range is [0.5, 1]. This refers to the slope of the terrain.

[0046] With preset base grid size (Based on the patrol accuracy setting, such as 10m×10m) the grid size for each location is calculated using the terrain complexity factor. ,formula: .

[0047] Based on the calculated mesh size The cropped digital elevation model base map is adaptively subdivided to obtain multiple irregular grid cells (or regular grid cells, selected according to actual needs), and each grid cell is uniquely numbered. , ; The total number of grid cells is recorded. The boundary coordinates, center coordinates, and average elevation of each grid cell are recorded. The spatial topology and typical semantic distribution of the initial environmental cognition map are associated to complete the grid division and obtain multiple grid cells. For each grid cell, statistically analyze the visual cognitive uncertainty index corresponding to all observation locations falling within that cell. A set of visual cognitive uncertainty indicators was obtained. The arithmetic mean was then calculated to obtain the average visual cognitive uncertainty index for that grid cell. Based on a preset confidence transformation function, the initial cognitive confidence value of the grid cell is calculated. ;in The adjustment factor is set to the default value. . The larger the value, the faster the initial cognitive confidence value decreases, making it easier to accurately screen areas to be filled in.

[0048] The spatial cognitive confidence map is composed of all grid cells and their corresponding initial cognitive confidence values.

[0049] In this embodiment, the process of calculating the cognitive confidence value recorded by each grid cell at the current moment includes: Based on the set of visual cognitive uncertainty indicators And based on a preset time decay factor , The closer to 1, the slower the time decay; the closer to 0, the faster the time decay. (Default) Different weights are assigned to indicators at different historical moments in the set of visual cognitive uncertainty indicators. ,in Current moment and the The time difference between moments; where the indicator weight at the current moment is higher than the indicator weight at historical moments; using a weighted average method, the indicator values ​​from the set of visual cognitive uncertainty indicators are fused to obtain the fused uncertainty index of the grid cell. ;in The total number of historical moments. For the corresponding number Indicators of visual perception uncertainty at any given moment. For the corresponding number The weight of indicators at any given time.

[0050] Based on the fusion uncertainty index, the cognitive confidence value of the grid cell at the current moment is calculated using a preset monotonically decreasing transformation function. .

[0051] In this embodiment, the process of selecting grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value to form a set of observation areas to be filled includes: Traverse each grid cell in the spatial cognition confidence map; set the current cognition confidence value. Below the initial cognitive confidence value The grid cells are marked as grid cells to be filled; the spatial extent of all grid cells to be filled is aggregated to generate the set of observation areas to be filled.

[0052] In this embodiment, the process of constructing the active perception optimization objective function based on the set of observation areas to be filled, combined with the current position of the UAV, remaining energy, and observation angle constraints, includes: Mark the grid cells to be filled as ; For each grid cell in the set of observation areas to be filled, a cognitive cost sub-item is defined to quantify the expected increase in the cognitive confidence value of the grid cell after the UAV performs a valid observation above the grid cell. That is, the lower the current confidence and the higher the uncertainty, the greater the expected cognitive cost.

[0053] Among them, the cognitive cost sub-item for: ;in To preset the cognitive completion threshold; For blind grid cells The cognitive confidence score.

[0054] Based on the current location of the UAV and the location distribution of the grid cells to be filled, a path cost sub-item is constructed. ;in This represents the total length of the drone's flight path. This represents the number of observation nodes; For the first The coordinates of each observation node; It is the first The observation node and the first The Euclidean distance between the observation nodes. The path cost sub-item is positively correlated with the total length of the UAV flight trajectory; Based on the remaining energy of the UAV and the preset energy consumption model, an energy cost sub-item is constructed. ;in Total energy consumption includes, but is not limited to, flight energy consumption and observation energy consumption; The remaining energy of the drone is currently available; wherein the energy cost sub-item is used to constrain the solution space of the active perception optimization objective function to not exceed the executable range of the drone's current remaining energy.

[0055] Based on the physical rotation range of the UAV's onboard gimbal and obstacle avoidance safety constraints, an observation angle constraint sub-item is constructed. ;in No. The elevation angle of each observation node; , These are the maximum and minimum values ​​of the airborne gimbal's pitch angle, respectively; and , ; No. The heading angle of each observation node; , These are the maximum and minimum values ​​of the heading angle of the airborne gimbal, respectively; and , Used to limit the range of pitch and heading angles for candidate observation nodes.

[0056] To further explain, when the pitch angle and yaw angle are within the preset range, the penalty is 0; when they exceed the range, the penalty increases as the excess increases, ensuring that the observation angle meets the physical constraints and obstacle avoidance requirements of the equipment.

[0057] For the cognitive cost sub-item Path cost sub-item Energy cost sub-item and observation angle constraint sub-items Standardize the cognitive cost items and then process them accordingly. Path cost sub-item Energy cost sub-item and observation angle constraint sub-items The active sensing optimization objective function is generated by performing weighted combination.

[0058] The objective function is solved using a heuristic optimization algorithm (such as genetic algorithm or particle swarm optimization), and the optimal sequence of observed nodes is output, which is the updated patrol path. Specifically: The Particle Swarm Optimization (PSO) algorithm searches for the optimal solution by simulating the flight and updating of particles in the solution space. It has the advantages of fast convergence speed and simple parameter settings, and is suitable for scenarios with a small number of blind spots to be filled and where patrol paths need to be generated quickly. The specific process is as follows: Each particle corresponds to an observation node sequence (i.e., a candidate patrol path), employing a combination of real-number encoding and integer encoding: the particle dimension is p (the total number of blind spots to be filled), and each dimension takes the value of an integer in the range [1, p]. The values ​​of each dimension for the same particle are unique (each blind spot to be filled is observed only once). The solution space is the set of all observation node sequences that conform to the encoding rules.

[0059] M particles are randomly generated (in this embodiment, the particle swarm size M is 40 by default; when p≤10, M=30), and each particle corresponds to a random sequence of observation nodes; at the same time, the path constraints (i.e. energy constraints and observation angle constraints) corresponding to each particle are checked, infeasible particles are eliminated, and the particles are regenerated to ensure that the initial particle swarm consists of feasible solutions.

[0060] use ;in To proactively perceive and optimize the objective function value; This is the penalty coefficient, the default value. , As a constraint penalty term, when an individual satisfies all constraints (energy, observation angle), If not satisfied, To constrain the total amount exceeding the limit, individuals that do not meet the constraint have extremely low fitness and are eliminated.

[0061] Calculate the fitness value for each particle. fitness value The higher the value, the better the path of the particle.

[0062] Each particle tracks its optimal position. That is, the sequence of observation nodes corresponding to its own historical best fitness and its global best position. That is, the sequence of observation nodes corresponding to the best fitness in the entire particle swarm's history, updating its own velocity and position, with the velocity update formula as follows: ; in This is the inertia weight, which defaults to 0.8 and decreases linearly to 0.4 during iteration to balance global exploration and local convergence. , This is the acceleration factor, with a default value of 2.0, which controls the speed at which particles approach their individual and global optimum. , Add random numbers between [0, 1] to increase the randomness of the search; For the first The generation The first particle The speed of the dimension, with a value range of [-2, 2], is used to avoid excessive speed causing search chaos. For the first The generation The first particle The location of the dimension, i.e., the observation node number.

[0063] When the number of iterations reaches the preset maximum value, or the change in the global optimal fitness value is less than or equal to 8 consecutive iterations. After the iteration terminates, the global optimal position is found. The corresponding sequence of observation nodes is the optimal solution.

[0064] By stitching together the starting node, i.e. the current position of the UAV, a complete updated patrol path is obtained, and the three-dimensional coordinates and observation angles of each observation node are output for adaptation to the flight control system for execution.

[0065] It should be further noted that if there are many blind areas to be filled, the genetic algorithm should be preferred, as it provides a more comprehensive search and avoids local optima.

[0066] In this embodiment, the process of controlling the target patrol drone to execute the updated patrol path and simultaneously updating the spatial cognitive confidence map and the initial environmental cognitive map during the execution includes: According to the order of the node sequence in the updated patrol path, the target patrol drone is controlled to fly to each observation node in sequence; At each observation node, the airborne AI vision module is controlled to collect the current multi-view visual observation data, and the above calculation process is repeated to obtain the cognitive consistency assessment of the current multi-view visual observation data, as well as the updated visual cognitive uncertainty index corresponding to the current observation position. Based on the updated visual cognitive uncertainty index, locate the grid cell to which it belongs, and recalculate the cognitive confidence value of the grid cell at the current moment to complete the update of the corresponding grid cell in the spatial cognitive confidence map. The updated multi-view visual observation data and its corresponding deep semantic features are integrated into the normal visual feature statistics of the corresponding position in the initial environmental cognition map to achieve dynamic correction of the initial environmental cognition map. After completing the execution of all observation nodes in the updated patrol path, the updated spatial cognition confidence map and the initial environmental cognition map are output as the input basis for the next round of patrol tasks.

[0067] like Figure 2 As shown, an AI vision-based drone automatic patrol system includes: an environmental mapping module, a patrol execution module, a visual evaluation module, a confidence generation module, an area selection module, a path planning module, and an update and optimization module. The environmental mapping module is used to acquire static environmental prior data of the target patrol area and construct an initial environmental cognitive map; The patrol execution module is used to control the target patrol drone to perform the first round of patrol missions based on a preset initial path or to perform updated patrol missions based on an updated patrol path, and to collect corresponding multi-view visual observation data during flight. The visual evaluation module is used to evaluate the cognitive consistency of the multi-view visual observation data through the airborne AI vision module and calculate the visual cognitive uncertainty index corresponding to each observation position. The confidence generation module divides the target patrol area into grids based on the visual cognitive uncertainty index and generates a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. The region filtering module filters out grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value, forming a set of observation regions to be filled in; The path planning module constructs an active perception optimization objective function based on the set of observation areas to be filled, combined with the current position of the UAV, remaining energy, and observation angle constraints; solves the active perception optimization objective function to generate an updated patrol path; The update and optimization module is used to update the spatial cognitive confidence map and the initial environmental cognitive map during the execution of updating the patrol path. Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk, or optical disk, etc.

[0068] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0069] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more electronic devices to execute all or part of the steps of the methods described in the various embodiments of this application.

[0070] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed application can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0074] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for automated patrolling of drones based on AI vision, characterized in that, The method includes: Acquire static environmental prior data of the target patrol area and construct an initial environmental cognition map, wherein the initial environmental cognition map includes: spatial topology, typical semantic distribution and statistical information of normal visual features; The target patrol drone is controlled to perform the first round of patrol mission based on a preset initial path, and collects multi-view visual observation data during the flight; The onboard AI vision module is used to evaluate the cognitive consistency of the multi-view visual observation data and calculate the visual cognitive uncertainty index corresponding to each observation position. Based on the aforementioned visual cognitive uncertainty index, the target patrol area is divided into grids to generate a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. Grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value are selected to form a set of observation areas to be filled in; Based on the set of observation areas to be filled, and combined with the constraints of the UAV's current position, remaining energy, and observation angle, an active perception optimization objective function is constructed; the active perception optimization objective function is solved to generate an updated patrol path; The target patrol drone is controlled to execute the updated patrol path, and the spatial cognition confidence map and the initial environmental cognition map are updated synchronously during the execution.

2. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 1, characterized in that, The calculation process of the visual cognitive uncertainty index includes: Extract deep semantic features from the current multi-view visual observation data and align their distribution with the normal visual features at the corresponding locations in the initial environmental cognition map; Calculate the KL divergence or JS divergence between semantic feature distributions as the first uncertainty component; Using a Bayesian neural network, the prediction entropy of the current observation result in semantic classification is calculated as the second uncertainty component. The first uncertainty component and the second uncertainty component are weighted and fused to obtain the visual cognitive uncertainty index.

3. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 2, characterized in that, The process of dividing the target patrol area into grids includes: Using the digital elevation model of the target patrol area as the base map, and combining the spatial distribution characteristics of the visual perception uncertainty index, the target patrol area is divided into grids to obtain multiple grid units. For each grid cell, the visual cognitive uncertainty index corresponding to all observation positions falling within that cell is statistically analyzed, and the initial cognitive confidence value of that grid cell is calculated based on a preset confidence transformation function. The spatial cognitive confidence map is composed of all grid cells and their corresponding initial cognitive confidence values.

4. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 3, characterized in that, The process of calculating the cognitive confidence value recorded in each grid cell at the current moment includes: Obtain the visual cognitive uncertainty indexes corresponding to all observation positions falling within the grid cell, and construct a set of visual cognitive uncertainty indices. Based on a preset time decay factor, different weights are assigned to indicators at different historical moments in the set of visual cognitive uncertainty indicators, wherein the weight of the indicator at the current moment is higher than that of the indicator at a historical moment; and a weighted average method is used to fuse the values ​​of each indicator in the set of uncertainty indicators to obtain the fused uncertainty index of the grid cell. Based on the fusion uncertainty index, the cognitive confidence value of the grid cell at the current moment is obtained by calculating using a preset monotonically decreasing transformation function.

5. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 4, characterized in that, The process of selecting grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value to form a set of observation areas to be filled includes: Traverse each grid cell in the spatial cognitive confidence map; mark grid cells whose current cognitive confidence value is lower than the initial cognitive confidence value as grid cells to be filled in; aggregate the spatial range of all grid cells to be filled in to generate the set of observation areas to be filled in.

6. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 5, characterized in that, Based on the set of observation areas to be filled, and combined with the constraints of the UAV's current position, remaining energy, and observation angle, the process of constructing the active perception optimization objective function includes: For each grid cell in the set of observation areas to be filled, a cognitive cost sub-item is defined to quantify the expected increase in the cognitive confidence value of the grid cell after the UAV performs one effective observation above the grid cell. Based on the current location of the UAV and the location distribution of the grid cells to be filled, a path cost sub-item is constructed, wherein the path cost sub-item is positively correlated with the total length of the UAV's flight trajectory; Based on the remaining energy of the UAV and the preset energy consumption model, an energy cost sub-item is constructed, wherein the energy cost sub-item is used to constrain the solution space of the active perception optimization objective function to not exceed the executable range of the current remaining energy of the UAV; Based on the physical rotation range of the UAV's onboard gimbal and obstacle avoidance safety constraints, an observation angle constraint sub-item is constructed to limit the range of pitch and heading angle values ​​for candidate observation nodes. The cognitive cost sub-item, path cost sub-item, energy cost sub-item, and observation angle constraint sub-item are weighted and combined to generate the active perception optimization objective function.

7. The method for automatic patrol of unmanned aerial vehicles based on AI vision according to claim 6, characterized in that, The process of controlling the target patrol drone to execute the updated patrol path and simultaneously updating the spatial cognition confidence map and the initial environmental cognition map during the execution includes: According to the order of the node sequence in the updated patrol path, the target patrol drone is controlled to fly to each observation node in sequence; At each observation node, the airborne AI vision module is controlled to collect current multi-view visual observation data, and to evaluate the cognitive consistency of the current multi-view visual observation data, and to calculate the updated visual cognitive uncertainty index corresponding to the current observation position. Based on the updated visual cognitive uncertainty index, locate the grid cell to which it belongs, and recalculate the cognitive confidence value of the grid cell at the current moment to complete the update of the corresponding grid cell in the spatial cognitive confidence map. The updated multi-view visual observation data and its corresponding deep semantic features are integrated into the normal visual feature statistics of the corresponding position in the initial environmental cognition map to achieve dynamic correction of the initial environmental cognition map. After completing the execution of all observation nodes in the updated patrol path, the updated spatial cognition confidence map and the initial environmental cognition map are output as the input basis for the next round of patrol tasks.

8. An AI vision-based drone automatic patrol system, implementing the AI ​​vision-based drone automatic patrol method described in any one of claims 1 to 7, characterized in that, include: The system includes modules for environmental mapping, patrol execution, visual assessment, confidence generation, area filtering, route planning, and update and optimization. The environmental mapping module is used to acquire static environmental prior data of the target patrol area and construct an initial environmental cognitive map; The patrol execution module is used to control the target patrol drone to perform the first round of patrol missions based on a preset initial path or to perform updated patrol missions based on an updated patrol path, and to collect corresponding multi-view visual observation data during flight. The visual evaluation module is used to evaluate the cognitive consistency of the multi-view visual observation data through the airborne AI vision module and calculate the visual cognitive uncertainty index corresponding to each observation position. The confidence generation module divides the target patrol area into grids based on the visual cognitive uncertainty index and generates a spatial cognitive confidence map, wherein each grid cell records the cognitive confidence value at the current moment. The region filtering module filters out grid cells with cognitive confidence values ​​lower than the initial cognitive confidence value, forming a set of observation regions to be filled in; The path planning module constructs an active perception optimization objective function based on the set of observation areas to be filled, combined with the current position of the UAV, remaining energy, and observation angle constraints; solves the active perception optimization objective function to generate an updated patrol path; The update and optimization module is used to update the spatial cognition confidence map and the initial environmental cognition map during the process of updating the patrol path.

Images