A cognitive-enhanced urban space unmanned aerial vehicle visual target search method

Abstract

The application relates to a cognitive-enhanced urban space unmanned aerial vehicle visual target search method. The method comprises the following steps: inputting a target image and a target text description into a multimodal large model to generate a target-related semantic element set and a gravitational weight; performing semantic segmentation based on the semantic elements, combining depth information to construct and dynamically update a target gravity map; updating a target probability map through a distance attenuation function according to the unmanned aerial vehicle pose and the field of view angle; constructing an occupancy grid map by projecting an obstacle; performing weighted clustering on the gravity map and the probability map to generate gravity peak points and probability peak points, and merging to form a candidate target point set; planning a path for each candidate target point and calculating a free energy change, and selecting an optimal target point and path; and converting the optimal search path into a flight control instruction to execute target search. The method can improve the autonomous visual target search efficiency and robustness of the unmanned aerial vehicle in a complex urban environment.

Description

Technical Field

[0001] This application relates to the field of UAV visual navigation technology, and in particular to a cognitively enhanced UAV visual target search method for urban space. Background Technology

[0002] With the continuous development of unmanned aerial vehicle (UAV) platforms, miniaturized visual sensors, edge computing devices, and low-altitude communication technologies, UAVs, due to their advantages of high maneuverability, flexible deployment, wide field of view, and high mission adaptability, have been widely applied in scenarios such as urban logistics and distribution, disaster relief, security patrols, emergency reconnaissance, traffic monitoring, and facility inspection. In these applications, visual target search is one of the crucial foundational capabilities of UAV autonomous intelligence. It typically requires UAVs to continuously perceive the external environment using onboard cameras, depth sensors, and positioning units, autonomously completing tasks such as target search, path planning, obstacle avoidance, and target confirmation, even under conditions of partial lack of prior knowledge and incomplete environmental observation. Especially in open urban environments, search targets may manifest as vehicles, shops, billboards, signs, or other static objects, while complex factors such as tall buildings, road networks, trees, pedestrians, and dynamic vehicles further increase the difficulty of target search.

[0003] In recent years, with the development of embodied intelligence, visual-language navigation, and multimodal large-scale model technology, target search research has gradually shifted from "search in geometric space" to "embodied search combining semantic understanding and environmental interaction." In the field of indoor object navigation, existing research has proposed methods such as target-oriented semantic exploration, zero-shot object target navigation, and fine-grained target cues, improving search performance through visual-language joint embedding or semantic map construction. Furthermore, research such as AerialVLN and EmbodiedCity has begun to extend visual-language navigation and embodied intelligence frameworks to urban spaces, providing new support for UAV task modeling, simulation evaluation, and benchmark construction in open urban environments. Meanwhile, research such as NEUSIS and Say-REAPEx has attempted to introduce neural symbolic reasoning, large-scale language models, or multimodal models into UAV search tasks, enabling agents to utilize common-sense knowledge, visual semantics, and language descriptions to assist in path recognition and online decision-making. These studies demonstrate that utilizing the cross-modal alignment and knowledge reasoning capabilities of large models to provide task priors and semantic understanding for UAV target search has become a new development trend.

[0004] However, existing technologies still have significant shortcomings in real-world UAV visual target search in urban spaces. First, while some methods have incorporated visual language models or large-scale models, most remain at the level of task instruction parsing, target category matching, or zero-shot detection, failing to stably transform the deep relationships between "target image—text description—environmental semantic entities" into quantifiable and computable search priorities. In other words, existing technologies generally lack a unified mechanism to quantify urban semantic objects strongly related to the target into navigation decision-making criteria. Second, existing methods often focus on single map representations, such as establishing only geometric occupancy maps, single-layer probability maps, or local semantic representations, making it difficult to simultaneously form a unified cognitive memory of target-related high-value areas, unexplored areas, and the spatial distribution of obstacles. Therefore, it is difficult to support continuous reasoning and dynamic updates during long-range search processes. Third, in large-scale urban open environments, search tasks inherently involve a contradiction between "exploring unknown areas" and "developing high-value areas," and most existing methods lack a unified decision-making and evaluation framework, making it difficult to adaptively balance these two types of behaviors according to task progress.

[0005] Therefore, there is an urgent need to propose a new visual target search technology for urban space drones that can comprehensively utilize target images and text descriptions to obtain target-related semantic priors during the task input stage, combine visual observation with spatial semantic relationships to form multi-layered cognitive memory during the environmental perception stage, and conduct a unified evaluation of exploration and development benefits during the decision-making and planning stage. Summary of the Invention

[0006] Based on this, it is necessary to provide a cognitively enhanced urban space UAV visual target search method that can transform the implicit semantic relationship between the target image and the text description into quantified gravity, construct a three-layer spatial cognitive memory, and achieve a dynamic balance between exploration and development through the free energy mechanism, in order to address the above-mentioned technical problems.

[0007] A cognitively enhanced method for visual target search by unmanned aerial vehicles (UAVs) in urban space, the method comprising: Acquire the target image and target text description for the search task, as well as the RGB images, depth images and pose information collected in real time during the drone's flight; The target image and target text description are input into a multimodal large model to generate a set of target-related semantic elements. The spatial correlation between each semantic element in the set and the target is quantified to obtain the gravitational weight of each semantic element. Semantic segmentation is performed on the current RGB image based on the set of target-related semantic elements to obtain the segmentation result; the pixel-level depth information in the segmentation result is converted into world coordinates based on the depth image, pose information and intrinsic and extrinsic parameters of the UAV vision sensor, and the world coordinates are discretized into a three-dimensional grid; the target gravity map is constructed and dynamically updated based on the semantic label with the highest frequency in each grid and the gravity weight corresponding to the semantic label. The current visible grid set is determined based on the current position, orientation, and field of view of the UAV. For each grid in the visible grid set, the probability of the target presence is updated using a decay function based on the distance between the grid center and the current position of the UAV. The target probability map is constructed and dynamically updated. Project the three-dimensional world coordinates onto the two-dimensional horizontal plane, determine the state of each two-dimensional grid based on the projection results, including occupied state, idle state and unknown state, construct and dynamically update the occupied grid map. The target gravity map at the current moment is subjected to weighted clustering to obtain several gravity clusters. The weighted centroid of each gravity cluster is calculated as the gravity peak point. The target probability map at the current moment is subjected to weighted clustering to obtain several probability clusters. The weighted centroid of each probability cluster is calculated as the probability peak point. The gravity peak points and probability peak points are merged to form a set of candidate target points. For each candidate target point in the candidate target point set, a candidate path from the current position of the UAV to the candidate target point is planned based on the current occupied grid map; the change of free energy after executing each candidate path is calculated, the candidate target point with the largest change of free energy is selected as the optimal target point, and the candidate path corresponding to the optimal target point is taken as the optimal search path; The optimal search path is translated into UAV flight control commands to control the UAV to perform target search.

[0008] The aforementioned cognitively enhanced UAV visual target search method for urban space first utilizes a multimodal large model to jointly understand target images and text descriptions. With carefully designed cue words, it directly transforms the semantic relevance inherent in human common sense into calculable numerical gravity weights. This enables the UAV to accurately identify high-value semantic regions strongly related to the target from a seemingly chaotic urban environment, converting abstract semantic understanding into quantifiable search priorities, thus fundamentally solving the technical bottleneck of semantic information not driving navigation decisions. Secondly, by constructing and dynamically updating a target gravity map, a target probability map, and an occupancy grid map in parallel, a three-layer collaborative spatial cognitive memory is formed. The target gravity map records the semantic relevance of each region to the target—development value; the target probability map dynamically marks under-observed areas—exploration value—using a distance decay function; and the occupancy grid map represents obstacle distribution and passable space—safety constraints. These three maps complement each other and are maintained uniformly, forming a complete spatial cognitive structure similar to a human mental model. This allows the UAV to continuously reason and update during long-range searches, ensuring that high-value areas are not overlooked and that blind exploration does not lead to danger. Finally, by introducing a free energy decision model with temperature parameter adjustment, development gains, exploration gains, path costs, and flight risks are incorporated into a unified evaluation system, and temperature parameters are used to achieve adaptive weighting of the two. The temperature parameter dynamically changes with exploration gains. In the early stages of the search, there is a lot of unknown space, and the entropy term dominates, so the system tends to choose probability peak points for exploration. As the mission progresses and the environment becomes more familiar, the system automatically switches to choosing gravity peak points for development. This adaptive decision-making mechanism allows the UAV to intelligently adjust its behavior strategy according to the mission progress without the need for manual setting of fixed weights, achieving a dynamic balance between exploration and development. It also achieves cognitive enhancement, integrating semantic cognitive guidance with the free energy decision-making mechanism, effectively balancing development and exploration strategies in target search, and significantly improving the efficiency and robustness of autonomous visual target search for UAVs in complex urban environments. Attached Figure Description

[0009] Figure 1 A flowchart of a cognitively enhanced urban space UAV visual target search method in one embodiment; Figure 2 This is a schematic diagram illustrating the generation of target-related semantic elements based on a multimodal large model in one embodiment; Figure 3 This is a schematic diagram of gravity weight quantization prompts in one embodiment; Figure 4 This is a schematic diagram of target estimation based on three-layer spatial cognitive memory in another embodiment; Figure 5 This is a flowchart of a decision-making process based on free energy in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In one embodiment, such as Figure 1 As shown, a cognitively enhanced method for UAV visual target search in urban space is provided, including the following steps: Step 102: Obtain the target image and target text description for the search task, as well as the RGB images, depth images, and pose information collected in real time during the drone's flight.

[0012] The input for the search task is the target image. Target text description And the initial pose of the drone; the drone continuously acquires RGB images during flight. Depth images and pose information ,in Represents the rotation matrix. Represents the translation vector. Indicates the current location of the drone. This indicates the drone's orientation.

[0013] Step 104: Input the target image and target text description into the multimodal large model to generate a set of target-related semantic elements, and quantify the spatial correlation between each semantic element in the set and the target to obtain the gravitational weight of each semantic element.

[0014] The drone received the target image and target text description Then, the two are input into a multimodal large model to infer a set of semantic elements related to the target. Furthermore, the spatial relevance between each semantic element in the semantic element set and the target is quantified to obtain the gravitational weight of each semantic element. The multimodal large model first outputs target-related semantic elements based on the target image and target text description, such as parking lot, vehicle, parking space, and illuminated sign. Then, it scores the relevance between the above semantic elements and the search target by combining prompt words, forming a semantic gravity weight. This gravity weight is input into the subsequent 3D grid memory update process to guide the UAV to prioritize semantic regions where the target is more likely to appear.

[0015] Step 106: Perform semantic segmentation on the current RGB image based on the target-related semantic element set to obtain the segmentation result; convert the pixel-level depth information in the segmentation result into world coordinates based on the depth image, pose information and intrinsic and extrinsic parameters of the UAV visual sensor, and discretize the world coordinates into a three-dimensional grid; construct and dynamically update the target gravity map based on the semantic label that appears most frequently in each grid and the gravity weight corresponding to the semantic label.

[0016] At time step The drone acquires the current RGB image. Depth images and pose information Using the target-related semantic elements obtained in step 104 as input prompts, the segmentation model performs semantic segmentation on the current RGB image to obtain the segmentation result. The segmentation result includes segmentation masks and semantic labels for each semantic entity. The semantic segmentation model in this application is based on GroundSAM.

[0017] Then, let the intrinsic parameter matrix of the UAV visual sensor be... The extrinsic parameter matrix is For depth images any pixel The corresponding world coordinates are calculated using the world coordinate back projection formula. Then, the world coordinates are mapped to the three-dimensional mesh index using a discretization formula. For each grid cell, the semantic labels of the pixels falling within that grid are counted, and the semantic label with the highest frequency is used as the semantic representation of that grid cell. Then, a mapping function is used to map the semantic labels to corresponding gravity weights. To enhance temporal stability, a sliding fusion method is used to update the target gravity map.

[0018] Step 108: Determine the set of visible grids at the current moment based on the current position, orientation and field of view of the UAV. For each grid in the set of visible grids, update the probability of the target presence based on the distance between the grid center and the current position of the UAV using a decay function, and construct and dynamically update the target probability map.

[0019] Target Probability Map The same size as the target gravity map, each of its grids Probability of the corresponding target During initialization, it can be set to... This indicates that the target may exist anywhere at the start of the search.

[0020] At time step According to the current location of the drone Orientation and field of view Define the current set of visible meshes. A distance decay function is applied to the visible grid. Update the probability value, where denoted as Euclidean distance between the grid center and the drone's current position. As the drone moves through the scene, the probability of observed areas gradually decreases, while unexplored areas maintain a higher probability value, thus forming a target probability map representing the "degree of unexploredness".

[0021] Step 110: Project the three-dimensional world coordinate points onto the two-dimensional horizontal plane, determine the state of each two-dimensional grid based on the projection results, including occupied state, idle state and unknown state, and construct and dynamically update the occupied grid map.

[0022] Projecting three-dimensional world coordinates onto a two-dimensional horizontal plane yields a two-dimensional occupied grid index. i , j The grid occupancy status is defined as follows: 1 indicates occupied (obstacle area), 0 indicates empty (passable area), and -1 indicates unknown (unobserved area). As a preferred implementation, a logarithmic probability update formula can also be used to enhance map robustness: ; in, This represents the logarithmic probability value of the raster. For current observations, This is a priori.

[0023] Step 112: Perform weighted clustering on the target gravity map at the current moment to obtain several gravity clusters, and calculate the weighted centroid of each gravity cluster as the gravity peak point; perform weighted clustering on the target probability map at the current moment to obtain several probability clusters, and calculate the weighted centroid of each probability cluster as the probability peak point; merge the gravity peak points and the probability peak points to form a set of candidate target points.

[0024] Based on the target gravity map Weighted clustering was performed to obtain several gravitational clusters. And calculate the weighted centroid of each cluster as the gravitational peak point. Similarly, based on the target probability map... Several probability clusters were obtained The weighted centroids of each cluster are used as probability peak points. The final set of candidate target points is: ; This results in a set of candidate target points that simultaneously take into account both development and exploration.

[0025] Step 114: For each candidate target point in the candidate target point set, plan a candidate path from the current position of the UAV to the candidate target point based on the current occupied grid map; calculate the change in free energy after executing each candidate path, select the candidate target point with the largest change in free energy as the optimal target point, and take the candidate path corresponding to the optimal target point as the optimal search path; convert the optimal search path into UAV flight control commands to control the UAV to fly and perform target search.

[0026] For each candidate target point First, plan the arrival path based on the occupied grid map. Then, the free energy gain corresponding to the candidate point is calculated. The free energy form given in this application is: ; in, Indicates a development inclination. Indicates an exploratory tendency. This refers to the temperature parameter.

[0027] To make the implementation details more specific, potential energy and entropy can be written in discrete form: ; in, Indicates the target point The nearby local neighborhood, To prevent constants with a denominator of zero.

[0028] Temperature parameters can be defined as: ; in, These are the initial proportional control parameters. This is the temperature change rate control parameter.

[0029] Potential energy term The term ε is used to characterize the benefits of developing high-gravity regions, while the entropy term ε is used to characterize the benefits of exploring high-probability unknown regions. In the early stages of the search, due to the large amount of unknown space, the entropy term has a higher proportion, and the system tends to select probability peak points. As the task progresses and the amount of unknown space decreases, the system gradually tends to select gravity peak points to increase the degree of development of the target-related regions.

[0030] Furthermore, to clearly define the "obstacle avoidance cost" and "maneuvering cost," it is preferable to include a path cost term: ; Therefore, the joint decision function can be written as: ; The search mission is considered successful when the distance between the drone and the target is less than a preset threshold; the search mission is considered to have failed when the target is not found after reaching the maximum number of search steps or the maximum endurance constraint.

[0031] In the aforementioned cognitively enhanced UAV visual target search method for urban space, the multimodal large model first outputs target-related semantic elements based on the target image and target text description, such as parking lot, vehicle, parking space, and illuminated sign. Then, it scores the relevance between these semantic elements and the search target using prompt words, forming a semantic gravity weight. This gravity weight is input into the subsequent 3D grid memory update process to guide the UAV to prioritize semantic regions where the target is more likely to appear. Further, an open-vocabulary semantic segmentation model segments the current environment RGB image based on the semantic elements. Pixel values ​​in the depth image, along with camera intrinsic and extrinsic parameters, are used to calculate the world coordinates corresponding to each pixel. The world coordinates are discretized into a 3D grid according to a preset grid resolution, and the semantic label with the highest frequency in each grid is used as the semantic representation of that grid. The gravity weight corresponding to the semantic representation is then used to write a gravity value to that grid, thus obtaining a target gravity map. Simultaneously, the system determines the set of visible grids based on the current UAV position, orientation, and field of view, and updates the target probability values ​​in the visible grids using a distance decay function. As the drone moves through the scene, the probability of observed areas gradually decreases, while insufficiently explored areas maintain a higher probability value, thus forming a target probability map representing the "degree of unexploredness." Additionally, the system projects 3D spatial points onto a 2D horizontal grid, forming an occupied grid map. Grids with projected obstacles are marked as occupied, grids within the current view that do not contain obstacles are marked as idle, and other areas remain unknown. The path planner uses this occupied grid map to avoid obstacles and generate feasible paths to candidate target points.

[0032] In the candidate target point generation phase, weighted density clustering is performed on high-gravity regions in the target gravity map to obtain multiple gravity clusters and calculate their gravity peak points; weighted density clustering is also performed on high-probability regions in the target probability map to obtain multiple probability clusters and calculate their probability peak points. This forms a set of candidate target points that simultaneously considers development and exploration. In the action decision phase, a path is planned for each candidate target point, and the change in free energy after executing the path is estimated. The potential energy term represents the benefit of developing high-gravity regions, the entropy term represents the benefit of exploring high-probability unknown regions, and the temperature parameter controls the weight ratio between the two. In the early stages of the search, due to the large amount of unknown space, the entropy term has a higher proportion, and the system tends to select probability peak points; as the task progresses and the unknown space decreases, the system gradually tends to select gravity peak points to improve the development of the target-related regions. When the distance between the UAV and the target is less than a preset threshold, the search task is considered successful; when the maximum search steps or maximum endurance constraint is reached and the target is still not found, the search task is considered a failure.

[0033] In one embodiment, such as Figure 2 and Figure 3 As shown, the target image and target text description are input into a multimodal large model to generate a set of target-related semantic elements. The spatial relevance of each semantic element in the set to the target is quantified to obtain the gravitational weight of each semantic element, including: target image With target text description Input a large multimodal model and generate a set of target-related semantic elements as follows: ; in, Generate prompt words for target-related semantic elements; The spatial relevance of each semantic element in the semantic element set to the target is quantified to obtain the gravitational weight of each semantic element: ; in, For gravity weight quantification prompts, This represents the set of gravitational weights for each semantic element.

[0034] Specifically, this embodiment guides a multimodal large model to complete two sub-tasks—semantic element extraction and gravity weight scoring—by designing specialized prompts. Target-related semantic element generation prompts guide the model to output urban environmental semantic entities that have spatial relationships with the search target; gravity weight quantification prompts guide the model to numerically score the spatial relevance between each semantic element and the target. Through this method, the common-sense knowledge and cross-modal understanding capabilities inherent in the large model are transformed into quantifiable navigation decision-making criteria, solving the problem of difficulty in quantifying target semantics in existing technologies.

[0035] In one embodiment, semantic segmentation is performed on the current RGB image based on a set of target-related semantic elements to obtain a segmentation result, including: Using the set of target-related semantic elements as input prompts, the segmentation model performs semantic segmentation on the current RGB image to obtain the segmentation result. : ; in, It includes segmentation masks and semantic labels for each semantic entity. For the current RGB image, For the set of semantic elements related to the target.

[0036] Specifically, this embodiment employs an open-vocabulary segmentation model (such as GroundSAM) for semantic segmentation. Unlike traditional segmentation models that can only identify fixed categories, open-vocabulary segmentation models can segment images based on arbitrary input text prompts (i.e., a set of target-related semantic elements), identifying the semantic entities specified in the prompt words. This allows the UAV to dynamically adjust the environmental semantic categories it needs to focus on for the current search task, improving the flexibility and task relevance of environmental perception.

[0037] In one embodiment, based on the depth image, pose information, and intrinsic and extrinsic parameters of the UAV vision sensor, the pixel-level depth information in the segmentation result is converted into world coordinates, and the world coordinates are discretized into a 3D mesh, including: Let the intrinsic parameter matrix of the UAV visual sensor be... The extrinsic parameter matrix is For depth images any pixel Corresponding world coordinates for: ; in, Indicates the horizontal index. Indicates the vertical index. Represents the translation vector. Represents the rotation matrix; Furthermore, world coordinates are mapped to a 3D mesh index: ; in, For spatial boundaries, It is a three-dimensional voxel resolution.

[0038] Specifically, this embodiment uses an intrinsic parameter matrix. and extrinsic parameter matrix The pixel coordinates in the depth image are converted into 3D points in the camera coordinate system, and then the camera coordinates are converted into world coordinates using the inverse of the rotation matrix and the translation vector. This world coordinate back-projection formula is the foundation for constructing gravity maps, probabilistic maps, and occupancy raster maps. A discretization formula maps continuous world coordinates to discrete 3D grid indices, facilitating subsequent rasterized storage and computation.

[0039] In one embodiment, a target gravity map is constructed and dynamically updated based on the most frequent semantic label in each grid and the gravity weight corresponding to the semantic label, including: For each grid The semantic labels of pixels falling within the grid are counted, and the semantic label with the highest frequency is used as the semantic representation of the grid: ; in, Indicates mapping to a grid The set of pixels, Represents pixels Semantic tags.

[0040] Correspondingly, the gravitational value of this grid can be expressed as: ; in, This represents a lookup table or mapping function used to map semantic labels to corresponding gravity weights. In this application, voxels are assigned values ​​according to the gravity weights corresponding to the semantic elements.

[0041] To enhance temporal stability, a sliding fusion approach is preferred for updating the target gravity map. ; in, Represents a grid The gravitational value, This represents a lookup or mapping function used to convert semantic tags. Mapped to corresponding gravitational weights For historical memory coefficient, This represents the gravity diagram at the previous moment.

[0042] Specifically, in this embodiment, for each grid The semantic labels of pixels falling within the grid are counted, and the semantic label with the highest frequency is used as the semantic representation of the grid. Then, through the mapping function Find the gravity weight corresponding to the semantic label and assign a gravity value to the grid. To enhance temporal stability, a sliding fusion method is used to update the target gravity map, where the history memory coefficient... The fusion weights of historical and current observations are controlled. Through this method, the UAV can memorize and continuously update the distribution of high-value semantic regions in the environment.

[0043] In one embodiment, such as Figure 4 As shown, the visible grid set at the current moment is determined based on the UAV's current position, orientation, and field of view. For each grid in the visible grid set, the target presence probability is updated using a decay function based on the distance between the grid center and the UAV's current position. A target probability map is constructed and dynamically updated, including: At time step According to the current location of the drone Orientation and field of view Define the current visible mesh set as: ; in, The coordinates of the grid center; The probability values ​​are updated using a distance decay function on the visible mesh: ; The target probability map is then updated as follows: ; in, Indicates the attenuation coefficient. This represents the Euclidean distance between the grid center and the drone's current position. Represents the distance decay function. Represents the grid at the previous time step. The probability of the corresponding target existing.

[0044] Specifically, in this embodiment, the target probability map Same size as the target gravity map, during initialization This indicates that the target may exist at any location at the start of the search. Visual grid set The criterion is whether the cosine of the angle between the grid center's direction vector relative to the UAV and the UAV's orientation is greater than or equal to 1. This refers to whether the grid falls within the UAV's field of view. For a visible grid, the probability of a target's presence decays exponentially. The probability is attenuated, becoming more pronounced with increasing distance because distant observations are less reliable. For invisible grids, the probability remains constant. As the drone moves, the probability of observed areas gradually decreases, while insufficiently explored areas maintain a higher probability value, forming a target probability map representing the "degree of unexploredness."

[0045] In one embodiment, a three-dimensional world coordinate point is projected onto a two-dimensional horizontal plane, and the state of each two-dimensional grid is determined based on the projection result, including: Projecting the 3D world coordinates onto a 2D horizontal plane yields the 2D occupied grid index: ; in, The horizontal component of world coordinates. For two-dimensional occupancy of raster resolution, Index in the X direction. Index in the Y direction, This represents the minimum boundary value in the X direction for a two-dimensional occupied raster map. This represents the minimum boundary value in the Y direction of a two-dimensional occupied raster map. Occupied grid state is defined as: ; Among them, occupied grids are used to represent obstacle areas, empty grids represent passable areas, and unknown grids represent areas that have not yet been observed.

[0046] Specifically, this embodiment achieves dimensionality reduction from a 3D environment to a 2D path planning space by projecting 3D spatial points onto a 2D horizontal plane (i.e., ignoring Z-axis height information). The grid index of a point is obtained by subtracting the minimum map boundary, dividing by the resolution, and rounding down. For valid points in the depth image with a distance less than a threshold, their corresponding 2D grids are marked as occupied (1); grids within the current visible range that do not contain obstacles are marked as idle (0); other areas remain in an unknown state (-1). Using this occupied grid map, the path planner can avoid obstacles and generate feasible paths to candidate target points.

[0047] In one embodiment, the weighted centroid of each gravitational cluster is calculated as the gravitational peak point: ; in, Represents a gravitational cluster. Represents a grid The gravitational value, These are the coordinates of the grid center.

[0048] Specifically, in this embodiment, based on the target gravity map Weighted density clustering was performed to obtain several gravitational clusters. For each gravitational cluster, with the gravitational value... As a weight, the weighted average of the coordinates of all grid centers within the cluster is calculated to obtain the weighted centroid of the cluster as the gravitational peak point. Gravitational peaks represent the centers of regions in the environment where target relevance is strongest, and are candidate targets for "development" activities.

[0049] In one embodiment, the weighted centroid of each probability cluster is calculated as the probability peak point as follows: ; in, Represents a probability cluster. Represents the grid at the current time. The probability that the corresponding target exists. These are the coordinates of the grid center.

[0050] Specifically, in this embodiment, based on the target probability map Perform weighted density clustering to obtain several probability clusters. For each probability cluster, the probability of the target existing is... As a weight, the weighted average of the coordinates of all grid centers within the cluster is calculated to obtain the weighted centroid of the cluster as the probability peak point. Probability peaks represent the centers of unexplored areas in the environment where the presence of a target is highly probable; they are candidate target points for "exploration" activities. Merging gravitational peaks with probability peaks forms a set of candidate target points that simultaneously considers both development and exploration. .

[0051] In one embodiment, such as Figure 5 As shown, the change in free energy after executing each candidate path is calculated, including: The change in free energy after executing each candidate path is calculated as follows: ; in, This indicates that the drone's execution path will reach the candidate target point. Afterwards, the system's new free energy value, This represents the current free energy value of the drone system in its current state.

[0052] Specifically, in this embodiment, the complete form of the free energy F is: ; in, Indicates a development inclination. Indicates an exploratory tendency. This refers to the temperature parameter.

[0053] To make the implementation details more specific, potential energy and entropy can be written in discrete form: ; in, Indicates the target point The nearby local neighborhood, To prevent constants with a denominator of zero.

[0054] Temperature parameters can be defined as: ; in, These are the initial proportional control parameters. This is a parameter controlling the rate of temperature change. This application provides a parameter sensitivity analysis and presents examples of optimal combinations in experiments.

[0055] Define the change in free energy after executing the candidate path as: ; The optimal target point then satisfies: ; The corresponding optimal search path is This section corresponds to the free energy decision-making and path planning mechanism in this application.

[0056] Furthermore, to clearly define the "obstacle avoidance cost" and "maneuvering cost," it is preferable to include a path cost term: ; Therefore, the joint decision function can be written as: ; In the early stages of the search, due to the large amount of unknown space, the entropy term... Larger, temperature parameters The corresponding value is relatively large, and the system tends to explore points with peak probabilities; as the task progresses, the unknown space decreases, and the entropy term... Decrease, temperature parameter As the gravity decreases accordingly, the system gradually shifts towards selecting gravitational peak points for development, thereby increasing the degree of development of the target-related region.

[0057] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0058] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cognitively enhanced method for visual target search by unmanned aerial vehicles (UAVs) in urban space, characterized in that, The method includes: Acquire the target image and target text description for the search task, as well as the RGB images, depth images and pose information collected in real time during the drone's flight; The target image and the target text description are input into a multimodal large model to generate a set of target-related semantic elements. The spatial correlation between each semantic element in the set of target-related semantic elements and the target is quantified to obtain the gravitational weight of each semantic element. Based on the target-related semantic element set, the current RGB image is semantically segmented to obtain the segmentation result; according to the depth image, the pose information, and the intrinsic and extrinsic parameters of the UAV visual sensor, the pixel-level depth information in the segmentation result is converted into world coordinates, and the world coordinates are discretized into a three-dimensional grid; according to the semantic label that appears most frequently in each grid and the gravity weight corresponding to the semantic label, a target gravity map is constructed and dynamically updated. The current visible grid set is determined based on the current position, orientation, and field of view of the UAV. For each grid in the visible grid set, the probability of the target presence is updated using a decay function based on the distance between the grid center and the current position of the UAV, and a target probability map is constructed and dynamically updated. Projecting three-dimensional world coordinates onto a two-dimensional horizontal plane, determining the state of each two-dimensional grid based on the projection results, including occupied state, idle state, and unknown state, and constructing and dynamically updating the occupied grid map; The target gravity map at the current moment is subjected to weighted clustering to obtain several gravity clusters, and the weighted centroid of each gravity cluster is calculated as the gravity peak point; the target probability map at the current moment is subjected to weighted clustering to obtain several probability clusters, and the weighted centroid of each probability cluster is calculated as the probability peak point; the gravity peak points and the probability peak points are merged to form a set of candidate target points; For each candidate target point in the candidate target point set, a candidate path from the current position of the UAV to the candidate target point is planned based on the current occupied grid map; the change in free energy after executing each candidate path is calculated, the candidate target point with the largest change in free energy is selected as the optimal target point, and the candidate path corresponding to the optimal target point is selected as the optimal search path; The optimal search path is converted into UAV flight control commands to control the UAV to perform target search.

2. The method according to claim 1, characterized in that, The target image and the target text description are input into a multimodal large model to generate a set of target-related semantic elements. The spatial relevance of each semantic element in the set to the target is quantified to obtain the gravitational weight of each semantic element, including: The target image With the target text description Input a large multimodal model and generate a set of target-related semantic elements. in, Generate prompt words for target-related semantic elements; The spatial relevance of each semantic element in the target-related semantic element set to the target is quantified to obtain the gravitational weight of each semantic element. in, For gravity weight quantification prompts, This represents the set of gravitational weights for each semantic element.

3. The method according to claim 1, characterized in that, Based on the target-related semantic element set, semantic segmentation is performed on the current RGB image to obtain the segmentation result, including: Using the set of target-related semantic elements as input prompts, the segmentation model performs semantic segmentation on the current RGB image to obtain the segmentation result. : in, It includes segmentation masks and semantic labels for each semantic entity. For the current RGB image, For the set of semantic elements related to the target.

4. The method according to claim 1, characterized in that, Based on the depth image, the pose information, and the intrinsic and extrinsic parameters of the UAV's visual sensor, the pixel-level depth information in the segmentation result is converted into world coordinates, and the world coordinates are discretized into a 3D mesh, including: Let the intrinsic parameter matrix of the UAV visual sensor be... The extrinsic parameter matrix is For depth images any pixel Corresponding world coordinates for: in, Indicates the horizontal index. Indicates the vertical index. Represents the translation vector. Represents the rotation matrix; Furthermore, world coordinates are mapped to a 3D mesh index: in, For spatial boundaries, It is a three-dimensional voxel resolution.

5. The method according to claim 1, characterized in that, Based on the most frequent semantic label in each grid and the corresponding gravity weight, a target gravity map is constructed and dynamically updated, including: Based on the most frequent semantic label in each grid and the corresponding gravity weight, a target gravity map is constructed and dynamically updated. in, Represents a grid The gravitational value, This represents a lookup or mapping function used to convert semantic tags. Mapped to corresponding gravitational weights For historical memory coefficient, This represents the gravity diagram at the previous moment.

6. The method according to claim 1, characterized in that, Based on the UAV's current position, orientation, and field of view, a set of visible grids is determined for the current moment. For each grid in the set, the probability of target presence is updated using a decay function based on the distance between the grid center and the UAV's current position. A target probability map is constructed and dynamically updated, including: At time step According to the current location of the drone Orientation and field of view Define the current visible mesh set as: in, The coordinates of the grid center; The probability values ​​are updated using a distance decay function on the visible mesh: The target probability map is then updated as follows: in, Indicates the attenuation coefficient. This represents the Euclidean distance between the grid center and the drone's current position. Represents the distance decay function. Represents the grid at the previous time step. The probability of the corresponding target existing.

7. The method according to claim 1, characterized in that, Projecting 3D world coordinates onto a 2D horizontal plane, and determining the state of each 2D grid cell based on the projection results, including: Projecting the 3D world coordinates onto a 2D horizontal plane yields the 2D occupied grid index. in, The horizontal component of world coordinates. For two-dimensional occupancy of raster resolution, Index in the X direction. Index in the Y direction, This represents the minimum boundary value in the X direction for a two-dimensional occupied raster map. This represents the minimum boundary value in the Y direction of a two-dimensional occupied raster map. The state of a two-dimensional raster is defined as follows: Among them, occupied grids are used to represent obstacle areas, empty grids represent passable areas, and unknown grids represent areas that have not yet been observed.

8. The method according to claim 1, characterized in that, Calculate the weighted center of mass of each gravitational cluster as the gravitational peak point. in, Represents a gravitational cluster. Represents a grid The gravitational value, These are the coordinates of the grid center.

9. The method according to claim 1, characterized in that, Calculate the weighted centroid of each probability cluster as the probability peak point. in, Represents a probability cluster, Represents the grid at the current time. The probability of the corresponding target existing. These are the coordinates of the grid center.

Images