Unmanned aerial vehicle body cognition alignment method based on man-machine cooperation

Abstract

The invention relates to an unmanned aerial vehicle body cognition alignment method based on man-machine cooperation. The method comprises the following steps: collecting an entity observation picture, and constructing an entity alignment task by taking the entity observation picture as a navigation graph discrete node; an unmanned aerial vehicle body cognitive alignment model constructed for the task comprises four modules. The problem modeling module converts tasks into POMDP containing elements such as states, actions and observation. The observation module questions a target fact with a self-reflection mechanism based on POMDP, and generates a suspected entity description. And the prediction module adopts zero sample learning, inputs the suspected entity description and the target fact into VLM, and outputs a prediction result. The action module depends on a navigation-questioning mechanism: during navigation, selecting an unexplored entity and getting close to obtain a picture, and controlling the distance through a bounding box proportion; during question asking, explored entity pictures are input into the VLM to generate distinguishing questions, and answers related to target facts are obtained through conversation with human beings. By adopting the method, accurate entity alignment in a complex urban environment can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration. BACKGROUND

[0002] With the rapid development of artificial intelligence and robotics, embodied agents are increasingly applied in urban environments, such as unmanned aerial vehicle inspection, intelligent monitoring, search and rescue tasks, etc. In these applications, the agent needs to accurately identify and locate specific target entities in a complex three-dimensional environment according to natural language instructions from humans, which is referred to as the problem of embodied interactive entity alignment (EIEA).

[0003] The existing technology has the following main problems: due to physical factors such as distance, occlusion, and lighting, the entity information obtained by the visual perception system of the agent has natural uncertainty, especially when observing at a distance, the key features of the entity may not be accurately identified. Current visual language models (VLM) still have hallucination phenomena when dealing with complex visual scenes, which may produce inaccurate entity descriptions, leading to entity alignment failure. Human initial instructions often contain semantic ambiguity, which is not sufficient to support accurate final decision-making, and the agent needs to actively obtain additional information. In an open environment, there are usually multiple suspected entities similar to the target description, which poses a challenge to the discrimination ability of the agent. Existing methods lack effective human-computer collaboration mechanisms and cannot minimize human and movement costs while ensuring accuracy.

[0004] Existing interactive target navigation methods mainly focus on indoor scenes, and usually model the problem as an interactive sequential confirmation problem, i.e., the agent searches and confirms suspected targets one by one. However, these methods have obvious deficiencies when facing open scenes in urban environments: first, urban environments are usually open, and the agent can discover multiple suspected entities at the same time, which need to be compared and distinguished; second, existing methods lack a theoretically guaranteed uncertainty quantification mechanism; finally, there is a lack of efficient human-computer collaboration strategies to balance accuracy and efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration, which can achieve efficient and accurate entity alignment in complex urban environments.

[0006] A method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration, the method comprising: The EIEA-EC dataset is developed based on the EmbodiedCity simulator; the EIEA-EC dataset includes observation pictures of entities collected from different angles and different distances, and the observation pictures are taken as discrete nodes on a navigation map to construct an entity alignment task; An unmanned aerial vehicle embodiment cognitive alignment model is constructed; the unmanned aerial vehicle embodiment cognitive alignment model includes a problem modeling module, an observation module, a prediction module, and an action module; The entity alignment task is input into the unmanned aerial vehicle embodiment cognitive alignment model, and the embodiment interactive entity alignment problem is modeled into a partially observable Markov decision process including a state space, an action space, an observation space, a state transition function, an observation function, a reward function, and a discount factor according to the problem modeling module; Based on the partially observable Markov decision process, the self-reflection mechanism is used according to the observation module to ask questions about each known target fact to generate descriptions of each suspected entity; According to the prediction module, a zero-shot learning method is used to input the descriptions of each suspected entity and the mastered target fact into a visual language model to output a target prediction result; According to the action module, a navigation-question mechanism is used to switch between navigation or questioning actions based on the observation of the entity in the target prediction result; if the navigation action is performed, an entity that has not been explored is selected and navigated to the vicinity to obtain observation pictures, and adaptive distance control is performed based on the proportion of the entity bounding box in the view; if the questioning action is performed, a comparative questioning method is used to select observation pictures of two explored entities and input them into the visual language model to generate a distinguishing question, and an answer about the target fact is obtained through a dialogue with a human user.

[0007] The above-mentioned unmanned aerial vehicle body cognitive alignment method based on human-computer cooperation is based on the EmbodiedCity simulator, and entity observation pictures are collected at different angles and distances and taken as discrete nodes of a navigation map. The pictures at multiple angles and distances cover the characteristics of entities in different physical conditions, providing more comprehensive reference samples for the model and reducing the bias of single observation. At the same time, the navigation map structure of discrete nodes makes the entity exploration path clearer and reduces the complexity of entity positioning in an open environment. The unmanned aerial vehicle body cognitive alignment model is constructed, the problem modeling module converts the task into a POMDP, which matches the partially observable characteristics of the urban environment, and through the definition of elements such as state, action, and observation, the entity alignment is converted into a dynamic decision and probability optimization problem, providing a theoretical framework for uncertain scenarios and making up for the defects of the existing methods lacking a quantitative mechanism for uncertainty. The observation module generates a suspected entity description by actively asking questions about the target fact, which is equivalent to letting the model autonomously fill in the information gap, avoiding the limitations of relying on ambiguous initial instructions. The prediction module uses zero-shot learning combined with VLM, which not only utilizes the cross-modal capabilities of VLM but also reduces the hallucination phenomenon of VLM when processing complex scenes alone by using dual inputs of suspected entity descriptions and target facts, thereby improving the accuracy of entity discrimination. The navigation questioning mechanism of the action module is based on adaptive distance control of the entity bounding box ratio to ensure that the unmanned aerial vehicle can obtain clear observations at an appropriate distance, solving the problem of long-distance feature recognition; the priority exploration strategy of unexplored entities avoids repeated movement and reduces the cost of movement. The comparative questioning method during questioning generates discriminative questions through VLM and interacts with humans to obtain key discriminative information at the lowest human cost, balancing the accuracy and cost in human-computer cooperation. The model uses POMDP as a framework to combine observation, prediction, and action, updates the belief state with each observation, uses the prediction result to guide the next action, and optimizes the judgment with new observations after the action, which is a dynamic adjustment mechanism that perfectly adapts to the scenario of multiple entities existing simultaneously in an open urban environment and the need for dynamic comparison, breaking through the inefficient mode of traditional indoor navigation sequential confirmation and solving the core limitations of existing technologies in complex urban environments, and finally achieving efficient and accurate entity alignment. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A flowchart of a method for aligning the body cognition of an unmanned aerial vehicle based on human-computer cooperation in an embodiment; Figure 2 A framework diagram of an uncertainty-driven collaborative entity alignment in an embodiment; Figure 3 A self-reflection prompt word template diagram in another embodiment; Figure 4 A comparative scoring prompt word template diagram in an embodiment; Figure 5This is a diagram of a template for comparing question prompt words in an embodiment. DETAILED DESCRIPTION

[0009] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0010] In one embodiment, Figure 1 As shown in FIG, a UAV embodied cognitive alignment method based on human-machine collaboration is provided, which includes the following steps: Step 102 : Develop an EIEA-EC dataset based on the EmbodiedCity simulator. The EIEA-EC dataset includes observation images of entities collected from different angles and distances. The observation images are used as discrete nodes on a navigation graph to construct an entity alignment task.

[0011] The construction of the EIEA-EC dataset is mainly divided into two steps, namely entity data collection and task data construction. First, in the entity data collection stage, observation images of entities are collected from different angles and distances. Then, in the task data construction stage, these observation images are used as discrete nodes on the navigation map to construct tasks. In the entity data collection stage, entity models such as humans and vehicles are placed in the simulator and the initial observation angles are randomly generated. and observation distance The angles are generated in a uniform distribution. , the generation of distance also obeys uniform distribution. Considering the performance limitation of VLM, the range of initial distance is set to Collect the initial observation pictures of the agent based on the entity’s position, initial observation angle, and distance ,in and are the height and width of the picture respectively, and the position where the initial observation picture is collected is recorded as In addition, a human annotator collected an image in the simulator as a record of the entity's true state; For entities , and the data collection results can be expressed as ,in Including the initial observation image and its corresponding coordinates, the rest Includes pictures and coordinates corresponding to different observation distances; Based on the collected entity data, we further construct EIEA tasks based on the navigation graph. Each task can be modeled as ,in and a node set and an edge set of a navigation graph respectively, target entity; In constructing the task, 3-5 entities of the same type are first randomly selected from the entity set, and one of them is randomly selected as the target entity. Further, the observation pictures of each entity at different distances in the task are taken as nodes, and the nodes are fully connected to form a navigation graph.

[0012] Step 104, constructing a UAV embodied cognitive alignment model; the UAV embodied cognitive alignment model includes a problem modeling module, an observation module, a prediction module, and an action module.

[0013] The UAV embodied cognitive alignment model is constructed as shown in Figure 2 .

[0014] Step 106, inputting the entity alignment task into the UAV embodied cognitive alignment model, and modeling the embodied interactive entity alignment problem as a partially observable Markov decision process including a state space, an action space, an observation space, a state transition function, an observation function, a reward function, and a discount factor according to the problem modeling module.

[0015] The initialization process includes the following specific steps: first, an initial observation view set is established, which contains initial observation images of each entity, and the resolution of each image is set to 480x640 pixels; then, a target fact set is extracted from the initial instruction provided by the human, and the key feature description is identified through natural language processing technology; next, an empty entity evaluation set is initialized, which is used to store the observation evaluation results of each entity subsequently; at the same time, an exploration identifier set is set, and the exploration state of all entities is initialized to 0 (unexplored); finally, an empty score set is initialized, which is used to store the matching scores of each entity.

[0016] For the embodied interactive entity alignment (EIEA) problem, given a set of entities and a human instruction , the task of the intelligent agent is to identify and confirm the target entity specified by the human from the entity set , that is . Considering the decision uncertainty of the intelligent agent due to the inability to obtain the complete state of the world, the problem can be modeled as a POMDP, which is mainly defined by a seven-tuple .

[0017] State space represents all the real states existing in the environment, among which some states can be directly obtained by the agent, such as the positions of the agent and each suspected entity. However, there are also some states that the agent cannot directly obtain, such as the visual observations and real attributes of the suspected entity and the target entity, and all the knowledge in the human mind about the target entity. Given a real environment state , since is not fully observable, the agent cannot make decisions based on , so the agent maintains a belief state at time . In theory, the belief state is a probability distribution about the real environment state , which represents the probability that the current real state is after experiencing the historical observation sequence and the action sequence . In the EIEA problem, considering that this probability is difficult to obtain directly, this application adopts a structured and approximate belief state representation, which contains the known information required for the agent to make decisions, and can be represented as a triple . Among them represents the current observation view of each entity, which is composed of RGB pictures of entities taken by the agent in the first-person perspective; represents the set of facts about the target entity extracted from the initial instruction and subsequent dialogues; records the evaluation of the current evaluation of each entity, for example, based on , the evaluation of the consistency of each entity with the relevant facts in .

[0018] The action space contains all the actions that the agent can perform, mainly containing three types of actions. At each time step , the agent can choose to perform one of the three types of actions, namely : navigation : the agent moves to the entity , aiming to reduce observation uncertainty by obtaining a closer perspective; questioning : the agent asks a human user a question aimed at eliminating ambiguity or obtaining new facts . Reporting : the agent actively ends the alignment process and reports to the human user that the entity is the target entity.

[0019] The observation space contains the information that the agent can obtain from the environment. At time step , the observation of the agent contains two channels, i.e. When the navigation action is selected, the agent will get a new observation view of the entity , i.e. When the question action is selected, the agent will get an answer from the human, i.e. .

[0020] The state transition function defines the change of the state after the execution of an action. Considering that the real state is static in the EIEA problem, only the change of the belief state is considered, i.e. When the navigation action is selected, the observation view of the entity is updated, and the entity evaluation will be updated accordingly. When the question action is selected, the agent will extract new facts from the question-answering with the human user and update the fact set. Based on the new fact set, the evaluation of all entities will be updated. When the report action is selected, the alignment process ends, and thus there is no need to update the belief state.

[0021] The observation function represents the probability of the agent obtaining the observation in the state , which is the root of the “partial observability”. Considering that the observation probability cannot be explicitly modeled in the EIEA problem, a noisy observation function is introduced to represent the observation process, i.e. . The noise mainly comes from two aspects: Visual noise : when the agent observes the entity, it is limited by the objective conditions such as field of view, occlusion, illumination, distance, etc., which may cause serious degradation of the visual features of the entity in the view, thus leading to the uncertainty of visual observation.

[0022] Language noise : when the agent dialogues with the human user to obtain new target facts, it is limited by the inherent ambiguity and incompleteness of natural language, which may cause the agent to obtain facts that deviate from or are incomplete with respect to the real state, thus constituting another form of noise.

[0023] The reward function provides a theoretical quantitative description of the final goal of the problem, and the discount factor The importance of balancing short-term rewards and long-term rewards. Generally, cumulative discount functions can be explicitly optimized using methods such as reinforcement learning. Considering that the present application does not use reinforcement learning methods, but proposes a zero-shot method, the reward function is used as an analysis tool rather than a learnable target. Specifically, for the EIEA task, the reward function can be constructed from the following three aspects: ① alignment results, when the agent performs the report action, and the alignment is successful, a larger positive reward is obtained, otherwise a negative reward or zero reward is obtained. ② Navigation cost, when the agent performs the navigation action , the movement cost will be generated, the negative number of movement distance or movement time can be used to introduce the punishment for navigation action. ③ Dialogue cost, when the agent performs the question action , the dialogue cost will be generated, the negative number of dialogue times can be used to introduce the punishment for navigation action. To encourage the agent to consider efficiency in decision-making and complete the task with less movement and questioning.

[0024] Step 108, based on the partially observable Markov decision process, the self-reflection mechanism of the observation module is used to generate a description of each suspected entity by separately asking questions about each known target fact.

[0025] The observation module generates a description for each entity by comparing the observation view of the entity with the existing facts, thereby providing a basis for the subsequent prediction stage. In this module, accurately identifying key features and obtaining detailed entity descriptions are key to successfully distinguishing similar entities. Considering the complexity and diversity of entities in real-world environments, previous work usually uses VLM to extract entity information from views. However, the current VLM model still has hallucination phenomenon, which may lead to deviation in identifying and describing key features of entities, thereby causing failure in entity alignment. Especially when the distance between entities is far, the proportion of entity pixels in the view is small, and VLM may not be able to accurately judge the characteristics of the entity. In order to alleviate the inherent hallucination of VLM, it is important to estimate the uncertainty of VLM when responding, that is, to estimate the degree of hesitation of VLM when outputting. When the degree of hesitation is high, the output of VLM may be hallucinated or unreliable.

[0026] Therefore, the output probability distribution of VLM is used to judge the uncertainty of VLM output. Given an autoregressive VLM, for picture representation, for prompt word representation, for historical reasoning representation, in the reasoning stage, VLM will generate a conditional probability distribution defined on the vocabulary at each time step: ; The output is then generated on the conditional probability. However, it is difficult to estimate the uncertainty of VLM using the conditional probability distribution in the above formula, because the vocabulary used by VLM usually has a large scale , which will result in a huge output space of VLM, making it difficult to calculate its uncertainty. To this end, the present application introduces a key self-evaluation link, which generates entity observations from open descriptions to closed judgments on specific facts through a carefully designed question and answer framework. Specifically, for a given set of target facts and entity views , a pre-defined prompt word (such as Figure 3 ) is used to ask each fact in the fact set individually, and only three answers are allowed for VLM output: yes, no, and I don't know. Such a design can force VLM to evaluate its cognitive state, and when there is insufficient visual evidence to support a clear judgment, it can acknowledge its limitations by choosing "I don't know" instead of being forced to produce illusory content.

[0027] Based on the information entropy theory, the normalized entropy is calculated to measure the uncertainty of VLM output; wherein represents the normalized entropy, and at the same time represents the uncertainty of the model output, and the greater the entropy, the higher the uncertainty. In addition, since VLM can only generate three answers, the size of the vocabulary . Further, a threshold is set to divide the uncertainty of VLM output into two levels, namely certain and uncertain: ; Given the facts and views , the self-reflection process is formalized as follows, wherein is the answer of VLM: ; For each entity, the observation module finally outputs a set for evaluation and description, which contains VLM's description of whether it meets all the facts and whether it is certain, wherein each element is composed of a triple , and the process is repeated until all target facts are processed, and finally a description set containing complete evaluation information is returned.

[0028] Step 110, according to the prediction module, a zero-shot learning method is used to input the description of each suspected entity and the mastered target facts into the visual language model, and output the target prediction result.

[0029] For target prediction in EIEA problems, given a list of target facts and observations of suspected entities , together form the input of the prediction model . Each entity is a prediction label, and only one of them is the true label, i.e., the target entity. The output of the prediction model is the probability of each prediction label. Based on this, the non-conformity score is calculated, and the conformal prediction method is used to obtain the prediction set of the target , where is the number of entities in the prediction set. Conformal prediction is a framework for constructing prediction sets with marginal coverage guarantees, which can generate a prediction set that contains the true label of the test sample with a certain probability at a pre-specified confidence level. This guarantee holds under finite samples and is usually independent of the true distribution of the data. Specifically, given

[0030] and are the input and output spaces of the prediction model, respectively, represents an unknown mapping distribution between and . A calibrated set is sampled from with independent and identical distribution. The conformal prediction method uses the calibrated set to calibrate the prediction model, so it also needs to define a non-conformity function to quantify the deviation of the true label of the sample from the “typical” or “conforming” behavior represented in the output of the prediction model. The non-conformity score can usually be calculated using the output probability of the prediction model: ; where represents the probability that the label predicted by the model for the sample is the true label , i.e., the correct prediction probability. The higher this probability, the lower the non-conformity score , meaning that this prediction is not “strange” and is “conforming” to expectations. Using the non-conformity function, a set of non-conformity scores

[0031] is calculated for all data in the calibrated set. These scores constitute the empirical distribution of the non-conformity of the prediction model on known answer data, which can be used to construct the prediction set for new test samples.

[0032] ​When a new test sample is given , whose true label is unknown, we first utilize the prediction model to compute the incompatibility score for each possible candidate label : ; where is an indicator function, which equals to 1 when the input condition is satisfied, otherwise 0. measures the extremity of the test sample when its true label is in the calibration score set. The smaller the value is, the more extreme the test sample is compared to the calibration set, and the less likely the true label is. In other words, the lower the error rate of excluding from the prediction set is. Given an error rate , if , we can consider that in order to guarantee the error rate does not exceed , the label needs to be included in the prediction set : ; In practice, to facilitate the calculation, we can first compute the empirical quantile for the calibration score set : ; where denotes the ceiling function, then the prediction set can be defined as: .

[0033] To obtain the probability of all predicted labels, we can refer to the observation module, limit the output of the LLM to a finite vocabulary that only contains all the predicted labels, the length of the vocabulary is the number of labels , and use the output distribution of the LLM on the vocabulary as the prediction probability. However, in the preliminary experiment, we found that due to the randomness of the LLM output, it is difficult to control the LLM output within the predefined vocabulary, especially when the length of the vocabulary is large. In addition, the LLM always shows excessive confidence before it has obtained enough information, concentrating the prediction probability on a few suspected entities, which makes the prediction accuracy cannot be guaranteed.

[0034] ​​Therefore, in order to ensure the stability and accuracy of the prediction result, the present application adopts a prompt template , which makes the LLM conduct in-depth analysis and comparison on each suspected entity based on the input, and finally provides a normalized score of each suspected entity . The prompt template is shown in Figure 4 , and the higher the score of a certain suspected entity, the higher the possibility that the LLM considers the entity to be the target. Based on the score , the non-compliance score can be defined as . Considering the performance limitation of the LLM, there are at most 5 suspected entities in an alignment task. In order to maintain the unity of the problem, when there are less than 5 suspected entities in the task, virtual entities are supplemented, and their feature descriptions are set to completely not comply with the target fact, thereby excluding the possibility of being the target.

[0035] In a single prediction, the normalized score distribution of each entity reflects the uncertainty of the LLM in prediction. In the early stage of the entity alignment process, the prediction set may contain more entities, because the LLM has greater uncertainty about the prediction result. With the continuous updating of the suspected entity view and the mastered facts, the prediction set at each time step may also change. When the prediction set contains only one suspected entity, i.e. single prediction, it can be considered that the LLM is confident to report the suspected entity as the entity specified by the human user, and at this time the entity alignment process can be actively ended. On the contrary, when there is not only one entity in the prediction set, more movements or dialogues are needed to help the LLM update its prediction result.

[0036] In addition, the human user can also control the process of entity alignment by giving different confidence levels. When it is desired to quickly complete the alignment while accepting a certain loss of accuracy, a lower confidence level can be provided to reduce the size of the prediction set, so as to quickly reach single prediction.

[0037] Step 112, according to the action module, adopt the navigation-question mechanism, based on the observation of the entity in the target prediction result, switch the navigation or question action, if the navigation action is executed, select an entity that has not been explored and navigate to the vicinity to obtain the observation picture, at the same time, based on the proportion of the entity bounding box in the view, adaptive distance control is carried out, if the question action is executed, the comparative question mode is adopted, the observation pictures of two explored entities are selected to input the visual language model to generate a distinguishing question, and the answer about the target fact is obtained through the dialogue with the human user.

[0038] The suspected entities included in the prediction set are all likely to be the target entity. When multiple entities appear in the set, it means that the LLM has uncertainty in prediction. To reduce uncertainty, navigation or questioning actions can be taken. According to the problem modeling, the navigation action can obtain the close-up observation view of the entity, thereby alleviating the noise in visual observation. The questioning action can obtain more facts about the target, thereby alleviating the noise in language observation. As shown in Algorithm 1 in Table 1, when there are at least two explored entities in the prediction set , the two entities with the highest scores are selected to perform the comparative questioning action; otherwise, the unexplored entity with the highest score is selected to perform the adaptive distance navigation action; the comparative questioning method is used to analyze the difference between the two entities and generate a discriminative question.

[0039] Table 1

[0040] The entire process of the method is shown in Algorithm 2 in Table 2. Given a task , the method can align the agent with human cognition and confirm the target entity specified by the human . First, initialize each element in the method (line 1), and then start the loop. Use the conformal prediction method to obtain a prediction set according to the score . If the size of the prediction set is reduced to 1, it means that the uncertainty is completely eliminated, and the method takes the unique entity as the final result and terminates the loop (lines 4-6). However, when there are still multiple candidate entities in the prediction set, the algorithm enters the core action decision phase (line 8), which is described in detail in Algorithm 1. If the questioning action is selected, a discriminative question is generated and new target facts are obtained through dialogue, and then the estimates of each entity are updated (lines 9-13). If the navigation action is selected, the navigation map is used to obtain the close-up observation picture of the currently explored entity, and the estimate of the entity is updated (lines 15-18). After completing any action and obtaining new information, the UCE method immediately re-evaluates the scores of all candidate entities (line 20) to prepare for more accurate prediction in the next round.

[0041] Table 2

[0042] In summary, the specific content includes: Initialization phase: system initialization according to input task , establishing entity observation view set , target fact set , entity evaluation set , exploration identifier and a set of scores The calibration set and confidence level parameters are set to prepare for the subsequent conformal prediction.

[0043] Main loop execution: Enter the main alignment loop and repeat the following steps until the unique target is found: First, perform the conformal prediction step, based on the current entity evaluations and fact set, generate a prediction set containing multiple candidate entities using conformal prediction theory At the same time, update the matching scores of each entity The size of the prediction set reflects the current level of uncertainty.

[0044] Then make a termination judgment, check if the size of the prediction set is equal to 1. If the prediction set contains only one entity, it means that the system has enough confidence to determine the target, at this time report the entity to the human user as the target entity, and end the entire alignment process.

[0045] If the prediction set contains multiple entities, perform the action decision step, determine the type of next action and target entity according to the navigation-question mechanism. The decision process considers the number of explored entities in the prediction set and the scores of each entity.

[0046] In the question branch, select the two highest scored explored entities, generate a discriminative question using the contrast question method, obtain new target facts through human-computer dialogue, update the fact set And re-evaluate all entities based on new facts.

[0047] In the navigation branch, select the highest scored unexplored entity as the navigation target, use the adaptive distance control method to move to the best observation position, obtain high-quality entity observation images, update the corresponding observation view and evaluation results, and mark the entity as explored.

[0048] Finally, perform the score update step, recalculate the matching scores of each entity based on the updated entity evaluations and fact set, provide accurate input data for the next round of prediction.

[0049] The entire algorithm loop executes the above steps, through continuous observation, prediction, action and update, gradually narrows the size of the prediction set, and finally realizes accurate entity alignment.

[0050] The above-mentioned unmanned aerial vehicle body cognitive alignment method based on human-computer cooperation is based on the EmbodiedCity simulator, and entity observation pictures are collected at different angles and distances, and are taken as discrete nodes of the navigation map. The pictures of multiple perspectives and multiple distances cover the characteristics of entities in different physical conditions, providing more comprehensive reference samples for the model and reducing the bias of single observation. At the same time, the navigation map structure of the discrete nodes makes the entity exploration path clearer and reduces the complexity of the entity positioning in the open environment. The unmanned aerial vehicle body cognitive alignment model is constructed, the problem modeling module converts the task into POMDP, which is consistent with the partially observable characteristics of the urban environment, and the entity alignment is converted into a dynamic decision and probability optimization problem through the definition of state, action, observation and other elements, providing a theoretical framework for uncertain scenarios and making up for the defects of the existing methods lacking of uncertainty quantization mechanism. The observation module generates a suspected entity description by actively asking questions about the target fact, which is equivalent to letting the model autonomously supplement the information gap, avoiding the limitations of relying on ambiguous initial instructions. The prediction module uses zero-shot learning combined with VLM, which not only utilizes the cross-modal ability of VLM, but also reduces the hallucination phenomenon of VLM when processing complex scenes alone by using the dual input of suspected entity description and target fact, thereby improving the entity discrimination accuracy. The navigation questioning mechanism of the action module is based on the adaptive distance control of the entity bounding box ratio, which ensures that the unmanned aerial vehicle can obtain clear observation at a suitable distance and solve the problem of long-distance feature recognition; the priority exploration strategy of unexplored entities avoids repeated movement and reduces the movement cost. The comparative questioning method during questioning generates a distinguishing question through VLM and interacts with humans, obtaining key discriminant information with the lowest human cost and balancing the accuracy and cost in human-computer cooperation. The model uses POMDP as the framework to combine observation, prediction and action, updates the belief state with each observation, uses the prediction result to guide the next action, and optimizes the judgment with the new observation after the action, which is a dynamic adjustment mechanism that perfectly adapts to the scenario of multiple entities existing simultaneously in the open urban environment and the need for dynamic comparison, breaks through the low-efficiency mode of traditional indoor navigation sequential confirmation, solves the core limitations of existing technologies in complex urban environments, and finally realizes efficient and accurate entity alignment.

[0051] In one embodiment, the state space includes known information required for agent decision-making, represented as a triple , where represents the current observation view of each entity, which is composed of RGB pictures of the entity taken by the agent in the first-person perspective; represents a set of facts about the target entity extracted from the initial instruction and subsequent dialogue; records the evaluation of the current observation of each entity: the action space includes three types of actions, and at each time step , the agent can choose to perform one of the three types of actions, i.e. , Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask Move, which aims to reduce observation uncertainty by obtaining a closer view; Ask

[0052] In one embodiment, the EIEA-EC dataset is developed based on the EmbodiedCity simulator, including: In the entity data collection phase, the entity model is placed in the simulator, and the initial observation angle and observation distance are randomly generated; According to the entity position, the initial observation angle and the observation distance, the initial observation picture of the entity by the agent is collected , wherein and are the height and width of the image, respectively, and the position of the initial observation image is denoted as ; For an entity , the data collection results can be represented as , where includes the initial observation image and its corresponding coordinates, and the rest includes images and coordinates corresponding to different observation distances. Based on the collected entity data, the EIEA task based on the navigation graph is constructed, and each task is modeled as , where and are the node set and edge set of the navigation graph, respectively, is the target entity. In constructing the task, first, 3-5 entities of the same type are randomly selected from the entity set, and one of them is randomly selected as the target entity. The observation images of each entity at different distances are used as nodes, and the nodes are fully connected to form the entity alignment task.

[0053] In specific embodiments, the detailed process of entity data collection includes the following steps: Parameter initialization: randomly generate initial observation parameters for each entity, including observation angle (conforming to uniform distribution from 0 to 360 degrees) and observation distance (conforming to uniform distribution from 5 to 30 meters). This randomization design ensures the diversity and generalization ability of the data set.

[0054] Multi-distance observation collection: for each entity, observation image collection is performed at six different distance positions, including the initial distance, the adaptive distance, and the fixed distances of 2 meters, 5 meters, 10 meters, and 15 meters. For initial distance observation, the observation position of the agent is calculated according to the randomly generated angle and distance parameters to ensure that a complete view of the entity can be obtained. For adaptive distance observation, the optimal observation position that satisfies the bounding box constraint condition is found by iterative adjustment using the adaptive distance control method described above. For fixed distance observation, the initial angle is kept unchanged, and only the observation distance is adjusted to the specified value.

[0055] Image acquisition and storage: at each determined observation position, the RGB observation image of the entity is obtained using the image acquisition function of the simulator, and the image resolution is set to 480x640 pixels. At the same time, the corresponding observation position coordinates, including three-dimensional space coordinates and observation angle information, are recorded.

[0056] Ground Truth Collection: Real state pictures of entities are collected by human annotators in a simulator as ground truth references, which are used to verify observation quality and algorithm performance.

[0057] Data Structure Organization: All collected data are organized into a structured format, including dictionary structures for real state pictures and each distance observation data, each observation data containing image and position information, facilitating subsequent task construction and algorithm training.

[0058] The detailed process of task data construction includes the following steps: Entity Selection Phase: Randomly select 3 to 5 entities from the same type of entity set as suspected entities in the task, and the randomization of the number of entities ensures the diversity of task difficulty. Then randomly select one of the selected entities as the target entity, and the other entities as interference entities.

[0059] Navigation Graph Node Construction: Create multiple navigation graph nodes for each selected entity, each node corresponding to the observation data of the entity at different observation distances. Specifically, each node contains four key information: entity identification, observation distance identification, corresponding observation image, and observation position coordinates. Such design allows the agent to observe the same entity at different distances.

[0060] Navigation Graph Edge Construction: The edges of the navigation graph are constructed in a fully connected manner, i.e. there is a reachable path between any two nodes. This design simplifies the path planning problem, allowing the agent to move directly between any two observation positions and focus on the core challenge of entity alignment.

[0061] Task Structure Organization: Organize the constructed navigation graph node set, edge set, target entity identification and all suspected entity list into a complete task data structure. This structure provides a complete task environment description for the UCE algorithm, including accessible observation positions, entity information and target settings.

[0062] Task Verification and Optimization: Check whether the constructed task meets the basic requirements, such as ensuring that the target entity and other entities have distinguishable feature differences, verifying the connectivity of the navigation graph, and ensuring the solvability and reasonableness of the task.

[0063] In one embodiment, the observation module uses a self-reflection mechanism to ask questions about each known target fact to generate descriptions of each suspected entity, including: For a given set of target facts And entity views Using predefined prompt words Each fact in the target fact set is asked individually, limiting the visual language model to output only yes, no, and I don't know, forcing the visual language model to evaluate the cognitive state of each fact and generate a description of each suspected entity.

[0064] In one embodiment, the prediction module employs a zero-shot learning method, inputs the description of each suspected entity and the mastered target facts into the visual language model, and outputs the target prediction result, including: Based on the conformal prediction theory, a target prediction method is constructed, and a calibration set is used to calibrate the model, define a non-conformity function to quantify the true label of the sample , and the deviation of the typical or conforming behavior represented in the output of the prediction model, wherein, represents a sample set, represents a label set, represents a real number set; Calculate the empirical quantile of the calibration set score set; construct test samples from the description of each suspected entity and the mastered target facts, use the prediction model to calculate the non-conformity score of each possible candidate label Based on the empirical quantile and the non-conformity score, guide the generation of the prediction set, i.e. the target prediction result The target prediction result contains the true label with at least confidence level, satisfying .

[0065] In one embodiment, for target prediction in embodied interactive entity alignment problems, given a list of target facts and observation descriptions of suspected entities, which together constitute the input of the prediction model, each entity is treated as a prediction label, and only one entity is the true label, i.e. the target entity, and the output of the prediction model is the probability of all prediction labels.

[0066] In one embodiment, the empirical quantile of the calibration set score set is calculated as: ; where, represents the empirical quantile, represents the non-conformity score of the th sample, represents the size of the calibration set, represents the pre-set error rate.

[0067] In one embodiment, the action module adopts a navigation-questioning mechanism to switch between navigation and questioning actions based on the observation of entities in the target prediction results, including: Given a set of predictions and their exploration identifiers and ratings When there are at least two explored entities in the prediction set, the two entities with the highest scores are selected to perform the question action. ; When there are many unexplored entities in the prediction set, select the unexplored entity with the highest score to perform navigation actions ; The observation distance is adjusted through the adaptive distance control method to make the entity bounding box meet the constraints, and the contrast questioning method is used to generate discriminative questions, and the target facts are obtained by comparing the differences between two entities in multiple aspects; multiple aspects include color, style and structure.

[0068] In a specific embodiment, the number of explored entities in the prediction set is counted. By checking the exploration identifier corresponding to each entity in the prediction set, the total number of entities with an identifier value of 1 (explored) is calculated. The action type is then determined based on the number of explored entities: if there are at least two explored entities in the prediction set, the question action mode is triggered; otherwise, the navigation action mode is triggered.

[0069] In the question-action mode, the two entities with the highest scores are selected from the explored entities in the prediction set as the comparison targets. These two entities will be used to generate discriminative questions. The selection process is based on the matching scores of the entities, and the entities most likely to be the target are prioritized for comparison analysis.

[0070] In the navigation action mode, the entity with the highest score is selected from the unexplored entities in the prediction set as the navigation target, and the agent moves to the vicinity of the entity for close observation. This strategy ensures that the most promising candidate entities are explored first, improving alignment efficiency.

[0071] Finally, the determined action type, question or navigation and the corresponding target entity are returned to provide clear guidance for the subsequent specific action execution.

[0072] In one embodiment, a given agent has Observation view First, the GroundingDino model is used to detect the bounding box of the entity , the coordinates of the bounding box are Then, the relationship between the bounding box and the entire observation view is controlled through discrete actions. To improve the observation quality, the target should be located in the center of the view and occupy a moderate proportion [1, 2]. Therefore, the observation distance is adjusted to meet the following constraints to obtain a high-quality observation view. The constraints are: and ; in, and are the upper left corner coordinates and lower right corner coordinates of the bounding box, respectively. Controls the minimum proportion of entities in the view. Controls the entity's border position in the view. and are the height and width of the observed image respectively.

[0073] In order to obtain the required observation view, the agent starts from a predetermined observation angle and initial position, and then selects an action from the action set {forward, backward, look up, look down} according to the state of the entity bounding box until the bounding box satisfies the constraints. The step size of forward and backward is set to 0.2m, and the step size of looking up and looking down is set to 5°. In the task execution phase, all suspected entities are first The status of is marked as "unexplored", and the corresponding identifier is set to When the agent chooses to navigate to a suspected entity When observing more closely, the agent goes directly from the current node to the suspected entity The observation node of the adaptive distance is found and the corresponding observation view is obtained. The entity is marked as "explored" and its identifier is updated. .

[0074] In one embodiment, the comparative questioning method includes a difference recognition stage, a question construction stage, a duplicate filtering stage, a human-computer interaction stage, and a fact update stage; the difference recognition stage is to compare the observation images of two suspected entities and Input visual language model and use difference analysis prompt words Output entity's specific difference description in multi-dimensional features; multi-dimensional features include color, style, structure, posture and gender; the problem construction stage is to describe the difference and the set of known facts Input a large language model and use questions to generate prompt words Generate binary choice questions that can distinguish between two entities ; The deduplication filtering stage ensures that the generated questions do not overlap with the fact set There are already repeated features in the system, so as to avoid repeated questions about the same feature; the human-computer interaction stage is to use natural language dialogue Obtain answers from human users and extract new target facts from them ; The fact update phase is execution Update the fact collection and update the evaluation of all entities accordingly .likeFigure 5 Fig. 4B shows a template graph of a contrast question prompt word.

[0075] It should be understood that, although Figure 1 The steps in the flowcharts of the above embodiments are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 At least part of the steps in the above embodiments can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0076] The technical features of the above embodiments can be combined in any manner. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present disclosure.

[0077] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A UAV embodied cognitive alignment method based on human-machine collaboration, characterized by: The method comprises: The EIEA-EC dataset is developed based on the EmbodiedCity simulator. The EIEA-EC dataset includes observation images of entities collected at different angles and distances. The observation images are used as discrete nodes on the navigation graph to construct the entity alignment task. Constructing a UAV embodied cognitive alignment model; the UAV embodied cognitive alignment model includes a problem modeling module, an observation module, a prediction module, and an action module; Inputting the entity alignment task into the UAV embodied cognitive alignment model, the embodied interactive entity alignment problem is modeled as a partially observable Markov decision process including a state space, an action space, an observation space, a state transfer function, an observation function, a reward function, and a discount factor according to a problem modeling module; Based on the partially observable Markov decision process, the observation module uses a self-reflection mechanism to ask questions about each known target fact and generate a description of each suspected entity; The prediction module uses a zero-shot learning method to input the description of each suspected entity and the grasped target facts into the visual language model and output a target prediction result; According to the action module, a navigation-questioning mechanism is adopted. Based on the observation of the entity in the target prediction result, the navigation or questioning action is switched. If the navigation action is performed, an unexplored entity is selected and navigated to the nearby area to obtain an observation image. At the same time, adaptive distance control is performed based on the proportion of the entity bounding box in the view. If the questioning action is performed, a comparative questioning method is adopted. The observation images of two explored entities are selected and input into the visual language model to generate discriminative questions, and answers about the target facts are obtained through dialogue with human users.

2. The method according to claim 1, characterized in that The state space includes the known information required for the agent to make decisions, represented as a triple ,in Represents the current observation view of each entity, consisting of an RGB image of the entity taken from the first-person perspective of the agent; represents the set of facts about the target entity extracted from the initial command and subsequent dialogue; Records the current evaluation of each entity: The action space includes three types of actions, at each time step , the agent can choose to perform one of three types of actions, namely , Represents the agent to the entity Movement, aimed at reducing observation uncertainty by obtaining a closer perspective; asking questions Indicates that the agent asks a question to the human user that aims to disambiguate or obtain new facts ;Report Indicates that the agent proactively ends the alignment process and reports the entity to the human user is the target entity; the observation space includes the time step , the observation of the intelligent agent includes two channels: vision and language, namely , when selecting navigation action When the agent obtains the entity New viewing view ,Right now ; When you select the question action When the agent receives a response from a human ,Right now ; The state transfer function is defined as , when selecting navigation action When the entity The observation view is updated and its entity evaluation is updated accordingly. When the agent extracts new facts from the questions and answers with human users And update the fact set, update the evaluation results of all entities based on the new fact set, when the report action is selected When , the alignment process ends and there is no need to update the belief state; the reward function includes alignment result reward, navigation cost reward and dialogue cost reward; The alignment result reward is when the agent performs the report Action, and When , it indicates that the alignment is successful and a positive reward is obtained, otherwise a negative reward or zero reward is obtained; the navigation cost reward is when the agent performs the navigation action When the agent performs the questioning action, the negative number of the moving distance or moving time is used to introduce the penalty for the navigation action; the dialogue cost reward is When , the negative number of dialogues is used to introduce a penalty for navigation actions.

3. The method according to claim 1, characterized in that The EIEA-EC dataset is developed based on the EmbodiedCity simulator, including: In the entity data collection phase, the entity model is placed in the simulator and the initial observation angle is randomly generated. and observation distance ; Collect the agent's initial observation pictures of the entity based on the entity's position, initial observation angle, and observation distance ,in and are the height and width of the picture respectively, and the position where the initial observation picture is collected is recorded as ; For entities , and the data collection results can be expressed as ,in Including the initial observation image and its corresponding coordinates, the rest Includes pictures and coordinates corresponding to different observation distances; Based on the collected entity data, EIEA tasks based on the navigation graph are constructed. Each task is modeled as ,in and are the node set and edge set of the navigation graph respectively, is the target entity; When constructing a task, we first randomly select 3-5 entities of the same type from the entity set, and randomly select one of them as the target entity. We use the observation images of different distances of each entity in the task as nodes, and fully connect the nodes to form an entity alignment task.

4. The method according to claim 1, wherein The observation module uses a self-reflection mechanism to ask questions about each known target fact and generate descriptions of each suspected entity, including: For a given set of target facts and entity views , using predefined prompt words Ask each fact in the target fact set a separate question, restricting the visual language model to output only three answers: yes, no, and don't know. This forces the visual language model to evaluate the cognitive state of each fact and generate a description of each suspected entity.

5. The method according to claim 1, wherein The prediction module uses a zero-shot learning method to input the description of each suspected entity and the grasped target facts into the visual language model, and outputs the target prediction result, including: Based on the conformal prediction theory, a target prediction method is constructed by using the calibration set Perform model calibration and define non-compliance function , used to quantify samples The true label The degree of deviation from typical or consistent behavior represented in the output of the predictive model, where represents a sample set, Represents a tag set, represents the set of real numbers; Calculate the empirical quantile of the calibration set score set; construct test samples based on the description of each suspected entity and the target facts , using the prediction model for each possible candidate label Calculating non-conformance scores Based on the empirical quantile and the non-conformity score, a prediction set is generated, i.e., the target prediction result. , the target prediction result is at least The confidence level includes the true label, satisfying .

6. The method according to claim 5, characterized in that The target prediction method comprises: For target prediction in the embodied interactive entity alignment problem, given a list of target facts and Observational description of suspected entities , which together constitute the prediction model input , each entity is regarded as a predicted label, where there is only one entity is the true label, i.e. the target entity, and the output of the prediction model is the probability of all predicted labels.

7. The method according to claim 5, characterized in that The empirical quantile of the set of calibration set scores is calculated as: in, represents the empirical quantile, Indicates the The non-conformity score of samples, represents the calibration set size, Indicates the preset error rate.

8. The method according to claim 1, characterized in that The action module adopts a navigation-questioning mechanism, and switches navigation or questioning actions based on the observation of entities in the target prediction results, including: Given a set of predictions and their exploration identifiers and ratings When there are at least two explored entities in the prediction set, the two entities with the highest scores are selected to perform the question action. ; When there are many unexplored entities in the prediction set, select the unexplored entity with the highest score to perform navigation actions ; The observation distance is adjusted through the adaptive distance control method so that the entity bounding box meets the constraints, and the contrast questioning method is used to generate discriminative questions, and the target facts are obtained by comparing the differences between two entities in multiple aspects; the multiple aspects include color, style and structure.

9. The method according to claim 8, characterized in that The constraints are: in, and are the upper left corner coordinates and lower right corner coordinates of the bounding box, respectively. Controls the minimum proportion of entities in the view. Controls the entity's border position in the view. and are the height and width of the observed image respectively.

10. The method according to claim 8, characterized in that The comparative questioning method includes a difference recognition stage, a question construction stage, a duplicate removal and filtering stage, a human-computer interaction stage, and a fact update stage; the difference recognition stage is to compare the observation images of two suspected entities and Input visual language model and use difference analysis prompt words Output entity's specific difference description in multi-dimensional features; the multi-dimensional features include color, style, structure, posture and gender; the problem construction stage is to describe the difference and the set of known facts Input a large language model and use questions to generate prompt words Generate binary choice questions that can distinguish between two entities The deduplication filtering stage is to ensure that the generated questions do not conflict with the fact set. There are already repeated features in the system, so as to avoid repeated questions about the same feature; the human-computer interaction stage is to conduct dialogue through natural language Obtain answers from human users and extract new target facts from them ; The fact update phase is executed Update the fact collection and update the evaluation of all entities accordingly .

Images