Method for unmanned machine body cognitive alignment based on human-machine cooperation

Abstract

The application relates to a kind of unmanned equipment body cognitive alignment methods based on human-computer cooperation.The method comprises: collecting entity observation picture, which is used as the discrete node of navigation map to construct entity alignment task.The unmanned equipment body cognitive alignment model constructed for the task contains four modules.Problem modeling module converts the task into POMDP containing state, action, observation and other elements.Observation module based on POMDP, with self-reflection mechanism to question the target fact, generates suspected entity description.Prediction module uses zero sample learning, inputs suspected entity description and target fact into VLM, and outputs prediction result.Action module relies on navigation-question mechanism: when navigating, select unexplored entity and approach to obtain pictures, and control distance by boundary box ratio;When asking questions, input the picture of the explored entity into VLM to generate a distinctive question, and obtain the relevant answer of the target fact through human dialogue.The method can accurately align entities in complex urban environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration. BACKGROUND

[0002] With the rapid development of artificial intelligence and robotics, embodied agents are increasingly applied in urban environments, such as unmanned aerial vehicle inspection, intelligent monitoring, search and rescue tasks, etc. In these applications, the agent needs to accurately identify and locate specific target entities in a complex three-dimensional environment according to natural language instructions from humans, which is referred to as the problem of embodied interactive entity alignment (EIEA).

[0003] The existing technology has the following main problems: due to physical factors such as distance, occlusion, and lighting, the entity information obtained by the visual perception system of the agent has natural uncertainty, especially when observing at a distance, the key features of the entity may not be accurately identified. Current visual language models (VLM) still have hallucination phenomena when dealing with complex visual scenes, which may produce inaccurate entity descriptions, leading to entity alignment failure. Human initial instructions often contain semantic ambiguity, which is not sufficient to support accurate final decision-making, and the agent needs to actively obtain additional information. In an open environment, there are usually multiple suspected entities similar to the target description, which poses a challenge to the discrimination ability of the agent. Existing methods lack effective human-computer collaboration mechanisms and cannot minimize human and movement costs while ensuring accuracy.

[0004] Existing interactive target navigation methods mainly focus on indoor scenes, and usually model the problem as an interactive sequential confirmation problem, i.e., the agent searches and confirms suspected targets one by one. However, these methods have obvious deficiencies when facing open scenes in urban environments: first, urban environments are usually open, and the agent can discover multiple suspected entities at the same time, which need to be compared and distinguished; second, existing methods lack a theoretically guaranteed uncertainty quantification mechanism; finally, there is a lack of efficient human-computer collaboration strategies to balance accuracy and efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration, which can achieve efficient and accurate entity alignment in complex urban environments.

[0006] A method for embodied cognitive alignment of unmanned aerial vehicles based on human-computer collaboration, the method comprising:

[0007] The EIEA-EC dataset is developed based on the Embodied City simulator; the EIEA-EC dataset includes observation pictures of entities collected from different angles and different distances, and the observation pictures are taken as discrete nodes on a navigation map to construct an entity alignment task;

[0008] An unmanned aerial vehicle embodiment cognitive alignment model is constructed; the unmanned aerial vehicle embodiment cognitive alignment model includes a problem modeling module, an observation module, a prediction module, and an action module;

[0009] The entity alignment task is input into the unmanned aerial vehicle embodiment cognitive alignment model, and the embodiment interactive entity alignment problem is modeled into a partially observable Markov decision process including a state space, an action space, an observation space, a state transition function, an observation function, a reward function, and a discount factor according to the problem modeling module;

[0010] Based on the partially observable Markov decision process, the self-reflection mechanism is used to ask questions about each known target fact to generate descriptions of each suspected entity according to the observation module;

[0011] According to the prediction module, the descriptions of each suspected entity and the mastered target fact are input into the visual language model to output a target prediction result using a zero-shot learning method;

[0012] According to the action module, a navigation-question mechanism is used to switch between navigation or questioning actions based on the observation of the entity in the target prediction result; if the navigation action is performed, an entity that has not been explored is selected and navigated to the vicinity to obtain observation pictures, and adaptive distance control is performed based on the proportion of the entity bounding box in the view; if the questioning action is performed, a comparative questioning method is used to select observation pictures of two explored entities and input them into the visual language model to generate a distinguishing question, and answers about the target fact are obtained through a dialogue with a human user.

[0013] The above-mentioned unmanned aerial vehicle body cognitive alignment method based on human-computer cooperation is based on the EmbodiedCity simulator, and entity observation pictures are collected at different angles and distances, and are taken as discrete nodes of a navigation map. The pictures of multiple perspectives and multiple distances cover the characteristics of entities in different physical conditions, providing more comprehensive reference samples for the model and reducing the bias of single observation. At the same time, the navigation map structure of the discrete nodes makes the entity exploration path clearer and reduces the complexity of entity positioning in an open environment. The unmanned aerial vehicle body cognitive alignment model is constructed, the problem modeling module converts the task into POMDP, which is consistent with the partially observable characteristics of the urban environment, and through the definition of elements such as state, action and observation, the entity alignment is converted into a dynamic decision and probability optimization problem, which provides a theoretical framework for uncertain scenarios and makes up for the defects of the existing methods lacking of uncertainty quantization mechanism. The observation module generates a suspected entity description by actively asking questions about the target fact, which is equivalent to letting the model autonomously supplement the information gap, avoiding the limitations of relying on ambiguous initial instructions. The prediction module uses zero-shot learning combined with VLM, which not only utilizes the cross-modal ability of VLM, but also reduces the hallucination phenomenon of VLM when processing complex scenes alone by using the dual input of suspected entity description and target fact, thereby improving the entity discrimination accuracy. The navigation questioning mechanism of the action module is based on the adaptive distance control of the entity bounding box ratio, which ensures that the unmanned aerial vehicle can obtain clear observation at a suitable distance and solve the problem of long-distance feature recognition; the priority exploration strategy of unexplored entities is used to avoid repeated movement and reduce the movement cost. The comparative questioning method during questioning generates a distinguishing question through VLM and interacts with humans, obtaining key discriminant information with the lowest human cost, and balancing the accuracy and cost in human-computer cooperation. The model uses POMDP as a framework to combine observation, prediction and action, updates the belief state with each observation, uses the prediction result to guide the next action, and optimizes the judgment with new observation after the action, which is a dynamic adjustment mechanism that perfectly adapts to the scenario of multiple entities existing simultaneously and needing dynamic comparison in an open urban environment, breaks through the low-efficiency mode of traditional indoor navigation sequential confirmation, solves the core limitations of existing technologies in complex urban environments, and finally realizes efficient and accurate entity alignment. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of a human-computer cooperation-based unmanned aerial vehicle body cognitive alignment method in an embodiment;

[0015] Figure 2 An uncertainty-driven collaborative entity alignment framework diagram in an embodiment;

[0016] Figure 3 A self-reflection prompt word template diagram in another embodiment;

[0017] Figure 4 A comparative scoring prompt word template diagram in an embodiment;

[0018] Figure 5 Fig. 1 shows a comparison of question prompt word templates for an embodiment. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0020] In one embodiment, as shown in Fig. 1, a human-robot collaborative unmanned robot body cognitive alignment method is provided, comprising the following steps: Figure 1

[0021] Step 102, developing an EIEA-EC dataset based on an EmbodiedCity simulator; the EIEA-EC dataset includes observation pictures of entities collected at different angles and different distances, and the observation pictures are used as discrete nodes on a navigation map to construct an entity alignment task.

[0022] The construction of the EIEA-EC dataset mainly includes two steps, namely entity data collection and task data construction. First, in the entity data collection stage, observation pictures of entities are collected at different angles and different distances, and then in the task data construction stage, these observation pictures are used as discrete nodes on a navigation map to construct a task. In the entity data collection stage, entity models such as humans and vehicles are placed in the simulator, and initial observation angles and observation distances are randomly generated. The generation of angles obeys a uniform distribution , and the generation of distances also obeys a uniform distribution. Considering the performance limitation of VLM, the range of initial distances is set to . According to the entity position, initial observation angle and distance, the initial observation picture of the agent to the entity is collected , where and are the height and width of the picture respectively, and the position of collecting the initial observation picture is recorded as . In addition, a picture is collected by a human annotator in the simulator as a record of the real state of the entity;

[0023] For an entity , the data collection result can be represented as , where includes the initial observation picture and its corresponding coordinates, and the remaining includes pictures and coordinates corresponding to different observation distances;

[0024] ​On the basis of the collected entity data, an EIEA task based on a navigation graph is further constructed. Each task can be modeled as wherein and are a node set and an edge set of the navigation graph respectively, is a target entity;

[0025] In constructing the task, 3-5 entities of the same type are first randomly selected from the entity set, and one of them is randomly selected as the target entity. Further, observation pictures of different distances of each entity in the task are taken as nodes, and the nodes are fully connected, thereby forming the navigation graph.

[0026] Step 104, constructing a UAV embodied cognitive alignment model; the UAV embodied cognitive alignment model includes a problem modeling module, an observation module, a prediction module and an action module.

[0027] The UAV embodied cognitive alignment model is constructed as shown in Figure 2 .

[0028] Step 106, inputting the entity alignment task into the UAV embodied cognitive alignment model, and modeling the embodied interactive entity alignment problem as a partially observable Markov decision process including a state space, an action space, an observation space, a state transition function, an observation function, a reward function and a discount factor according to the problem modeling module.

[0029] The initialization process includes the following specific steps: first, an initial observation view set containing initial observation images of each entity is established, and the resolution of each image is set to 480x640 pixels; then, a target fact set is extracted from the initial instruction provided by the human, and the key feature description is identified through natural language processing technology; next, an empty entity evaluation set is initialized, which is used to store the observation evaluation results of each entity in the subsequent process; at the same time, a set of exploration identifiers is set, and the exploration state of all entities is initialized to 0 (unexplored); finally, an empty score set is initialized, which is used to store the matching scores of each entity.

[0030] For the embodied interactive entity alignment (EIEA) problem, given a set of entities and a human instruction , the task of the intelligent agent is to identify and confirm the target entity specified by the human from the entity set , that is Considering the decision-making uncertainty of an agent due to its inability to know the complete state of the world, this problem can be modeled as a POMDP, mainly defined by a seven-tuple. .

[0031] state space This represents all existing real states in the environment. Some states can be directly obtained by the agent, such as the agent's position and the positions of various suspected entities. However, some states cannot be directly obtained by the agent, such as the visual observations and real attributes of suspected and target entities, as well as all human knowledge about the target entity. Given a real environmental state... ,because Incompletely observable, intelligent agents cannot rely on... To make decisions, the agent must act at any given moment. Maintaining a state of belief In theory, belief states are about the state of the real environment. A probability distribution It indicates that after experiencing the historical observation sequence and action sequence After that, the current true state is The probability of this belief state is difficult to obtain directly in the EIEA problem. Therefore, this application employs a structured, approximate belief state representation, which contains the known information required for the agent's decision-making and can be represented as a triple. .in This represents the current view of each entity, consisting of RGB images of the entities taken by the agent from a first-person perspective; This represents the set of facts about the target entity extracted from the initial instructions and subsequent dialogue; It records the current evaluation of each entity, for example, based on For each entity and An assessment of the degree of conformity with the relevant facts.

[0032] Action space It includes all actions that the agent can perform, mainly comprising three types of actions, at each time step. The intelligent agent can choose to perform one of three types of actions, namely :navigation Agent to Entity Moving around aims to reduce observational uncertainty by gaining a closer perspective; asking questions The intelligent agent poses a question to the human user aimed at dispelling ambiguity or acquiring new facts. .Report The intelligent agent proactively terminates the alignment process and reports the entity to the human user. is the target entity.

[0033] Observation Space contains the information that the agent can obtain from the environment. At time step , the agent's observation contains both visual and linguistic channels, i.e. When a navigation action is selected, the agent will obtain a new observation view of the entity , i.e. When a question action is selected, the agent will obtain an answer from the human, i.e. .

[0034] State Transition Function defines the change of state after performing an action. Considering the EIEA problem, the real state is static, thus only the change of belief state is considered, i.e. When a navigation action is selected, the observation view of the entity is updated, and the entity evaluation will be updated accordingly. When a question action is selected, the agent will extract new facts from the question-answering with the human user and update the fact set. Based on the new fact set, the evaluation of all entities will be updated. When a report action is selected, the alignment process ends, thus there is no need to update the belief state.

[0035] Observation Function denotes the probability of the agent obtaining an observation in state , which is the root of "partial observability". Considering that the observation probability cannot be explicitly modeled in the EIEA problem, a noisy observation function is introduced to represent the observation process, i.e. . The noise mainly comes from two aspects:

[0036] Visual Noise : When the agent observes an entity, it is limited by objective conditions such as field of view, occlusion, illumination, distance, etc., which may cause serious degradation of the visual features of the entity in the view, thus leading to the uncertainty of visual observation.

[0037] Linguistic Noise : When the agent dialogues with the human user to obtain new target facts, it is limited by the inherent ambiguity and incompleteness of natural language, which may cause the agent to obtain facts that deviate from or are incomplete with respect to the real state, thus constituting another form of noise.

[0038] reward function provides a theoretical quantitative description of the final goal of the problem, the discount factor balances the importance of short-term rewards and long-term rewards. Generally, cumulative discount functions can be explicitly optimized using methods such as reinforcement learning. Considering that the present application does not use a reinforcement learning method, but proposes a zero-shot method, the reward function is used as an analysis tool rather than a learnable target. Specifically, for the EIEA task, the reward function can be constructed from the following three aspects: ① alignment result, when the agent performs the report action, and if the alignment is successful, a larger positive reward is obtained, otherwise a negative reward or zero reward is obtained. ② navigation cost, when the agent performs the navigation action , a movement cost will be generated, which can be introduced by using the negative number of movement distance or movement time to punish the navigation action. ③ dialogue cost, when the agent performs the question action , a dialogue cost will be generated, which can be introduced by using the negative number of dialogue times to punish the navigation action. To encourage the agent to consider efficiency in decision-making and complete the task with less movement and questioning.

[0039] Step 108, based on the partially observable Markov decision process, the observation module uses a self-reflection mechanism to generate a description of each suspected entity by separately asking questions about each known target fact.

[0040] The observation module generates a description for each entity by comparing the observation view of the entity with existing facts, thereby providing a basis for the subsequent prediction stage. In this module, accurately identifying key features and obtaining detailed entity descriptions are key to successfully distinguishing similar entities. Considering the complexity and diversity of entities in real-world environments, previous work usually uses VLM to extract entity information from views. However, the current VLM model still has hallucination phenomenon, which may lead to deviation in identifying and describing key features of entities, thereby causing failure in entity alignment. Especially when the distance between entities is far, the proportion of entity pixels in the view is small, and the VLM may not be able to accurately judge the characteristics of the entity. In order to alleviate the inherent hallucination of VLM, it is important to estimate the uncertainty of VLM when responding, that is, to estimate the degree of hesitation of VLM when outputting. When the degree of hesitation is high, the output of the VLM may be hallucinated or unreliable.

[0041] Therefore, the output probability distribution of the VLM is used to judge the uncertainty of the VLM output. Given an autoregressive VLM, for picture representation, for prompt word representation, To characterize the historical reasoning, at the inference stage, the VLM generates a conditional probability distribution over the vocabulary at each time step:

[0042]

[0043] and then generates the output over this conditional probability. However, it is difficult to estimate the uncertainty of the VLM using the conditional probability distribution in the above formula, because the vocabulary used by the VLM usually has a large scale , which will lead to a huge output space of the VLM, making it difficult to calculate its uncertainty. To this end, the present application introduces a key self-evaluation link, and through a carefully designed question and answer framework, the entity observation is generated from an open description to a closed judgment of specific facts. Specifically, for a given set of target facts and entity views , a pre-defined prompt word is used (as shown in Figure 3 ), and each fact in the fact set is asked individually, and the VLM is only allowed to output three answers: yes, no, and I don't know. Such a design can force the VLM to evaluate its cognitive state, and when the visual evidence is not sufficient to support a clear judgment, it can admit its limitations by choosing "I don't know" instead of being forced to produce illusory content.

[0044] Based on the information entropy theory, the normalized entropy is calculated to measure the uncertainty of the VLM output; wherein represents the normalized entropy, and at the same time represents the uncertainty of the model output, and the greater the entropy, the higher the uncertainty. In addition, since the VLM can only generate three answers, the size of the vocabulary . Further, a threshold is set, which can divide the uncertainty of the VLM output into two levels, namely certain and uncertain:

[0045]

[0046] Given the facts and views , the self-reflection process is formalized as follows, wherein is the answer of the VLM:

[0047]

[0048] For each entity, the observation module finally outputs a set ​​​​The set is used to evaluate and describe it, containing VLM's description of whether it conforms to all facts and whether it is certain, where each element consists of triples. Repeat this process until all target facts have been processed, and finally return a set of descriptions containing complete evaluation information.

[0049] Step 110: Using the zero-shot learning method in the prediction module, the descriptions of each suspected entity and the known target facts are input into the visual language model, and the target prediction results are output.

[0050] For target prediction in EIEA problems, given a list of target facts... and Observational description of a suspected entity Together, they constitute the input of the prediction model. Each entity is treated as a prediction label, with exactly one entity. The true label represents the target entity. The output of the prediction model is the probability of all predicted labels. Based on this, a non-conformity score is calculated, and a conformal prediction method is used to obtain the prediction set of the target. ,in It predicts the number of entities in a set.

[0051] Conformal prediction is a framework for constructing prediction sets with a marginal coverage guarantee. It generates a prediction set at a pre-defined confidence level that contains the true label of the test sample with a certain probability. This guarantee holds even with finite samples and typically does not depend on the true distribution of the data. Specifically, given... and These are the input and output spaces of the prediction model, respectively. Represent a and The unknown mapping distribution between them. From Sampling an independent and identically distributed calibration set Conformal prediction methods utilize calibration sets. To provide calibration for the prediction model, a non-compliance function also needs to be defined. Used for quantifying samples Authentic Labels The degree of deviation from the "typical" or "consistent" behavior represented in the output of the predictive model. The non-consistency score is typically calculated using the output probability of the predictive model.

[0052] ;

[0053] in, Indicates the model predicts samples The label is a real label the probability of being correct, i.e. the probability of correct prediction. The higher this probability, the lower the non-conformance score , meaning that this prediction is not "strange" and is "conforming" to the expectation.

[0054] A set of non-conformance scores is computed for all data in the calibration set using the non-conformance function , which forms an empirical distribution of the non-conformance of the prediction model on the known answer data. This distribution is used to construct the prediction set for new test samples.

[0055] When a new test sample is given, whose true label is unknown, the non-conformance score is first computed for each possible candidate label using the prediction model , and then the empirical quantile of this score in the calibration score set is computed:

[0056] ;

[0057] where is an indicator function that takes value 1 when the input condition is satisfied, and 0 otherwise. measures the extremeness of in the calibration score set when is the true label. The smaller this value, the more extreme is than more data in the calibration set, and the less likely is the true label. In other words, the lower the error rate of excluding from the prediction set. Given an error rate , if , then it can be considered that in order to guarantee the error rate does not exceed , the label needs to be included in the prediction set :

[0058] ;

[0059] In practice, for ease of computation, the empirical quantile is first computed for the calibration score set :

[0060] ;

[0061] where denotes the ceiling function, then the prediction set can be defined as:

[0062] .

[0063] To obtain the probability of all predicted labels, the output of LLM can be restricted to a limited vocabulary containing only all predicted labels by referring to the observation module, the length of the vocabulary is the number of labels , and the output distribution of LLM on the vocabulary is used as the predicted probability. However, in preliminary experiments, it is found that due to the randomness of LLM output, the output of LLM is difficult to control on the pre-defined vocabulary, especially when the length of the vocabulary is large. In addition, LLM always shows excessive confidence before obtaining enough information, concentrating the predicted probability on part of the suspected entities, so that the prediction accuracy cannot be guaranteed.

[0064] Therefore, in order to ensure the stability and accuracy of the prediction result, the present application adopts a prompt word template , so that LLM can conduct in-depth analysis and comparison on each suspected entity based on the input, and finally provide a normalized score of each suspected entity . The prompt word template is as shown in Figure 4 , and the higher the score of a suspected entity is, the higher the possibility that LLM considers the entity to be the target. Based on the score , the non-compliance score can be defined as . Considering the performance limitation of LLM, there are at most 5 suspected entities in an alignment task. In order to maintain the unity of the problem, when there are less than 5 suspected entities in the task, virtual entities are supplemented, and their feature descriptions are set to be completely inconsistent with the target fact, so as to exclude the possibility of being the target.

[0065] In a single prediction, the normalized score distribution of each entity reflects the uncertainty of LLM in prediction. In the early stage of the entity alignment process, the prediction set may contain more entities, because LLM has greater uncertainty about the prediction result. With the continuous updating of the suspected entity view and the mastered facts, the prediction set at each time step may also change. When the prediction set contains only one suspected entity, i.e. single prediction, it can be considered that LLM is confident to report the suspected entity as the entity specified by the human user at the confidence level required by the human user, at which time the entity alignment process can be actively ended. On the contrary, when the prediction set contains more than one entity, more movements or dialogues are needed to help LLM update its prediction result.

[0066] In addition, the human user can also control the process of entity alignment by giving different confidence levels. When it is desired to quickly complete the alignment while accepting a certain loss of accuracy, a lower confidence level can be provided to reduce the size of the prediction set, so as to reach single prediction more quickly.

[0067] Step 112, according to the action module, the navigation-question mechanism is adopted, based on the observation of the entity in the target prediction result, the navigation or the question action is switched, if the navigation action is executed, an entity which has not been explored is selected and navigated to the vicinity to obtain observation pictures, at the same time, adaptive distance control is carried out based on the proportion of the entity bounding box in the view, if the question action is executed, the comparative question method is adopted, the observation pictures of two explored entities are selected to input the visual language model to generate a distinguishing question, and the answer about the target fact is obtained through the dialogue with the human user.

[0068] The suspected entities contained in the prediction set may be the target entity, when multiple entities appear in the set, it means that the LLM has uncertainty in prediction. In order to reduce the uncertainty, navigation or question action can be adopted. According to the problem modeling, the navigation action can obtain the close observation view of the entity, so as to alleviate the noise in visual observation. The question action can obtain more information about the target, so as to alleviate the noise in language observation. As shown in Table 1, algorithm 1, when there are at least two explored entities in the prediction set , the two entities with the highest scores are selected to execute the comparative question action; otherwise, the unexplored entity with the highest score is selected to execute the adaptive distance navigation action; the comparative question method is adopted to analyze the difference between the two entities and generate a distinguishing question.

[0069] Table 1

[0070]

[0071] The whole process of the method of the present application is shown in Table 2, algorithm 2. Given a task , the method can align the agent with human cognition, and confirm the target entity specified by the human . First, each element in the method is initialized (line 1), and then the loop is started. The conformal prediction method is adopted to obtain the prediction set according to the score , if the size of the prediction set is reduced to 1, it indicates that the uncertainty is completely eliminated, and the method takes the only entity as the final result and terminates the loop (lines 4-6). However, when there are still multiple candidate entities in the prediction set, the algorithm will enter the core action decision stage (line 8), the process of which has been described in detail in algorithm 1. If the question action is selected, a distinguishing question is generated and a new target fact is obtained through the dialogue, and then the estimation of each entity is updated (lines 9-13). If the navigation action is selected, the close observation picture of the currently explored entity is obtained by using the navigation map, and the estimation of the entity is updated (lines 15-18). After completing any action and obtaining new information, the UCE method will immediately reevaluate the scores of all candidate entities (line 20), in preparation for a more accurate prediction in the next round.

[0072] Table 2

[0073]

[0074] In summary, the specific content includes:

[0075] Initialization phase: according to the input task System initialization, establish entity observation view set , target fact set , entity evaluation set , exploration identifier And the scoring set . Set the calibration set and confidence level parameter, prepare for the subsequent conformal prediction.

[0076] Main loop execution: enter the main alignment loop, repeat the following steps until a unique target is found:

[0077] First, perform the conformal prediction step, based on the current entity evaluation and fact set, use conformal prediction theory to generate a prediction set containing multiple candidate entities , while updating the matching score of each entity . The size of the prediction set reflects the current level of uncertainty.

[0078] Then make a termination judgment, check if the size of the prediction set is equal to 1. If the prediction set contains only one entity, it means that the system has enough confidence to determine the target, at this time report the entity to the human user as the target entity, and end the entire alignment process.

[0079] If the prediction set contains multiple entities, perform the action decision step, determine the next action type and target entity according to the navigation-question mechanism. The decision process takes into account the number of explored entities in the prediction set and the score of each entity.

[0080] In the question branch, select the two highest-scoring explored entities, use the contrast question method to generate a discriminative question, obtain new target facts through human-computer dialogue, update the fact set , and re-evaluate all entities based on the new facts.

[0081] In the navigation branch, select the highest-scoring unexplored entity as the navigation target, use the adaptive distance control method to move to the best observation position, obtain high-quality entity observation images, update the corresponding observation view and evaluation results, and mark the entity as explored.

[0082] Finally, the score update step is performed to recalculate the matching scores of each entity based on the updated entity assessment and fact set, providing accurate input data for the next round of prediction.

[0083] The entire algorithm loop performs the above steps, gradually narrowing down the prediction set size through continuous observation, prediction, action, and update, and ultimately achieving accurate entity alignment.

[0084] The above-mentioned unmanned aerial vehicle body cognitive alignment method based on human-computer cooperation is based on the EmbodiedCity simulator, and entity observation pictures are collected at different angles and distances and used as discrete nodes of the navigation map. The pictures from multiple angles and distances cover the characteristics of entities in different physical conditions, providing more comprehensive reference samples for the model and reducing the bias of single observation. At the same time, the navigation map structure of discrete nodes makes the entity exploration path clearer and reduces the complexity of entity positioning in an open environment. The construction of the unmanned aerial vehicle body cognitive alignment model converts the task into POMDP in the problem modeling module, which fits the partially observable characteristics of the urban environment. Through the definition of state, action, and observation elements, the entity alignment is converted into a dynamic decision and probability optimization problem, providing a theoretical framework for uncertain scenarios and making up for the lack of uncertainty quantification mechanism in existing methods. The observation module generates suspected entity descriptions by actively asking questions about target facts, which is equivalent to letting the model autonomously fill information gaps and avoids the limitations of relying on ambiguous initial instructions. The prediction module uses zero-shot learning combined with VLM, which not only utilizes the cross-modal capabilities of VLM but also reduces the hallucination phenomenon of VLM when dealing with complex scenarios alone by using dual inputs of suspected entity descriptions and target facts, thereby improving entity discrimination accuracy. The navigation questioning mechanism of the action module is based on adaptive distance control of the entity bounding box ratio, ensuring that the unmanned aerial vehicle can obtain clear observations at an appropriate distance and solving the problem of long-distance feature recognition. The priority exploration strategy of unexplored entities avoids repeated movement and reduces movement costs. The comparative questioning method during questioning generates distinctive questions through VLM and interacts with humans to obtain key discriminant information at the lowest human cost, balancing accuracy and cost in human-computer cooperation. The model uses POMDP as a framework to combine observation, prediction, and action. Each observation updates the belief state, the prediction result guides the next action, and the new observation after the action optimizes the judgment. This dynamic adjustment mechanism perfectly adapts to scenarios where multiple entities exist simultaneously and need to be dynamically compared in an open urban environment, breaking through the inefficient mode of traditional indoor navigation sequential confirmation and solving the core limitations of existing technologies in complex urban environments. Finally, it realizes efficient and accurate entity alignment.

[0085] In one embodiment, the state space includes known information required for agent decision-making, represented as a three-tuple wherein This represents the current view of each entity, consisting of RGB images of the entities taken by the agent from a first-person perspective; This represents the set of facts about the target entity extracted from the initial instructions and subsequent dialogue; The current evaluation of each entity is recorded: the action space includes three types of actions, at each time step. The intelligent agent can choose to perform one of three types of actions, namely , Indicates the agent's interaction with the entity. Moving around aims to reduce observational uncertainty by gaining a closer perspective; asking questions This means that the intelligent agent poses a question to the human user aimed at dispelling ambiguity or obtaining new facts. ;Report This indicates that the intelligent agent actively terminates the alignment process and reports the entity to the human user. That is, the target entity; the observation space includes the time step. The agent's observation includes two channels: vision and language. When selecting navigation action At that time, the intelligent agent obtains the entity New viewing view ,Right now When selecting the question action At that time, the intelligent agent receives answers from humans. ,Right now The state transition function is defined as follows: When selecting navigation action At that time, entity The view of observation is updated, and its entity evaluation is updated accordingly when the question action is selected. At the same time, the AI ​​can extract new facts from its interactions with human users. And update the fact set, and based on the new fact set, update the evaluation results of all entities when the reporting action is selected. At this point, the alignment process ends, and there is no need to update the belief state; the reward function includes the alignment result reward, the navigation cost reward, and the dialogue cost reward; the alignment result reward is the reward given when the agent executes the report. Actions, and If the alignment is successful, a positive reward is awarded; otherwise, a negative or zero reward is awarded. The navigation cost reward is the amount of time the agent performs the navigation action. In this case, negative numbers of the distance or time traveled are used to introduce a penalty for the navigation action; the dialogue cost reward is the reward when the agent performs a questioning action. In this case, a negative number of dialogues is used to introduce a penalty for navigation actions.

[0086] In one embodiment, the EIEA-EC dataset is developed based on the Embodied City simulator, including:

[0087] In the entity data collection phase, the physical model is placed in the simulator and the initial observation angle is randomly generated and observation distance ;

[0088] According to the entity position, initial observation angle and observation distance, the initial observation picture of the agent to the entity is collected , where and are the height and width of the picture respectively, and the position of the initial observation picture is recorded as ;

[0089] For entity , the data collection result can be represented as , where includes the initial observation picture and its corresponding coordinates, and the rest includes pictures and coordinates corresponding to different observation distances;

[0090] Based on the collected entity data, the EIEA task based on the navigation graph is constructed, each task is modeled as , where and are the node set and edge set of the navigation graph respectively, is the target entity;

[0091] When constructing the task, first randomly select 3-5 entities of the same type from the entity set, and randomly select one as the target entity, and the observation pictures of each entity at different distances in the task are used as nodes, and the nodes are fully connected to form the entity alignment task.

[0092] In specific embodiments, the detailed process of entity data collection includes the following steps:

[0093] Parameter initialization: randomly generate initial observation parameters for each entity, including observation angle (conforming to uniform distribution from 0 to 360 degrees) and observation distance (conforming to uniform distribution from 5 to 30 meters). This random design ensures the diversity and generalization ability of the dataset.

[0094] Multi-distance observation collection: For each entity, observation image collection is conducted at six different distance positions, including initial distance, adaptive distance, and fixed distances of 2 meters, 5 meters, 10 meters, and 15 meters. For initial distance observation, the observation position of the agent is calculated according to the randomly generated angle and distance parameters to ensure that a complete view of the entity can be obtained. For adaptive distance observation, the optimal observation position that meets the bounding box constraint condition is found by iterative adjustment using the adaptive distance control method described above. For fixed distance observation, the initial angle is kept unchanged, and only the observation distance is adjusted to the specified value.

[0095] Image collection and storage: At each determined observation position, the RGB observation image of the entity is obtained using the image collection function of the simulator, and the image resolution is set to 480x640 pixels. At the same time, the corresponding observation position coordinates, including three-dimensional space coordinates and observation angle information, are recorded.

[0096] Real state recording: The human annotator collects the real state picture of the entity in the simulator as the ground truth reference, which is used to verify the observation quality and algorithm performance.

[0097] Data structure organization: All collected data are organized into a structured format, including a dictionary structure of real state pictures and distance observation data, each observation data containing image and position information, facilitating subsequent task construction and algorithm training.

[0098] The detailed process of task data construction includes the following steps:

[0099] Entity selection phase: Randomly select 3 to 5 entities from the same type of entity set as suspected entities in the task, and the randomization of the number of entities ensures the diversity of task difficulty. Then randomly select one of the selected entities as the target entity, and the other entities as interference entities.

[0100] Navigation graph node construction: Create multiple navigation graph nodes for each selected entity, each node corresponding to the observation data of the entity at different observation distances. Specifically, each node contains four key information: entity identification, observation distance identification, corresponding observation image, and observation position coordinates. Such design allows the agent to observe the same entity at different distances.

[0101] Navigation graph edge construction: The edges of the navigation graph are constructed in a fully connected manner, i.e., there is a reachable path between any two nodes. This design simplifies the path planning problem, allowing the agent to move directly between any two observation positions and focusing on the core challenge of entity alignment.

[0102] Task structure organization: The constructed navigation graph node set, edge set, target entity identifiers, and list of all suspected entities are organized into a complete task data structure. This structure provides the UCE algorithm with a complete description of the task environment, including accessible observation locations, entity information, and target settings.

[0103] Task verification and optimization: Check whether the constructed tasks meet the basic requirements, such as ensuring that the target entity has distinguishable feature differences from other entities, verifying the connectivity of the navigation graph, and ensuring the solvability and rationality of the tasks.

[0104] In one embodiment, the observation module uses a self-reflection mechanism to generate descriptions of each suspected entity by asking questions about each known target fact, including:

[0105] For a given set of target facts and entity view Using predefined prompt words Each fact in the target fact set is asked a separate question. The visual language model is restricted to only outputting three answers: yes, no, and I don't know. The visual language model is forced to evaluate the cognitive state of each fact and generate a description of each suspected entity.

[0106] In one embodiment, the prediction module employs a zero-shot learning method, inputting descriptions of each suspected entity and the known target facts into the visual language model, and outputting target prediction results, including:

[0107] A target prediction method is constructed based on conformal prediction theory, and a calibration set is used. Perform model calibration and define non-compliance functions. Used for quantifying samples Authentic Labels The degree of deviation from typical or consistent behavior represented in the output of the prediction model, where, Represents the sample set, Represents a set of tags. Represents the set of real numbers;

[0108] Calculate the empirical quantiles of the calibration set score set; construct test samples based on the descriptions of each suspected entity and the known facts about the target. Using a prediction model for each possible candidate label Calculate non-compliance score The prediction set, i.e., the target prediction result, is generated based on empirical quantiles and non-compliance scores. The target prediction result is at least The confidence level includes the true label and satisfies .

[0109] In one embodiment, for target prediction in embodied interactive entity alignment problem, given a list of target facts and observations of suspect entities , which collectively constitute the input of the prediction model Each entity is treated as a prediction label, and only one entity is the true label, i.e., the target entity. The output of the prediction model is the probability of all prediction labels.

[0110] In one embodiment, the empirical quantile of the calibration set score set is calculated as:

[0111] ;

[0112] where denotes the empirical quantile, denotes the non-compliance score of the th sample, denotes the calibration set size, denotes the pre-set error rate.

[0113] In one embodiment, the navigation-question mechanism is adopted according to the action module, based on the observation of the entities in the target prediction result, the navigation or question action is switched, including:

[0114] Given the prediction set and its exploration identifier and score When there are at least two explored entities in the prediction set, select the two entities with the highest score to perform the question action ; When there are many unexplored entities in the prediction set, select the unexplored entity with the highest score to perform the navigation action ; Adjust the observation distance through the adaptive distance control method, so that the entity bounding box meets the constraint condition, generate a distinctive question through the comparative question method, and obtain the target fact by comparing the differences of the two entities in multiple aspects, including color, style and structure.

[0115] In specific embodiments, the number of explored entities in the prediction set is counted, and the total number of entities with an exploration identifier value of 1 (explored) is calculated by checking the exploration identifier corresponding to each prediction set entity. Then, according to the number of explored entities, the action type is determined: if there are at least two explored entities in the prediction set, the question action mode is triggered; otherwise, the navigation action mode is triggered.

[0116] In the question-and-action mode, the two entities with the highest scores are selected from the explored entities in the prediction set as comparison targets. These two entities will be used to generate discriminative questions. The selection process is based on the matching scores of the entities, prioritizing the entities most likely to be the targets for comparative analysis.

[0117] In navigation action mode, the highest-rated entity is selected from the unexplored entities in the prediction set as the navigation target, and the agent moves to the vicinity of that entity for close observation. This strategy ensures that the most promising candidate entities are explored first, improving alignment efficiency.

[0118] Ultimately, the system returns the identified action type, prompts or navigation, and the corresponding target entity, providing clear guidance for the execution of subsequent actions.

[0119] In one embodiment, a given agent pairs with entities View of observation First, the GroundingDino model is used to detect the bounding boxes of entities. Let the coordinates of the bounding box be... Then, the relationship between the bounding box and the entire viewing view is controlled through discrete actions. To improve viewing quality, the target should be located in the center of the view and occupy a moderate proportion [1, 2]. Therefore, the following constraints are satisfied by adjusting the viewing distance to obtain a high-quality viewing view:

[0120] and ;

[0121] in, and These are the coordinates of the top-left and bottom-right corners of the bounding box, respectively. Control the minimum proportion of an entity in the view. Controls the boundary position of entities in the view. and Observe the height and width of the image respectively.

[0122] To obtain a satisfactory view, the agent starts from a pre-determined viewing angle and initial position. Then, based on the state of the entity bounding box, it selects an action from the action set {forward, backward, look up, look down} to execute until the bounding box satisfies the constraints. The step size for forward and backward is set to 0.2m, and the step size for looking up and looking down is set to 5°. During the task execution phase, all suspected entities are first... The status is marked as "unexplored", and the corresponding identifier is set to When the agent chooses to navigate to a suspected entity Upon closer observation, the agent can directly reach the suspected entity from its current node. The system adaptively distances the observation nodes, obtains the corresponding observation view, marks the entity as "explored," and updates its identifier. .

[0123] In one embodiment, the comparative questioning method includes a difference identification stage, a question construction stage, a deduplication filtering stage, a human-computer interaction stage, and a fact update stage; the difference identification stage involves comparing the observed images of two suspected entities. and Input visual language model and use differential analysis prompt words The output entity provides a detailed description of the differences in its multi-dimensional features; these features include color, style, structure, pose, and gender; the problem construction phase involves describing these differences. and the set of facts already known Input a large language model and generate prompt words using questions. Generate a binary choice problem that can distinguish between two entities. The deduplication filtering stage ensures that the generated questions do not match the set of facts. The system already contains repetitive features, so avoid asking the same question repeatedly about the same feature; the human-computer interaction stage involves dialogue using natural language. Obtain answers from human users and extract new target facts from them. The fact update phase is the execution phase. Update the fact set and update the evaluation of all entities accordingly. .like Figure 5 The image shown is a template for comparing question prompts.

[0124] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above embodiments only express several implementation ways of the present application, and the description is more specific and detailed, but it should not be understood as a limitation to the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A human-robot collaborative unmanned implement body cognitive alignment method, characterized in that, The method comprises: Developing an EIEA-EC dataset based on an EmbodiedCity simulator; the EIEA-EC dataset comprises observation pictures of entities collected at different angles and different distances, which are taken as discrete nodes on a navigation map to construct an entity alignment task; Constructing a UAV embodied cognitive alignment model; the UAV embodied cognitive alignment model comprises a problem modeling module, an observation module, a prediction module, and an action module; Inputting the entity alignment task into the UAV embodied cognitive alignment model, modeling the embodied interactive entity alignment problem into a partially observable Markov decision process comprising a state space, an action space, an observation space, a state transition function, an observation function, a reward function, and a discount factor according to the problem modeling module; Based on the partially observable Markov decision process, generating descriptions of each suspected entity by respectively asking each known target fact according to the observation module using a self-reflection mechanism; The prediction module uses a zero-shot learning method to input the descriptions of the suspected entities and the known target facts into a visual language model to output a target prediction result; The action module uses a navigation-question mechanism to switch between navigation and questioning actions based on the observation of the entities in the target prediction result; if the navigation action is performed, an entity that has not been explored is selected and navigated to the vicinity to obtain observation pictures, and adaptive distance control is performed based on the proportion of the entity bounding box in the view; if the questioning action is performed, a comparative questioning method is used to select observation pictures of two explored entities and input them into the visual language model to generate a distinguishing question, and an answer about the target fact is obtained through a dialogue with a human user; The prediction module uses a zero-shot learning method to input the descriptions of the suspected entities and the known target facts into a visual language model to output a target prediction result, comprising: A target prediction method is constructed based on conformal prediction theory, and a calibration set is used. Perform model calibration and define non-compliance functions. Used for quantifying samples Authentic Labels The degree of deviation from typical or consistent behavior represented in the output of the prediction model, where, Represents the sample set, Represents a set of tags. Represents the set of real numbers; calculating an empirical quantile of the calibration set score set; constructing a test sample according to the description of each suspected entity and the target fact mastered , using a prediction model to predict each possible candidate label Calculate non-compliance score Based on the empirical quantile and the non-compliance score, guide the generation of the prediction set, that is, the target prediction result , the target prediction result contains the true label at least Confidence level, meet , wherein, Indicates the empirical quantile, Indicates the preset error rate.

2. The method of claim 1, wherein, The state space includes the known information required for the agent's decision-making, represented as a triple. ,in This represents the current view of each entity, consisting of RGB images of the entities taken by the agent from a first-person perspective; This represents the set of facts about the target entity extracted from the initial instructions and subsequent dialogue; The current evaluation of each entity is recorded; the action space includes three types of actions, at each time step. The agent chooses to perform one of three types of actions, namely , Indicates the agent's interaction with the entity. Moving around aims to reduce observational uncertainty by gaining a closer perspective; asking questions This means that the intelligent agent poses a question to the human user aimed at dispelling ambiguity or obtaining new facts. ;Report This indicates that the intelligent agent actively terminates the alignment process and reports the entity to the human user. That is, the target entity; the observation space includes the time step The agent's observation includes two channels: vision and language. When selecting navigation action At that time, the intelligent agent obtains the entity New viewing view ,Right now When selecting the question action At that time, the intelligent agent receives answers from humans. ,Right now The state transition function is defined as follows: When selecting navigation action At that time, entity The view of observation is updated, and its entity evaluation is updated accordingly when the question action is selected. At the same time, the AI ​​can extract new facts from its interactions with human users. And update the fact set, and based on the new fact set, update the evaluation results of all entities when the reporting action is selected. At this point, the alignment process ends, and there is no need to update the belief state; the reward function includes alignment result reward, navigation cost reward, and dialogue cost reward; The alignment result reward is given when the agent performs a report action, and a positive reward if the alignment is successful, otherwise a negative reward or zero reward; the navigation cost reward is given when the agent performs a navigation action , a negative number of moving distance or moving time is used to introduce a penalty for the navigation action; the dialogue cost reward is given when the agent performs a question action , a negative number of dialogue times is used to introduce a penalty for the navigation action.

3. The method of claim 1, wherein, Developing an EIEA-EC dataset based on an EmbodiedCity simulator, comprising: In the physical data collection phase, the physical model is placed in the simulator and initial observation angles are randomly generated and observation distances ; According to the entity position, an initial observation angle, and an observation distance, an initial observation picture of the entity by the agent is collected , wherein , and are a height and a width of the picture respectively, and a position at which the initial observation picture is collected is recorded as ; For the entity The data collection result is expressed as Wherein The initial observation picture and its corresponding coordinates are included, and the rest The pictures and coordinates corresponding to different observation distances are included; On the basis of the collected entity data, EIEA tasks based on navigation graphs are constructed, each task is modeled as wherein and are a node set and an edge set of the navigation graph respectively, is a target entity; When constructing the task, first randomly select 3-5 entities of the same type from the entity set, and randomly select one as the target entity; observation pictures of each entity at different distances are taken as nodes, and the nodes are fully connected to form an entity alignment task.

4. The method of claim 1, wherein, The observation module uses a self-reflection mechanism to respectively ask each known target fact to generate descriptions of each suspected entity, comprising: For a given target fact set and entity view , each fact in the target fact set is individually asked with a predefined prompt , limiting the visual language model to output only yes, no, and I don't know answers, forcing the visual language model to evaluate the cognitive state of each fact, generating a description of each suspected entity.

5. The method of claim 1, wherein, The target prediction method comprises: For target prediction in embodied interactive entity alignment problem, given a list of target facts and observation descriptions of K suspected entities , which collectively constitute the input of the prediction model Each entity is treated as a prediction label, and only one entity is the true label, i.e., the target entity, and the output of the prediction model is the probability of all prediction labels.

6. The method of claim 1, wherein, Calculate the empirical quantile of the calibration set score set as: wherein, represents the non-conformance score for the th sample, represents the calibration set size.

7. The method of claim 1, wherein, The action module uses a navigation-question mechanism to switch between navigation and questioning actions based on the observation of the entities in the target prediction result, comprising: Given a prediction set and its exploration identifier and scores When there are at least two explored entities in the prediction set, select the two entities with the highest scores to perform the questioning action When there are more unexplored entities in the prediction set, select the unexplored entity with the highest score to perform the navigation action Adjust the observation distance by an adaptive distance control method to make the entity bounding box meet the constraint condition, generate a distinguishing question by a comparative questioning method, and obtain the target fact by comparing the differences of the two entities in multiple aspects, including color, style, and structure.

8. The method of claim 7, wherein, The constraint condition is: wherein, and are respectively the left-top corner coordinate and the right-bottom corner coordinate of the bounding box, the minimum proportion of the control entity in the view, the boundary position of the control entity in the view, and are respectively the height and the width of the observed picture.

9. The method of claim 7, wherein, The comparative question method comprises a difference identification stage, a question construction stage, a duplicate filtering stage, a human-computer interaction stage and a fact updating stage; the difference identification stage is for identifying differences between observation images of two suspected entities and inputting a visual language model and adopting a difference analysis prompt word outputting specific difference descriptions of entities in multidimensional features; the multidimensional features comprise color, style, structure, posture and gender; the question construction stage is for constructing difference descriptions and a set of mastered facts inputting a large language model and adopting a question generation prompt word generating a binary selection question capable of distinguishing between two entities ; the duplicate filtering stage is for ensuring that the generated question is not repeated with the features in the set of facts , so as to avoid repeated questioning of the same feature; the human-computer interaction stage is for obtaining answers of a human user through natural language dialogue , and extracting new target facts therefrom ; the fact updating stage is for updating the set of facts and updating the evaluation of all entities accordingly .​

Images