A continuous visual language navigation method based on causal reasoning and cognitive thinking

Abstract

The present invention discloses a continuous visual language navigation method based on causal reasoning and cognitive thinking, comprising the following steps: S1. receiving a natural language instruction of a current navigation task, and capturing a view when executing the task as an original view; S2. utilizing a style counterfactual adjustment module to generate a counterfactual style view with a different style from the original view based on the original view; S3. inputting the original view and the generated counterfactual style view into a visual encoder to extract visual features; performing feature encoding on the natural language instruction through a text encoder to extract text features; S4. calculating the layout deviation between the current environment and the prior knowledge through spatial semantic analysis and in combination with prior knowledge, and then eliminating the deviation using a causal reasoning mechanism; S5. combining the encoded text features and visual features, and using the debiased prior knowledge to predict the action of the navigation agent at the current time step; S6. repeating the above steps until the navigation task is completed or the maximum number of moving steps is reached.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal visual language navigation technology, and in particular to a continuous visual language navigation method based on causal reasoning and cognitive thinking. Background Art

[0002] In the field of vision-language navigation (VLN), navigation agents need to understand natural language instructions and combine visual information to navigate effectively. With the deepening of research, VLN tasks have been extended from discrete environments to continuous environments (VLN-CE), which requires navigation agents to make decisions in more complex spaces. However, the introduction of continuous environments leads to many uncertainties in the navigation process, which makes the actions of navigation agents in this situation relatively difficult. Therefore, it is necessary to improve the capabilities of navigation agents in continuous environments, especially in terms of generalization ability and decision-making efficiency.

[0003] In the early stages of development, VLN tasks [1-7] were mainly performed in simulation environments (such as the Matterport 3D simulator [8]) to execute R2R [9], RxR instructions

[10] , etc. Due to its heavy computational cost and vast search space, early research work was mainly focused on discrete environments. Although this reduces the complexity of the actual environment by giving a series of assumptions and has a certain degree of abstraction, it still played a great role in promoting the development of the VLN community. Among these works, COAT

[11] proposed to significantly improve the performance of navigation agents in complex indoor environments according to natural language instructions by fusing and aligning visual and language object information; by changing the navigation environment of the navigation agent, the navigation agent can receive more diverse visual views[12,13] and learn more navigation knowledge, which can essentially be regarded as a "data enhancement" method; sometimes the navigation ability of the navigation agent alone may not be enough, and the introduction of additional auxiliary tasks or measures can give the navigation agent a chance to "regret"

[14] and think about wrong actions again

[15] . With the assistance of additional tasks

[16] , it can handle previous navigation failures and greatly reduce the navigation error rate; the Speaker-Follower navigation agent

[17] significantly improves the navigation success rate of existing navigation agents by executing the processes of "instruction generation-instruction interpretation" and "generating candidate paths and reasoning in the panoramic action space" through two modules; introducing graph networks into VLN to construct entity relationship graphs

[18] is also an effective method with good robustness and interpretability.

[0004] In order to get closer to navigation in a realistic environment, VLN-CE

[19] removes some abstract assumptions of VLN in a discrete environment, so that the navigation agent no longer relies on a predefined connectivity graph for navigation, and allows the navigation agent to be controlled with more fine-grained actions. Compared with the previous discrete VLN, the complexity of VLN-CE is greatly increased, and a more effective algorithm is needed to handle it. Krantz et al. further proposed a type of language-conditioned waypoint prediction network, namely Waypoint Model

[20] , which can preliminarily solve the problem of continuous navigation, but the computational cost of its training is relatively high; unlike the research that focuses only on VLN-CE, Hong et al. proposed a method to narrow the performance gap between discrete VLN and continuous VLN

[21] . Their method mainly converts the waypoint prediction function of continuous VLN into a form suitable for discrete VLN, thereby effectively improving navigation performance. In addition, this method can also enable discrete VLN to play a greater role in VLN-CE tasks. Moreover, compared with the waypoint prediction network given by the previous VLN-CE proposer, this work not only has a significant improvement in performance. It has greatly improved the training resources and time, making it a classic work in the research community at that time; some works predict the context semantics through cross-modal decoupling and modular training [22,23], and then predict the navigation path to the target; MLANet

[24] proposes to start from the encoder-decoder architecture, through instruction segmentation and global perception, to overcome the challenges of complex instruction understanding and long action sequences, and further improve the performance of continuous VLN; some methods [25,26,27,28] further improve the spatiotemporal reasoning ability of navigation agents by focusing on the spatial structure or semantic relationship of the navigation environment.

[0005] As an analytical method that can reveal the causal relationship behind the data, causal reasoning [29,30,31] has been gradually introduced into various fields (medicine, social science, etc.). Its great ability has aroused widespread interest in researchers of visual language tasks. Recently, causal reasoning has been used in the field of visual question answering to learn more complex problems [32,33]. In terms of visual navigation, [34,35,36] proposed to generate instructions, paths and observation environments that are different from the original data in a counterfactual way, thereby supporting more robust training.

[37] proposed to apply counterfactual causal analysis to navigation reasoning, making the role of causal reasoning in promoting navigation more direct.

[0006] At present, although the work on VLN tasks is relatively mature, it is not easy to migrate from discrete environments to continuous environments for navigation. The work on VLN-CE still needs to be supplemented, and most people do not realize the importance of "active thinking" of navigation agents for their navigation planning. Since most of the previous work was based on passive learning by navigation agents, this defect limits the open acquisition of knowledge and curbs the development of navigation performance. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a continuous visual language navigation method based on causal reasoning and cognitive thinking. This method injects "thinking ability" into the navigation agent, so that the prediction of the navigation agent is no longer solely dependent on the likelihood relationship between the data, but further explores the causal relationship behind the data and performs relational reasoning before predicting the results.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] A continuous visual language navigation method based on causal reasoning and cognitive thinking, comprising the following steps:

[0010] S1. Obtain the natural language instructions of the current navigation task, and capture the view of the current position of the navigation agent as the original view;

[0011] S2. Based on the original view obtained in S1, a counterfactual style view different from the original view style is generated through a style counterfactual adjustment module;

[0012] S3. Input the counterfactual style view and the original view generated in S2 into the visual encoder to learn visual features, and extract text features from the natural language instructions obtained in S1 through the text encoder;

[0013] S4. Perform spatial semantic analysis on the current environment, extract the current spatial layout, and calculate the layout deviation between the current spatial layout and the prior knowledge in combination with the prior knowledge. Then, use the causal reasoning mechanism to eliminate the layout deviation and obtain the debiased prior knowledge.

[0014] S5. Combine the encoded text features and visual features and use the debiased prior knowledge to predict the action of the navigation agent at the current time step;

[0015] S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of moving steps is reached.

[0016] Furthermore, the specific workflow of the style counterfactual adjustment module in step S2 is as follows:

[0017] First, the view image is encoded and the original view is encoded using a convolutional neural network architecture to ensure the effective extraction of the original view visual features;

[0018] Secondly, style feature sampling is performed to obtain style embedding by sampling from a multivariate normal distribution based on the PBN dataset;

[0019] Finally, the style embedding integration is completed. The style embedding is combined with the visual features through the conditional instance normalization method, so that the generated counterfactual style view can show the counterfactual style characteristics while maintaining the original content; the formal expression is:

[0020]

[0021] Among them, δ and θ are the sampled style embeddings calculated by two independent fully connected layers, and μ and σ are the original view img old The mean and standard deviation of img new Represents a counterfactual style view.

[0022] Furthermore, in step S3, the visual encoder encodes the RGB view and depth map in the counterfactual style view and the original view respectively. The encoder used for the RGB view is the ResNet50 network pre-trained on ImageNet; for the depth map, the ResNet50 encoder trained for the point-goal navigation task is used; for the natural language instructions received by the navigation agent, Bi-LSTM is used as the instruction encoder to obtain text features and keep track of word meanings.

[0023] Furthermore, in step S4, when performing spatial semantic analysis, objects in the environment are classified into 101 types, of which 100 are the most frequently appearing in natural language instructions and the environment, and the rest are classified as "others".

[0024] Further, in step S4, the current spatial layout is defined as a set S = {P i |1≤i≤K}, where K is the number of object types in the current scene. For each P i , which is defined as P i ={p i,j |1≤j≤K}, which represents the object category c i The context distribution of each p i,j =p(c j |c i ) indicates that in the current observation, the object category c j Able to be in category c iThe probability of surrounding observations, where 0≤i,j,K≤101, the context distribution follows the Dirichlet distribution; for the object layout in the scene, the specific calculation process is:

[0025] First, the prior parameters for each object category The initialization value will vary depending on the environment in which the navigation agent is located;

[0026] Second, the navigation agent transforms the received visual observation O into a count vector V, where V i Corresponding to the object category c in the current observation i The number of occurrences is implemented by the object detection network; then the likelihood function is calculated for the object category c i , whose corresponding context observation O i Follows a multinomial distribution, that is, O i ~Mult(V i- ,P i ), where V i- Indicates the category count V i The count of the object itself is further excluded from the calculation; then, the object category c is calculated based on Bayesian inference i The posterior distribution of is calculated as:

[0027]

[0028] Since the navigation agent iterates at time step t, the posterior distribution at time t will be sent to time t+1 as the prior distribution for calculation, so as to continuously update the layout knowledge;

[0029] Finally, we need to measure the layout deviation ΔG, which is expressed as the KL divergence between the posterior distribution and the prior distribution. It is used to evaluate the difference between the current environment and the prior knowledge. It is formally expressed as:

[0030]

[0031] Among them, the function KL(·) represents the calculation of KL-divergence, and tanh(·) represents the tanh activation function;

[0032] The prior knowledge refers to the previous knowledge or experience about the problem domain that is already possessed before learning or reasoning; including but not limited to the expertise of experts in the corresponding field, the results of previous research, rules or common sense knowledge. Prior parameters are the initial estimates or probability distributions of model parameters before considering the data.

[0033] Furthermore, in step S5, it is necessary to eliminate the layout deviation from the prior knowledge, and the calculation formula is:

[0034]

[0035] Among them, a represents the fact prediction, represents the counterfactual prediction, δ represents the threshold hyperparameter adjusted relative to the layout bias, TCNE(a) represents the causal reasoning module that debiased the factual prediction, and ΔG is the layout bias.

[0036] Furthermore, the navigation agent does different things during causal analysis during training and inference:

[0037] During the training phase, the navigation agent first initializes the prior parameters of the object layout by randomly exploring the environment. Subsequently, in each training cycle, the navigation agent receives visual observations and converts them into count vectors of object categories. Based on these observations and prior parameters, the navigation agent uses Bayesian inference to update the posterior distribution of the object layout. The navigation strategy of the navigation agent is optimized through reinforcement learning to maximize the cumulative reward of reaching the goal.

[0038] During the reasoning phase, the navigation agent calculates the layout deviation between the current environment and the training environment. The layout deviation is normalized by the KL divergence between the posterior distribution and the prior distribution combined with the tanh function. According to the size of the layout deviation, the navigation agent adjusts its prediction to eliminate potential deviations caused by prior knowledge; among them, the current environment refers to the environment where the navigation agent is currently located, and the training environment refers to the environment of the navigation agent during the training period.

[0039] Furthermore, in step S6, the navigation agent uses an action set including four actions, namely, moving forward 0.25m, turning left 15°, turning right 15° and stopping, and the navigation agent navigates in a realistic and continuous environment; the maximum number of moving steps is a hyperparameter that can be adjusted according to actual conditions.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking are implemented.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking.

[0042] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0043] 1. The method uses a training strategy that is mainly based on reinforcement learning and supplemented by imitation learning. It also uses data aggregation technology to iteratively integrate the decision data of the navigation agent in the environment and fuse it with historical data sets to enhance the agent's adaptability to the environment and the accuracy of its decisions, so as to ensure that the navigation agent can make decisions based on accumulated experience. In reinforcement learning, the navigation agent trains the navigation strategy by maximizing the reward in the action; in imitation learning, the action sequence is optimized by using a teacher forcing strategy with inflection weighting to adapt to repetitive actions in long sequences.

[0044] 2. By giving the navigation agent cognitive thinking ability in navigation tasks, its performance is significantly improved, and a new research idea is introduced to the VLN-CE community. The technical solution of the present invention introduces advanced causal analysis technology to address the limitations of existing navigation agents in interpretability and causal interpretation, aiming to build a more transparent and efficient navigation system. Specifically, the navigation agent first enhances its in-depth understanding of spatial semantics and improves the adaptability of the knowledge base by acquiring observation data from a counterfactual perspective. Then, by strategically intervening in the decision-making process of the navigation agent, it ensures that the navigation prediction is more in line with the actual environment after eliminating bias. Finally, the navigation agent adopts an iterative "observation-action" thinking mechanism to continuously optimize the navigation results of each step and effectively identify the optimal solution space of the task. Through empirical evaluation and detailed analysis, the performance of the navigation agent of the present invention has been fully verified.

[0045] 3. In addition, since the navigation agent conducts in-depth causal analysis before making each decision, this feature is highly similar to the human decision-making process, making the reasoning process of visual language navigation closer to reality. This design not only enhances the interpretability of decisions, but also significantly improves the generalization ability of the navigation agent, enabling it to make more accurate and reasonable judgments in different situations.

[0046] 4. The invention has low hardware requirements and good algorithm performance. Its core causal analysis process not only forms the basis for implementation, but also enables the technology to be easily integrated into other comprehensive algorithm software systems. Therefore, it can be integrated into existing unmanned intelligent devices such as driverless cars and smart home robots as a modular component, thereby enhancing the functions of these embodied robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention.

[0048] Figure 2 It is the overall framework structure diagram of the method of the present invention.

[0049] FIG. 3 is a diagram of causal reasoning navigation adjustment analysis in the method of the present invention.

[0050] Figure 4 It is an architecture diagram involving training and reasoning in the method of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is described in detail through the accompanying drawings and specific embodiments, aiming to show its basic concept and operation process. The schematic diagrams provided in the accompanying drawings are only used to assist understanding and do not represent the actual product size or limit the scope of the patent. In the absence of conflict, the features in the embodiments can be combined with each other to explore more implementation possibilities. Those skilled in the art should understand that in order to more clearly convey the concept of the present invention, some parts may be omitted, enlarged or reduced in the drawings, and known structures and their descriptions may be omitted.

[0052] Please refer to the attached figure for details. Figure 1 and Figure 2 This embodiment relates to a continuous visual language navigation method based on causal reasoning and cognitive thinking. The detailed steps are as follows:

[0053] S1. Obtain the natural language instructions for the current navigation task, and capture the input view of the current location as the original view. The instructions for this step are issued before the navigation step begins. They can be either detailed or roughly described. In particular, when the description is rough, it is more difficult for the navigation agent to navigate, and the navigation agent needs to have good reasoning ability. In this embodiment, there is no limit on the degree of detail of the instruction description; the original view here needs to be actively obtained, which mainly depends on the current environment location. The original views obtained at different locations are different.

[0054] S2. Based on the original view obtained in S1, a counterfactual style view with different styles is generated through the style counterfactual adjustment module. This step is to enable the navigation agent to learn the essential spatial semantic relationship in different view styles, rather than only focusing on inefficient visual associations such as object size and model in the view. In this step, the specific process of the style counterfactual adjustment module is as follows: 1) View image encoding: The view image is encoded using a convolutional neural network (CNN) architecture to ensure the effective extraction of image content features. The introduction of residual connections enhances the deep learning ability of the network and improves the generalization performance of the navigation agent on new data; 2) Style feature sampling: The style embedding is obtained by sampling from a multivariate normal distribution based on the PBN (Painter By Numbers) dataset. This process ensures that the generated style features are consistent with the target style. This method implies a counterfactual reasoning, that is, the navigation agent can infer the corresponding style features in the absence of direct reference; 3) Style embedding integration: The style embedding is combined with the content features through the conditional instance normalization method, so that the generated image shows the counterfactual style features while maintaining the original content. This step theoretically supports the application of counterfactual thinking, that is, the navigation agent can simulate a view image with a specific style without direct reference. The formal expression is:

[0055] Among them, δ and θ are the sampled style embeddings calculated by two independent fully connected layers, and μ and σ are the original view img old The mean and standard deviation of img new Represents a counterfactual style view.

[0056] S3. Compare the counterfactual style view generated in S2 with the original view Figure 1 The natural language instructions obtained in S1 are input into the visual encoder to learn visual features, and the text features are extracted through the text encoder; the visual encoder encodes the RGB view using the ResNet50 network pre-trained on ImageNet, and the depth map uses the ResNet50 encoder trained by the point-goal navigation task, while the natural language instructions are obtained through the Bi-LSTM encoder to obtain low-level instruction features and keep track of the word meaning.

[0057] S4. Combined with causal reasoning methods, Figure 3a and Figure 3b The entire analysis process of the navigation agent is explained in detail in the form of a cause-effect diagram, as follows:

[0058] Figure 3aThe fact prediction process of causal reasoning is shown. In fact prediction, the action prediction of the current time step depends not only on the current navigation agent input, but also on the prior knowledge (also called "experience") learned by the navigation agent, represented by the variable E. It is the most important part of the entire causal analysis, because the input data is only related to the current time step, while the experience is formed in a cumulative form. Therefore, the causal relationship chain O→A and I→A represent the current exploration-based effect, while E→A represents the experience-based effect. In addition, the formation of experiential knowledge is due to the navigation agent's learning of the input data, so the relationship chain O→E and I→E are inevitable.

[0059] Figure 3b Demonstrate the counterfactual analysis process of causal reasoning. First, an intervention operation is performed on the input data Observation and Instruction. The implementation of this operation is mainly achieved by assigning a certain value to the corresponding variable. Specifically, in this embodiment, two schemes are set for this intervention value: a zero vector or a random vector. This intervention operation weakens the influence of the observation value O and the instruction I on the subsequent action prediction to a certain extent, and will also change the value of the intermediate variable (ie, E); secondly, because it is necessary to explore the influence of experience E on action prediction, the invariance of E is maintained when performing counterfactual operations; finally, after reasoning operations such as intervention and counterfactual operations, the navigation agent will predict the counterfactual action.

[0060] After the above cause-effect diagram analysis, this embodiment explains how to represent an environment layout and measure the differences between environment layouts as follows:

[0061] This embodiment focuses on the objects in the environment and their relative spatial relationships, and uses contextual relationships to represent the object layout. Similar to "Language and visual entity relationship graph for agent navigation[J].

[38] ", the objects in the space are classified into 101 types, of which 100 are the most frequent in instructions and environments, and the remaining rare object types are classified as "others". In a scene, the object layout can be defined as a set S = {P i |1≤i≤K}, where K is the number of object types in the current scene. For each P i , which is defined as P i ={p i,j |1≤j≤K}, which represents the object category c i The context distribution of each p i,j =p(c j |c i) indicates that in the current observation, the object category c j Can be in category c i The probability of surrounding observations (where 0≤i,j,K≤101) is obtained, and the context distribution follows the Dirichlet distribution.

[0062] For the object layout in the scene, the calculation process is as follows: First, the prior parameters of each category are calculated. The initialization value will vary depending on the environment in which the navigation agent is located; secondly, the navigation agent converts the received visual observation O into a count vector V, where V i Corresponding to the object category c in the current observation i The number of occurrences can be implemented by related object detection networks (including Faster R-CNN and Mask R-CNN). Then the likelihood function is calculated. For category c i , whose corresponding context observation O i Follows a multinomial distribution, that is, O i ~Mult(V i- ,P i ), where V i- Indicates the category count V i Its own count is further excluded from ; then, the category c is calculated based on Bayesian inference i The posterior distribution of is calculated as:

[0063]

[0064] It should be noted that since the navigation agent is iteratively calculated at time step t, the posterior distribution at time t will be sent to time t+1 as the prior distribution for calculation, thereby continuously updating the layout knowledge; finally, the layout deviation ΔG of the environment is calculated, which is expressed as the KL divergence between the posterior distribution and the prior distribution, which is used to evaluate the difference between the current environment and the prior knowledge, and can be expressed in the form of:

[0065]

[0066] The function KL(·) represents the calculation of KL-divergence, and tanh(·) represents the tanh activation function.

[0067] S5: By removing the layout deviation with negative effects from the original factual prediction a∈A, the final action decision is made more in line with the current situation. In fact, this part requires the joint support of the environment layout difference and counterfactual prediction, because when the environment layout gap is small, the counterfactual prediction of causal analysis through experience E should not be harmful to the final decision, and a part of it can be retained to maintain the positive effect of experience knowledge on the current prediction; on the contrary, when the environment layout gap is large, the experience knowledge that has been learned at this time cannot be fully applied to the current environment, and some negative effects need to be removed. Formally, the debiasing process of prior knowledge can be expressed as:

[0068]

[0069] in, δ represents the threshold hyperparameter for adjusting the bias term based on the relative environment layout difference, and TCNE(a) represents the causal reasoning module for debiasing fact prediction. Furthermore, the encoded text features and visual features are combined, and the debiased prior knowledge and softmax function are used to predict the action of the navigation agent at the current time step. The action set is: {move forward 0.25m, turn left 15°, turn right 15°, stop}.

[0070] S6: Repeat the above steps until the navigation agent algorithm determines that the navigation task is completed or the maximum number of moving steps is reached, where the maximum number of moving steps is a predetermined hyperparameter that can be adjusted according to actual conditions.

[0071] This embodiment uses a training strategy that is mainly based on reinforcement learning and supplemented by imitation learning. In reinforcement learning, the navigation agent trains the navigation strategy by maximizing the reward in the action. In imitation learning, the action sequence is optimized by using a teacher forcing strategy with inflection weighting to adapt to repetitive actions in long sequences; in addition, the integrated use of data aggregation technology ensures that the navigation agent can make decisions based on accumulated experience.

[0072] What the navigation agent does during training and during inference is different in terms of what it does during causal analysis (e.g. Figure 4As shown in the figure): During the training phase, the navigation agent first initializes the prior parameters of the object layout by randomly exploring the environment. Subsequently, in each training cycle, the navigation agent receives visual observations and converts them into count vectors of object categories. Based on these observations and prior parameters, the navigation agent uses Bayesian inference to update the posterior distribution of the object layout, and the navigation strategy of the navigation agent is optimized through reinforcement learning to maximize the cumulative reward for reaching the goal; during the reasoning phase, the navigation agent calculates the layout deviation between the current environment and the training environment. The layout deviation is normalized by the KL divergence between the posterior distribution and the prior distribution and combined with the tanh function. According to the size of the layout deviation, the navigation agent adjusts its prediction to eliminate potential deviations caused by experience.

[0073] Preferably, in order to comprehensively evaluate the performance of the navigation agent, this embodiment uses the VLN-CE dataset to conduct experiments in the Habitat simulator, which includes 16,844 path-instruction pairs distributed in 90 scenes, including 5,611 unique paths. Specifically, the experimental process involves using the train split, val-seen split, and val-unseen split in the dataset. The experimental process mainly focuses on the following four performance evaluation indicators:

[0074] 1) TL: represents the trajectory length of the agent’s navigation path (in meters);

[0075] 2) NE: represents the average distance between the agent’s final position and the goal (in meters);

[0076] 3) SR: indicates the ratio of agents stopping within 3 meters of the target range (navigation is considered successful as long as the agent is within 3 meters of the target range);

[0077] 4) SPL: Path-weighted success rate that measures both navigation accuracy and efficiency.

[0078] In the implementation method, a GPU server equipped with multiple GetForce RTX 3090 graphics cards is used for parallel computing to improve data processing efficiency. In order to optimize the performance of the navigation agent in visual and language navigation tasks, the Adam optimizer was selected, and the learning rate of 2.5e-4 and the batch size of 5 were set. In addition, an inflection point weight coefficient of 3.2 was introduced to promote the navigation agent to make more diverse decisions in complex environments. The training of the navigation agent lasted 45 cycles, with an evaluation every 5 cycles, and finally the best navigation agent was selected from the unseen data in the validation set for analysis.

[0079] This embodiment is compared with the current classic and popular strong baseline navigation agent in terms of performance, and the results are shown in Table 1. As can be seen from the table, on the Val-Seen split, except for the TL indicator, the navigation agent does not achieve the best performance, and the navigation agent is 3% higher than the best performance in the baseline method in terms of SR and 2% SPL. The reason for the relatively poor TL indicator may be that the complexity of the navigation agent causes the navigation agent to consider success more when navigating and ignores those paths that have short trajectories but may fail in navigation. On the Val-Unseen split dataset, the navigation agent improves the navigation performance by 2% SR and 2% SPL relative to the best comparison method ERG when the TL and NE indicators are relatively good, which fully demonstrates the effectiveness of the navigation agent. In a horizontal comparison, although the ranking of the navigation agent in the NE indicator on Val-Unseen lags behind that on Val-Seen, it still performs well among the compared methods, which means that the navigation agent still has a very strong generalization ability.

[0080] In addition to the above comparison, an ablation experiment is also conducted on the algorithm proposed in this embodiment. For specific results, please refer to Table 2. It can be seen from the table that whether it is to adjust the style counterfactually on the view or to directly perform causal reasoning on the navigation prediction, the cognitive thinking ability of the navigation agent can be improved, and the causal reasoning adjustment directly before the navigation prediction has a better effect than the counterfactual adjustment on the view, which further illustrates that direct operation can improve the performance of the navigation agent more than indirect influence, and when the two act at the same time, the performance improvement of the navigation agent is maximized.

[0081] Table 1. Performance comparison. Bold fonts indicate the best value of the current indicator, "↓" indicates the lower the better, and "↑" indicates the higher the better.

[0082]

[0083] Table 2. Ablation study: The impact of different modules and design choices on the proposed navigation agent on the Val-Seen and Val-Unseen datasets. The best results are shown in bold. “↓” means lower is better, “↑” means higher is better

[0084]

[0085] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps in the continuous visual language navigation method based on causal reasoning and cognitive thinking in the above embodiments, and the electronic device specifically includes the following contents:

[0086] Processor, memory, communications interface and bus;

[0087] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0088] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the continuous visual language navigation method based on causal reasoning and cognitive thinking in the above embodiment are implemented.

[0089] Preferably, an embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking in the above-mentioned embodiment, and a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements all the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking in the above-mentioned embodiment.

[0090] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0092] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0095] It should be stated that the positional relationships and detailed descriptions shown in the accompanying drawings are intended to provide an exemplary illustration of the technical solution, so as to facilitate the understanding of the core concepts and operating principles of the invention. These examples are not intended to limit the scope of protection of the patent, but rather serve as an auxiliary means of explaining the technical solution. Within the framework of the present invention, ordinary technicians can, based on the prior art knowledge and the guidance of the present invention, through logical analysis, reasoning and experiments, appropriately modify, replace or innovate the technical solution to adapt to different application scenarios and needs. These changes and adjustments, as long as they do not deviate from the basic principles and spirit of the invention, should be deemed to be within the scope of protection defined in the claims.

[0096] The detailed description of this embodiment shows many aspects of the technical solution, but they do not exclude other possible implementation forms. Ordinary technicians can make various forms of specific changes based on the enlightenment of the present invention without departing from the scope of protection of the invention and the claims, including but not limited to adjustments in hardware configuration, software implementation, system architecture, operating procedures, etc. These changes are intended to optimize the performance, efficiency or reliability of the technical solution, or expand its scope of application, and all belong to the scope of protection of the present invention.

[0097] In addition, the terms and expressions used in the present invention are not intended to restrictively define the technical solution, but to provide clarity and consistency in the description. In the description, the use of terms is intended to facilitate the understanding and description of the technical content, and should not be regarded as a limitation on the scope of patent protection. Therefore, any reasonable interpretation and application of the terms, as long as it is consistent with the purpose of the invention and the essence of the claims, should be included in the scope of protection of the present invention.

[0098] References:

[0099] [1]Hong Y,Wu Q,Qi Y,et al.Vln bert:A recurrent vision-and-languagebert for navigation[C] / / Proceedings of the IEEE / CVF conference on ComputerVision and Pattern Recognition.2021:1643-1653.

[0100] [2]Magassouba A,Sugiura K,Kawai H.CrossMap transformer:A crossmodalmasked path transformer using double back-translation for vision-and-languagenavigation[J].IEEE Robotics and Automation Letters,2021,6(4):6258-6265.

[0101] [3]Zhang Y,Tan H,Bansal M.Diagnosing the environment bias in vision-and-language navigation[J].arXiv preprint arXiv:2005.03086,2020.

[0102] [4]Chen S,Guhur P L,Schmid C,et al.History aware multimodaltransformer for vision-and-language navigation[J].Advances in neuralinformation processing systems,2021,34:5834-5847.

[0103] [5]Li X,Wang Z,Yang J,et al.KERM:Knowledge Enhanced Reasoning forVision-and-Language Navigation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:2583-2592.

[0104] [6]Wu W,Chang T,Li X,et al.Vision-language navigation:A survey andtaxonomy[J].Neural Computing and Applications,2024,36(7):3291-3316.

[0105] [7]Gu J,Stefani E,Wu Q,et al.Vision-and-language navigation:A surveyof tasks,methods,and future directions[J].arXiv preprint arXiv:2203.12667,2022.

[0106] [8]Chang A,Dai A,Funkhouser T,et al.Matterport3d:Learning from rgb-ddata in indoor environments[J].arXiv preprint arXiv:1709.06158,2017.

[0107] [9]Anderson P,Wu Q,Teney D,et al.Vision-and-language navigation:Interpreting visually-grounded navigation instructions in real environments[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2018:3674-3683.

[0108]

[10] Ku A,Anderson P,Patel R,et al.Room-across-room:Multilingualvision-and-language navigation with dense spatiotemporal grounding[J].arXivpreprint arXiv:2010.07954,2020.

[0109]

[11] Ni H,Chen J,Zhu D Y,et al.A Cross-Modal Object-Aware Transformerfor Vision-and-Language Navigation[C] / / 2022 IEEE 34th InternationalConference on Tools with Artificial Intelligence(ICTAI).IEEE,2022:976-981.

[0110]

[12] Li J,Tan H,Bansal M.Envedit:Environment editing for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2022:15407-15417.

[0111]

[13] Tan H,Yu L,Bansal M.Learning to navigate unseen environments:Backtranslation with environmental dropout[J].arXiv preprint arXiv:1904.04195,2019.

[0112]

[14] Ma C Y,Wu Z,AlRegib G,et al.The regretful agent:Heuristic-aidednavigation through progress estimation[C] / / Proceedings of the IEEE / CVFconference on Computer Vision and Pattern Recognition.2019:6732-6740.

[0113]

[15] Zhou X,Liu W,Mu Y.Rethinking the spatial route prior in vision-and-language navigation[J].arXiv preprint arXiv:2110.05728,2021.

[0114]

[16] Zhu F,Zhu Y,Chang X,et al.Vision-language navigation with self-supervised auxiliary reasoning tasks[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2020:10012-10022.

[0115]

[17] Fried D,Hu R,Cirik V,et al.Speaker-follower models for vision-and-language navigation[J].Advances in Neural Information Processing Systems,2018,31.

[0116]

[18] Hong Y,Rodriguez C,Qi Y,et al.Language and visual entityrelationship graph for agent navigation[J].Advances in Neural InformationProcessing Systems,2020,33:7685-7696.

[0117]

[19] Krantz J,Wijmans E,Majumdar A,et al.Beyond the nav-graph:Vision-and-language navigation in continuous environments[C] / / Computer Vision–ECCV2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,PartXXVIII 16.Springer International Publishing,2020:104-120.

[0118]

[20] Krantz J,Gokaslan A,Batra D,et al.Waypoint models forinstruction-guided navigation in continuous environments[C] / / Proceedings ofthe IEEE / CVF International Conference on Computer Vision.2021:15162-15171.

[0119]

[21] Hong Y,Wang Z,Wu Q,et al.Bridging the gap between learning indiscrete and continuous environments for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:15439-15449.

[0120]

[22] Georgakis G,Schmeckpeper K,Wanchoo K,et al.Cross-modal maplearning for vision and language navigation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2022:15460-15470.

[0121]

[23] Irshad M Z,Ma C Y,Kira Z.Hierarchical cross-modal agent forrobotics vision-and-language navigation[C] / / 2021 IEEE InternationalConference on Robotics and Automation(ICRA).IEEE,2021:13238-13246.

[0122]

[24] He Z,Wang L,Li S,et al.MLANet:Multi-Level Attention Network withSub-instruction forContinuous Vision-and-Language Navigation[J].arXivpreprint arXiv:2303.01396,2023.

[0123]

[25] Wang T,Wu Z,Yao F,et al.Graph based Environment Representationfor Vision-and-Language Navigation in Continuous Environments[J].arXivpreprint arXiv:2301.04352,2023.

[0124]

[26] Wang Z,Li X,Yang J,et al.Gridmm:Grid memory map for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision.2023:15625-15636.

[0125]

[27] Hong Y,Zhou Y,Zhang R,et al.Learning navigational visualrepresentations with semantic map supervision[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:3055-3067.

[0126]

[28] Zubair Irshad M,Chowdhury Mithun N,Seymour Z,et al.SASRA:Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-LanguageNavigation in Continuous Environments[J].arXiv e-prints,2021:arXiv:2108.11945.

[0127]

[29] Pearl J.Causal inference[J].Causality:objectives and assessment,2010:39-58.

[0128]

[30] Pearl J.Causal inference in statistics:An overview[J].2009.

[0129]

[31] Yao L,Chu Z,Li S,et al.A survey on causal inference[J].ACMTransactions on Knowledge Discovery from Data(TKDD),2021,15(5):1-46.

[0130]

[32] Liu Y,Li G,Lin L.Cross-modal causal relational reasoning forevent-level visual question answering[J].IEEE Transactions on PatternAnalysis and Machine Intelligence,2023.

[0131]

[33] Abbasnejad E,Teney D,Parvaneh A,et al.Counterfactual vision andlanguage learning[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2020:10044-10054.

[0132]

[34] Wang H,Liang W,Shen J,et al.Counterfactual cycle-consistentlearning for instruction following and generation in vision-languagenavigation[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:15471-15481.

[0133]

[35] Fu T J,Wang X E,Peterson M F,et al.Counterfactual vision-and-language navigation via adversarial path sampler[C] / / Computer Vision–ECCV2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,PartVI 16.Springer International Publishing,2020:71-86.

[0134]

[36] Parvaneh A,Abbasnejad E,Teney D,et al.Counterfactual vision-and-language navigation:Unravelling the unseen[J].Advances in Neural InformationProcessing Systems,2020,33:5296-5307.

[0135]

[37] Asano H,Yonetani R,Nishimura M,et al.Counterfactual FairnessFilter for Fair-Delay Multi-Robot Navigation[J].arXiv preprint arXiv:2305.11465,2023.

[0136]

[38] Hong Y,Rodriguez C,Qi Y,et al.Language and visual entityrelationship graph for agent navigation[J].Advances in Neural InformationProcessing Systems,2020,33:7685-7696.

[0137]

[39] Irshad M Z,Mithun N C,Seymour Z,et al.Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuousenvironments[C] / / 2022 26th International Conference on Pattern Recognition(ICPR).IEEE,2022:4065-4071.

[0138]

[40] Raychaudhuri S,Wani S,Patel S,et al.Language-aligned waypoint(law)supervision for vision-and-language navigation in continuousenvironments[J].arXiv preprint arXiv:2109.15207,2021.

Claims

1. A continuous visual language navigation method based on causal reasoning and cognitive thinking, characterized in that: The following steps are involved: S1. Obtain the natural language instructions of the current navigation task, and capture the view of the current position of the navigation agent as the original view; S2. Based on the original view obtained in S1, a counterfactual style view different from the original view style is generated through a style counterfactual adjustment module; S3. Input the counterfactual style view and the original view generated in S2 into the visual encoder to learn visual features, and extract text features from the natural language instructions obtained in S1 through the text encoder; S4. Perform spatial semantic analysis on the current environment, extract the current spatial layout, and calculate the layout deviation between the current spatial layout and the prior knowledge in combination with the prior knowledge. Then, use the causal reasoning mechanism to eliminate the layout deviation and obtain the debiased prior knowledge. S5. Combine the encoded text features and visual features and use the debiased prior knowledge to predict the action of the navigation agent at the current time step; S6. Repeat the above steps until the navigation agent determines that the navigation task is completed or the maximum number of moving steps is reached.

2. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: The specific workflow of the style counterfactual adjustment module in step S2 is: First, the view image is encoded and the original view is encoded using a convolutional neural network architecture to ensure the effective extraction of the original view visual features; Secondly, style feature sampling is performed to obtain style embedding by sampling from a multivariate normal distribution based on the PBN dataset; Finally, the style embedding integration is completed. The style embedding is combined with the visual features through the conditional instance normalization method, so that the generated counterfactual style view can show the counterfactual style characteristics while maintaining the original content; the formal expression is: Among them, δ and θ are the sampled style embeddings calculated by two independent fully connected layers, and μ and σ are the original view img old The mean and standard deviation of img new Represents a counterfactual style view.

3. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: In step S3, the visual encoder encodes the RGB view and depth map in the counterfactual style view and the original view respectively. The encoder used for the RGB view is the ResNet50 network pre-trained on ImageNet; for the depth map, the ResNet50 encoder trained by the point-goal navigation task is used; for the natural language instructions received by the navigation agent, Bi-LSTM is used as the instruction encoder to obtain text features and keep track of word meanings.

4. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: In step S4, when performing spatial semantic analysis, objects in the environment are classified into 101 types, of which 100 are the most frequently appearing in natural language instructions and the environment, and the rest are classified as "others".

5. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: In step S4, the current spatial layout is defined as a set S = {P i |1≤i≤K}, where K is the number of object types in the current scene. For each P i , which is defined as P i ={p i，j |1≤j≤K}, which represents the object category c i The context distribution of each p i，j = p(c j |c i ) indicates that in the current observation, the object category c j Able to be in category c i The probability of surrounding observations, where 0≤i, j, K≤101, and the context distribution follows the Dirichlet distribution; for the object layout in the scene, the specific calculation process is: First, a priori parameters are given for each object category The initialization value will vary depending on the environment in which the navigation agent is located; Second, the navigation agent transforms the received visual observation O into a count vector V, where V i Corresponding to the object category c in the current observation i The number of occurrences is implemented by the object detection network; then the likelihood function is calculated for the object category c i , whose corresponding context observation O i Follows a multinomial distribution, that is, O i ~Mult(V i -, P i ), where V i- Indicates the category count V i The count of the object itself is further excluded from the calculation; then, the object category c is calculated based on Bayesian inference i The posterior distribution of is calculated as: Since the navigation agent iterates at time step t, the posterior distribution at time t will be sent to time t+1 as the prior distribution for calculation, so as to continuously update the layout knowledge; Finally, we need to measure the layout deviation ΔG, which is expressed as the KL divergence between the posterior distribution and the prior distribution. It is used to evaluate the difference between the current environment and the prior knowledge. It is formally expressed as: Among them, the function KL(·) represents the calculation of KL-divergence, and tanh(·) represents the tanh activation function; Prior knowledge refers to the previous knowledge or experience about the problem domain that is already possessed before learning or reasoning; including but not limited to the expertise of experts in the corresponding field, the results of previous research, rules or common sense knowledge.

6. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: In step S5, it is necessary to eliminate the layout deviation from the prior knowledge, and the calculation formula is: Among them, a represents the fact prediction, represents the counterfactual prediction, δ represents the threshold hyperparameter adjusted relative to the layout bias, TCNE(a) represents the causal reasoning module that debiased the factual prediction, and ΔG is the layout bias.

7. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: The navigation agent does different things during causal analysis during training and inference: During the training phase, the navigation agent first initializes the prior parameters of the object layout by randomly exploring the environment. Subsequently, in each training cycle, the navigation agent receives visual observations and converts them into count vectors of object categories. Based on these observations and prior parameters, the navigation agent uses Bayesian inference to update the posterior distribution of the object layout. The navigation strategy of the navigation agent is optimized through reinforcement learning to maximize the cumulative reward of reaching the goal. During the reasoning phase, the navigation agent calculates the layout deviation between the current environment and the training environment. The layout deviation is normalized by the KL divergence between the posterior distribution and the prior distribution combined with the tanh function. According to the size of the layout deviation, the navigation agent adjusts its prediction to eliminate potential deviations caused by prior knowledge; among them, the current environment refers to the environment where the navigation agent is currently located, and the training environment refers to the environment of the navigation agent during the training period.

8. The continuous visual language navigation method based on causal reasoning and cognitive thinking according to claim 1 is characterized in that: In step S6, the navigation agent uses an action set including four actions, namely moving forward 0.25m, turning left 15°, turning right 15° and stopping, and the navigation agent navigates in a realistic and continuous environment; the maximum number of moving steps is a hyperparameter that can be adjusted according to actual conditions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the continuous visual language navigation method based on causal reasoning and cognitive thinking as described in any one of claims 1 to 8 are implemented.

Images