Visual language navigation and visual language navigation model training method and device, equipment and medium

Abstract

The invention provides a visual language navigation and model training method, device, equipment and medium, and the method comprises the steps: obtaining a training sample, processing a to-be-executed instruction and environment information in the training sample through a core model, obtaining a navigation feature corresponding to the to-be-executed instruction, and inputting the navigation feature into an action prediction head. Obtaining predicted action information corresponding to the navigation features, and inputting the navigation features into a thinking chain generation head to obtain predicted thinking chain information corresponding to the navigation features; and then, based on the predicted action information, the reference action information in the training sample, the predicted thinking chain information and the reference thinking chain information in the training sample, training the core model, the action prediction head and the thinking chain generation head so as to take the trained core model and the trained action prediction head as a visual language navigation model. According to the method, the reasoning efficiency and the reasoning accuracy can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of visual language navigation technology, and in particular to a visual language navigation method, apparatus, device, and medium for training its model. Background Technology

[0002] Vision-and-Language Navigation (VLN) integrates natural language processing, computer vision, multimodal information, and robot navigation. Its core is training an intelligent agent to understand instructions and navigate to a target location in a real or simulated visual environment.

[0003] In related technologies, visual language navigation tasks require robots to make real-time decisions in complex environments based on natural language instructions. Before making a final decision, visual language navigation models explicitly generate intermediate reasoning processes through thought chain reasoning techniques, which often result in low reasoning efficiency and accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a visual language navigation method, apparatus, device, and medium for training its model.

[0005] According to a first aspect of the present disclosure, a visual language navigation model training method is provided, comprising:

[0006] Acquire training samples, which include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed;

[0007] By processing the instructions to be executed and environmental information through the core model, the navigation features corresponding to the instructions to be executed are obtained;

[0008] The navigation features are input into the action prediction head to obtain the predicted action information corresponding to the navigation features;

[0009] Input the navigation features into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features;

[0010] Based on predicted action information, reference action information, predicted thought chain information, and reference thought chain information, the core model, action prediction head, and thought chain generation head are trained.

[0011] The trained core model and action prediction head are used as a visual language navigation model to predict the actions of the agent.

[0012] According to a second aspect of the present disclosure, a visual language navigation model training apparatus is provided, comprising:

[0013] The first processing module is configured to acquire training samples, which include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed.

[0014] The second processing module is configured to process the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed.

[0015] The third processing module is configured to input navigation features into the action prediction head to obtain predicted action information corresponding to the navigation features.

[0016] The fourth processing module is configured to input navigation features into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features;

[0017] The fifth processing module is configured to train the core model, action prediction head, and thought chain generation head based on predicted action information, reference action information, predicted thought chain information, and reference thought chain information.

[0018] The sixth processing module is configured to use the trained core model and action prediction head as a visual language navigation model to predict the actions of the agent.

[0019] According to a third aspect of the present disclosure, a visual language navigation method is provided, comprising:

[0020] Determine the instructions to be executed and the environment information;

[0021] A visual language navigation model is used to predict actions based on the instructions to be executed and environmental information to obtain predicted action information; wherein, the visual language navigation model is trained using any of the above-mentioned visual language navigation model training methods;

[0022] The agent is controlled to move based on predicted action information.

[0023] According to a fourth aspect of the present disclosure, a visual language navigation device is provided, comprising:

[0024] The tenth processing module is configured to determine the instructions to be executed and the environment information;

[0025] The eleventh processing module is configured to use a visual language navigation model to predict actions based on the instructions to be executed and environmental information, thereby obtaining predicted action information; wherein, the visual language navigation model is trained using any of the aforementioned visual language navigation model training devices.

[0026] The twelfth processing module is configured to control the agent's movement based on predicted action information.

[0027] A fifth aspect of this disclosure provides an electronic device, comprising:

[0028] Memory, used to store computer programs;

[0029] A processor is configured to execute a computer program stored in a memory, wherein, when the computer program is executed, it implements the method of any embodiment of the visual language navigation model training method of the first aspect of this disclosure.

[0030] A sixth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any embodiment of the visual language navigation model training method of the first aspect described above.

[0031] A seventh aspect of this disclosure provides a computer program including computer-readable code, wherein the computer program instructions, when executed by a processor, implement the method of any embodiment of the visual language navigation model training method of the first aspect described above.

[0032] Based on the visual language navigation and its model training method, apparatus, device, and medium provided in the above embodiments of this disclosure, training samples (including instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed) can be obtained. Then, the core model processes the instructions to be executed and the environmental information to obtain navigation features corresponding to the instructions to be executed. Then, the navigation features are input into the action prediction head to obtain predicted action information corresponding to the navigation features, and the navigation features are input into the thought chain generation head to obtain predicted thought chain information corresponding to the navigation features. Subsequently, the core model, action prediction head, and thought chain generation head are trained based on the predicted action information, reference action information, predicted thought chain information, and reference thought chain information. Finally, the trained core model and action prediction head are used as a visual language navigation model for action prediction of intelligent agents. Therefore, using predicted action information and corresponding reference action information obtained from navigation features for model training helps improve the accuracy of action prediction results of the trained visual language navigation model in visual language navigation scenarios. Furthermore, simultaneously using predicted action information and predicted thought chain information obtained from navigation features, along with corresponding reference action information and reference thought chain information, for model training, and using the reference thought chain information as an auxiliary signal to guide the model's internalized reasoning pattern during training, allows the trained visual language navigation model to avoid explicit reasoning and generate a clear reasoning process. Instead, it directly predicts actions based on the internalized reasoning learned during training, reducing reasoning overhead and improving the reasoning efficiency of the visual language navigation model. Moreover, since action prediction does not require the aid of thought chain information, it helps reduce the influence of reasoning illusions caused by thought chain information, further improving the accuracy of action prediction in visual language navigation scenarios.

[0033] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0034] Figure 1 This is an exemplary scenario diagram to which this disclosure applies.

[0035] Figure 2 This is a schematic diagram of the first process of a visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0036] Figure 3 This is a schematic diagram of a second process of a visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0037] Figure 4 This is a schematic diagram of a third process for training a visual language navigation model provided in an exemplary embodiment of this disclosure.

[0038] Figure 5 This is a schematic diagram of the fourth process of the visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0039] Figure 6 This is a fifth flowchart illustrating the visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0040] Figure 7 This is a sixth flowchart illustrating a visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0041] Figure 8 This is a seventh flowchart illustrating a visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0042] Figure 9 This is a schematic diagram of a first type of visual language navigation method provided in an exemplary embodiment of this disclosure.

[0043] Figure 10 This is a schematic diagram of a second flowchart of a visual language navigation method provided in an exemplary embodiment of this disclosure.

[0044] Figure 11 This is a schematic diagram of the training process of the visual language navigation model in the visual language navigation model training method provided in an exemplary embodiment of this disclosure.

[0045] Figure 12 This is a schematic diagram illustrating the reasoning process of the visual language navigation model in the visual language navigation method provided by an exemplary embodiment of this disclosure.

[0046] Figure 13 This is a schematic diagram of the first structure of a visual language navigation model training device provided in an exemplary embodiment of the present disclosure.

[0047] Figure 14 This is a schematic diagram of a second structure of a visual language navigation model training device provided in an exemplary embodiment of this disclosure.

[0048] Figure 15 This is a schematic diagram of the first structure of a visual language navigation device provided in an exemplary embodiment of the present disclosure.

[0049] Figure 16 This is a schematic diagram of a second structure of a visual language navigation device provided in an exemplary embodiment of the present disclosure.

[0050] Figure 17 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0051] To explain this disclosure, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0052] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0053] It should be noted that the acquisition, storage, transmission, and other processing of data (such as instructions to be executed and environmental information) in this disclosure comply with relevant laws, regulations, social ethics, and provisions concerning public interests. For example, the acquisition, storage, transmission, and other processing of data in this disclosure are all carried out with the knowledge and consent of the users and have obtained full authorization from the relevant users.

[0054] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0055] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0056] Application Overview

[0057] In visual language navigation tasks, robots need to make real-time decisions in complex environments based on natural language instructions. Before generating a final decision, thought chain reasoning techniques, which incorporate intermediate reasoning processes, have successfully enhanced reasoning capabilities in various visual language-driven tasks. However, the role of reasoning in VLN tasks has not been fully explored.

[0058] In developing this disclosure, the inventors discovered that existing reasoning strategies (such as Pre-Think and Post-Think) actually lead to poor robot performance during the testing phase due to the intermediate reasoning process, causing navigation failures. This phenomenon can be termed "reasoning collapse." For example, errors or illusions during reasoning can lead to incorrect predicted actions; mismatch between training and reasoning can cause error accumulation along the trajectory and cascade navigation failures. This results in low reasoning efficiency and accuracy.

[0059] Exemplary Overview

[0060] This disclosure utilizes predicted action information and corresponding reference action information obtained from navigation features during the training of a visual language navigation model. This helps improve the accuracy of action prediction results in visual language navigation scenarios. Furthermore, by simultaneously using predicted action information and predicted thought chain information obtained from navigation features, along with corresponding reference action information and reference thought chain information, for model training, and using the reference thought chain information as an auxiliary signal to guide the model's internalized reasoning pattern during training, the trained visual language navigation model can avoid explicit reasoning and generate a clear reasoning process. Instead, it directly predicts actions based on internalized reasoning learned during training, reducing reasoning overhead and improving the reasoning efficiency of the visual language navigation model. Moreover, since action prediction does not rely on thought chain information, it helps reduce the influence of reasoning illusions caused by thought chain information, further improving the accuracy of action prediction in visual language navigation scenarios.

[0061] The visual language navigation model trained according to the embodiments of this disclosure can be applied to any visual language navigation scenario in which an intelligent agent navigates based on language instructions and visual information. Figure 1 This is an exemplary scenario diagram to which this disclosure applies. Figure 1 In this context, the intelligent agent 100 can be a physical or virtual electronic device such as a robot, drone, or vehicle that has command recognition (e.g., natural language command recognition), environmental information acquisition (e.g., environmental image acquisition), and navigation functions. For example, the intelligent agent 100 can be equipped with a visual language navigation model.

[0062] The following example illustrates how an agent 100 acquires an instruction to be executed (e.g., a natural language instruction) and environmental information (e.g., a first image acquired at a first moment and at least one second image sequence acquired at a second moment, the second moment being before the first moment).

[0063] First, after the agent 100 determines the instruction to be executed and the environmental information, the agent 100 can use the visual language navigation model to predict the action based on the instruction to be executed and the environmental information, and obtain the predicted action information.

[0064] Afterwards, agent 100 can control the agent to move based on the predicted action information.

[0065] For example, if the instruction to be executed determined by agent 100 is "Please walk from the entrance to the kitchen and then get me the milk from the refrigerator," and the environmental information is "the environmental image collected by agent 100 when receiving the above instruction, and the environmental image collected before receiving the above instruction," then agent 100 can use a visual language navigation model to predict the action based on the above instruction and the above environmental information, thereby obtaining the predicted action information "walk to the kitchen, walk towards the refrigerator, open the refrigerator, and take out the milk." Next, agent 100 can execute the first action in the action sequence represented by the predicted action information, namely, "walk to the kitchen."

[0066] Optionally, during the execution of the action represented by the predicted action information corresponding to the instruction to be executed, the agent 100 can continuously collect environmental information and generate new predicted action information accordingly, thereby adjusting its unexecuted actions until the aforementioned instruction to be executed is completed.

[0067] Exemplary methods

[0068] Figure 2 This is a schematic flowchart of a first embodiment of a visual language navigation model training method provided by this disclosure. This embodiment can be applied to any electronic device, such as a terminal device, server, cloud, etc. The aforementioned terminal device can be, for example, a computer, vehicle, robot, or robotic vacuum cleaner capable of supporting visual language navigation model training. Figure 2 As shown, this embodiment includes the following steps:

[0069] Step 210: Obtain training samples. The training samples include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed.

[0070] Training samples can provide use cases and supervision information for training the visual language navigation model. For example, training samples can provide use cases such as "instructions to be executed" and "environmental information" for training the visual language navigation model, and can provide supervision information such as "reference action information" and "reference thought chain information" for training the visual language navigation model.

[0071] The instruction to be executed can be a natural language description that guides the agent to complete a navigation task, and it can serve as the target basis for action decisions in visual language navigation tasks. As an example, the instruction to be executed can be "go to the kitchen" or "approach the refrigerator".

[0072] Environmental information can be visual data reflecting the navigation environment state of the agent, such as images. This can include single-frame images, multi-frame image sequences, etc. Visual data can be used to determine spatial layout, object distribution, obstacle positions, and other information. In specific implementations, visual data can be acquired using visual sensors (e.g., cameras) configured on the agent. Environmental information can include the current environmental image captured by the agent at the first moment corresponding to the instruction to be executed, and can further include visual environmental images captured during historical time periods prior to this first moment. The first moment can be within a preset time window (e.g., 0.1 seconds) based on the moment the agent receives the instruction to be executed.

[0073] Action information can represent actions (or sequences of actions) that match the instruction to be executed and environmental information. For example, if the instruction to be executed is "move forward to the desk," the action information could be "move forward 0.8 meters, stop"; if the instruction to be executed is "turn left towards the window," the action information could be "turn left 90 degrees." In some cases, action information can represent navigation actions, such as moving forward, turning left, turning right, and stopping.

[0074] Reference action information can be standard action information that matches the instruction to be executed and the environmental information, and it can be determined by manual or automatic annotation.

[0075] The thought chain information can be structured reasoning information that matches the instruction to be executed and environmental information. It can make implicit reasoning processes explicit. As an example, if the instruction to be executed is "go back to the living room from the balcony", the thought chain information could be "the instruction is 'go back to the living room from the balcony', the balcony door is currently in front of us, historical images show that the door connects to the living room and there are no obstacles → we should move 0.5 meters toward the door".

[0076] Reference thought chain information can be standard thought chain information that matches the instructions to be executed and environmental information, and it can be determined by manual or automatic annotation.

[0077] Step 220: Process the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed.

[0078] The core model can be the core processing unit of the visual-language navigation model to be trained. It is used to encode, extract features, and fuse features from the instructions to be executed and environmental information, thereby generating navigation features that include the relationship between the two and their respective characteristics. The input data to the core model can be the instructions to be executed and environmental information, or the processing results obtained from processing these instructions and environmental information. The output data of the core model can be the navigation features corresponding to the instructions to be executed.

[0079] Navigation features can be feature vectors or feature matrices that incorporate the fusion and correlation between the semantics of the instruction to be executed and environmental information. For example, if the instruction to be executed is "Go to the kitchen and find the milk in the refrigerator," and the environmental information is "the current visual observation image, such as an image captured when currently in the living room; and navigation history observation images, such as images captured at trajectory points from the living room to the kitchen and images inside the refrigerator during historical navigation," the core model can process the above instruction to be executed and environmental information to obtain navigation features that instruct the agent to go from its current location in the living room to the kitchen, open the refrigerator, and take out the milk.

[0080] Step 230: Input the navigation features into the action prediction head to obtain the predicted action information corresponding to the navigation features.

[0081] The action prediction head can be a model structure that outputs predicted actions (or action sequences) based on navigation features. For example, the action prediction head can contain fully connected layers and activation functions. The input data of the action prediction head can be navigation features. The output data of the action prediction head can be the predicted action information corresponding to the navigation features.

[0082] Action information can represent a single action or a sequence of actions arranged in a temporal order. An action sequence includes actions at least one time step. For example, action information can represent distance traveled, turning angle, etc.

[0083] Predicted action information can be action information determined by an action prediction head. Predicted action information can represent a single predicted action or a sequence of predicted actions consisting of multiple actions. The predicted action sequence includes predicted actions at at least one time step.

[0084] Step 240: Input the navigation features into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features.

[0085] The thought chain generator head can be a model structure that outputs a predicted thought chain (description of the reasoning process) based on navigation features. The input data of the thought chain generator head can be navigation features, and the output data can be the predicted thought chain information corresponding to the navigation features.

[0086] Thought chain information can be used to explain the model's reasoning process when executing a specific instruction, helping to understand how the model combines the instruction with environmental information to make corresponding decisions. Thought chain information can include at least one of the following: a predicted navigation trajectory point sequence, or predicted thought process information. The navigation trajectory point sequence includes navigation trajectory points at at least one time step.

[0087] Predicting thought chain information can be the thought chain information output by the thought chain generator.

[0088] Step 250: Train the core model, action prediction head, and thought chain generation head based on predicted action information, reference action information, predicted thought chain information, and reference thought chain information.

[0089] In some optional implementations of this embodiment, the SGD (Stochastic Gradient Descent) optimizer can be used, with the joint loss of action loss (difference between predicted action information and reference action information) and thought chain loss (difference between predicted thought chain information and reference thought chain information) as the objective, to iteratively adjust the model parameters of the core model, action prediction head, and thought chain generation head; alternatively, the Adam (Adaptive Moment Estimation) optimizer can be used, with a learning rate decay strategy (e.g., decaying to 0.9 of the original every 10 rounds), to minimize the joint loss through batch training and validation set tuning, thereby completing the training of the core model, action prediction head, and thought chain generation head.

[0090] Step 260: The trained core model and action prediction head are used as a visual language navigation model to predict the actions of the agent.

[0091] Among them, the visual language navigation model can be a model that integrates visual perception and language understanding capabilities, and outputs navigation actions based on the instructions to be executed and environmental information.

[0092] An intelligent agent can be an intelligent entity with the ability to perceive, make decisions and execute, and can be a physical entity (such as a robot) or a virtual entity (such as a navigation role in a simulation environment).

[0093] Thus, the agent can collect environmental information (such as the current frame image) and receive instructions to be executed in real time, and input the environmental information and instructions to be executed into the visual language navigation model. The visual language navigation model outputs the next single action (such as "move forward 1 meter") or multiple future actions. After the agent executes one or more actions, it repeatedly collects environmental information, and the visual language navigation model outputs the next single action (such as "move forward 1 meter") or multiple future actions based on the latest collected environmental information and instructions to be executed, until the instructions to be executed are completed.

[0094] In some optional implementations of this embodiment, during training, thought chain reasoning can be used as auxiliary supervision, while explicit reasoning is avoided during the reasoning process, resulting in a visual language navigation agent with auxiliary reasoning capabilities. This agent takes received natural language navigation instructions as instructions to be executed, uses visual observations from the environment (e.g., images) as environmental information, and determines the next navigation action (e.g., move forward, turn left, turn right, stop) based on this information, thereby obtaining predicted action information. Thus, a thought chain generation head is integrated into the agent's decision-making process as an "Aux-Think" module. This module does not directly output predicted action information but assists the action prediction head, which is the main decision-making module, in performing deeper understanding, prediction, or planning, thereby improving the agent's generalization ability and navigation success rate in complex instructions and unknown environments, and reducing dependence on large-scale labeled training data.

[0095] In this embodiment, training samples (including instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed) can be obtained. Then, the core model processes the instructions to be executed and the environmental information to obtain navigation features corresponding to the instructions to be executed. Then, the navigation features are input into the action prediction head to obtain the predicted action information corresponding to the navigation features, and the navigation features are input into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features. Subsequently, the core model, action prediction head, and thought chain generation head are trained based on the predicted action information, reference action information, predicted thought chain information, and reference thought chain information. Finally, the trained core model and action prediction head are used as a visual language navigation model for action prediction of the intelligent agent. Therefore, using predicted action information and corresponding reference action information obtained from navigation features for model training helps improve the accuracy of action prediction results of the trained visual language navigation model in visual language navigation scenarios. Furthermore, simultaneously using predicted action information and predicted thought chain information obtained from navigation features, along with corresponding reference action information and reference thought chain information, for model training, and using the reference thought chain information as an auxiliary signal to guide the model's internalized reasoning pattern during training, allows the trained visual language navigation model to avoid explicit reasoning and generate a clear reasoning process. Instead, it directly predicts actions based on the internalized reasoning learned during training, reducing reasoning overhead and improving the reasoning efficiency of the visual language navigation model. Moreover, since action prediction does not require the aid of thought chain information, it helps reduce the influence of reasoning illusions caused by thought chain information, further improving the accuracy of action prediction in visual language navigation scenarios.

[0096] In some optional implementations of this embodiment, the training samples may further include a reference navigation trajectory point sequence and reference thinking process information. Inputting navigation features into the thought chain generation head yields a predicted navigation trajectory point sequence and predicted thinking process information corresponding to the navigation features. Therefore, step 250 may include: training the core model, action prediction head, and thought chain generation head based on predicted action information, reference action information, predicted navigation trajectory point sequence, reference navigation trajectory point sequence, predicted thinking process information, and reference thinking process information. As an example, the difference between the predicted thinking process information and the reference thinking process information can be calculated using the cross-entropy loss function, and the difference between the reference navigation trajectory point sequence and the predicted navigation trajectory point sequence can be calculated to obtain the thought chain loss; simultaneously, the difference between the predicted navigation trajectory point sequence and the reference navigation trajectory point sequence can be calculated using the mean squared error loss function as the navigation trajectory loss; then, combined with the action loss, the three losses are weighted and summed to form the final joint loss. Updating the model parameters through the backpropagation algorithm gradually approximates the reference information, thereby improving the model's inference ability and the accuracy of navigation actions.

[0097] The navigation trajectory point sequence may include multiple trajectory points that match the instructions to be executed and environmental information, arranged in the order of arrival during navigation.

[0098] The predicted navigation trajectory point sequence can be a navigation trajectory point sequence predicted by the thought chain generator head.

[0099] The reference navigation trajectory point sequence can be a standard navigation trajectory point sequence that matches the command to be executed and environmental information, and it can be marked and determined manually or automatically.

[0100] The thought process information represents the thought process of an intelligent agent that is matched with the instructions to be executed and environmental information.

[0101] Predictive thinking process information can be information about the thinking process predicted by the thinking chain generator.

[0102] Reference thought process information can be standard thought process information that matches the instructions to be executed and environmental information, and it can be identified through manual or automatic annotation.

[0103] In some alternative embodiments, such as Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 210 may include the following steps:

[0104] Step 2110: Obtain the instruction to be executed at the first moment.

[0105] The first moment can be the moment corresponding to the instruction to be executed. For example, the first moment can be the start time of the instruction to be executed, the end time of the instruction to be executed, or the intermediate moment between the start time and the end time of the instruction.

[0106] In some cases, the first moment can be taken as the current moment.

[0107] In some cases, user voice commands can be received through the voice interaction module of an electronic device, thereby identifying the user voice command as an instruction to be executed, and the moment when the instruction to be executed is received is identified as the first moment.

[0108] Step 2120: Obtain a first image acquired at a first time and at least one second image sequence acquired at a second time, wherein the second time is prior to the first time.

[0109] The first image can be environmental information acquired at the first moment and represented in the form of an image. The first image can be used to reflect the real-time environmental state synchronized with the instruction to be executed.

[0110] The second moment can be a point in time preceding the first moment, and it can serve as a temporal basis for obtaining historical environmental information. The second moment can be different for each second image in the second image sequence.

[0111] The second image sequence may contain multiple second images arranged in chronological order of acquisition time.

[0112] As an example, a visual sensor mounted on an intelligent agent can capture a first image in real time at the first moment, and at the same time read a second image sequence captured before the first moment from a memory or cloud.

[0113] Step 2130: Use the first image and the second image sequence as environmental information.

[0114] The environmental information in the training samples of this embodiment includes historical trajectories. In this way, the trained visual language navigation model can perceive environmental information more completely, reduce position misjudgment, and improve the accuracy of action prediction.

[0115] Optionally, environmental information can be determined by combining the first image at the first moment and a third image sequence (predicted short-term images) taken after the first moment for a preset time (e.g., 1 second). Thus, the first image is captured in real-time by a camera, and environmental changes (the third image sequence) are predicted based on the first image over a preset time, with both used as environmental information. This allows the model to anticipate short-term environmental changes (e.g., "an obstacle will appear in 1 second"), thereby improving the foresight of action prediction.

[0116] In some alternative embodiments, such as Figure 4 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 220 may include the following steps:

[0117] Step 2210: The instruction to be executed is encoded through the instruction encoding module in the core model to obtain the instruction code.

[0118] The instruction encoding module is the core model structure used to encode the instructions to be executed. It can convert the instructions to be executed into a machine-understandable form (instruction encoding) and extract key semantic information (such as keywords like "balcony" and "shelf") from the instructions.

[0119] Instruction encoding can be data output by the instruction encoding module that contains key semantic information of the instruction to be executed.

[0120] In some optional implementations of this embodiment, the instruction encoding module can be implemented using a BERT (Bidirectional Encoder Representations from Transformers) architecture. After word segmentation and embedding layer processing of the instruction to be executed, the instruction encoding can be output through the Transformer encoder. Alternatively, the instruction encoding module can also be implemented using an LSTM (Long Short-Term Memory) architecture. After performing temporal modeling on the word embedding sequence of the instruction to be executed, the instruction encoding is output.

[0121] Step 2220: Encode the environmental information using the visual encoding module in the core model to obtain the visual encoding.

[0122] The visual encoding module can be a model structure in the core model used to encode the visual features of environmental information (e.g., images / image sequences). It can be used to convert images in environmental information into a machine-understandable form (visual encoding), for example, extracting spatial features (such as object shape, position, and color) and temporal features (such as the motion trajectory of image sequences) from images.

[0123] Visual encoding can be data output by a visual encoding module that contains key visual features from environmental information.

[0124] In some optional implementations of this embodiment, the visual encoding module can be implemented using a ResNet (Residual Neural Network) architecture, which outputs visual encoding after performing convolution and pooling processing on the image in the environmental information; or, the visual encoding module can also be implemented using a ViT (Vision Transformer) architecture, which outputs visual encoding after performing block embedding and Transformer encoding on the environmental information.

[0125] Step 2230: The instruction encoding and visual encoding are processed by the feature extraction module in the core model to obtain the navigation features corresponding to the instruction to be executed.

[0126] The feature extraction module can be a model structure within the core model used to fuse instruction encoding and visual encoding to generate navigation features. The input data for the feature extraction module can be instruction encoding and visual encoding, and the output data can be the navigation features corresponding to the instruction to be executed.

[0127] In some optional implementations of this embodiment, the feature extraction module can be implemented using a cross-attention mechanism, where the instruction encoding is used as the query, the visual encoding is used as the key and value, and the attention weights are calculated and then fused to generate navigation features; or, the feature extraction module can directly concatenate the instruction encoding and the visual encoding, and output navigation features through a fully connected layer and layer normalization processing.

[0128] In this embodiment, the core model processing is split into an instruction encoding module and a visual encoding module. Dedicated modules are set up to better suit their respective processing objectives, making the semantic extraction of instruction encoding more accurate and the spatial features of visual encoding more complete. In this way, the fused navigation features contain more accurate information on the association between instructions and the environment, thereby improving the prediction accuracy of the downstream prediction head.

[0129] Alternatively, the instructions to be executed and environmental information can be combined and input into the core model to directly obtain the navigation features corresponding to the instructions to be executed. This simplifies the model structure of the core model, thereby reducing computational costs and improving the processing efficiency of the core model.

[0130] In some alternative embodiments, such as Figure 5 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 250 may include the following steps:

[0131] Step 2510: Determine the first loss value based on the predicted action information and the reference action information.

[0132] The first loss value can be a loss value calculated based on the difference between the predicted action information and the reference action information. It can be used to measure the deviation between the action prediction result and the standard action, thereby evaluating the accuracy of the action prediction head output.

[0133] In some optional implementations of this embodiment, methods such as cross-entropy loss and mean squared error loss can be used to determine the first loss value based on the predicted action information and the reference action information.

[0134] Step 2520: Determine the second loss value based on the predicted thought chain information and the reference thought chain information.

[0135] The second loss value can be a loss value calculated based on the difference between the predicted thought chain information and the reference thought chain information. It can be used to measure the deviation between the predicted thought chain and the standard reasoning logic, thereby assessing the accuracy of the thought chain generation head output.

[0136] In some optional implementations of this embodiment, methods such as cross-entropy loss and mean squared error loss can be used to determine the second loss value based on the predicted thought chain information and the reference thought chain information.

[0137] Step 2530: Train the core model, action prediction head, and thought chain generation head based on the first loss value and the second loss value.

[0138] In some optional implementations of this embodiment, the first loss value and the second loss value can be weighted and summed according to weights (e.g., action loss weight 0.6, thought chain loss weight 0.4) to obtain the total loss. The Adam optimizer is then used to minimize the total loss, and the model parameters are iteratively adjusted. Alternatively, the total loss value can be determined first based on the first loss value and the second loss value. Then, based on the total loss value, the core model, action prediction head, and thought chain generation head are trained. The total loss is obtained by directly summing the first loss value and the second loss value. The training is iteratively trained until the total loss converges, with the goal of minimizing the total loss.

[0139] In this embodiment, the training loss is based on both the first loss value (action loss) and the second loss value (inference loss), which can more accurately measure the deviation between action prediction and inference logic, avoid the one-sidedness of a single loss, avoid neglecting one aspect for another, and make the loss calculation more accurate.

[0140] Alternatively, a Bayesian model can be used to train the core model, action prediction head, and thought chain generation head by predicting action information, referencing action information, predicting thought chain information, and referencing thought chain information. This allows for the introduction of uncertainty modeling, further enhancing the model's generalization ability and robustness in complex scenarios, thus better adapting to diverse task requirements.

[0141] In some alternative embodiments, such as Figure 6 As shown above, in the above Figure 2 Based on the illustrated embodiment, after step 220, the following steps may also be included:

[0142] Step 310: Reconstruct the instructions to be executed by inputting navigation features into the instruction reconstruction head to obtain the reconstructed instructions.

[0143] The instruction reconstruction head can be a model structure that reconstructs the instruction to be executed based on navigation features. The input data of the instruction reconstruction head can be navigation features, and the output data can be the reconstructed instruction.

[0144] In some optional implementations of this embodiment, the instruction refactoring header may include a Transformer decoder and a fully connected layer.

[0145] Reconstruction instructions can be instructions output from the instruction reconstruction header or instructions generated in reverse based on navigation features.

[0146] In some optional implementations of this embodiment, the instruction reconstruction head can adopt an autoregressive generation method, starting from the first word, predicting and generating reconstruction instructions word by word based on the generated words and navigation features (e.g., first predicting "go", then predicting "yang", and finally predicting "tai"); or, it can adopt a non-autoregressive generation method, predicting the probability distribution of all words simultaneously based on navigation features, and selecting the combination of words with the highest probability as the reconstruction instructions.

[0147] Based on this, step 250 may include the following steps:

[0148] Step 251: Based on the predicted action information, reference action information, predicted thought chain information, reference thought chain information, reconstructed instructions and instructions to be executed, adjust the parameters of the core model, action prediction head, thought chain generation head and instruction reconstruction head.

[0149] In some optional implementations of this embodiment, a Bayesian model can be used to adjust the parameters of the core model, action prediction head, thought chain generation head, and instruction reconstruction head by predicting action information, referencing action information, predicting thought chain information, referencing thought chain information, reconstructing instructions, and instructions to be executed. Therefore, by introducing a Bayesian model, the correlation between various tasks can be modeled during training, thereby better utilizing shared information between tasks and improving the model's generalization ability and robustness.

[0150] In this embodiment, by adding an instruction reconstruction head, a reverse supervision task from navigation features to instructions is introduced. This allows the model to deduce the instructions to be executed from the navigation features, facilitating a deeper understanding of which environmental features are strongly correlated with instruction semantics, thereby improving the model's semantic alignment ability. Furthermore, the reverse supervision data provides the model with more comprehensive semantic supervision, enabling it to accurately associate instruction semantics based on features even when faced with changes in instruction expression (e.g., "go to the balcony" becomes "walk to the balcony") or "local environmental changes" (e.g., minor adjustments to the balcony door position), thus improving the model's generalization ability.

[0151] In some alternative embodiments, such as Figure 7 As shown above, in the above Figure 6 Based on the illustrated embodiment, step 251 may include the following steps:

[0152] Step 2511: Determine the first loss value based on the predicted action information and the reference action information.

[0153] Step 2512: Determine the second loss value based on the predicted thought chain information and the reference thought chain information.

[0154] For the implementation of steps 2511 and 2512 above, please refer to steps 2510 and 2520 above respectively, and will not be repeated here.

[0155] Step 2513: Determine the third loss value based on the reconstruction instructions and the instructions to be executed.

[0156] The third loss value can be a loss value calculated based on the difference between the reconstructed instruction and the instruction to be executed. It can measure the semantic deviation between the reconstructed instruction and the original instruction, thereby evaluating the accuracy of the instruction reconstruction header output.

[0157] In some optional implementations of this embodiment, methods such as cross-entropy loss and mean squared error loss can be used to determine the third loss value based on the reconstruction instruction and the instruction to be executed.

[0158] Step 2514: Based on the first loss value, the second loss value, and the third loss value, train the core model, the action prediction head, the thought chain generation head, and the instruction reconstruction head.

[0159] In some optional implementations of this embodiment, the first loss value, the second loss value, and the third loss value can be weighted and summed according to weights (e.g., action loss weight 0.4, thought chain loss weight 0.3, instruction loss weight 0.3) to obtain the total loss, and the Adam optimizer can be used to minimize the total loss and iteratively adjust the model parameters; or, the first loss value, the second loss value, and the third loss value can be directly summed to obtain the total loss, and iterative training can be performed until the total loss converges with the goal of minimizing the total loss.

[0160] In some optional implementations of this embodiment, such as Figure 11 As shown, Aux-Think utilizes reasoning specifically through auxiliary tasks during training. During training, two reasoning-based auxiliary tasks were designed, including a thought chain reasoning task and an action reasoning task, as well as a primary action-based task, namely, an instruction reasoning task.

[0161] The three tasks mentioned above will be described in detail below:

[0162] Task 1: Scrolling time-domain motion planning.

[0163] Receding-Horizon Action Planning is introduced as the main task. During training, the model calculates the target instruction I and the current visual observation o. t and navigation history observation O t Predict the next n action sequences (a t a t+1 , ..., a t+n-1 This setup encourages short-term predictions while maintaining responsiveness to new observations. The training objective for each trajectory τ is defined as:

[0164]

[0165] In formula (1), This represents the first loss value. This represents the actual action or reference action information at future time step t+k. t Indicates the current visual observation, O t This indicates that the navigation history observations, current visual observations, and navigation history observations together constitute the aforementioned environmental information. τ This represents the instruction to be executed corresponding to trajectory τ.

[0166] Task 2: CoT-based inference.

[0167] The given natural language instruction I10 is used as the instruction to be executed, and the current visual observation o is used. t 11 and navigation history observation O t 12 serves as environmental information. The model is trained to generate CoT (Chain-of-Thought) inference trajectories under the aforementioned instructions to be executed and environmental information, and uses these trajectories as information for predicting thought chains. This facilitates the learning of structured reasoning patterns and strengthens the connection between language, vision, and action. The CoT-based inference loss for each trajectory τ is:

[0168]

[0169] In formula (2), This represents the second loss value, with the base of the logarithm being e, and o. t Indicates the current visual observation, O t This represents navigation history observations. The combination of current visual observations and navigation history observations can serve as the aforementioned environmental information. τ This represents the instruction to be executed corresponding to trajectory τ. This represents the actual reasoning trajectory at time step t, i.e., the reference thought chain information.

[0170] Task 3: Instruction-based reasoning.

[0171] Given a visual observation sequence The model is trained to reconstruct the corresponding instruction I. This inverse reasoning task provides supplementary supervision beyond the CoT signal, further enriching the model's semantic alignment capabilities. Its training loss is:

[0172]

[0173] In formula (3), This represents the third loss value, with the base of the logarithm being e. I represents the sequence of visual observations, i.e., environmental information. τ This represents the instruction to be executed corresponding to trajectory τ.

[0174] During training, three tasks were jointly trained, and the tasks were switched by changing the cues. The final loss function is:

[0175]

[0176] In formula (4), L represents the total loss, and D represents the set of training trajectories. This represents the first loss value. This represents the second loss value. This represents the third loss value.

[0177] Therefore, during the training process, the natural language instruction I10 is first input as the instruction to be executed into the instruction encoding module 14, and the navigation history observation O is then processed. t 12 is input to sampling processing module 13, and the sampling result is compared with the current visual observation. t11 is used as environmental data input to the visual encoding module 15. Then, the output data from both the instruction encoding module 14 and the visual encoding module 15 are input to the core VLN model 16 for feature extraction, obtaining the output representation of navigation features. Afterwards, task switching is achieved through the task switching module 17 by switching prompts. For example, the prompt could be, "You are an intelligent navigation robot. You have received the following image sequence and a navigation instruction. Please carefully consider the next action you should take and express your thought process in natural language." Guided by the prompt, the model generates multi-sentence reasoning text containing spatial cognition, instruction analysis, and path judgment. The generated results possess natural language features, often including expressions such as "Let me think," "I see... therefore I think...", reflecting a step-by-step analysis and judgment process.

[0178] In the instruction inference task, navigation features are input to the instruction reconstruction head 18 to obtain reconstructed instructions, and instruction loss 19 is calculated based on the reconstructed instructions and the instructions to be executed to obtain the third loss value. In the thought chain inference task, navigation features are input to the thought chain generation head 20 to obtain predicted thought chain information, and CoT loss 21 is calculated based on the predicted thought chain information and the reference thought chain information to obtain the second loss value. In the action inference task, navigation features are input to the action prediction head 22 to obtain predicted action information, and action loss 23 is calculated based on the predicted action information and the reference action information to obtain the first loss value.

[0179] Here, to ensure standardization and task separation in subsequent training, the generated content can be encapsulated in the following structure:

[0180] <think> (Inference Text)< / think>

[0181] <answer> (Action Tag)< / answer>

[0182] Alternatively, adjustments can be made as needed, retaining only [the original text]. <think>Partially used for inference supervision.

[0183] Optionally, heuristic rules or small models can be used to evaluate the quality of the generated text. For example, judgments can be made on language fluency, whether it contains visual keywords relevant to the environment, and whether it reasonably explains the basis for the current action decision. This helps to eliminate verbose, out-of-context, or action-inconsistent text.

[0184] Optionally, the diversity of generated samples can be expanded by modifying the prompts, switching different visual sampling strategies, and adjusting language style templates, thereby enhancing the breadth of scenarios covered by the thought chain and the richness of expression.

[0185] Next, the sum of instruction Loss19, CoT Loss21 and action Loss23 is calculated to obtain the total loss L24. Based on the total loss L24, the model parameter update operation shown in label 25 is performed on instruction encoding module 14, visual encoding module 15, core VLN module 16, instruction reconstruction head 18, thought chain generation head 20 and action prediction head 22.

[0186] Joint training across the three tasks enriches the model’s semantic foundation and decision-making capabilities, enabling it to generalize better from limited data; the inference-induced supervision signals are more closely integrated with the high-level semantic structure of the instructions, making each training example more informative; and the model’s internalized reasoning behavior, learned in CoT training, encourages broader exploration.

[0187] In this embodiment, a third loss value (instruction reconstruction loss) is introduced, which, together with the first and second loss values, constitutes a three-dimensional supervision system. This system constrains the model from both forward (action, reasoning) and backward (semantics) perspectives, resulting in more comprehensive loss supervision and more complete information learned by the model. Furthermore, the third loss value clarifies the sources of bias in the semantic backward task, allowing for targeted adjustments to the instruction reconstruction head (e.g., a large third loss value necessitates optimization of the decoder's attention layer). Simultaneously, it works in synergy with the first and second loss values ​​to optimize the core model, ensuring that parameter adjustments consider the three objectives of action, reasoning, and semantics, thereby improving the accuracy of reasoning.

[0188] In some alternative embodiments, such as Figure 8 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 220 may include the following steps:

[0189] Step 2240: Decompose the instruction to be executed into a sequence of sub-instructions by using the instruction decomposition model.

[0190] The instruction decomposition model can be a model structure used to decompose complex, long-step instructions to be executed into simple, short-step sub-instructions.

[0191] In some optional implementations of this embodiment, the instruction decomposition model can focus on decomposing complex, long-distance instructions into a series of simpler, more easily executable intermediate sub-goals or semantic milestones, resulting in a sequence of sub-instructions to be executed. For example, the instruction "Please walk from the entrance to the kitchen, and then find the milk in the refrigerator" can be decomposed into a sequence of sub-instructions to be executed: "Arrive at the kitchen," "Find the refrigerator," "Open the refrigerator," and "Find the milk." The sub-instructions in the sequence can be text descriptions or predefined semantic tags. During training, the model compares the sub-instructions with manually labeled reference sub-instructions and calculates the loss.

[0192] In some optional implementations of this embodiment, the instruction decomposition model can be implemented based on architectures such as sequence labeling, text summarization, or dialogue generation. For example, the instruction decomposition model can adopt a combination of rules and sequence labeling, first dividing steps based on preset rules (such as keywords like "passed through" and "arrived"), and then determining the sub-instruction boundaries through the BERT sequence labeling model to output the sub-sequence; or, the instruction decomposition model can also adopt a "dialogue generation" approach, inputting the instruction to be executed into the T5 model, using "Please break down the instruction into sub-steps" as the prompt, and having the model generate a sub-instruction sequence.

[0193] Step 2250: The core model is used to process the sequence of sub-instructions to be executed and the environmental information to obtain the navigation feature sequence corresponding to the sequence of sub-instructions to be executed.

[0194] The sequence of sub-instructions to be executed can be a sequence of multiple simple sub-instructions output by the instruction decomposition model, and each sub-instruction can correspond to an independent navigation step target.

[0195] The navigation feature sequence can be a sequence of multiple sub-navigation features generated by the core model based on the sequence of sub-instructions to be executed and environmental information. Each sub-navigation feature corresponds to the contextual and environmental information of a sub-instruction.

[0196] In some optional implementations of this embodiment, the core model can encode each sub-instruction separately, and simultaneously encode the corresponding sub-environment information (such as the "study-living room" environment image corresponding to the sub-instruction "walk from the study to the living room") separately, and then fuse them to generate the sub-navigation features corresponding to each sub-instruction to form a sequence; or, the core model can first perform temporal encoding on the entire sequence of sub-instructions to be executed, perform visual encoding on the complete environment information, and then fuse the corresponding regions of the temporal encoding of the sub-instructions and the environment encoding through an attention mechanism to generate a sub-navigation feature sequence.

[0197] Step 2260: Based on the navigation feature sequence, determine the navigation features corresponding to the instruction to be executed.

[0198] In some optional implementations of this embodiment, all sub-navigation features in the navigation feature sequence can be concatenated in the order of sub-instructions and mapped to a final navigation feature of a unified dimension through a fully connected layer; or, an attention fusion mechanism can be used, with the contribution of each sub-navigation feature to the final navigation target as a weight (e.g., the weight of the sub-feature "near the shelf" is higher than that of "enter the warehouse area"), and the final navigation feature can be obtained by weighted summation.

[0199] In this embodiment, a command decomposition model breaks down complex commands into sub-sequences, which in turn break down complex targets into simpler sub-targets. The core model can understand the semantics of each sub-command step by step, thereby reducing the difficulty of model understanding and improving the accuracy of semantic extraction. In addition, each sub-feature in the navigation feature sequence corresponds to the association between sub-commands and sub-environments. The core model can generate more targeted features for each sub-target (e.g., the sub-command "find meeting room 302" corresponds to the features "3rd floor corridor, 302 sign"). The integrated final navigation features contain more accurate step-by-step association information, and the actions and inference predictions of the downstream prediction head are more aligned with the step-by-step targets, thereby improving the overall navigation accuracy.

[0200] Alternatively, instead of decomposing the instruction to be executed, the core model can be used to process the instruction to be executed and the environmental information to obtain the navigation feature sequence corresponding to the instruction to be executed, thereby improving the inference efficiency of the model.

[0201] Any of the visual language navigation model training methods provided in this disclosure can be executed by any suitable device with visual language navigation model training capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the visual language navigation model training methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the visual language navigation model training methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0202] Figure 9 This is a schematic flowchart of a first embodiment of a visual language navigation method provided in this disclosure. This embodiment can be applied to intelligent agents, such as various types of robots. Figure 9 As shown, it includes the following steps:

[0203] Step 410: Determine the instruction to be executed and the environment information.

[0204] The descriptions of the instructions to be executed and the environment information are provided above and will not be repeated here.

[0205] Step 420: Use the visual language navigation model to predict actions based on the instructions to be executed and environmental information to obtain predicted action information; wherein, the visual language navigation model is trained using any of the above-mentioned visual language navigation model training methods.

[0206] The relevant description of the predicted action information can be found above, and will not be repeated here.

[0207] Step 430: Control the agent to move based on the predicted action information.

[0208] The relevant description of the intelligent agent can be found above, and will not be repeated here.

[0209] Here, after determining the predicted action information, the agent can be controlled to move according to the predicted action information. As an example, the agent can convert the predicted action information into an electrical signal that can be recognized by the agent's motion controller (such as a signal that "move forward 1 meter" corresponds to a specific number of motor rotations), and drive the agent to perform the action through the controller.

[0210] In this embodiment, the visual language navigation model trained using any of the aforementioned visual language navigation model training methods can predict actions based on the instructions to be executed and environmental information, obtaining predicted action information. Then, the agent is controlled to move based on this predicted action information. Therefore, using the predicted action information obtained based on navigation features and the corresponding reference action information for model training helps improve the accuracy of action prediction results in visual language navigation scenarios. Furthermore, simultaneously using the predicted action information and predicted thought chain information obtained based on navigation features, along with the corresponding reference action information and reference thought chain information, for model training, and using the reference thought chain information as an auxiliary signal to guide the model's internalized reasoning mode during training, allows the trained visual language navigation model to avoid explicit reasoning and generate no explicit reasoning process. Instead, it directly predicts actions based on the internalized reasoning learned during training, reducing reasoning overhead and improving the reasoning efficiency of the visual language navigation model. Moreover, since action prediction does not require the aid of thought chain information, it helps reduce the influence of reasoning illusions caused by thought chain information, further improving the accuracy of action prediction in visual language navigation scenarios.

[0211] In some optional implementations of this embodiment, step 430 may include: determining the first action in the predicted action sequence represented by the predicted action information, wherein the predicted action sequence includes predicted actions for at least one time step, and then controlling the mobile device to execute the first action. For example, when the predicted action sequence is "move forward 1 meter → turn left 90 degrees → move forward 0.5 meters", the mobile device is first controlled to execute the action of "move forward 1 meter". After executing the action of the current time step, the environmental information is reacquired, and step 420 is executed again to update the predicted action information, thereby realizing dynamic navigation control of the agent. In this way, the agent can be gradually guided to complete high-precision action execution in complex environments, improving the overall navigation flexibility and reliability.

[0212] In some alternative embodiments, such as Figure 10 As shown above, in the above Figure 9 Based on the illustrated embodiment, step 420 may include the following steps:

[0213] Step 4210: Using the core model in the visual language navigation model, based on the instruction to be executed and environmental information, obtain the navigation features corresponding to the instruction to be executed.

[0214] The specific implementation of step 4210 above can be found in the relevant description of step 220 above. To avoid repetition, it will not be repeated here.

[0215] Step 4220: Using the action prediction head in the visual language navigation model, action prediction is performed based on navigation features to obtain the predicted action sequence.

[0216] The specific implementation of step 4220 above can be found in the relevant description of step 230 above. To avoid repetition, it will not be repeated here.

[0217] In this embodiment, an action prediction head predicts actions based on navigation features to obtain a predicted action sequence. This sequence allows the agent to be controlled to move. For example, the agent can be controlled to execute the first action in the predicted action sequence, or to execute the first preset number of actions in the predicted action sequence sequentially. Therefore, multiple actions in the predicted action sequence can be combined to comprehensively control the agent to execute each action, thereby improving the agent's navigation accuracy and adaptability in complex environments.

[0218] In some optional implementations of this embodiment, see [link to relevant documentation]. Figure 12 During the inference phase, only action prediction is enabled. The model predicts the next n actions, but only executes the first one. This ensures fast, real-time responsive navigation while avoiding additional inference overhead.

[0219] Specifically, firstly, the natural language instruction I30 is input as the instruction to be executed to the instruction encoding module 31, and the navigation history observation O is... t 33 is input to the sampling processing module 34, and the sampling result is compared with the current visual observation. t 32 is used as environmental data input to the visual encoding module 35. Then, the output data of both the instruction encoding module 31 and the visual encoding module 35 are input to the pre-trained core VLN model 36 for feature extraction to obtain the output representation of navigation features. Then, based on the output representation, the action task 37 is executed to predict the next n actions. Specifically, the output representation is input to the action prediction head 38 to obtain the output action sequence a. t a t+1 a t+2 ......a t+n-1 Afterwards, the agent can perform the operation shown in label 29, that is, perform the first action a. t .

[0220] Therefore, the above-mentioned optional implementation proposes and implements the "Aux-Think" multi-task training framework, which jointly trains the active action prediction task in traditional visual-language navigation (VLN) with multiple auxiliary reasoning tasks (such as thought chain generation and instruction reconstruction). This framework goes beyond simply improving the model architecture; instead, it enhances the model's inherent reasoning ability by designing different types of supervision signals to train the model more effectively. Compared to focusing primarily on model architecture optimization or data augmentation, Aux-Think emphasizes how to guide the model to learn deeper, generalizable reasoning patterns through auxiliary tasks. The above implementation introduces "thought chain-based (CoT) reasoning" as an auxiliary training task, bringing the concept of "thought chain" from large-scale language models to the VLN domain. The model is explicitly supervised during training to generate intermediate reasoning steps or text descriptions (CoT trajectories). This allows the model to learn structured, logically clear reasoning processes, rather than simply performing end-to-end black-box mapping. Compared to the implicit reasoning process within a VLN model, the above implementation makes it explicit and supervised through Aux-Think, enabling more effective training of the model to understand complex instructions and environments. Furthermore, by introducing "instruction-based reconstruction reasoning" as an auxiliary training task, this implementation requires reconstructing the original natural language instructions from the visual observation sequence of the entire navigation trajectory. This forces the model to deeply understand the bidirectional semantic relationship between visual information and language instructions. Compared to a unidirectional (instruction-to-action) VLN task, this inverse task provides complementary supervision signals, significantly enhancing the model's visual grounding capabilities and comprehensive semantic understanding of instructions. By separating complex reasoning during training from efficient execution during inference, this implementation allows the model to learn complex thinking and understanding capabilities during training using auxiliary reasoning tasks, improving data efficiency and generalization. However, during the inference (actual navigation) phase, these auxiliary reasoning modules are disabled, and the model only performs lightweight and fast action prediction. This approach resolves the conflict between training complex models and real-time navigation efficiency, enabling the model to benefit from richer and more complex training signals without increasing online inference overhead. Furthermore, the implementation improves data efficiency and generalization ability: through this innovative multi-task assisted training strategy, Aux-Think allows the model to achieve navigation performance comparable to or better than existing methods requiring large amounts of data, even with limited data. Moreover, the deep reasoning capabilities learned by the model enable it to exhibit stronger adaptability and generalization ability when faced with unfamiliar environments and complex, novel instructions.Thus, by introducing an auxiliary reasoning mechanism, the above implementation can improve data training efficiency: achieving navigation performance comparable to or better than existing methods that require more data, using the same or less training data; enhancing command understanding: better parsing complex, multi-step, or obscure natural language navigation commands; improving environmental perception and path planning capabilities: more intelligently utilizing visual information for more effective path planning, avoiding dead ends or unnecessary detours; improving generalization ability: exhibiting stronger adaptability in unseen environments or when faced with novel command combinations; and improving navigation efficiency: in the model inference stage of practical applications, the model only performs lightweight and fast action predictions, ensuring real-time performance and navigation efficiency.

[0221] Any of the visual language navigation methods provided in this disclosure can be executed by any suitable device with visual language navigation capabilities, including but not limited to: robots, robot vacuum cleaners, vehicles, etc. Alternatively, any of the visual language navigation methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the visual language navigation methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0222] Exemplary device

[0223] Figure 13 This is a schematic diagram of a first structural embodiment of a visual language navigation model training apparatus provided in this disclosure. The visual language navigation model training apparatus of this disclosure can be used to implement the visual language navigation model training method of any of the above embodiments. Figure 13 As shown, the visual language navigation model training device in this embodiment includes:

[0224] The first processing module 510 is configured to acquire training samples, which include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed.

[0225] The second processing module 520 is configured to process the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed.

[0226] The third processing module 530 is configured to input navigation features into the action prediction head to obtain predicted action information corresponding to the navigation features.

[0227] The fourth processing module 540 is configured to input navigation features into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features;

[0228] The fifth processing module 550 is configured to train the core model, action prediction head, and thought chain generation head based on predicted action information, reference action information, predicted thought chain information, and reference thought chain information.

[0229] The sixth processing module 560 is configured to use the trained core model and action prediction head as a visual language navigation model for action prediction of the agent.

[0230] Figure 14 This is a schematic diagram of a second structure of a visual language navigation model training device provided in an exemplary embodiment of this disclosure. For example... Figure 14 As shown, in Figure 13 Based on the illustrated embodiment, in some possible implementations, the first processing module 510 may include:

[0231] The first processing submodule 5110 is configured to acquire the instruction to be executed at the first moment;

[0232] The second processing submodule 5120 is configured to acquire a first image acquired at a first time and at least one sequence of second images acquired at a second time, wherein the second time is prior to the first time.

[0233] The third processing submodule 5130 is configured to use the first image and the second image sequence as environmental information.

[0234] In some possible implementations, the second processing module 520 may include:

[0235] The fourth processing submodule 5210 is configured to encode the instruction to be executed through the instruction encoding module in the core model to obtain the instruction encoding;

[0236] The fifth processing submodule 5220 is configured to encode environmental information through the visual encoding module in the core model to obtain visual encoding;

[0237] The sixth processing submodule 5230 is configured to process the instruction encoding and visual encoding through the feature extraction module in the core model to obtain the navigation features corresponding to the instruction to be executed.

[0238] In some possible implementations, the fifth processing module 550 may include:

[0239] The seventh processing submodule 5510 is configured to determine a first loss value based on the predicted action information and the reference action information;

[0240] The eighth processing submodule 5520 is configured to determine the second loss value based on the predicted thought chain information and the reference thought chain information;

[0241] The ninth processing submodule 5530 is configured to train the core model, action prediction head, and thought chain generation head based on the first loss value and the second loss value.

[0242] In some possible implementations, the apparatus may further include:

[0243] The seventh processing module 570 is configured to process the instruction to be executed and environmental information through the core model, obtain the navigation features corresponding to the instruction to be executed, input the navigation features into the instruction reconstruction head to reconstruct the instruction to be executed, and obtain the reconstructed instruction.

[0244] The eighth processing module 580 is configured to train the core model, action prediction head, and thought chain generation head based on predicted action information, reference action information, predicted thought chain information, and reference thought chain information, including:

[0245] The ninth processing module 590 is configured to adjust the parameters of the core model, action prediction head, thought chain generation head, and instruction reconstruction head based on predicted action information, reference action information, predicted thought chain information, reference thought chain information, reconstruction instructions, and instructions to be executed.

[0246] In some possible implementations, the ninth processing module 590 may include:

[0247] The tenth processing submodule 5910 is configured to determine a first loss value based on the predicted action information and the reference action information;

[0248] The eleventh processing submodule 5920 is configured to determine the second loss value based on the predicted thought chain information and the reference thought chain information.

[0249] The twelfth processing submodule 5930 is configured to determine the third loss value based on the reconstruction instructions and the instructions to be executed;

[0250] The thirteenth processing submodule 5940 is configured to train the core model, action prediction head, thought chain generation head, and instruction reconstruction head based on the first loss value, the second loss value, and the third loss value.

[0251] In some possible implementations, the seventh processing module 570 may include:

[0252] The fourteenth processing submodule 5710 is configured to decompose the instruction to be executed into a sequence of sub-instructions to be executed by using an instruction decomposition model.

[0253] The fifteenth processing submodule 5720 is configured to process the sequence of sub-instructions to be executed and the environmental information through the core model to obtain the navigation feature sequence corresponding to the sequence of sub-instructions to be executed;

[0254] The sixteenth processing submodule 5730 is configured to determine the navigation features corresponding to the instruction to be executed based on the navigation feature sequence.

[0255] The visual language navigation model training device provided in this embodiment can execute the corresponding steps of the above-mentioned visual language navigation model training methods, thereby achieving the technical effects of the above-mentioned visual language navigation model training methods. The visual language navigation model training device and the visual language navigation model training methods can refer to and cite each other in terms of specific implementation and technical effects. For the sake of brevity, they will not be elaborated here.

[0256] Figure 15 This is a schematic diagram of a first structural embodiment of a visual language navigation device provided by this disclosure. The visual language navigation device of this disclosure can be used to implement the visual language navigation method of any of the above embodiments. Figure 15 As shown, the visual language navigation device in this embodiment includes:

[0257] The tenth processing module 610 is configured to determine the instruction to be executed and the environment information;

[0258] The eleventh processing module 620 is configured to use a visual language navigation model to predict actions based on the instructions to be executed and environmental information, thereby obtaining predicted action information; wherein the visual language navigation model is trained using any of the aforementioned visual language navigation model training devices.

[0259] The twelfth processing module 630 is configured to control the agent's movement based on predicted motion information.

[0260] Figure 16 This is a schematic diagram of a second structure of a visual language navigation device provided in an exemplary embodiment of this disclosure. For example... Figure 16 As shown, in Figure 15 Based on the illustrated embodiment, in some possible implementations, the eleventh processing module 620 may include:

[0261] The seventeenth processing submodule 6210 is configured to use the core model in the visual language navigation model to obtain the navigation features corresponding to the instruction to be executed based on the instruction to be executed and environmental information.

[0262] The eighteenth processing submodule 6220 is configured to use the action prediction head in the visual language navigation model to predict actions based on navigation features and obtain a predicted action sequence.

[0263] The visual language navigation device provided in this embodiment can execute the corresponding steps of the above visual language navigation methods, thereby achieving the technical effects of the above visual language navigation methods. The visual language navigation device and the visual language navigation methods can refer to and cite each other in terms of specific implementation and technical effects. For the sake of brevity, they will not be elaborated here.

[0264] Exemplary electronic devices

[0265] Figure 17 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0266] The processor 91 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0267] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of this disclosure described above.

[0268] In one example, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0269] The input device 93 may also include, for example, a touchscreen, a microphone, various sensors, etc. The microphone can be used to acquire commands to be executed, such as natural language navigation commands. Sensors may include, for example, image sensors (e.g., cameras, webcams, etc.). Image sensors can be used for environmental perception, i.e., acquiring environmental information. Environmental information may include a first image acquired at a first moment and a sequence of at least one second image acquired at a second moment, the second moment preceding the first moment.

[0270] The output device 94 can output various information to the outside, including, for example, a display, a speaker, a communication network and its connected remote output devices, etc.

[0271] Of course, for the sake of simplicity, Figure 17 Only some of the components of the electronic device 90 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 90 may include any other suitable components depending on the specific application.

[0272] Exemplary computer program products and computer-readable storage media

[0273] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the visual language navigation model training method or visual language navigation method according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0274] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0275] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the visual language navigation model training method or visual language navigation method according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0276] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0277] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0278] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0279] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0280] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0281] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0282] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0283] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.< / think>

Claims

1. A method for training a visual language navigation model, comprising: Acquire training samples, which include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed; The core model processes the instruction to be executed and the environmental information to obtain the navigation features corresponding to the instruction to be executed. The navigation features are input into the action prediction head to obtain the predicted action information corresponding to the navigation features; The navigation features are input into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features; Based on the predicted action information, the reference action information, the predicted thought chain information, and the reference thought chain information, the core model, the action prediction head, and the thought chain generation head are trained. The trained core model and action prediction head are used as a visual language navigation model to predict the actions of the agent.

2. The method according to claim 1, wherein obtaining training samples comprises: Obtain the instruction to be executed at the first moment; Acquire a first image captured at the first time point and at least one sequence of second images captured at a second time point, wherein the second time point is prior to the first time point; The first image and the second image sequence are used as the environmental information.

3. The method according to claim 1, wherein, The process of processing the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed includes: The instruction to be executed is encoded using the instruction encoding module in the core model to obtain the instruction encoding; The environmental information is encoded using the visual encoding module in the core model to obtain visual encoding; The feature extraction module in the core model processes the instruction encoding and visual encoding to obtain the navigation features corresponding to the instruction to be executed.

4. The method according to claim 1, wherein, The training of the core model, the action prediction head, and the thought chain generation head based on the predicted action information, the reference action information, the predicted thought chain information, and the reference thought chain information includes: Based on the predicted action information and the reference action information, a first loss value is determined; Based on the predicted thought chain information and the reference thought chain information, a second loss value is determined; The core model, the action prediction head, and the thought chain generation head are trained based on the first loss value and the second loss value.

5. The method according to claim 1, wherein, After processing the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed, the process further includes: The navigation feature input instruction reconstruction head is used to reconstruct the instruction to be executed, and the reconstructed instruction is obtained; The training of the core model, the action prediction head, and the thought chain generation head based on the predicted action information, the reference action information, the predicted thought chain information, and the reference thought chain information includes: Based on the predicted action information, the reference action information, the predicted thought chain information, the reference thought chain information, the reconstruction instruction, and the instruction to be executed, the parameters of the core model, the action prediction head, the thought chain generation head, and the instruction reconstruction head are adjusted.

6. The method according to claim 5, wherein, The step of adjusting parameters of the core model, the action prediction head, the thought chain generation head, and the instruction reconstruction head based on the predicted action information, the reference action information, the predicted thought chain information, the reference thought chain information, the reconstruction instruction, and the instruction to be executed includes: Based on the predicted action information and the reference action information, a first loss value is determined; Based on the predicted thought chain information and the reference thought chain information, a second loss value is determined; Based on the reconstruction instruction and the instruction to be executed, a third loss value is determined; Based on the first loss value, the second loss value, and the third loss value, the core model, the action prediction head, the thought chain generation head, and the instruction reconstruction head are trained.

7. The method according to claim 1, wherein, The process of processing the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed includes: The instruction to be executed is decomposed using an instruction decomposition model to obtain a sequence of sub-instructions to be executed. The core model is used to process the sequence of sub-instructions to be executed and the environmental information to obtain the navigation feature sequence corresponding to the sequence of sub-instructions to be executed. Based on the navigation feature sequence, the navigation features corresponding to the instruction to be executed are determined.

8. A visual language navigation method, comprising: Determine the instructions to be executed and the environment information; A visual language navigation model is used to predict the action of the instruction to be executed and the environmental information to obtain predicted action information; wherein, the visual language navigation model is trained using the visual language navigation model training method described in any one of claims 1-7. The agent is controlled to move based on the predicted action information.

9. The method according to claim 8, wherein, The step of using a visual language navigation model to predict actions based on the instruction to be executed and the environmental information to obtain predicted action information includes: Using the core model in the visual language navigation model, based on the instruction to be executed and the environmental information, the navigation features corresponding to the instruction to be executed are obtained; Using the action prediction head in the visual language navigation model, action prediction is performed based on the navigation features to obtain a predicted action sequence.

10. A visual language navigation model training device, comprising: The first processing module is configured to acquire training samples, which include: instructions to be executed, environmental information, and reference action information and reference thought chain information corresponding to the instructions to be executed. The second processing module is configured to process the instruction to be executed and the environmental information through the core model to obtain the navigation features corresponding to the instruction to be executed; The third processing module is configured to input the navigation features into the action prediction head to obtain the predicted action information corresponding to the navigation features. The fourth processing module is configured to input the navigation features into the thought chain generation head to obtain the predicted thought chain information corresponding to the navigation features; The fifth processing module is configured to train the core model, the action prediction head, and the thought chain generation head based on the predicted action information, the reference action information, the predicted thought chain information, and the reference thought chain information. The sixth processing module is configured to use the trained core model and action prediction head as a visual language navigation model to predict the actions of the agent.

11. A visual language navigation device, comprising: The tenth processing module is configured to determine the instructions to be executed and the environment information; The eleventh processing module is configured to use a visual language navigation model to predict the action of the instruction to be executed and the environmental information to obtain predicted action information; wherein the visual language navigation model is trained using the visual language navigation model training method described in any one of claims 1-7. The twelfth processing module is configured to control the agent to move based on the predicted action information.

12. A computer-readable storage medium storing a computer program, which, when executed, performs the visual language navigation model training method according to any one of claims 1-7, or performs the visual language navigation method according to any one of claims 8-9.

13. An electronic device, the electronic device comprising: processor; Memory used to store instructions to be executed by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the visual language navigation model training method according to any one of claims 1-7, or to execute the visual language navigation method according to any one of claims 8-9.

Images