Navigation model training method and apparatus

Abstract

The application provides a navigation model training method and device, wherein the method comprises the following steps: performing multi-task training on an initial navigation model to obtain a first navigation strategy output by the initial navigation model; training the initial navigation model according to a deviation correction action to obtain an intermediate navigation model and a second navigation strategy output by the intermediate navigation model; controlling an agent to move based on the second navigation strategy, determining an entropy value of the agent performing each navigation action, determining a candidate state sample according to the entropy value, generating a plurality of candidate action sequences corresponding to the candidate state sample, and determining a reward value corresponding to each candidate action sequence; obtaining a first expert action, calculating a cross-entropy of the navigation action based on the first expert action, determining a joint loss based on the reward value and the cross-entropy, updating model parameters according to the joint loss, and obtaining a target navigation model. The embodiment can enable the navigation model to have error recovery capability and reward-driven exploration capability, thereby improving the training efficiency and the generalization capability of the navigation model.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation, and more specifically, to a navigation model training method and apparatus. Background Technology

[0002] Visual language navigation requires embodied agents to navigate autonomously in physical environments based on natural language instructions. Currently, incorporating large-scale visual language models into navigation decision-making typically employs imitation learning or reinforcement learning training methods. However, imitation learning essentially memorizes demonstrated actions, limiting its generalization ability in unfamiliar environments. When the agent enters states unseen during the training phase during inference, errors accumulate, and the lack of proactive exploration capabilities negatively impacts the accuracy and stability of the final navigation model. Reinforcement learning, on the other hand, usually explores existing trajectories, failing to cover out-of-distribution states that actually occur during inference, and is prone to overconfidence in its output, resulting in low training efficiency. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide at least one navigation model training method, apparatus, electronic device, and storage medium, which enable the navigation model to have error recovery capability and reward-driven exploration capability, thereby improving training efficiency and generalization capability of the navigation model.

[0004] In a first aspect, embodiments of the present invention provide a navigation model training method, comprising: The initial navigation model is trained on a multi-task basis using an expert demonstration dataset and navigation video sampling frames. The first navigation strategy output by the initial navigation model after the multi-task training is obtained based on the navigation environment. The initial navigation model is trained according to the first navigation strategy and the corresponding correction action to obtain the intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction action is obtained. The agent moves in the navigation environment based on the second navigation strategy. The entropy value of each navigation action performed by the agent during the movement is determined. Candidate state samples are determined based on the entropy value, and multiple candidate action sequences corresponding to the candidate state samples are generated. The reward value corresponding to each candidate action sequence is determined. Obtain the first expert action corresponding to the state of each agent in the second navigation strategy. Calculate the cross-entropy based on the navigation action indicated by the second navigation strategy based on the first expert action. Determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy. Update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

[0005] Optionally, the initial navigation model is trained on a multi-task basis using an expert demonstration dataset and sampled frames from navigation videos to obtain the first navigation strategy output by the initial navigation model after multi-task training, based on the navigation environment, including: The navigation instructions, navigation sampling frames, and second expert actions extracted from the expert demonstration dataset are input into the initial navigation model so that the initial navigation model outputs predicted actions. The initial navigation model is then trained on the action prediction task based on the predicted actions and the second expert actions. The navigation video sampling frames are input into the initial navigation model so that the initial navigation model outputs trajectory description information based on the navigation video sampling frames, and the initial navigation model is trained on the trajectory description task based on the trajectory description information and the corresponding navigation instructions. Based on the training results of action prediction task training and trajectory description task training, the initial navigation model is jointly trained, so that the initial navigation model that has completed joint training outputs the first navigation strategy based on the navigation environment.

[0006] Optionally, the initial navigation model is trained based on the first navigation strategy and the corresponding correction action to obtain an intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction action is obtained, including: The agent is controlled to move within the navigation environment based on a first navigation strategy; In response to the agent meeting the preset correction conditions during movement, the expert network obtains the correction actions for the agent's state output. The initial navigation model is trained based on the correction actions corresponding to the states of each agent during the movement process to obtain the intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction actions is obtained.

[0007] Optionally, the reward value corresponding to each candidate action sequence is determined, including: Based on a preset sequence length threshold, the first reward of the candidate action sequence in the format dimension is calculated; Based on the reference action sequence corresponding to the candidate state sample in the second navigation strategy, calculate the second reward of the candidate action sequence in the consistency dimension. Determine the first pose information after the agent executes the candidate action sequence, and the second pose information after the agent executes the reference action sequence. Calculate the cosine similarity between the first pose information and the second pose information to obtain the third reward of the candidate action sequence in the position dimension. Determine the first displacement distance after the agent executes the candidate action sequence and the second displacement distance after the agent executes the reference action sequence. Calculate the normalized distance between the first displacement distance and the second displacement distance to obtain the fourth reward of the candidate action sequence in the distance dimension. The reward value corresponding to each candidate action sequence is determined based on the first reward, second reward, third reward, and fourth reward.

[0008] Optionally, obtaining the first expert action corresponding to the state of each agent in the second navigation strategy, and calculating the cross-entropy based on the first expert action for the navigation action indicated by the second navigation strategy, including: Determine the proportion of the number of correction actions in the second navigation strategy to the total number of navigation actions corresponding to each navigation trajectory, and sort each navigation trajectory from high to low according to the proportion; According to the sorting results, obtain the agent state corresponding to each navigation trajectory, input the agent state into the expert network, and obtain the first expert action corresponding to the agent state; The cross-entropy is calculated based on the navigation actions indicated by the second navigation strategy and the actions of the first expert.

[0009] Optionally, the joint loss of the intermediate navigation model is determined based on the reward value and cross-entropy, including: Based on the reward values ​​of multiple candidate action sequences corresponding to the same candidate state sample, the advantage value corresponding to each candidate action sequence is determined, the reinforcement learning loss is determined based on the advantage value, and the cross-entropy is used as the imitation learning loss. The joint loss of the intermediate navigation model is determined by weighted summation of the reinforcement learning loss and the imitation learning loss.

[0010] Secondly, embodiments of the present invention provide a navigation model training device, comprising: The initial training module is used to perform multi-task training on the initial navigation model based on the expert demonstration dataset and navigation video sampling frames, and to obtain the first navigation strategy output by the initial navigation model after completing multi-task training based on the navigation environment. The intermediate training module is used to train the initial navigation model based on the first navigation strategy and the corresponding correction action to obtain the intermediate navigation model, and to obtain the second navigation strategy output by the intermediate navigation model based on the correction action. The reward determination module is used to control the agent to move in the navigation environment based on the second navigation strategy, determine the entropy value of each navigation action performed by the agent during the movement, determine candidate state samples based on the entropy value, generate multiple candidate action sequences corresponding to the candidate state samples, and determine the reward value corresponding to each candidate action sequence. The parameter update module is used to obtain the first expert action corresponding to the state of each agent in the second navigation strategy, calculate the cross-entropy based on the navigation action indicated by the second navigation strategy based on the first expert action, determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy, and update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

[0011] Thirdly, embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the first aspect or any optional implementation of the first aspect are performed.

[0012] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the first aspect or any optional implementation thereof.

[0013] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0014] In any of the above aspects or any implementation of any aspect, the initial navigation model is trained on a multi-task basis using an expert demonstration dataset and sampled frames from navigation videos. This allows the initial navigation model to simultaneously learn to make action decisions based on the navigation environment and understand trajectory information in the navigation video, thereby forming a first navigation strategy with basic navigation capabilities and visual understanding. Based on this, corresponding correction actions are obtained according to the execution of the first navigation strategy in the navigation environment. The initial navigation model is then further trained using the first navigation strategy and correction actions, enabling the resulting intermediate navigation model to identify and correct its own errors during navigation, thus achieving strong error recovery capabilities. Furthermore, the agent's movement is controlled based on a second navigation strategy, and candidate state samples are determined based on the entropy values ​​of each navigation action. This focuses the training process on states with high uncertainty in the navigation model's predictions. Multiple candidate action sequences are then generated for each candidate state, and corresponding reward values ​​are determined. This allows the navigation model to explore various actions under states with significant decision-making uncertainty, and the navigation effect of different action sequences is evaluated through reward values, thereby enhancing the reward-driven exploration capability. Simultaneously, the first expert actions corresponding to the states of each agent in the second navigation strategy are acquired, and offline imitation learning is performed using the cross-entropy between the navigation actions output by the second navigation strategy and the first expert actions. This allows the navigation model to further correct prediction biases using expert knowledge. Finally, the exploration ability reflected in the reward value and the expert imitation ability reflected in the cross-entropy are used together to determine the joint loss, and the model parameters of the intermediate navigation model are updated accordingly. This enables the navigation model to simultaneously possess error recovery, active exploration, and expert imitation capabilities based on existing navigation knowledge. By using targeted high-uncertainty states as candidate state samples and combining them with expert actions for joint training, the repetitive training on low-value, low-uncertainty samples can be reduced, improving the utilization efficiency of training samples. At the same time, the navigation model can adapt to different navigation states and environmental changes, reducing dependence on specific navigation trajectories, thereby improving training efficiency and enhancing the generalization ability of the navigation model.

[0015] The beneficial effects of the aforementioned navigation model training device, electronic equipment, and storage medium are described in the description of the navigation model training method above, and will not be repeated here. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to the present invention and, together with the specification, serve to explain the technical solutions of the present invention. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart of a navigation model training method provided by an embodiment of the present invention is shown; Figure 2 A flowchart illustrating a navigation model training method provided by an embodiment of the present invention is shown; Figure 3 A navigation diagram of the multi-task training phase provided in an embodiment of the present invention is shown; Figure 4 A navigation diagram illustrating the correction phase provided in an embodiment of the present invention is shown; Figure 5 A navigation diagram for online reinforcement learning provided in an embodiment of the present invention is shown; Figure 6 This diagram illustrates a navigation system based on offline imitation learning provided in an embodiment of the present invention. Figure 7 This diagram illustrates a navigation model training device provided in an embodiment of the present invention. Figure 8 An exemplary system architecture in which embodiments of the present invention can be applied is shown; Figure 9 A schematic diagram of the structure of a computer system used to implement embodiments of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0020] The above problems and solutions are the result of the inventor's practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be considered as the inventor's contribution to the invention.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] To facilitate understanding of this embodiment, a detailed description of the navigation model training method disclosed in this invention will be provided first. The execution entity of the navigation model training method provided in this invention is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this navigation model training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0023] See Figure 1 The diagram shows a flowchart of a navigation model training method provided in an embodiment of the present invention. The method includes steps S101 to S104, wherein: S101: Perform multi-task training on the initial navigation model based on the expert demonstration dataset and navigation video sampling frames, and obtain the first navigation strategy output by the initial navigation model based on the navigation environment after completing multi-task training.

[0024] In this embodiment of the invention, the expert demonstration dataset can be understood as a collection of navigation example data used for supervised training of the initial navigation model. It records relevant information generated by experts or pre-set expert strategies when completing navigation processes under different navigation tasks and environments. The expert demonstration dataset may include navigation instructions corresponding to the navigation process, navigation sampling frames, expert actions performed by the agent in different navigation states, and corresponding navigation trajectories. Expert actions can represent the actions that should be performed correctly in the corresponding navigation state, such as moving forward, turning left, turning right, or stopping. Through the expert demonstration dataset, examples with correct navigation behavior can be provided to the initial navigation model, enabling it to learn the correspondence between navigation instructions, navigation environment information, and navigation actions, thereby acquiring the ability to perform basic navigation tasks.

[0025] The navigation state mentioned above and the agent state mentioned below can be understood as the specific state of the agent in the navigation environment at a certain moment and the navigation information related to that state. For example, an agent is navigating in an indoor environment. At a certain moment, the agent is in the middle of a corridor, facing forward with a fork in the road ahead. It has acquired a first-person perspective image, has historical action information, and needs to continue moving towards the target location according to navigation instructions. All of this information constitutes the agent's navigation state at that moment, i.e., the agent state.

[0026] Navigation video sampling frames refer to video image frames acquired from navigation videos recording the movement of an intelligent agent or other agent in a navigation environment, according to preset time intervals, frame intervals, or sampling rules. Each navigation video sampling frame can be used to represent the environmental visual information corresponding to a certain moment in the navigation process, such as a first-person view of the agent's current position. The image may contain roads, passages, obstacles, the surrounding environment of the target location, or other navigation-related visual content. Multiple navigation video sampling frames can collectively reflect the environmental changes and movement process of the intelligent agent during navigation, enabling the initial navigation model to learn to visually understand the navigation environment and combine environmental visual information at different time points to understand the corresponding navigation process and trajectory changes, thereby enhancing the initial navigation model's ability to understand different navigation environments and scenarios.

[0027] In embodiments of the present invention, such as Figure 2As shown, an initial navigation model can be trained on an expert demonstration dataset and navigation video sampling frames to perform action prediction and trajectory description tasks. Through joint training of these two tasks, the initial navigation model can simultaneously acquire navigation action decision-making capabilities and navigation environment understanding capabilities. Specifically, the initial navigation model is trained on an expert demonstration dataset and navigation video sampling frames to obtain a first navigation strategy output by the trained initial navigation model based on the navigation environment. This includes: inputting navigation instructions extracted from the expert demonstration dataset, navigation sampling frames, and second expert actions into the initial navigation model to enable it to output predicted actions; training the initial navigation model on the action prediction task based on the predicted actions and the second expert actions; inputting navigation video sampling frames into the initial navigation model to enable it to output trajectory description information; training the initial navigation model on the trajectory description information and corresponding navigation instructions; and jointly training the initial navigation model based on the training results of the action prediction and trajectory description tasks to enable the jointly trained initial navigation model to output a first navigation strategy based on the navigation environment.

[0028] In practical implementation, the expert demonstration dataset can be used to provide training samples with correct navigation behavior for the initial navigation model. This dataset can include navigation instructions, navigation sampling frames, and second-expert actions. Navigation instructions can characterize the goal or requirements of the navigation task, such as "move from the current position to the end of the corridor and then turn left." Navigation sampling frames can represent environmental visual information at different navigation moments during the expert's execution of the navigation task. Each frame can be a three-channel RGB image acquired from the agent's first-person perspective and can be combined with corresponding historical action information to form the input for the navigation model to predict actions. Specifically, at each navigation moment, the initial navigation model can receive navigation instructions, multiple frames of long-term memory uniformly sampled from the historical navigation process, and short-term memory formed during the most recent navigation process. The short-term memory can consist of alternating images and actions, forming an "image-action-image-action" sequence, enabling the initial navigation model to simultaneously utilize environmental information over a longer time span and recent continuous navigation behavior information to determine the action to be performed. The initial navigation model can output predicted actions based on the above information. Predicted actions can be in the form of action blocks, each action block including no more than a preset number of consecutive discrete actions, such as one or more actions from forward, left, right, and stop. By comparing the predicted action with the corresponding second expert action in the expert demonstration dataset, the initial navigation model can be trained on the action prediction task, enabling the initial navigation model to learn to select appropriate navigation actions under different navigation commands and visual environments.

[0029] During the training process of the trajectory description task, such as Figure 3As shown, navigation video sampling frames can be input into the initial navigation model. The initial navigation model outputs trajectory description information based on the environmental visual information in the sampling frames, and compares the generated trajectory description information with the corresponding navigation commands. This allows the initial navigation model to learn how to understand the corresponding navigation tasks and movement processes based on visual changes during navigation. Action prediction and trajectory description tasks train the initial navigation model from the perspectives of action decision-making and visual understanding, respectively. The action prediction task helps the initial navigation model establish the association between the navigation environment, navigation commands, and navigation actions, while the trajectory description task enhances the initial navigation model's understanding of the navigation process and environmental changes. After completing the training of both tasks, the initial navigation model can be jointly trained based on the training results of the action prediction and trajectory description tasks. After joint training, the initial navigation model can not only understand the visual information in the navigation environment and the task requirements expressed by the navigation commands, but also output corresponding navigation actions by combining learned expert navigation behavior. This allows the jointly trained initial navigation model to output a first navigation strategy based on the navigation environment.

[0030] It should be noted that while both navigation video sampling frames and navigation sampling frames extracted from expert demonstration datasets can represent visual information in the navigation environment as image frames, they can differ in their training tasks and data sources. Navigation sampling frames extracted from expert demonstration datasets are part of expert navigation behavior data, corresponding to navigation instructions, second expert actions, etc., and are primarily used for action prediction tasks. For example, in a navigation sampling frame, the agent is located at a passageway intersection; combining the corresponding navigation instructions and the second expert action executed by the expert, the initial navigation model can learn whether to turn left or right in that visual environment and navigation task. Navigation video sampling frames, on the other hand, can be obtained from videos used to record the navigation process. They are mainly used for trajectory description tasks, focusing on reflecting environmental changes and the continuous relationships between different positions during navigation. Navigation video sampling frames can be multiple image frames obtained from a complete navigation video according to certain rules. Through the visual changes between multiple sampling frames, the initial navigation model understands the agent's direction of movement, environmental changes, and trajectory development during navigation. For example, continuous navigation video sampling frames can reflect the process of an agent moving from a room into a corridor, moving along the corridor, and reaching a corner. The initial navigation model can generate corresponding trajectory description information based on this continuously changing visual information.

[0031] S102: Train the initial navigation model according to the first navigation strategy and the corresponding correction action to obtain the intermediate navigation model, and obtain the second navigation strategy output by the intermediate navigation model based on the correction action.

[0032] In this embodiment of the invention, after completing the multi-task training of the initial navigation model and obtaining the first navigation strategy, the initial navigation model can be further trained using an expert-intervention-based imitation learning method. This enables the initial navigation model to learn and correct errors it is prone to make during navigation, thereby obtaining an intermediate navigation model with error recovery capabilities. Specifically, the initial navigation model is trained according to the first navigation strategy and the corresponding correction actions to obtain an intermediate navigation model. The second navigation strategy output by the intermediate navigation model based on the correction actions is then obtained, including: controlling the agent to move in the navigation environment based on the first navigation strategy; in response to the agent meeting preset correction conditions during movement, obtaining the correction actions output by the expert network for the agent's state; and training the initial navigation model based on the correction actions corresponding to the states of each agent during movement to obtain the intermediate navigation model, and obtaining the second navigation strategy output by the intermediate navigation model based on the correction actions.

[0033] In practical implementation, a first navigation strategy can be deployed in the navigation environment. This strategy continuously outputs navigation action blocks based on current navigation commands, the current first-person perspective three-channel RGB image, long-term memory, and short-term memory, among other navigation history information. The agent then executes these action blocks to move within the navigation environment. After each action block, the agent obtains a new first-person perspective three-channel RGB image and adds the new visual observations and executed actions to the current navigation history, allowing the first navigation strategy to continue making subsequent action decisions based on the updated history. During this process, the first navigation strategy may output incorrect navigation actions due to inaccurate visual understanding, insufficient utilization of historical information, or a complex current navigation state, causing the agent to gradually deviate from the correct navigation direction. At this point, the agent's current state during movement can be used to determine whether preset correction conditions are met. For example, the degree of deviation of the agent's current trajectory from the correct navigation trajectory can be used for judgment. When the deviation reaches a preset threshold, the current navigation state is considered to require expert correction. When an agent meets preset correction conditions in a certain navigation state, the expert network can output a correction action for the current agent state. This correction action can be the correct navigation action that the expert network believes should be performed in the current state. For example, if the first navigation strategy predicts moving right, but the current environment and navigation target determine that moving left is necessary, the expert network can output "move left" as the correction action. For multiple agent states accessed during the first navigation strategy's movement, corresponding correction actions can be obtained for each state that meets the correction conditions. The agent state and the corresponding correction action constitute training information, enabling the initial navigation model to learn the correct action to take in states prone to navigation errors. By training the initial navigation model using agent states and corresponding correction actions, the initial navigation model can focus on learning navigation situations where the original first navigation strategy is prone to errors, gradually developing the ability to reselect the correct action from erroneous states. Figure 3 and Figure 4 As shown, the correct action of the agent should be to turn left after passing through the first door. However, during the operation, the agent mistakenly chose to go straight forward when passing through the first door according to the first navigation strategy, resulting in entering the wrong channel. The expert network can output corrective actions such as backing away or turning around based on the agent's state at this time. Through the correspondence between this state and the corrective action, the training process enables the model to improve the probability of choosing the correct action when encountering similar visual environments and navigation situations in the future, thereby enhancing the error recovery capability in the navigation process.

[0034] After obtaining corrective actions, the initial navigation model can be further trained using different agent states and corresponding corrective actions during movement. This allows for parameter adjustments based on expert correction information, and the trained initial navigation model serves as an intermediate navigation model. The intermediate navigation model can continue to employ a multi-round navigation approach similar to the first navigation strategy, predicting continuous actions based on navigation instructions, long-term memory, short-term memory, and current visual observations. After the model outputs navigation action blocks, the agent executes them, and the navigation history is updated based on new observations after execution. Therefore, the second navigation strategy output by the intermediate navigation model can better handle error states that are prone to occur in actual navigation compared to the first navigation strategy. When the agent deviates from its course, the intermediate navigation model has the ability to reselect appropriate actions based on current environmental visual information and historical navigation information, rather than continuing to move in the wrong direction after an erroneous action has occurred.

[0035] S103: Based on the second navigation strategy, control the agent to move in the navigation environment, determine the entropy value of each navigation action performed by the agent during the movement, determine candidate state samples based on the entropy value, generate multiple candidate action sequences corresponding to the candidate state samples, and determine the reward value corresponding to each candidate action sequence.

[0036] In this embodiment of the invention, after obtaining the second navigation strategy, it can be deployed in the navigation environment, and the agent can be controlled to continuously execute navigation actions based on the second navigation strategy, allowing the agent to move gradually in the navigation environment. During continuous navigation, entropy values ​​can be calculated for the prediction results of the navigation actions output by the second navigation strategy in different navigation states. The entropy value of the action can be used to reflect the degree of uncertainty of the second navigation strategy when making action decisions in the current navigation state. The higher the entropy value, the more uncertain the second navigation strategy is about which navigation action to take in the current state, and the more dispersed the prediction probabilities among different candidate actions are usually. The lower the entropy value, the higher the certainty of the second navigation strategy in making the current action decision. Therefore, based on the entropy values ​​corresponding to each navigation action during the movement, states with higher entropy values ​​can be selected as candidate state samples from the multiple navigation states actually visited by the agent. For example, they can be sorted from high to low entropy values, and a portion of the top-ranked navigation states can be selected. This allows subsequent training to focus more on navigation states where the second navigation strategy itself is less uncertain, rather than exploring all navigation states evenly, thereby reducing repeated training on states that already have high decision certainty and improving the utilization efficiency of training samples.

[0037] After determining the candidate state samples, such as Figure 2As shown, multiple candidate action sequences can be generated for each candidate state sample using Group Relative Policy Optimization (GRPO). This means generating multiple sets of candidate action sequences for the same candidate state sample, and comparing them relative to each other within the group based on the reward value of each sequence to determine their superiority over other sequences in the same group. These multiple candidate action sequences can be understood as various possible consecutive action schemes generated for the same navigation state. Figure 5 As shown, the agent is currently at an intersection. Based on its candidate state samples, it can generate multiple candidate action sequences, including "turn left," "go straight," and "turn right." Each candidate action sequence can consist of no more than a preset number of discrete navigation actions, thus forming multiple different navigation decision schemes. The reward value corresponding to each candidate action sequence can then be determined, and the reward values ​​of multiple candidate action sequences corresponding to the same candidate state sample can be compared within a group based on GRPO. The specific method for determining the reward value will be explained in detail below and will not be elaborated here.

[0038] In this embodiment of the invention, determining the reward value corresponding to each candidate action sequence includes: calculating a first reward for the candidate action sequence in the format dimension based on a preset sequence length threshold; calculating a second reward for the candidate action sequence in the consistency dimension based on a reference action sequence corresponding to the candidate state sample in the second navigation strategy; determining the first pose information after the agent executes the candidate action sequence and the second pose information after the agent executes the reference action sequence, and calculating the cosine similarity between the first pose information and the second pose information to obtain a third reward for the candidate action sequence in the position dimension; determining the first displacement distance after the agent executes the candidate action sequence and the second displacement distance after the agent executes the reference action sequence, and calculating the normalized distance between the first displacement distance and the second displacement distance to obtain a fourth reward for the candidate action sequence in the distance dimension; and determining the reward value corresponding to each candidate action sequence based on the first reward, the second reward, the third reward, and the fourth reward.

[0039] In practical implementation, after generating multiple candidate action sequences for the same candidate state sample, each candidate action sequence can be evaluated from multiple dimensions to determine the corresponding reward value. This reward value serves as the basis for subsequent GRPO intra-group comparisons and strategy optimization. These multiple dimensions can include format, consistency, location, and distance dimensions. First, the format of the candidate action sequences can be evaluated based on a preset sequence length threshold. Since the navigation model outputs continuous navigation actions in the form of action blocks, each candidate action sequence needs to meet a preset action quantity limit. Therefore, it can be determined whether the number of discrete navigation actions contained in the candidate action sequence exceeds the preset sequence length threshold, and the first reward for the format dimension is determined based on the judgment result. For example, when the preset sequence length threshold is 4, the candidate action sequence "forward, left, forward" contains 3 discrete navigation actions, meeting the sequence length requirement, and the corresponding first reward can be determined as 1; if the candidate action sequence contains more than 4 actions, it can be determined that the candidate action sequence does not meet the output format requirement, and the corresponding first reward can be determined as 0. Alternatively, a higher first reward can be assigned to candidate action sequences that meet the length requirement, and a lower first reward to those that do not. This embodiment of the invention does not limit the specific value of the first reward and can set it according to actual needs. The first reward can eliminate candidate action sequences that do not conform to the navigation action output format, ensuring that subsequent reward evaluations are based on valid action sequences.

[0040] After determining the first reward for the format dimension, a consistency evaluation can be performed on the candidate action sequences based on the reference action sequences corresponding to the candidate state samples. The reference action sequence represents the original action sequence output by the second navigation strategy under the candidate state sample. For example, if the second navigation strategy outputs "forward, left" in the current navigation state, this action sequence can be used as the reference action sequence for the current candidate state sample. Different candidate action sequences regenerated for the same candidate state sample are then compared with the reference action sequence, and the second reward is determined based on the degree of consistency in action content. This consistency evaluation reflects the degree of correlation between the candidate action sequence and the original decision of the second navigation strategy. For example, if the candidate action sequence is completely consistent with the reference action sequence, the second reward can be set to 1; otherwise, it can be set to 0. Alternatively, a higher second reward can be assigned when they are completely consistent; a relatively lower consistency reward can be obtained when only some actions are consistent; and a further reduction can be made if they are completely different. This encourages the exploration of new action schemes while retaining the navigation behavior information already learned by the second navigation strategy, preventing complete deviation from the existing navigation strategy during online exploration.

[0041] Furthermore, the positional dimension of the candidate action sequence can be evaluated based on the actual spatial motion results after the agent executes the candidate action sequence. Specifically, the state after executing the candidate action sequence in a navigation environment can be simulated, and the first pose information after executing the candidate action sequence can be obtained; simultaneously, the second pose information after the agent executes the reference action sequence can be simulated. The pose information can be used to characterize the changes in position and orientation of the agent after executing the corresponding action sequence, such as lateral displacement, longitudinal displacement, and heading angle changes relative to the period before the action sequence was executed. Then, the cosine similarity between the first pose information and the second pose information can be calculated, and the obtained cosine similarity can be used as the third reward of the candidate action sequence in the positional dimension. The closer the first pose information and the second pose information are, the more consistent their corresponding motion direction and posture changes are, the higher the cosine similarity, and the higher the corresponding third reward; conversely, if the candidate action sequence causes a significant difference between the agent's orientation and the reference action sequence, the cosine similarity decreases, and the corresponding third reward also decreases. For example, if a reference action sequence causes the agent to turn left and move forward, while a candidate action sequence causes the agent to turn right, then even if the two action sequences have the same number of actions, their pose changes after execution may be significantly different. In this case, the candidate action sequence may receive a lower position reward.

[0042] In addition to pose changes, the displacement effect produced by candidate action sequences can be further evaluated. The first displacement distance after the agent executes the candidate action sequence, and the second displacement distance after executing the reference action sequence, can be simulated. A normalized distance is calculated based on the difference between the first and second displacement distances to obtain a fourth reward for the candidate action sequence in the distance dimension. Normalization reduces the impact of different navigation scenarios and action block lengths on the distance evaluation results, making the distance reward more suitable for comparison between different candidate action sequences. For example, if the reference action sequence moves the agent a longer distance towards the target, while the candidate action sequence, although in a similar direction, moves a significantly shorter distance, the normalized distance between them will be larger, resulting in a lower fourth reward. Conversely, if the displacement distance produced by the candidate action sequence is close to that of the reference action sequence, the normalized distance will be smaller, leading to a higher fourth reward. Therefore, the third reward primarily reflects the similarity between the pose change direction and posture result of the candidate action sequence and the reference action sequence, while the fourth reward primarily reflects the similarity between the displacement distance produced by the candidate action sequence and the reference action sequence.

[0043] After obtaining the first, second, third, and fourth rewards, the evaluation results from multiple dimensions can be integrated to determine the reward value corresponding to each candidate action sequence. For example, the four rewards can be weighted and summed according to preset weights, or other preset reward fusion methods can be used to obtain the comprehensive reward value. Thus, for multiple candidate action sequences generated from the same candidate state sample, their corresponding comprehensive reward values ​​can be obtained respectively. Furthermore, these candidate action sequences and their reward values ​​can be compared relatively as samples in the same group of GRPO, giving the candidate action sequence with the higher reward a greater advantage over other candidate action sequences in the same group, thereby guiding the second navigation strategy to increase the probability of generating high-quality candidate action sequences.

[0044] It should be noted that the four reward value calculation dimensions described above are merely illustrative examples of one possible implementation method in this invention and do not constitute an undue limitation on the invention. In practical applications, other reward value evaluation dimensions can be set according to actual needs and circumstances. For example, a target proximity dimension can also be introduced to evaluate whether the agent is closer to the target position after the candidate action sequence is executed. This invention does not impose specific limitations on this, but only on the ability to achieve its function.

[0045] S104: Obtain the first expert action corresponding to the state of each agent in the second navigation strategy, calculate the cross-entropy based on the navigation action indicated by the second navigation strategy based on the first expert action, determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy, and update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

[0046] In embodiments of the present invention, such as Figure 2 As shown, while performing online reinforcement learning training based on GRPO, offline imitation learning training can also be performed simultaneously, enabling the navigation model to continuously learn the correct decisions of experts under different navigation states. Specifically, the process involves obtaining the first expert actions corresponding to the states of each agent in the second navigation strategy, and calculating the cross-entropy of the navigation actions indicated by the second navigation strategy based on the first expert actions. This includes: determining the proportion of the number of correction actions in the second navigation strategy to the total number of navigation actions corresponding to each navigation trajectory, and sorting each navigation trajectory from high to low according to the proportion; obtaining the agent states corresponding to each navigation trajectory according to the sorting results, inputting the agent states into the expert network to obtain the first expert actions corresponding to the agent states; and calculating the cross-entropy between the navigation actions indicated by the second navigation strategy and the first expert actions.

[0047] In specific implementation, such as Figure 6As shown, during the movement of the agent controlled by the second navigation strategy in the navigation environment, multiple navigation trajectories can be formed. Each navigation trajectory records the states of multiple agents visited sequentially during a continuous navigation process, as well as the navigation actions output by the second navigation strategy in response to these agent states. In some navigation trajectories, the second navigation strategy may complete navigation relatively accurately, while in others, it may frequently produce erroneous actions, requiring expert correction. Therefore, the number of correction actions in each navigation trajectory can be counted, and the proportion of correction actions to the total number of corresponding navigation actions for that trajectory can be calculated. This proportion is used as an indicator to measure the degree of erroneous decisions in the navigation trajectory, and the navigation trajectories are ranked from high to low according to the proportion. The higher the proportion of correction actions, the more erroneous decisions the second navigation strategy makes in that navigation trajectory, and the higher the value of the current navigation trajectory for model learning and error correction. Therefore, navigation trajectories with a higher proportion of correction actions can be prioritized for subsequent offline imitation learning. For example, navigation trajectory 1 contains 100 navigation actions, of which 20 require expert correction, so the corresponding correction action ratio is 20%; navigation trajectory 2 contains 100 navigation actions, of which 5 require expert correction, so the corresponding ratio is 5%. Therefore, navigation trajectory A can be used first for training, so that training resources are more concentrated on navigation scenarios where the second navigation strategy is prone to errors.

[0048] After obtaining the agent states corresponding to each navigation trajectory according to the sorting results, the corresponding first expert action can be re-determined based on the obtained agent states. Specifically, the agent states actually accessed by the agent during navigation can be input into the expert network. The expert network then determines the more suitable navigation action for the current state based on information such as the current navigation command, the current visual environment, and navigation history, and uses the action output by the expert network as the first expert action. Here, the agent state is the state experienced by the agent during actual movement based on the second navigation strategy, rather than an independent navigation trajectory pre-generated by the second navigation strategy. For example, when the second navigation strategy controls the agent to reach an intersection with two passages (left and right), the current first-view RGB image and navigation command information jointly reflect the current agent state. Based on the second navigation strategy, the output might be right, while the expert network determines that the correct action should be left based on the current navigation task. Therefore, the leftward output by the expert network for this agent state can be used as the first expert action. By re-acquiring the first expert action from the actual access state in the navigation trajectory, the expert action can correspond to the navigation state actually experienced by the second navigation strategy. Even if a certain state does not trigger correction during the execution of the second navigation strategy, the expert network can still provide the corresponding first expert action for that state, thereby forming a more complete expert supervision information.

[0049] After obtaining the first expert action, the navigation action indicated by the second navigation strategy can be compared with the first expert action, and the cross-entropy between them can be calculated. Cross-entropy measures the difference between the action distribution output by the second navigation strategy in the current agent state and the target distribution corresponding to the expert action. A lower cross-entropy indicates that the intermediate navigation model has learned the expert's decision-making in that navigation state well; conversely, a higher cross-entropy indicates that the intermediate navigation model still has a significant decision-making bias in that state. By prioritizing the use of navigation trajectories sorted according to the proportion of corrective actions to obtain the agent state and the first expert action, and calculating the cross-entropy between the navigation action and the first expert action based on these states, offline imitation learning can focus more on navigation trajectories where the second navigation strategy has historically made errors. This improves the efficiency of training sample utilization and enhances the intermediate navigation model's ability to correct actions in erroneous navigation states.

[0050] In this embodiment of the invention, after obtaining the reward values ​​corresponding to each candidate action sequence and the cross-entropy between the navigation action indicated by the second navigation strategy and the first expert action, online reinforcement learning and offline imitation learning can be combined to jointly train the intermediate navigation model. Online reinforcement learning primarily utilizes the reward differences between candidate action sequences to guide the intermediate navigation model's exploration, while offline imitation learning primarily utilizes expert actions to constrain the intermediate navigation model's action decisions, enabling the intermediate navigation model to further explore navigation behaviors with higher rewards while retaining expert navigation experience. Specifically, determining the joint loss of the intermediate navigation model based on the reward values ​​and cross-entropy includes: determining the advantage value corresponding to each candidate action sequence based on the reward values ​​of multiple candidate action sequences corresponding to the same candidate state sample; determining the reinforcement learning loss based on the advantage value; and using the cross-entropy as the imitation learning loss; and weighted summing the reinforcement learning loss and the imitation learning loss to determine the joint loss of the intermediate navigation model.

[0051] In practice, the reward statistics for a group of candidate action sequences can be determined first by calculating the average and variance of the group's reward values. Then, the dominance value is determined based on the deviation of the reward value of each candidate action sequence from the group's overall reward statistics. Candidate action sequences with reward values ​​higher than the group's average have higher dominance values, indicating better navigation performance compared to other action sequences in the same group. Candidate action sequences with reward values ​​lower than the group's average have lower dominance values, indicating relatively poor navigation performance. After obtaining the dominance values ​​for each candidate action sequence, the reinforcement learning loss can be determined based on these values. The reinforcement learning loss measures the relationship between the probability of the current intermediate navigation model generating different candidate action sequences for candidate state samples and the corresponding dominance value. When a candidate action sequence has a high dominance value, the reinforcement learning loss can guide the intermediate navigation model to increase the probability of generating that candidate action sequence under the same or similar navigation states. When a candidate action sequence has a low dominance value, the positive contribution of that candidate action sequence to the model's policy update can be reduced, thus gradually leading the model to generate navigation action sequences with higher rewards. Furthermore, the cross-entropy calculated between the navigation action indicated by the second navigation strategy and the first expert action can be used as the imitation learning loss. Cross-entropy reflects the difference between the navigation action currently output by the intermediate navigation model and the first expert action determined by the expert network for the same agent state. The imitation learning loss is low when the model output is consistent with the first expert action or has a high probability of predicting the expert action; the imitation learning loss is high when the model output deviates from the first expert action. Therefore, the imitation learning loss can be used to constrain the model to maintain consistency with expert navigation behavior.

[0052] After determining the reinforcement learning loss and the imitation learning loss, the two losses can be weighted and summed according to preset weights to obtain the joint loss corresponding to the intermediate navigation model. For example, the reinforcement learning loss and the imitation learning loss can be multiplied by their respective weight coefficients and then summed. One of the weights can be dynamically changed during the training process, allowing the joint training to have different emphases at different training stages. In the early stages of training, a relatively large weight can be set for the imitation learning loss, causing the intermediate navigation model to refer more to the correct navigation behavior provided by the expert network, avoiding over-exploration before the model has developed stable navigation capabilities. As training progresses, the weight of the imitation learning loss can be reduced while the relative role of the reinforcement learning loss can be increased, allowing the model to explore high-reward action sequences based on its existing expert navigation capabilities. Therefore, the joint loss includes both the imitation learning constraint from expert supervision and the reinforcement learning optimization objective from the candidate action reward. When updating the model parameters, the intermediate navigation model can, on the one hand, correct the action deviation of the model in the actual navigation state based on the first expert action, so as to avoid the model deviating from the effective navigation behavior that has been learned; on the other hand, it can adjust the action generation probability of the model based on the relative reward difference between different candidate action sequences, so that the model can explore action schemes that are not fully covered in the expert demonstration data but have good effects in the actual navigation environment.

[0053] According to a second aspect of the embodiments of the present invention, such as Figure 7 As shown, a navigation model training device 700 is provided, comprising: The initial training module 701 is used to perform multi-task training on the initial navigation model based on the expert demonstration dataset and navigation video sampling frames, and to obtain the first navigation strategy output by the initial navigation model after completing multi-task training based on the navigation environment. The intermediate training module 702 is used to train the initial navigation model according to the first navigation strategy and the corresponding correction action to obtain the intermediate navigation model, and to obtain the second navigation strategy output by the intermediate navigation model based on the correction action. The reward determination module 703 is used to control the agent to move in the navigation environment based on the second navigation strategy, determine the entropy value of each navigation action performed by the agent during the movement, determine candidate state samples based on the entropy value, generate multiple candidate action sequences corresponding to the candidate state samples, and determine the reward value corresponding to each candidate action sequence. The parameter update module 704 is used to obtain the first expert action corresponding to the state of each agent in the second navigation strategy, calculate the cross-entropy based on the navigation action indicated by the second navigation strategy based on the first expert action, determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy, and update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

[0054] Optionally, the initial training module 701 is specifically used for: The navigation instructions, navigation sampling frames, and second expert actions extracted from the expert demonstration dataset are input into the initial navigation model so that the initial navigation model outputs predicted actions. The initial navigation model is then trained on the action prediction task based on the predicted actions and the second expert actions. The navigation video sampling frames are input into the initial navigation model so that the initial navigation model outputs trajectory description information based on the navigation video sampling frames, and the initial navigation model is trained on the trajectory description task based on the trajectory description information and the corresponding navigation instructions. Based on the training results of the action prediction task training and the trajectory description task training, the initial navigation model is jointly trained, so that the initial navigation model that has completed the joint training outputs the first navigation strategy based on the navigation environment.

[0055] Optionally, the intermediate training module 702 is specifically used for, including: The agent is controlled to move within the navigation environment based on a first navigation strategy; In response to the agent meeting the preset correction conditions during movement, the expert network obtains the correction actions for the agent's state output. The initial navigation model is trained based on the correction actions corresponding to the states of each agent during the movement process to obtain the intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction actions is obtained.

[0056] Optionally, the reward determination module 703 is specifically used for: Based on a preset sequence length threshold, the first reward of the candidate action sequence in the format dimension is calculated; Based on the reference action sequence corresponding to the candidate state sample in the second navigation strategy, calculate the second reward of the candidate action sequence in the consistency dimension. Determine the first pose information after the agent executes the candidate action sequence, and the second pose information after the agent executes the reference action sequence. Calculate the cosine similarity between the first pose information and the second pose information to obtain the third reward of the candidate action sequence in the position dimension. Determine the first displacement distance after the agent executes the candidate action sequence and the second displacement distance after the agent executes the reference action sequence. Calculate the normalized distance between the first displacement distance and the second displacement distance to obtain the fourth reward of the candidate action sequence in the distance dimension. The reward value corresponding to each candidate action sequence is determined based on the first reward, second reward, third reward, and fourth reward.

[0057] Optionally, the parameter update module 704 is specifically used for: Determine the proportion of the number of correction actions in the second navigation strategy to the total number of navigation actions corresponding to each navigation trajectory, and sort each navigation trajectory from high to low according to the proportion; According to the sorting results, obtain the agent state corresponding to each navigation trajectory, input the agent state into the expert network, and obtain the first expert action corresponding to the agent state; The cross-entropy is calculated based on the navigation actions indicated by the second navigation strategy and the actions of the first expert.

[0058] Optionally, the parameter update module 704 is specifically used for: Based on the reward values ​​of multiple candidate action sequences corresponding to the same candidate state sample, the advantage value corresponding to each candidate action sequence is determined, the reinforcement learning loss is determined based on the advantage value, and the cross-entropy is used as the imitation learning loss. The joint loss of the intermediate navigation model is determined by weighted summation of the reinforcement learning loss and the imitation learning loss.

[0059] According to a third aspect of the present invention, an electronic device for training a navigation model is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0060] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0061] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0062] Figure 8 An exemplary system architecture 800 is shown that can be applied to the navigation model training method or navigation model training apparatus implemented in this invention.

[0063] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0064] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0065] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0066] Server 805 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server can process received navigation model training requests and feed back the processing results (for example only) to the terminal devices.

[0067] It should be noted that the navigation model training method provided in this embodiment of the invention is generally executed by server 805, and correspondingly, the navigation model training device is generally located in server 805. The navigation model training method provided in this embodiment of the invention can also be executed by terminal devices 801, 802, and 803, and correspondingly, the navigation model training device can be located in terminal devices 801, 802, and 803.

[0068] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0069] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0070] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0071] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0072] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0073] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0075] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor. For example, a processor may include an initial training module, an intermediate training module, a reward determination module, and a parameter update module. The names of these modules do not necessarily limit the module itself. For instance, the initial training module can be described as "a module that performs multi-task training on an initial navigation model based on an expert demonstration dataset and navigation video sampling frames, and obtains the first navigation strategy output by the initial navigation model after multi-task training based on the navigation environment."

[0076] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist alone and not assembled into the device. The aforementioned computer-readable medium carries one or more programs. When one or more of these programs are executed by a device, the device implements the following method: multi-task training of an initial navigation model based on an expert demonstration dataset and navigation video sampling frames; obtaining a first navigation policy output by the initial navigation model based on the navigation environment after multi-task training; training the initial navigation model according to the first navigation policy and corresponding correction actions to obtain an intermediate navigation model; obtaining a second navigation policy output by the intermediate navigation model based on the correction actions; controlling an agent to move in the navigation environment based on the second navigation policy; determining the entropy value of each navigation action performed by the agent during the movement; determining candidate state samples based on the entropy values; generating multiple candidate action sequences corresponding to the candidate state samples; determining the reward value corresponding to each candidate action sequence; obtaining a first expert action corresponding to each agent state in the second navigation policy; calculating cross-entropy based on the first expert action for the navigation actions indicated by the second navigation policy; determining the joint loss of the intermediate navigation model based on the reward value and cross-entropy; updating the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

[0077] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A navigation model training method, characterized in that, include: The initial navigation model is trained using an expert demonstration dataset and navigation video sampling frames to obtain the first navigation strategy output by the initial navigation model based on the navigation environment after the completion of the multi-task training. The initial navigation model is trained according to the first navigation strategy and the corresponding correction action to obtain an intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction action is obtained. Based on the second navigation strategy, the agent moves in the navigation environment. The entropy value of each navigation action performed by the agent during the movement is determined. Candidate state samples are determined based on the entropy value, and multiple candidate action sequences corresponding to the candidate state samples are generated. The reward value corresponding to each candidate action sequence is determined. Obtain the first expert action corresponding to the state of each agent in the second navigation strategy, calculate the cross-entropy based on the first expert action for the navigation action indicated by the second navigation strategy, determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy, and update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

2. The method according to claim 1, characterized in that, The initial navigation model is trained on a multi-task basis using an expert demonstration dataset and sampled navigation video frames. The first navigation strategy output by the initial navigation model after completing the multi-task training, based on the navigation environment, is obtained, including: The navigation instructions, navigation sampling frames, and second expert actions extracted from the expert demonstration dataset are input into the initial navigation model so that the initial navigation model outputs predicted actions, and the initial navigation model is trained on the action prediction task based on the predicted actions and the second expert actions; The navigation video sampling frames are input into the initial navigation model so that the initial navigation model outputs trajectory description information based on the navigation video sampling frames, and the initial navigation model is trained on the trajectory description task based on the trajectory description information and the corresponding navigation instructions. Based on the training results of the action prediction task training and the trajectory description task training, the initial navigation model is jointly trained, so that the initial navigation model that has completed the joint training outputs the first navigation strategy based on the navigation environment.

3. The method according to claim 1, characterized in that, The initial navigation model is trained according to the first navigation strategy and the corresponding correction action to obtain an intermediate navigation model. The second navigation strategy output by the intermediate navigation model based on the correction action is then obtained, including: The agent is controlled to move within the navigation environment based on the first navigation strategy; In response to the agent meeting preset correction conditions during movement, the expert network obtains the correction action output for the agent's state. The initial navigation model is trained based on the correction actions corresponding to the states of each agent during the movement to obtain an intermediate navigation model, and the second navigation strategy output by the intermediate navigation model based on the correction actions is obtained.

4. The method according to claim 1, characterized in that, Determine the reward value corresponding to each candidate action sequence, including: Based on a preset sequence length threshold, the first reward of the candidate action sequence in the format dimension is calculated; Based on the reference action sequence corresponding to the candidate state sample in the second navigation strategy, calculate the second reward of the candidate action sequence in the consistency dimension; Determine the first pose information after the agent executes the candidate action sequence, and the second pose information after the agent executes the reference action sequence. Calculate the cosine similarity between the first pose information and the second pose information to obtain the third reward of the candidate action sequence in the position dimension. Determine the first displacement distance after the agent executes the candidate action sequence and the second displacement distance after the agent executes the reference action sequence, and calculate the normalized distance between the first displacement distance and the second displacement distance to obtain the fourth reward of the candidate action sequence in the distance dimension; Based on the first reward, the second reward, the third reward, and the fourth reward, determine the reward value corresponding to each candidate action sequence.

5. The method according to claim 3, characterized in that, Obtain the first expert action corresponding to the state of each agent in the second navigation strategy, and calculate the cross-entropy based on the first expert action for the navigation action indicated by the second navigation strategy, including: Determine the proportion of the number of correction actions in the second navigation strategy to the total number of navigation actions corresponding to each navigation trajectory, and sort each navigation trajectory from high to low according to the proportion; According to the sorting results, obtain the agent state corresponding to each navigation trajectory, input the agent state into the expert network, and obtain the first expert action corresponding to the agent state; The cross-entropy is calculated based on the navigation action indicated by the second navigation strategy and the first expert action.

6. The method according to any one of claims 1-5, characterized in that, The joint loss of the intermediate navigation model is determined based on the reward value and cross-entropy, including: Based on the reward values ​​of multiple candidate action sequences corresponding to the same candidate state sample, the advantage value corresponding to each candidate action sequence is determined, the reinforcement learning loss is determined based on the advantage value, and the cross-entropy is used as the imitation learning loss. The reinforcement learning loss and the imitation learning loss are weighted and summed to determine the joint loss of the intermediate navigation model.

7. A navigation model training device, characterized in that, include: The initial training module is used to perform multi-task training on the initial navigation model based on the expert demonstration dataset and navigation video sampling frames, and to obtain the first navigation strategy output by the initial navigation model based on the navigation environment after completing the multi-task training. The intermediate training module is used to train the initial navigation model according to the first navigation strategy and the corresponding correction action to obtain the intermediate navigation model, and to obtain the second navigation strategy output by the intermediate navigation model based on the correction action. The reward determination module is used to control the agent to move in the navigation environment based on the second navigation strategy, determine the entropy value of each navigation action performed by the agent during the movement, determine candidate state samples based on the entropy value, generate multiple candidate action sequences corresponding to the candidate state samples, and determine the reward value corresponding to each candidate action sequence. The parameter update module is used to obtain the first expert action corresponding to the state of each agent in the second navigation strategy, calculate the cross-entropy based on the first expert action for the navigation action indicated by the second navigation strategy, determine the joint loss of the intermediate navigation model based on the reward value and the cross-entropy, and update the model parameters of the intermediate navigation model according to the joint loss to obtain the trained target navigation model.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images