Robot navigation method and apparatus, electronic device, storage medium, and program product

Abstract

The application provides a robot navigation method and device, electronic equipment, storage medium and program product, and belongs to the technical field of robot visual language navigation. The method comprises the following steps: in response to a navigation instruction, at least one atomic action obtained by analyzing the navigation instruction is acquired; target knowledge data is screened from a knowledge database based on a current atomic action; node features associated with the current atomic action in a navigation graph are updated based on the target knowledge data; and the current atomic action is executed based on the updated node features. The node features associated with the current atomic action in the navigation graph are injected with features through the target knowledge data, so as to alleviate the semantic gap between the instruction and the low-level action, significantly improve the path success rate and the cross-environment generalization capability, and further improve the success rate of navigation.

Description

Technical Field

[0001] This invention relates to the field of robot visual language navigation technology, and in particular to a robot navigation method, device, electronic device, storage medium and program product. Background Technology

[0002] Vision-and-Language Navigation (VLN) tasks require robots to move autonomously in a three-dimensional environment based on natural language commands. Early research was mostly conducted in discrete environments, using predefined navigation maps for decision-making, which simplified environmental perception and action space to some extent. However, as research has expanded to continuous environments (VLN-CE), robots need to perform low-level actions on a three-dimensional mesh, resulting in a large and diverse action space, significantly increasing the complexity of perception and decision-making.

[0003] In a continuous environment, an instruction is often broken down into multiple atomic actions. Each atomic action is usually anchored to a specific landmark or spatial relationship and needs to be completed by multiple low-level action sequences. For example, the instruction "go to the kitchen" not only requires identifying the target area, but also adjusting orientation and executing a series of actions from a dynamic perspective.

[0004] Existing methods attempt to enhance navigation through single-frame images or textual representations, but these static representations can only provide landmark cues and are difficult to capture dynamic information during the robot's interaction with landmarks. Therefore, they lack generalization ability in continuous environments, resulting in a gap between instructions and actions, which can easily lead to navigation failure. Summary of the Invention

[0005] This invention provides a robot navigation method, device, electronic device, storage medium, and program product to solve the problem that there is a gap between instructions and actions in existing navigation methods, which easily leads to navigation failure.

[0006] This invention provides a robot navigation method, which includes the following steps: In response to a navigation command, at least one atomic action obtained by parsing the navigation command is acquired, wherein the at least one atomic action includes the current atomic action; Based on the current atomic action, target knowledge data is obtained by filtering from the knowledge database; Update the node features in the navigation graph associated with the current atomic action based on the target knowledge data; The current atomic action is performed based on the updated node characteristics.

[0007] According to the present invention, a robot navigation method is provided, wherein obtaining target knowledge data from a knowledge database based on the current atomic action specifically includes: Based on the current atomic action, candidate knowledge data is obtained by filtering from the knowledge database; The target knowledge data is obtained by filtering the candidate knowledge data from the images acquired by the image acquisition module.

[0008] According to the present invention, a robot navigation method is provided, wherein each piece of knowledge data in the knowledge database includes an atomic action, and the step of filtering candidate knowledge data from the knowledge database based on the current atomic action specifically includes: Obtain the action similarity between the current atomic action and the atomic actions contained in each knowledge data in the knowledge database; Based on the action similarity, candidate knowledge data is obtained by filtering from the knowledge database.

[0009] According to the present invention, a robot navigation method is provided, wherein each knowledge data in the knowledge database further includes a visual image sequence, and the target knowledge data is obtained by filtering the candidate knowledge data based on the images acquired by the image acquisition module, specifically including: Obtain the visual image sequence contained in each piece of knowledge data in the candidate knowledge data and the image acquired by the image acquisition module; The image acquired by the image acquisition module is input into the shared encoder to obtain the first extraction result output by the shared encoder; The visual image sequence contained in each of the candidate knowledge data is input into the shared encoder to obtain multiple second extraction results output by the shared encoder. Determine the average similarity between the first extraction result and each of the second extraction results; According to the average similarity from largest to smallest, the visual image sequence contained in each knowledge data in the candidate knowledge data is reordered to obtain the reordering result; Based on the re-ranking results, target knowledge data is obtained by filtering from the candidate knowledge data.

[0010] According to the present invention, a robot navigation method is provided, wherein the node features associated with the current atomic action in the navigation graph include basic text features; the step of updating the node features associated with the current atomic action in the navigation graph based on the target knowledge data specifically includes: The target knowledge data and the node features associated with the current atomic action are input into the first processing model to obtain the coarse-grained enhanced features output by the first processing model. The target knowledge data and the node features associated with the current atomic action are input into the second processing model to obtain the fine-grained enhanced features output by the second processing model; The basic text features, the coarse-grained enhancement features, and the fine-grained enhancement features are weighted and combined to obtain the node features associated with the current atomic action in the updated navigation graph; The first processing model processes data with less precision than the second processing model.

[0011] According to the present invention, a robot navigation method is provided, wherein the first processing model includes a visual language encoding model, and the step of inputting the target knowledge data and the node features associated with the current atomic action into the first processing model to obtain the coarse-grained enhanced features output by the first processing model specifically includes: The visual image sequence in the target knowledge data is input into the visual language coding model to obtain a frame feature sequence. The frame features in the frame feature sequence correspond one-to-one with the visual images in the visual image sequence in the target knowledge data. The frame feature sequence is aggregated using a guided attention mechanism to obtain a compressed vector; The coarse-grained enhancement feature is obtained based on the node features associated with the compression vector and the current atomic action.

[0012] According to the present invention, a robot navigation method is provided, wherein the second processing model includes a cross-attention mechanism model, and the step of inputting the target knowledge data and the node features associated with the current atomic action into the second processing model to obtain the fine-grained enhancement features output by the second processing model specifically includes: The first frame sequence is obtained by sequentially splicing the visual image sequences in the target knowledge data. The node features associated with the first frame sequence and the current atomic action are input into the cross-attention mechanism model to obtain fine-grained enhanced features.

[0013] According to the present invention, a robot navigation method is provided, the robot navigation method further comprising: Obtain a training dataset, which includes multiple demonstration instructions and trajectory image sequence samples corresponding to each demonstration instruction; The demonstration instruction is decomposed to obtain at least one atomic action sample corresponding to the demonstration instruction; The visual image sequence sample corresponding to each atomic action sample is determined based on at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction. The knowledge database is constructed based on at least one atomic action sample corresponding to the demonstration instruction and visual image sequence samples associated with at least one atomic action sample corresponding to the demonstration instruction.

[0014] According to the present invention, a robot navigation method is provided, wherein determining a visual image sequence sample corresponding to each atomic action sample based on at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction specifically includes: At least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction are respectively input into the visual language coding model to obtain the frame feature sample and text description feature sample output by the visual language coding model. A similarity matrix is ​​constructed based on the frame feature samples and the text description feature samples; Based on the similarity matrix, an alignment relationship is constructed between at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction; Based on the alignment relationship, keyframes related to at least one atomic action sample corresponding to the demonstration instruction are determined; Determine the adjacent frames corresponding to each keyframe; Based on the keyframes and adjacent frames, a visual image sequence sample is constructed that is associated with at least one atomic action sample corresponding to the demonstration instruction.

[0015] The present invention also provides a robot navigation device, the robot navigation device comprising the following modules: an acquisition module, configured to acquire at least one atomic action obtained by parsing the navigation command in response to a navigation command, wherein the at least one atomic action includes the current atomic action; The filtering module is used to filter target knowledge data from the knowledge database based on the current atomic action; The update module is used to update the node features in the navigation graph associated with the current atomic action based on the target knowledge data; An execution module is used to execute the current atomic action based on the updated node features.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot navigation method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot navigation method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot navigation method as described above.

[0019] The present invention provides a robot navigation method that, during the navigation process, retrieves a knowledge base, dynamically acquires the visual sequence most relevant to the current atomic action, i.e., target knowledge data, and injects features into the node features associated with the current atomic action in the navigation graph based on the target knowledge data, thereby alleviating the semantic gap between instructions and low-level actions.

[0020] By applying the above navigation methods, the path success rate and cross-environment generalization ability can be significantly improved for visual language navigation tasks in multiple continuous environments, thereby increasing the probability of navigation success. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the robot navigation method provided by the present invention; Figure 2 This is a schematic diagram illustrating the construction of the knowledge database provided by this invention; Figure 3 This is a schematic diagram of the overall process of the robot navigation method provided by the present invention; Figure 4 This is a schematic diagram illustrating the difference in navigation effects between the baseline model provided by this invention and the model enhanced with action sequence knowledge. Figure 5 This is a schematic block diagram of the robot navigation device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The following is combined with Figures 1-6 The robot navigation method, apparatus, electronic device, storage medium, and program product described in this invention aim to solve the problem of navigation failure caused by the gap between instructions and actions in existing navigation methods.

[0025] Figure 1 This is one of the flowcharts illustrating the robot navigation method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps: Step 101: In response to a navigation instruction, obtain at least one atomic action obtained from parsing the navigation instruction, wherein the at least one atomic action includes the current atomic action; In some embodiments, navigation instructions may be instructions received by the robot from the user. For example, if the user and the robot interact via voice technology, the navigation instructions may be instructions recognized by the robot through voice technology.

[0026] For example, a navigation instruction could be "Go to the kitchen and check the current temperature displayed on the water heater".

[0027] In some embodiments, navigation instructions may be triggered autonomously by the robot during its operation, when a certain scenario is met.

[0028] For example, when traveling from the bedroom to the kitchen, one needs to pass through the living room, and when a long table is detected in the living room, a navigation instruction to "walk along the long table" is automatically generated.

[0029] Among them, the series of actions obtained after parsing navigation instructions can be considered as atomic actions when a certain action cannot be further broken down into more detailed actions.

[0030] Step 102: Based on the current atomic action, filter the target knowledge data from the knowledge database; The knowledge database can be understood as a pre-built database containing multiple knowledge data. This knowledge data can be used to enrich the node features associated with atomic actions in the navigation graph, so that the enriched node features can more accurately express the atomic action, thus providing a foundation for visual navigation.

[0031] Step 103: Update the node features associated with the current atomic action in the navigation graph based on the target knowledge data; Step 104: Execute the current atomic action based on the updated node features.

[0032] During navigation, the knowledge database is searched to dynamically obtain the visual sequence most relevant to the current atomic action, i.e., the target knowledge data. Based on the target knowledge data, the features of the nodes associated with the current atomic action in the navigation map are injected to alleviate the semantic gap between instructions and low-level actions.

[0033] By applying the above navigation methods, the path success rate and cross-environment generalization ability can be significantly improved for visual language navigation tasks in multiple continuous environments, thereby increasing the probability of navigation success.

[0034] In some embodiments, target knowledge data is obtained by filtering from a knowledge database based on the current atomic action, specifically including: Candidate knowledge data is obtained by filtering from the knowledge database based on the current atomic action; The target knowledge data is obtained by filtering the images acquired by the image acquisition module from the candidate knowledge data.

[0035] In this embodiment, the knowledge database can be searched in two stages based on the current atomic action and the image acquired by the image acquisition module as filtering conditions, so as to obtain the visual sequence most relevant to the current atomic action.

[0036] In some embodiments, each piece of knowledge data in the knowledge database includes an atomic action. Candidate knowledge data is then filtered from the knowledge database based on the current atomic action, specifically including: Obtain the action similarity between the current atomic action and the atomic actions contained in each knowledge data in the knowledge database; Candidate knowledge data is obtained by filtering from the knowledge database based on action similarity.

[0037] In this embodiment, since each piece of knowledge data in the knowledge database includes an atomic action, the current atomic action can be used as a filtering condition during the process of filtering candidate knowledge data, thereby making the filtered candidate knowledge data related to the current atomic action.

[0038] Specifically, by obtaining the action similarity between the current atomic action and the atomic actions contained in each knowledge data in the knowledge database, knowledge data similar to the current atomic action can be selected from the knowledge database during the process of screening candidate knowledge data. This makes the updated node features more accurate when the target knowledge data obtained in the final screening updates the node features.

[0039] In some embodiments, the action similarity is sorted, and the top K knowledge data with the highest action similarity are selected as candidate knowledge data.

[0040] In some embodiments, action similarity is represented by cosine similarity.

[0041] In some embodiments, at least one atomic action obtained from parsing navigation instructions is a n This indicates that for each atom, an action is performed. Calculate the cosine similarity between the entry and all atomic actions in the knowledge base, and select the K most similar candidate entries to form a set. ; in, It is the set of candidate entries, that is, candidate knowledge data. The filtered knowledge data.

[0042] In some embodiments, each piece of knowledge data in the knowledge database further includes a sequence of visual images. Target knowledge data is obtained by filtering candidate knowledge data based on images acquired by the image acquisition module, specifically including: Obtain the visual image sequence contained in each piece of candidate knowledge data and the image acquired by the image acquisition module; The image acquired by the image acquisition module is input into the shared encoder to obtain the first extraction result output by the shared encoder; The visual image sequence contained in each knowledge data in the candidate knowledge data is input into the shared encoder to obtain multiple second extraction results output by the shared encoder. Determine the average similarity between the first extraction result and each second extraction result; Based on the average similarity from largest to smallest, the visual image sequences contained in each knowledge data in the candidate knowledge data are reordered to obtain the reordering results; Based on the re-ranking results, the target knowledge data is obtained from the candidate knowledge data.

[0043] In this embodiment, candidate knowledge data is obtained by filtering the current atomic action and the atomic actions contained in each knowledge data in the knowledge database. Therefore, the filtering of candidate knowledge data is more inclined to the similarity of atomic actions, and there may still be a mismatch between the target knowledge data and the scenario in which the current atomic action is located.

[0044] Based on this, the images acquired by the image acquisition module can be used to perform secondary screening of candidate knowledge data, thereby selecting the optimal knowledge data, i.e., the optimal target knowledge data, from the candidate knowledge data. When updating the node features associated with the current atomic action using the target knowledge data, the updated node features can be made more accurate.

[0045] Specifically, the images acquired by the image acquisition module and the visual image sequences contained in each piece of candidate knowledge data are input into a shared encoder, so that the visual image sequences can be transformed into corresponding features using the shared encoder, and the similarity between features can be used to filter the knowledge data.

[0046] For example, the image acquired by the image acquisition module is based on the current observation. This indicates that the current observations The frame images of the candidate entries (i.e., the visual image sequence contained in each knowledge data in the candidate knowledge data) are respectively input into the shared encoder E(·) for feature extraction, and their average similarity is calculated. Finally, the optimal sequence is selected, and its expression is as follows: ; in, This represents the optimal sequence. As with atomic action The paired action sequence knowledge, i.e., the target knowledge data. Indicates the current observation The corresponding features, Represents the first in the candidate knowledge data j The features corresponding to the visual image sequence contained in a knowledge data set. This indicates the number of images contained in a visual image sequence. Indicates the current observation The candidate space is formed by the sequence of visual images contained in each piece of knowledge data in the candidate knowledge data.

[0047] In some embodiments, the node features associated with the current atomic action in the navigation graph include basic text features; updating the node features associated with the current atomic action in the navigation graph based on target knowledge data specifically includes: Input the node features associated with the target knowledge data and the current atomic action into the first processing model to obtain the coarse-grained enhanced features output by the first processing model; Input the node features associated with the target knowledge data and the current atomic action into the second processing model to obtain the fine-grained enhanced features output by the second processing model; We perform a weighted combination of basic text features, coarse-grained enhancement features, and fine-grained enhancement features to obtain the node features associated with the current atomic action in the updated navigation graph; The first processing model processes data with less precision than the second processing model.

[0048] In this embodiment, the basic text features, coarse-grained enhancement features, and fine-grained enhancement features are weighted and combined so that the content obtained by weighted summation can replace the original basic text features, thereby realizing the update of node features.

[0049] For example, the node features associated with the previous atomic action, after being updated, are represented by the following formula: ⋅ ; in, This represents the updated representation of node features associated with previous atomic actions. and The parameters are non-negative learnable parameters. Based on basic text features, For coarse-grained enhancement features, This is a fine-grained enhancement feature.

[0050] In some embodiments, the first processing model includes a visual language encoding model, which inputs the target knowledge data and the node features associated with the current atomic action into the first processing model to obtain coarse-grained enhanced features output by the first processing model, specifically including: The visual image sequence in the target knowledge data is input into the visual language coding model to obtain the frame feature sequence. The frame features in the frame feature sequence correspond one-to-one with the visual images in the visual image sequence in the target knowledge data. A guided attention mechanism is used to aggregate the frame feature sequences to obtain a compressed vector; Coarse-grained enhanced features are obtained based on node features associated with the compression vector and the current atomic action.

[0051] In this embodiment, a visual language coding model is used to extract features from the visual image sequence in the target knowledge data to obtain a frame feature sequence corresponding to the visual image sequence in the target knowledge data. The frame feature sequence can be understood as a sequence composed of frame features.

[0052] The use of guided attention mechanisms can aggregate the sequence of frame features, i.e., the frame feature sequence, while selectively focusing on important parts of the frame feature sequence to form a compressed vector.

[0053] Specifically, in the coarse-grained path, let For atomic actions Text embedding, To retrieve sequence frames The feature representation is then aggregated through an instruction-guided attention mechanism to obtain a compressed vector, the formula of which is as follows: ; in, For the first frame feature sequence n Each frame feature It is a temperature parameter.

[0054] Then the compression vector As an action representation aligned with instructions, it is used in conjunction with navigation graph node features through a cross-attention mechanism. Merge, where M is the number of nodes in the current topology graph: ； in, This represents the cross-attention mechanism.

[0055] In some embodiments, the second processing model includes a cross-attention mechanism model, which inputs the target knowledge data and the node features associated with the current atomic action into the second processing model to obtain fine-grained enhanced features output by the second processing model, specifically including: The first frame sequence is obtained by sequentially splicing the visual image sequences in the target knowledge data. By inputting the node features associated with the first frame sequence and the current atomic action into the cross-attention mechanism model, fine-grained enhanced features are obtained.

[0056] In this embodiment, in the fine-grained path, all retrieved frame sequences are concatenated into a unified frame sequence according to the instruction order. Furthermore, by fusing cross-attention and node features, step-by-step temporal details are preserved: ; in, This represents the cross-attention mechanism.

[0057] In this process, the use of coarse-grained and fine-grained enhancement features enables high-level semantic alignment and progressive temporal guidance, thereby improving the accuracy of low-level action prediction.

[0058] In some embodiments, the robot navigation method further includes: Obtain the training dataset, which includes multiple demonstration instructions and trajectory image sequence samples corresponding to each demonstration instruction; The demonstration instructions are decomposed to obtain at least one atomic action sample corresponding to the demonstration instructions; The visual image sequence sample corresponding to each atomic action sample is determined based on at least one atomic action sample corresponding to the demonstration command and the trajectory image sequence sample corresponding to the demonstration command. A knowledge database is constructed based on at least one atomic action sample corresponding to the demonstration instruction and visual image sequence samples related to at least one atomic action sample corresponding to the demonstration instruction.

[0059] In this embodiment, the knowledge database adopts a pre-built database, which forms a "build-retrieve-inject" process architecture with other steps of the present invention. This can significantly alleviate the semantic gap between instructions and low-level actions, and significantly improve the path success rate and cross-environment generalization ability in multiple continuous environment visual language navigation tasks.

[0060] In some embodiments, the trajectory image sequence samples corresponding to the demonstration instructions can be obtained directly using existing simulators.

[0061] Specifically, the Matterport3D simulator is used to obtain the demonstration trajectory using the annotations of the training dataset, and an image sequence is generated using a fixed 60-degree field of view, which is to obtain the trajectory image sequence sample corresponding to the demonstration command.

[0062] In some embodiments, instruction decomposition prompts are pre-constructed, which guide the large language model to decompose complex navigation instructions into progressive subtasks, each consisting of an action and a landmark, thereby covering key locations and transitional scenarios in the instructions. The prompts impose constraints on the model, including: they must cover key locations and transitional scenarios; descriptions of independent actions without landmarks or combining multiple actions are prohibited; and the output must remain concise without altering the original semantics.

[0063] Specifically, the constraints on the model by the prompt words include: prohibiting the appearance of independent actions lacking landmarks or the combined description of multiple actions; requiring the generated results to be concise and clear and consistent with the original semantics; and requiring that reference landmarks be explicitly specified for turning actions.

[0064] Using the above design, the model can generate a standardized sequence of atomic actions for subsequent navigation planning and execution.

[0065] In this process, the demonstration instructions can be directly broken down into... , where N is the number of atomic actions.

[0066] In some embodiments, determining the visual image sequence sample corresponding to each atomic action sample based on at least one atomic action sample corresponding to the demonstration instruction and trajectory image sequence samples corresponding to the demonstration instruction specifically includes: Input at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction into the visual language coding model to obtain the frame feature sample and text description feature sample corresponding to the output of the visual language coding model. A similarity matrix is ​​constructed based on frame feature samples and text description feature samples; Based on the similarity matrix, an alignment relationship is constructed between at least one atomic action sample corresponding to the demonstration command and a trajectory image sequence sample corresponding to the demonstration command; Based on alignment, determine the keyframes associated with at least one atomic action sample corresponding to the demonstration instruction; Determine the adjacent frames corresponding to each keyframe; Based on keyframes and adjacent frames, construct and demonstrate visual image sequence samples related to at least one atomic action sample corresponding to the instruction.

[0067] In this embodiment, at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction are respectively input into the visual language coding model, and the atomic action sample and trajectory image sequence sample can be represented as frame feature samples and text description feature samples.

[0068] The representation of frame feature samples and text description feature samples can be used to construct a similarity matrix. Then, the similarity matrix can be used to construct the alignment relationship between at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction. In this process, a frame image corresponding to the atomic action sample, i.e., the keyframe, can be found in time sequence.

[0069] Specifically, for a trajectory image sequence of length T and atomic action sequences of length N Among them, atomic action sequence That is, at least one atomic action sample corresponding to the demonstration command mentioned above, a trajectory image sequence of length T. That is, the trajectory image sequence samples corresponding to the demonstration instructions mentioned above are used to obtain frame feature samples and text description feature samples through the shared visual language coding model E(·), where, .

[0070] The similarity matrix between all frame feature samples and text description feature samples is calculated. By gradually accumulating the optimal matching score through dynamic programming, a monotonic alignment relationship between atomic actions and frames is established. The recursive formula is as follows: in This represents the optimal cumulative similarity between the first n atomic actions and the first t frames of images. After considering all atomic actions, the maximum score is found by searching the last row of the expression. Then, tracing back the path from back to front, we obtain a strictly increasing sequence of frame indices. And each frame Specify as the corresponding atomic action The core frame ensures the consistency between the action sequence and the visual trajectory in the time dimension.

[0071] In the core frame determined for atomic actions Within a time neighborhood, a window size w is set, and similarity is calculated for adjacent frames within the range before and after the core frame. The similarity is calculated when the following condition is met: ∈ in, When a preset similarity threshold is set, the adjacent frames are included in the frame sequence of the corresponding atomic action to increase visual context information while maintaining temporal consistency. The expanded alignment result is stored as a knowledge entry in the action sequence knowledge base, i.e., the knowledge database.

[0072] Among them, the core frame, which is also the key frame in this application, is a frame that meets the conditions as an adjacent frame.

[0073] like Figure 2 As shown, the construction of a knowledge database can be divided into the following three steps: Step S1: Demonstrate the parsing of instruction-trajectory pairs into atomic action samples using a large language model. Each atomic action sample corresponds to a basic behavior anchored by a landmark.

[0074] For example, the demonstration instructions are: walk past the sofa and bookshelf, walk past the dining table, then turn left and stop at the refrigerator. Based on LLM task decomposition, atomic actions can be obtained, namely: 1. Passing by the sofa.

[0075] 2. Passing by the bookshelf.

[0076] 3. Turn left and pass the restaurant tables.

[0077] 4. Stop in front of the refrigerator.

[0078] Step S2: To establish the correspondence between atomic action samples and trajectory frame samples, the system uses a shared visual-language encoder to perform feature representation on image frame samples and atomic action samples, calculates a similarity matrix, and uses dynamic programming to perform global monotonic alignment, thereby selecting the core frame.

[0079] For example, the navigation trajectory can form a task-frame similarity matrix.

[0080] Step S3: Within the temporal neighborhood of the core frame, expand the selection of context frames based on local semantic similarity.

[0081] For example, the navigation trajectory can form a task-frame similarity matrix. Here, the context frame is also known as the adjacent frame in this invention, and the purpose of the adjacent frame is to supplement visual context information.

[0082] Based on the above steps, all action sequences are constructed from the training set trajectories, forming a knowledge base composed of "action-frame sequence" pairs, which is used for subsequent navigation reasoning and planning.

[0083] For example, an action sequence knowledge base is obtained, which includes passing the sofa, passing the bookshelf, turning left, passing the dining table, and stopping in front of the refrigerator.

[0084] In some embodiments, such as Figure 3 As shown, during the initialization phase, the input natural language commands are parsed into a series of atomic actions by a large language model. For example, the command "turn around and leave the room, pass the table, and wait by the light switch" is parsed into the series of atomic actions "turn around, pass through, experience, turn left to enter, and stop." Subsequently, during each navigation step, the current observation information is combined, such as updating the topology map based on the panoramic view and the parsed atomic actions, to retrieve the corresponding visual sequence from the Action Sequence Knowledge (ASK), i.e., the knowledge base in this invention. The retrieved visual sequence is then processed sequentially through coarse-grained (e.g., coarse-grained action interaction obtained from coarse-grained action knowledge) and fine-grained interaction modules (e.g., fine-grained action interaction obtained from fine-grained action knowledge) to generate action-enhanced node representations. These enhanced representations are further used for cross-modal planning, such as the navigation planner executing low-level actions. The navigation planner includes content obtained through multi-layered transformation and embedding of commands, thereby supporting the prediction and execution of low-level actions and improving the perception accuracy and decision-making ability during navigation. Specifically, robot navigation mainly includes the following steps: Step S1: Atomic Action Decomposition The demonstration instructions and trajectories in the training data are collected and parsed. The complex instructions are decomposed into multiple atomic action samples using a large language model. Each atomic action sample corresponds to a landmark-related basic behavior.

[0085] Step S2: Core Frame Alignment The trajectory image sequence samples are associated with the atomic action samples. The features of the two are represented by a shared visual-language encoding model. A monotonic matching relationship is established in time order using dynamic programming, thereby determining a core frame for each atomic action.

[0086] Step S3: Context Frame Expansion Within the temporal neighborhood of each core frame, adjacent image frames with high similarity to the core frame are selected to form a more complete action visual sequence sample. The context frame is also the adjacent frame in this invention.

[0087] Step S4: Action Sequence Knowledge Retrieval During the navigation process, for each atomic action obtained from the parsing, firstly, several candidate knowledge entries that are closest to the action are selected by the instruction similarity. Then, the candidate entries are reordered according to the current observed image. Finally, the most relevant visual sequence is selected as the knowledge supplement for the action.

[0088] Step S5: Action Sequence Knowledge Injection The retrieved action sequence knowledge is combined with the node features in the navigation graph through two fusion paths: coarse-grained and fine-grained, to obtain the final node representation. Then, the retrieval and fusion module is jointly trained end-to-end with the cross-modal planner, and low-level actions are output based on the enhanced node features to complete the navigation task.

[0089] Figure 4 This is a schematic diagram illustrating the difference in navigation effects between the baseline model provided by this invention and the model enhanced with action sequence knowledge.

[0090] like Figure 4 As shown, the instructions are: walk along the brown table to the porcelain display area, turn left in front of the display area and stop near the brown pillar.

[0091] The retrieved action sequence knowledge is: brown table, brown table, brown table, brown table, walk along the long table.

[0092] For the instruction "move along the brown table," the baseline model ETPNav initially headed in the correct direction but quickly deviated from the expected path, turning away from the table and crossing the room, failing to execute continuous and directional movement correctly. However, by introducing action sequence knowledge, ETPNav+ASK retrieves relevant visual sequences (such as "walk along the long table"), obtaining progressive visual contextual guidance that enables the agent to consistently maintain the correct orientation along the edge of the table, ultimately reaching the target location accurately.

[0093] It should be noted that the robot navigation device provided by the present invention can execute the robot navigation method of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0094] Figure 5 This is a schematic block diagram of the robot navigation device provided by the present invention. Figure 5 As shown, the robot navigation device includes: The acquisition module 501 is used to acquire at least one atomic action obtained by parsing the navigation instruction in response to the navigation instruction, wherein the at least one atomic action includes the current atomic action; The filtering module 502 is used to filter target knowledge data from the knowledge database based on the current atomic action; Update module 503 is used to update the node features associated with the current atomic action in the navigation graph based on the target knowledge data; Execution module 504 is used to execute the current atomic action based on the updated node features.

[0095] In this embodiment, during navigation, the knowledge base is retrieved to dynamically obtain the visual sequence most relevant to the current atomic action, i.e., target knowledge data. Based on the target knowledge data, feature injection is performed on the node features associated with the current atomic action in the navigation map, thereby alleviating the semantic gap between instructions and low-level actions.

[0096] By applying the above navigation methods, the path success rate and cross-environment generalization ability can be significantly improved for visual language navigation tasks in multiple continuous environments, thereby increasing the probability of navigation success.

[0097] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 660, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 660. The processor 610 can call logical instructions from the memory 630 to execute a robot navigation method. This method includes: in response to a navigation instruction, acquiring at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action including a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating the node features associated with the current atomic action in the navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node features.

[0098] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the robot navigation method provided in the above embodiments. The method includes: in response to a navigation instruction, acquiring at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action including a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating node features associated with the current atomic action in the navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node features.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the robot navigation method provided in the above embodiments. The method includes: in response to a navigation command, acquiring at least one atomic action obtained by parsing the navigation command, the at least one atomic action including a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating node features associated with the current atomic action in a navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node features.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot navigation method, characterized in that, include: In response to a navigation command, at least one atomic action obtained by parsing the navigation command is acquired, wherein the at least one atomic action includes the current atomic action; Based on the current atomic action, target knowledge data is obtained by filtering from the knowledge database; Update the node features in the navigation graph associated with the current atomic action based on the target knowledge data; Execute the current atomic action based on the updated node features; The node features associated with the current atomic action in the navigation graph include basic text features; The step of updating the node features associated with the current atomic action in the navigation graph based on the target knowledge data specifically includes: The target knowledge data and the node features associated with the current atomic action are input into the first processing model to obtain the coarse-grained enhanced features output by the first processing model. The target knowledge data and the node features associated with the current atomic action are input into the second processing model to obtain the fine-grained enhanced features output by the second processing model; The basic text features, the coarse-grained enhancement features, and the fine-grained enhancement features are weighted and combined to obtain the node features associated with the current atomic action in the updated navigation graph; The first processing model processes data with less precision than the second processing model.

2. The robot navigation method according to claim 1, characterized in that, The process of obtaining target knowledge data from the knowledge database based on the current atomic action specifically includes: Based on the current atomic action, candidate knowledge data is obtained by filtering from the knowledge database; The target knowledge data is obtained by filtering the candidate knowledge data from the images acquired by the image acquisition module.

3. The robot navigation method according to claim 2, characterized in that, Each piece of knowledge data in the knowledge database includes an atomic action. The process of filtering candidate knowledge data from the knowledge database based on the current atomic action specifically includes: Obtain the action similarity between the current atomic action and the atomic actions contained in each knowledge data in the knowledge database; Based on the action similarity, candidate knowledge data is obtained by filtering from the knowledge database.

4. The robot navigation method according to claim 2, characterized in that, Each piece of knowledge data in the knowledge database also includes a sequence of visual images. The target knowledge data is obtained by filtering the images acquired by the image acquisition module from the candidate knowledge data, specifically including: Obtain the visual image sequence contained in each piece of knowledge data in the candidate knowledge data and the image acquired by the image acquisition module; The image acquired by the image acquisition module is input into the shared encoder to obtain the first extraction result output by the shared encoder; The visual image sequence contained in each of the candidate knowledge data is input into the shared encoder to obtain multiple second extraction results output by the shared encoder. Determine the average similarity between the first extraction result and each of the second extraction results; According to the average similarity from largest to smallest, the visual image sequence contained in each knowledge data in the candidate knowledge data is reordered to obtain the reordering result; Based on the re-ranking results, target knowledge data is obtained by filtering from the candidate knowledge data.

5. The robot navigation method according to claim 1, characterized in that, The first processing model includes a visual language encoding model. The step of inputting the target knowledge data and the node features associated with the current atomic action into the first processing model to obtain the coarse-grained enhanced features output by the first processing model specifically includes: The visual image sequence in the target knowledge data is input into the visual language coding model to obtain a frame feature sequence. The frame features in the frame feature sequence correspond one-to-one with the visual images in the visual image sequence in the target knowledge data. The frame feature sequence is aggregated using a guided attention mechanism to obtain a compressed vector; The coarse-grained enhancement feature is obtained based on the node features associated with the compression vector and the current atomic action.

6. The robot navigation method according to claim 1, characterized in that, The second processing model includes a cross-attention mechanism model. The step of inputting the target knowledge data and the node features associated with the current atomic action into the second processing model to obtain the fine-grained enhanced features output by the second processing model specifically includes: The first frame sequence is obtained by sequentially splicing the visual image sequences in the target knowledge data. The node features associated with the first frame sequence and the current atomic action are input into the cross-attention mechanism model to obtain fine-grained enhanced features.

7. The robot navigation method according to any one of claims 1 to 6, characterized in that, The robot navigation method further includes: Obtain a training dataset, which includes multiple demonstration instructions and trajectory image sequence samples corresponding to each demonstration instruction; The demonstration instruction is decomposed to obtain at least one atomic action sample corresponding to the demonstration instruction; The visual image sequence sample corresponding to each atomic action sample is determined based on at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction. The knowledge database is constructed based on at least one atomic action sample corresponding to the demonstration instruction and visual image sequence samples associated with at least one atomic action sample corresponding to the demonstration instruction.

8. The robot navigation method according to claim 7, characterized in that, The step of determining the visual image sequence sample corresponding to each atomic action sample based on at least one atomic action sample corresponding to the demonstration instruction and trajectory image sequence samples corresponding to the demonstration instruction specifically includes: At least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction are respectively input into the visual language coding model to obtain the frame feature sample and text description feature sample output by the visual language coding model. A similarity matrix is ​​constructed based on the frame feature samples and the text description feature samples; Based on the similarity matrix, an alignment relationship is constructed between at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction; Based on the alignment relationship, keyframes related to at least one atomic action sample corresponding to the demonstration instruction are determined; Determine the adjacent frames corresponding to each keyframe; Based on the keyframe and the adjacent frames, construct a visual image sequence sample related to at least one atomic sample action corresponding to the demonstration instruction.

9. A robot navigation device, characterized in that, The robot navigation device includes: The acquisition module is configured to, in response to a navigation instruction, acquire at least one atomic action obtained by parsing the navigation instruction, wherein the at least one atomic action includes the current atomic action; The filtering module is used to filter target knowledge data from the knowledge database based on the current atomic action; The update module is used to update the node features in the navigation graph associated with the current atomic action based on the target knowledge data; The execution module is used to execute the current atomic action based on the updated node features; The node features associated with the current atomic action in the navigation graph include basic text features; The step of updating the node features associated with the current atomic action in the navigation graph based on the target knowledge data specifically includes: The target knowledge data and the node features associated with the current atomic action are input into the first processing model to obtain the coarse-grained enhanced features output by the first processing model. The target knowledge data and the node features associated with the current atomic action are input into the second processing model to obtain the fine-grained enhanced features output by the second processing model; The basic text features, the coarse-grained enhancement features, and the fine-grained enhancement features are weighted and combined to obtain the node features associated with the current atomic action in the updated navigation graph; The first processing model processes data with less precision than the second processing model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot navigation method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot navigation method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot navigation method as described in any one of claims 1 to 8.

Images