Robot navigation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a robot navigation method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of robot visual language navigation, and the robot navigation method comprises the following steps: in response to a navigation instruction, obtaining at least one atomic action obtained by analyzing the navigation instruction; screening from a knowledge database based on the current atomic action to obtain target knowledge data; updating node features associated with the current atomic action in the navigation map based on the target knowledge data; and executing the current atomic action based on the updated node features, and performing feature injection on the node features associated with the current atomic action in the navigation map through the target knowledge data so as to relieve a semantic gap between an instruction and a low-level action, so that the path success rate and the cross-environment generalization ability are remarkably improved, and the success probability of navigation is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision language navigation, and in particular to a robot navigation method and device, an electronic device, a storage medium and a program product. BACKGROUND

[0002] The vision-and-language navigation (VLN) task requires a robot to autonomously move in a three-dimensional environment according to natural language instructions. Early research is mostly carried out in discrete environments, and a pre-defined navigation graph is used for decision-making, which to some extent simplifies the environment perception and action space. However, as research extends to continuous environments (VLN-CE), the robot needs to perform low-level actions on a three-dimensional grid, and the action space is large and diverse, and the complexity of perception and decision-making increases significantly.

[0003] In a continuous environment, an instruction is often decomposed into multiple atomic actions, each of which is usually anchored to a specific landmark or spatial relationship and needs to be completed by multiple low-level action sequences. For example, the instruction "walk to the kitchen" not only requires the identification of the target area, but also needs to adjust the orientation in a dynamic view and perform a continuous action sequence.

[0004] Existing methods attempt to enhance navigation through single-frame images or textual representations, but these static representations can only provide landmark clues and are difficult to capture dynamic information in the process of robot interaction with landmarks, thus lacking generalization ability in continuous environments, resulting in a gap between instructions and actions, which easily causes navigation failure. SUMMARY

[0005] The present application provides a robot navigation method, device, electronic device, storage medium and program product to solve the problem of the gap between instructions and actions in existing navigation methods, which easily causes navigation failure.

[0006] The present application provides a robot navigation method, which comprises the following steps: In response to a navigation instruction, at least one atomic action obtained by parsing the navigation instruction is acquired, and the at least one atomic action includes a current atomic action; Target knowledge data is filtered from a knowledge database based on the current atomic action; The node features associated with the current atomic action in the navigation graph are updated based on the target knowledge data; The current atomic action is executed based on the updated node features.

[0007] According to the present application, a robot navigation method is provided, and the target knowledge data is filtered from a knowledge database based on the current atomic action, specifically comprising: filtering candidate knowledge data from the knowledge database based on the current atomic action; filtering target knowledge data from the candidate knowledge data based on an image collected by an image collection module.

[0008] According to the application, a robot navigation method is provided, each knowledge data in the knowledge database comprises an atomic action, the candidate knowledge data is filtered from the knowledge database based on the current atomic action, and specifically includes: obtaining an action similarity between the current atomic action and an atomic action contained in each knowledge data in the knowledge database; filtering the candidate knowledge data from the knowledge database based on the action similarity.

[0009] According to the application, a robot navigation method is provided, each knowledge data in the knowledge database further comprises a visual image sequence, the target knowledge data is filtered from the candidate knowledge data based on an image collected by an image collection module, and specifically includes: obtaining a visual image sequence contained in each knowledge data in the candidate knowledge data and an image collected by the image collection module; inputting the image collected by the image collection module into a shared encoder to obtain a first extraction result output by the shared encoder; inputting the visual image sequence contained in each knowledge data in the candidate knowledge data into the shared encoder to obtain a plurality of second extraction results output by the shared encoder; determining an average similarity between the first extraction result and each second extraction result; reordering the visual image sequence contained in each knowledge data in the candidate knowledge data according to an order of the average similarity from large to small to obtain a reordering result; filtering the target knowledge data from the candidate knowledge data based on the reordering result.

[0010] According to the application, a robot navigation method is provided, the node feature associated with the current atomic action in the navigation graph comprises a basic text feature, and the node feature associated with the current atomic action in the navigation graph is updated based on the target knowledge data, and specifically includes: inputting the node feature associated with the target knowledge data and the current atomic action into a first processing model to obtain a coarse-grained enhanced feature output by the first processing model; inputting the node feature associated with the target knowledge data and the current atomic action into a second processing model to obtain a fine-grained enhanced feature output by the second processing model. combining the base text features, the coarse-grained enhanced features and the fine-grained enhanced features to obtain updated node features associated with the current atomic action in the updated navigation graph; The first processing model has a lower processing precision than the second processing model.

[0011] According to the present application, a robot navigation method is provided, the first processing model comprises a visual language coding model, the node features associated with the target knowledge data and the current atomic action are input into the first processing model, coarse-grained enhanced features output by the first processing model are obtained, and the method specifically comprises: The visual image sequence in the target knowledge data is input into the visual language coding model to obtain a frame feature sequence, and each frame feature in the frame feature sequence corresponds to a visual image in the visual image sequence in the target knowledge data; The frame feature sequence is aggregated by using a guided attention mechanism to obtain a compressed vector; The coarse-grained enhanced features are obtained based on the compressed vector and the node features associated with the current atomic action.

[0012] According to the present application, a robot navigation method is provided, the second processing model comprises a cross-attention mechanism model, the node features associated with the target knowledge data and the current atomic action are input into the second processing model, fine-grained enhanced features output by the second processing model are obtained, and the method specifically comprises: The visual image sequence in the target knowledge data is sequentially spliced to obtain a first frame sequence; The first frame sequence and the node features associated with the current atomic action are input into the cross-attention mechanism model to obtain fine-grained enhanced features.

[0013] According to the present application, a robot navigation method is provided, and the robot navigation method further comprises: A training data set is obtained, the training data set comprises a plurality of demonstration instructions and a trajectory image sequence sample corresponding to each demonstration instruction; The demonstration instruction is decomposed to obtain at least one atomic action sample corresponding to the demonstration instruction; A visual image sequence sample corresponding to each atomic action sample is determined based on the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction; The knowledge database is constructed based on the at least one atomic action sample corresponding to the demonstration instruction and the visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instruction.

[0014] According to the present application, a robot navigation method is provided, which determines a visual image sequence sample corresponding to each atomic action sample based on at least one atomic action sample corresponding to the demonstration instruction and a trajectory image sequence sample corresponding to the demonstration instruction, and specifically comprises: The at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction are respectively input into a visual language coding model to obtain frame feature samples and text description feature samples output by the visual language coding model; A similarity matrix is constructed based on the frame feature samples and the text description feature samples; An alignment relationship between the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction is constructed based on the similarity matrix; Based on the alignment relationship, a key frame related to the at least one atomic action sample corresponding to the demonstration instruction is determined; A neighboring frame corresponding to each key frame is determined; A visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instruction is constructed based on the key frame and the neighboring frame.

[0015] The present application also provides a robot navigation device, which comprises the following modules: an acquisition module, configured to acquire at least one atomic action obtained by parsing a navigation instruction in response to the navigation instruction, wherein the at least one atomic action comprises a current atomic action; A screening module, configured to screen target knowledge data from a knowledge database based on the current atomic action; An updating module, configured to update a node feature associated with the current atomic action in a navigation graph based on the target knowledge data; An execution module, configured to execute the current atomic action based on the updated node feature.

[0016] The present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the robot navigation method of any of the above when executing the program.

[0017] The present application also provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the robot navigation method of any of the above.

[0018] The present application also provides a computer program product, which comprises a computer program, wherein the computer program is executable by a processor to implement the robot navigation method of any of the above.

[0019] The robot navigation method provided by the application retrieves the knowledge base during navigation, dynamically obtains a visual sequence most relevant to a current atomic action, that is, target knowledge data, and injects features of a node associated with the current atomic action in a navigation graph based on the target knowledge data, so as to alleviate the semantic gap between instructions and low-level actions.

[0020] By applying the above navigation method, the path success rate and cross-environment generalization capability can be significantly improved for multiple continuous environment visual language navigation tasks, thereby improving the success rate of navigation. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a flowchart of the robot navigation method provided by the application; Figure 2 is a schematic diagram of building a knowledge database provided by the application; Figure 3 is a schematic diagram of the overall flow of the robot navigation method provided by the application; Figure 4 is a schematic diagram of the difference effect of the baseline model and the action sequence knowledge enhanced model provided by the application in the navigation process; Figure 5 is a schematic block diagram of the robot navigation device provided by the application; Figure 6 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0024] The robot navigation method, device, electronic device, storage medium and program product provided by the application will be described below. Figures 1-6 The robot navigation method, device, electronic device, storage medium and program product provided by the application are described, aiming to solve the problem that there is a gap between instructions and actions in the existing navigation method, which easily causes navigation failure.

[0025] Figure 1 is one of the flowcharts of the robot navigation method provided by the present application, as shown in the figure, including but not limited to the following steps: Figure 1 Step 101, in response to a navigation instruction, at least one atomic action obtained by parsing the navigation instruction is acquired, and the at least one atomic action includes a current atomic action; In some embodiments, the navigation instruction can be an instruction received by the robot from a user. For example, the user interacts with the robot through voice technology, and the navigation instruction can be an instruction recognized by the robot through voice technology.

[0026] For example, the navigation instruction can be "go to the kitchen and check the temperature displayed by the water heater".

[0027] In some embodiments, the navigation instruction can be triggered autonomously by the robot when a certain scenario is met during the operation of the robot.

[0028] For example, during the process of going from the bedroom to the kitchen, it is necessary to pass through the living room, and when a long table in the living room is detected, a navigation instruction of "walking along the long table" is automatically generated.

[0029] Among them, a series of actions obtained by parsing the navigation instruction can be regarded as atomic actions when one action cannot be further divided into more detailed actions.

[0030] Step 102, target knowledge data is screened from a knowledge database based on the current atomic action; Among them, the knowledge database can be understood as a pre-constructed database, wherein the database contains a plurality of knowledge data, which can be used to enrich the node features associated with the atomic action in the navigation graph, so that the enriched node features can more accurately express the atomic action, thereby providing a basis for visual navigation.

[0031] Step 103, based on the target knowledge data, the node features associated with the current atomic action in the navigation graph are updated; Step 104, the current atomic action is executed based on the updated node features.

[0032] During navigation, the knowledge database is searched to dynamically obtain the most relevant visual sequence of the current atomic action, that is, the target knowledge data, and the node features associated with the current atomic action in the navigation graph are feature-injected based on the target knowledge data, so as to alleviate the semantic gap between the instruction and the low-level action.

[0033] ​By applying the above navigation method, for multiple continuous environment visual language navigation tasks, the path success rate and cross-environment generalization ability can be significantly improved, thereby improving the success rate of navigation.

[0034] In some embodiments, the target knowledge data is filtered from the knowledge database based on the current atomic action, specifically including: The candidate knowledge data is filtered from the knowledge database based on the current atomic action; The target knowledge data is filtered from the candidate knowledge data based on the image collected by the image collection module.

[0035] In this embodiment, the knowledge database can be searched in two stages based on the current atomic action and the image collected by the image collection module as the filtering condition, thereby obtaining the visual sequence most relevant to the current atomic action.

[0036] In some embodiments, each knowledge data in the knowledge database includes an atomic action, and the candidate knowledge data is filtered from the knowledge database based on the current atomic action, specifically including: The action similarity between the current atomic action and the atomic action included in each knowledge data in the knowledge database is obtained; The candidate knowledge data is filtered from the knowledge database based on the action similarity.

[0037] In this embodiment, since each knowledge data in the knowledge database includes an atomic action, the current atomic action can be used as a filtering condition in the process of filtering the candidate knowledge data, thereby making the candidate knowledge data filtered relevant to the current atomic action.

[0038] Specifically, by obtaining the action similarity between the current atomic action and the atomic action included in each knowledge data in the knowledge database, in the process of filtering the candidate knowledge data, the knowledge data similar to the current atomic action can be filtered from the knowledge database, so that when the target knowledge data is finally filtered to update the node feature, the updated node feature is more accurate.

[0039] In some embodiments, the action similarity is sorted, and the top K knowledge data with the highest action similarity is used as the candidate knowledge data.

[0040] In some embodiments, the action similarity is represented by cosine similarity.

[0041] In some embodiments, at least one atomic action obtained by analyzing the navigation instruction is represented as a n , then each atomic action Calculate the cosine similarity with all atomic actions in the knowledge base, and select the top K most similar candidate entries to form a set: ; wherein, is the set formed by the candidate entries, that is, the candidate knowledge data, the filtered knowledge data.

[0042] In some embodiments, each knowledge data in the knowledge database further includes a visual image sequence, and the target knowledge data is filtered from the candidate knowledge data based on the image collected by the image collection module, specifically including: Obtain the visual image sequence contained in each knowledge data in the candidate knowledge data and the image collected by the image collection module; Input the image collected by the image collection module into the shared encoder to obtain a first extraction result output by the shared encoder; Input the visual image sequence contained in each knowledge data in the candidate knowledge data into the shared encoder to obtain a plurality of second extraction results output by the shared encoder; Determine the average similarity of the first extraction result and each second extraction result; According to the order from large to small of the average similarity, reorder the visual image sequence contained in each knowledge data in the candidate knowledge data, and obtain a reordering result; Based on the reordering result, filter the target knowledge data from the candidate knowledge data.

[0043] In this embodiment, the candidate knowledge data is filtered using the current atomic action and the atomic action contained in each knowledge data in the knowledge database, so the filtering of the candidate knowledge data is more inclined to the similarity of the atomic action, and the target knowledge data and the current atomic action in the scene still do not match.

[0044] Based on this, the image collected by the image collection module can be used to perform secondary filtering on the candidate knowledge data, and then the optimal knowledge data, that is, the optimal target knowledge data, is filtered from the candidate knowledge data, and then the target knowledge data is used to update the node features associated with the current atomic action, which can make the updated node features more accurate.

[0045] Specifically, by inputting the image collected by the image collection module and the visual image sequence contained in each knowledge data in the candidate knowledge data into the shared encoder respectively, the shared encoder is used to convert the visual image sequence into corresponding features, and the similarity between the features is used to realize the filtering of the knowledge data.

[0046] Exemplarily, the image collected by the image collection module is a current observation The current observation and the frame image of the candidate item (i.e., the visual image sequence contained in each knowledge data in the candidate knowledge data) are respectively input into a shared encoder E(·) for feature extraction, and the average similarity is calculated, and finally the optimal sequence is selected, and the expression is as follows: ; Wherein, represents the optimal sequence, as an action sequence knowledge paired with the atomic action , that is, the target knowledge data, represents the feature corresponding to the current observation , represents the feature corresponding to the visual image sequence contained in the i-th knowledge data in the candidate knowledge data, j represents the number of images contained in the visual image sequence, represents the candidate space composed of the current observation and the visual image sequence contained in each knowledge data in the candidate knowledge data. In some embodiments, the node feature associated with the current atomic action in the navigation graph includes a basic text feature; updating the node feature associated with the current atomic action in the navigation graph based on the target knowledge data, specifically comprising:

[0047] inputting the target knowledge data and the node feature associated with the current atomic action into a first processing model to obtain a coarse-grained enhanced feature output by the first processing model; inputting the target knowledge data and the node feature associated with the current atomic action into a second processing model to obtain a fine-grained enhanced feature output by the second processing model; combining the basic text feature, the coarse-grained enhanced feature and the fine-grained enhanced feature by weighting to obtain the updated node feature associated with the current atomic action in the navigation graph; Wherein, the first processing model has a lower data processing precision than the second processing model. In this embodiment, the basic text feature, the coarse-grained enhanced feature and the fine-grained enhanced feature are combined by weighting, so as to replace the original basic text feature with the content obtained by weighted summation, thereby realizing the updating of the node feature.

[0048] Exemplarily, the node feature associated with the previous atomic action is updated and represented by the following formula:

[0049] ⋅ ; ​wherein, the node features associated with the previous atomic action are in an updated representation, and are non-negative learnable parameters, are base text features, are coarse-grained enhanced features, are fine-grained enhanced features.

[0050] In some embodiments, the first processing model comprises a visual language coding model, the target knowledge data and the node features associated with the current atomic action are input into the first processing model, and coarse-grained enhanced features output by the first processing model are obtained, specifically comprising: inputting a visual image sequence in the target knowledge data into the visual language coding model to obtain a frame feature sequence, wherein the frame features in the frame feature sequence correspond one-to-one to the visual images in the visual image sequence in the target knowledge data; aggregating the frame feature sequence using a guided attention mechanism to obtain a compressed vector; obtaining coarse-grained enhanced features based on the compressed vector and the node features associated with the current atomic action.

[0051] In this embodiment, the visual image sequence in the target knowledge data is extracted using the visual language coding model to obtain a frame feature sequence corresponding to the visual image sequence in the target knowledge data, wherein the frame feature sequence can be understood as a sequence composed of frame features.

[0052] The use of the guided attention mechanism can aggregate the sequence composed of frame features, i.e., the frame feature sequence, while selectively focusing on important parts of the frame feature sequence to form a compressed vector.

[0053] Specifically, in the coarse-grained path, let be the text embedding of the atomic action , and be the feature representation of the retrieval sequence frame , then the frame features are aggregated by the instruction-guided attention mechanism to obtain a compressed vector, which is expressed as follows: ; wherein, is the i-th frame feature in the frame feature sequence, n is a temperature parameter. The compressed vector

[0054] is an instruction-aligned action representation, which is fused with the navigation graph node features through a cross-attention mechanism, where M is the number of nodes in the current topological graph: ​； wherein, represents a cross-attention mechanism.

[0055] In some embodiments, the second processing model comprises a cross-attention mechanism model, the target knowledge data and the node features associated with the current atomic action are input into the second processing model, and the fine-grained enhanced features output by the second processing model are obtained, specifically comprising: sequentially splicing the visual image sequence in the target knowledge data to obtain a first frame sequence; inputting the first frame sequence and the node features associated with the current atomic action into the cross-attention mechanism model to obtain the fine-grained enhanced features.

[0056] In this embodiment, in the fine-grained path, all the retrieved frame sequences are spliced into a unified frame sequence according to the order of instructions , and are fused with the node features through cross-attention to retain the gradual timing details: ; wherein, represents a cross-attention mechanism.

[0057] In this process, the use of coarse-grained enhanced features and fine-grained enhanced features realizes high-level semantic alignment and gradual timing guidance, and improves the accuracy of low-level action prediction.

[0058] In some embodiments, the robot navigation method further comprises: obtaining a training data set, the training data set comprising a plurality of demonstration instructions and a trajectory image sequence sample corresponding to each demonstration instruction; decomposing the demonstration instruction to obtain at least one atomic action sample corresponding to the demonstration instruction; determining a visual image sequence sample corresponding to each atomic action sample based on the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction; constructing a knowledge database based on the at least one atomic action sample corresponding to the demonstration instruction and the visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instruction.

[0059] In this embodiment, the knowledge database adopts a pre-constructed database, which forms a "construction-retrieval-injection" process architecture with other steps of the present application, can significantly alleviate the semantic gap between instructions and low-level actions, and significantly improve the path success rate and cross-environment generalization ability in multiple continuous environment visual language navigation tasks.

[0060] In some embodiments, the trajectory image sequence sample corresponding to the demonstration instruction can be directly obtained using an existing simulator.

[0061] Specifically, in the Matterport3D simulator, a demonstration trajectory is obtained using the annotation of the training data set, and an image sequence is generated using a fixed 60-degree field of view, i.e., a trajectory image sequence sample corresponding to the demonstration instruction is obtained.

[0062] In some embodiments, a pre-constructed instruction decomposition prompt word is used to guide the large language model to decompose the complex navigation instruction into step-by-step sub-tasks, each step consisting of an action and a landmark, so as to cover the key positions and transition scenes in the instruction. The constraints of the prompt word on the model include: must cover the key positions and transition scenes, cannot appear independent actions without landmarks or descriptions of merging multiple actions, and requires the output to be concise without changing the original semantics.

[0063] Specifically, the constraints of the prompt word on the model include: prohibiting independent actions without landmarks or merged descriptions of multiple actions, requiring the generated result to be concise and consistent with the original semantics, and explicitly specifying the reference landmark for the turning action.

[0064] With the above design, the model can generate a standardized atomic action sequence for subsequent navigation planning and execution.

[0065] In this process, the demonstration instruction can be directly decomposed to obtain where N is the number of atomic actions.

[0066] In some embodiments, the visual image sequence sample corresponding to each atomic action sample is determined based on at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction, specifically including: inputting the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction into a visual language coding model to obtain frame feature samples and text description feature samples corresponding to the visual language coding model output; constructing a similarity matrix based on the frame feature samples and the text description feature samples; constructing an alignment relationship between the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction based on the similarity matrix; determining key frames related to the at least one atomic action sample corresponding to the demonstration instruction based on the alignment relationship; determining the adjacent frames corresponding to each key frame; constructing a visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instruction based on the key frames and the adjacent frames.

[0067] In this embodiment, at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction are respectively input into the visual language coding model, and the atomic action sample and the trajectory image sequence sample can be represented in the form of frame feature sample and text description feature sample.

[0068] The representation of the frame feature sample and the text description feature sample can utilize the frame feature sample and the text description feature sample to construct a similarity matrix, and then utilize the similarity matrix to construct the alignment relationship between the at least one atomic action sample corresponding to the demonstration instruction and the trajectory image sequence sample corresponding to the demonstration instruction. In this process, a frame image corresponding to the atomic action sample, i.e., a key frame, can be found in the time sequence.

[0069] Specifically, for a trajectory image sequence with a length of T and an atomic action sequence with a length of N , wherein the atomic action sequence is at least one atomic action sample corresponding to the demonstration instruction in the foregoing, the trajectory image sequence with a length of T is the trajectory image sequence sample corresponding to the demonstration instruction in the foregoing, and the frame feature sample and the text description feature sample are obtained by the shared visual language coding model E(·), respectively, wherein .

[0070] The similarity matrix of all frame feature samples and text description feature samples is calculated, and the optimal matching score is gradually accumulated by a dynamic programming method to establish a monotonic alignment relationship between the atomic action and the frame, and the recursive formula is: wherein represents the optimal cumulative similarity of the first n atomic actions and the first t frame images, and when all atomic actions are considered, the maximum score is found by the last row , and then the path is backtracked from back to front to obtain a strictly increasing frame index sequence . And each frame is designated as the core frame of the corresponding atomic action , thereby ensuring the consistency of the action sequence and the visual trajectory in the time dimension.

[0071] A window size w is set in the time neighborhood of the core frame of the atomic action determined, and the similarity of the adjacent frames in the range before and after the core frame is calculated, and when the condition: ∈ is met, the core frame is determined as the key frame of the atomic action. For a preset similarity threshold, the adjacent frame is included in the frame sequence of the corresponding atomic action to increase the visual context information while maintaining time consistency. The extended alignment result is stored in the action sequence knowledge base, i.e., the knowledge database.

[0072] Among them, the core frame, i.e., the key frame in the application, is the frame that meets the condition as the adjacent frame.

[0073] As Figure 2 shown, the construction of the knowledge database can be divided into the following three steps, namely: Step S1: The demonstration instruction-trajectory pair is parsed into atomic action samples via a large language model. Each atomic action sample corresponds to a basic behavior anchored by a landmark.

[0074] Exemplarily, the demonstration instruction is: walk through the sofa and bookshelf, walk through the dining table, then turn left, and stop in front of the refrigerator. Based on the task decomposition of the LLM, the atomic actions can be obtained, i.e.: 1. Walk through the sofa.

[0075] 2. Walk through the bookshelf.

[0076] 3. Turn left and walk through the dining table.

[0077] 4. Stop in front of the refrigerator.

[0078] Step S2: To establish the correspondence between the atomic action samples and the trajectory frame samples, the system performs feature representation on the image frame samples and the atomic action samples through a shared visual-linguistic encoder, calculates a similarity matrix, and uses a dynamic programming method for global monotonic alignment to select core frames.

[0079] Exemplarily, the navigation trajectory can form a task-frame similarity matrix.

[0080] Step S3: In the time neighborhood of the core frame, context frames are selected according to local semantic similarity.

[0081] Exemplarily, the navigation trajectory can form a task-frame similarity matrix. Among them, the context frame, i.e., the adjacent frame in the application, aims to supplement the visual context information.

[0082] Based on the above steps, all action sequences are constructed based on the training set trajectory to form a knowledge base composed of "action-frame sequence" pairs, which is used for subsequent navigation reasoning and planning.

[0083] Exemplarily, the action sequence knowledge base is obtained, i.e., it includes walking through the sofa, walking through the bookshelf, turning left, walking through the dining table, and stopping in front of the refrigerator.

[0084] In some embodiments, asFigure 3 As shown, in the initialization phase, the input natural language instruction is parsed into a series of atomic actions via a large language model. Exemplarily, the instruction such as "turn around and leave the room, pass the table, and wait by the light switch" is parsed into a series of atomic actions such as "turn around, pass, go through, turn left, and stop at". Subsequently, in the process of navigation at each step, the current observation information is combined, for example, the topological graph is updated according to the panoramic view, and the parsed atomic action is retrieved from the action sequence knowledge (ASK), i.e., the knowledge base in the present application, to retrieve the corresponding visual sequence. The retrieved visual sequence is sequentially passed through the coarse-grained, e.g., coarse-grained action knowledge, and the fine-grained interaction module, e.g., fine-grained action knowledge, to generate an action-enhanced node representation. These enhanced representations are further used for cross-modal planning, such as the execution of the underlying action by the navigation planner, wherein the navigation planner includes the content obtained by the instruction through multi-layer conversion and instruction embedding, thereby supporting the prediction and execution of low-level actions and improving the perception accuracy and decision-making ability in the navigation process. Specifically, robot navigation mainly includes the following steps: Step S1: Atomic action decomposition Collect the demonstration instructions and trajectories in the training data and perform parsing. Utilize a large language model to decompose complex instructions into multiple atomic action samples, each of which corresponds to a landmark-related basic behavior.

[0085] Step S2: Core frame alignment Correlate the trajectory image sequence sample and the atomic action sample. Perform feature representation on both through a shared vision-language encoding model, and utilize a dynamic programming method to establish a monotonic matching relationship in time sequence, thereby determining a core frame for each atomic action.

[0086] Step S3: Context frame expansion Within the time neighborhood of each core frame, select adjacent image frames with high similarity to the core frame to form a more complete action visual sequence sample, wherein the context frame is the adjacent frame in the present application.

[0087] Step S4: Action sequence knowledge retrieval During the navigation process, for each parsed atomic action, first filter a number of candidate knowledge items closest to the action through instruction similarity, then reorder the candidate items according to the current observation image, and finally select the most relevant visual sequence as the knowledge supplement for the action.

[0088] Step S5: Action sequence knowledge injection The retrieved action sequence knowledge is combined with the node features in the navigation graph through two fusion paths of coarse granularity and fine granularity to obtain the final node representation; then the retrieval and fusion module is trained end-to-end with the cross-modal planner, and based on the enhanced node features, low-level actions are output to complete the navigation task.

[0089] Figure 4 is a schematic diagram provided by the application for the difference effect of the baseline model and the action sequence knowledge enhanced model in the navigation process.

[0090] As Figure 4 shown, the instruction is: walk along the brown table to the porcelain display area, turn left in front of the display area and stop near the brown column.

[0091] The retrieved action sequence knowledge is: brown table, brown table, brown table, brown table, walk along the long table.

[0092] For the instruction "walk along the brown table", although the baseline model ETPNav initially moves in the correct direction, it soon deviates from the expected path, turns away from the table and crosses the room, failing to correctly perform continuous and directional movement. After introducing the action sequence knowledge, ETPNav+ASK obtains step-by-step visual context guidance by retrieving relevant visual sequences (such as "walk along the long table"), enabling the agent to always maintain the correct direction along the table edge and finally accurately reach the target position.

[0093] It should be noted that the robot navigation device provided by the application can execute the robot navigation method of any of the above embodiments when actually running, and this embodiment will not be repeated.

[0094] Figure 5 is a schematic block diagram of the robot navigation device provided by the application. As Figure 5 shown, the robot navigation device comprises: The acquisition module 501 is configured to acquire at least one atomic action obtained by parsing the navigation instruction in response to the navigation instruction, the at least one atomic action comprising a current atomic action. The screening module 502 is configured to screen target knowledge data from the knowledge database based on the current atomic action. The update module 503 is configured to update the node features associated with the current atomic action in the navigation graph based on the target knowledge data. The execution module 504 is configured to execute the current atomic action based on the updated node features.

[0095] In this embodiment, during navigation, the knowledge base is retrieved to dynamically obtain the visual sequence most relevant to the current atomic action, that is, the target knowledge data, and the node features associated with the current atomic action in the navigation graph are injected based on the target knowledge data, so as to alleviate the semantic gap between instructions and low-level actions.

[0096] By applying the above navigation method, for a plurality of continuous environment visual language navigation tasks, the path success rate and cross-environment generalization capability can be significantly improved, thereby improving the success rate of navigation.

[0097] Figure 6 is a structural schematic diagram of an electronic device provided by the application, as Figure 6 shown, the electronic device can include a processor (processor) 610, a communication interface (communications interface) 620, a memory (memory) 630 and a communication bus 660, wherein the processor 610, the communication interface 620, the memory 630 complete the communication between each other through the communication bus 660. The processor 610 can call the logic instructions in the memory 630 to execute the robot navigation method, which includes: in response to a navigation instruction, obtaining at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action including a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating the node features associated with the current atomic action in the navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node features.

[0098] In addition, the logic instructions in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage program codes.

[0099] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the robot navigation method provided by the above-mentioned embodiments, and the method comprises: in response to a navigation instruction, obtaining at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action comprising a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating a node feature associated with the current atomic action in a navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node feature.

[0100] In another aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement a robot navigation method provided by the above-mentioned embodiments, and the method comprises: in response to a navigation instruction, obtaining at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action comprising a current atomic action; filtering target knowledge data from a knowledge database based on the current atomic action; updating a node feature associated with the current atomic action in a navigation graph based on the target knowledge data; and executing the current atomic action based on the updated node feature.

[0101] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0102] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0103] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of robot navigation, characterized by, The method comprises the following steps: in response to a navigation instruction, obtaining at least one atomic action parsed by the navigation instruction, at least one of the atomic actions comprising a current atomic action; based on the current atomic action, screening target knowledge data from a knowledge database; based on the target knowledge data, updating node features associated with the current atomic action in a navigation graph; based on the updated node features, performing the current atomic action.

2. The method of claim 1, wherein, The method of screening target knowledge data from the knowledge database based on the current atomic action comprises the following steps: based on the current atomic action, screening candidate knowledge data from the knowledge database; based on images collected by an image collection module, screening target knowledge data from the candidate knowledge data.

3. The method of robot navigation of claim 2, wherein, Each knowledge data in the knowledge database comprises an atomic action, and the method of screening candidate knowledge data from the knowledge database based on the current atomic action comprises the following steps: obtaining an action similarity between the current atomic action and an atomic action contained in each knowledge data in the knowledge database; based on the action similarity, screening candidate knowledge data from the knowledge database.

4. The method of Claim 2, wherein, Each knowledge data in the knowledge database further comprises a visual image sequence, and the method of screening target knowledge data from the candidate knowledge data based on images collected by an image collection module comprises the following steps: obtaining a visual image sequence contained in each knowledge data in the candidate knowledge data and images collected by the image collection module; inputting the images collected by the image collection module into a shared encoder to obtain a first extraction result output by the shared encoder; inputting the visual image sequence contained in each knowledge data in the candidate knowledge data into the shared encoder to obtain a plurality of second extraction results output by the shared encoder; determining an average similarity between the first extraction result and each second extraction result; reordering the visual image sequence contained in each knowledge data in the candidate knowledge data according to an order of the average similarity from large to small to obtain a reordering result; based on the reordering result, screening target knowledge data from the candidate knowledge data.

5. The method of Claim 1, wherein, The node features associated with the current atomic action in the navigation graph comprise basic text features; The method of updating node features associated with the current atomic action in the navigation graph based on the target knowledge data comprises the following steps: inputting the target knowledge data and the node features associated with the current atomic action into a first processing model to obtain coarse-grained enhanced features output by the first processing model; inputting the target knowledge data and the node features associated with the current atomic action into a second processing model to obtain fine-grained enhanced features output by the second processing model; weighting and combining the basic text features, the coarse-grained enhanced features and the fine-grained enhanced features to obtain updated node features associated with the current atomic action in the navigation graph; wherein the first processing model processes data at a lower level of detail than the second processing model.

6. The method of robot navigation of claim 5, wherein, The first processing model comprises a visual language coding model, the node features associated with the target knowledge data and the current atomic action are input into the first processing model, coarse-grained enhanced features output by the first processing model are acquired, and the method specifically comprises the following steps: The visual image sequence in the target knowledge data is input into the visual language coding model to obtain a frame feature sequence, and the frame features in the frame feature sequence correspond to the visual images in the visual image sequence in the target knowledge data one by one. The frame feature sequence is aggregated by using a guided attention mechanism to obtain a compressed vector. The coarse-grained enhanced features are obtained based on the compressed vector and the node features associated with the current atomic action.

7. The method of robot navigation of claim 5, wherein, The second processing model comprises a cross-attention mechanism model, the node features associated with the target knowledge data and the current atomic action are input into the second processing model, fine-grained enhanced features output by the second processing model are acquired, and the method specifically comprises the following steps: The visual image sequence in the target knowledge data is sequentially spliced to obtain a first frame sequence. The first frame sequence and the node features associated with the current atomic action are input into the cross-attention mechanism model to obtain fine-grained enhanced features.

8. The method of any one of claims 1-7, wherein, The robot navigation method further comprises the following steps: A training data set is acquired, the training data set comprises a plurality of demonstration instructions and a trajectory image sequence sample corresponding to each demonstration instruction; The demonstration instructions are decomposed to obtain at least one atomic action sample corresponding to the demonstration instructions; Based on the at least one atomic action sample corresponding to the demonstration instructions and the trajectory image sequence sample corresponding to the demonstration instructions, a visual image sequence sample corresponding to each atomic action sample is determined; Based on the at least one atomic action sample corresponding to the demonstration instructions and the visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instructions, the knowledge database is constructed.

9. The method of robot navigation of claim 8, wherein, The method specifically comprises the following steps: The at least one atomic action sample corresponding to the demonstration instructions and the trajectory image sequence sample corresponding to the demonstration instructions are input into a visual language coding model respectively to obtain corresponding frame feature samples and text description feature samples output by the visual language coding model; Based on the frame feature samples and the text description feature samples, a similarity matrix is constructed; Based on the similarity matrix, an alignment relationship between the at least one atomic action sample corresponding to the demonstration instructions and the trajectory image sequence sample corresponding to the demonstration instructions is constructed; Based on the alignment relationship, key frames related to the at least one atomic action sample corresponding to the demonstration instructions are determined; Corresponding adjacent frames of each key frame are determined; Based on the key frames and the adjacent frames, a visual image sequence sample related to the at least one atomic action sample corresponding to the demonstration instructions is constructed.

10. A robot navigation device, characterized by The robot navigation device comprises: an acquisition module configured to acquire, in response to a navigation instruction, at least one atomic action obtained by parsing the navigation instruction, the at least one atomic action including a current atomic action; a screening module configured to screen target knowledge data from a knowledge database based on the current atomic action; an updating module configured to update a node feature associated with the current atomic action in a navigation graph based on the target knowledge data; an execution module configured to execute the current atomic action based on the updated node feature.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the robot navigation method in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the robot navigation method in any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the robot navigation method in any one of claims 1 to 9.

Images