Indoor autonomous navigation based on natural language

Abstract

A scalable solution for robot behavior navigation following natural language instructions is proposed. An example of the solution includes: receiving a navigation graph of a task environment, a natural language instruction, and an initial position of a robot in the navigation graph through a pre-trained sequence prediction model, wherein the navigation graph includes nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between positions; and sequentially predicting a series of single-step behaviors that the robot can perform to navigate the robot from the initial position to the destination through the pre-trained sequence prediction model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to indoor autonomous navigation, and more particularly to a method for navigating a robot in a task environment. Background Art

[0002] Currently, the autonomous navigation systems used in most mobile robots rely on a fine-grained map of the task environment that is pre-established by scanning. During navigation, in response to receiving coordinates, the robot will search for a global path through a search algorithm, and then optimize the global path based on local observations to obtain the final planned path. However, when located in a new environment, existing mobile robots cannot immediately perform autonomous navigation because the destination coordinates may be unknown or a fine-grained map is not available. Summary of the invention

[0003] Embodiments of the present disclosure provide a method and apparatus for navigating a robot in a task environment, and a non-transitory medium.

[0004] In a first aspect, some embodiments of the present disclosure provide a method for navigating a robot in a task environment. The method comprises: receiving a navigation map of the task environment, a natural language instruction, and an initial position of the robot in the navigation map through a pre-trained sequence prediction model, wherein the navigation map comprises nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between positions; and sequentially predicting a series of single-step actions that the robot can perform through the pre-trained sequence prediction model to navigate the robot from the initial position to the destination.

[0005] In a second aspect, some embodiments of the present disclosure provide an electronic device comprising at least one processor and a memory storing executable instructions that cause the at least one processor to execute a method for navigating a robot in a task environment according to any one of the embodiments in the first aspect.

[0006] In a third aspect, some embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is executable to cause a processor to perform a method for navigating a robot in a task environment according to any one of the embodiments in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings.

[0008] Figure 1 is a flow chart of a method for navigating a robot in a task environment according to an embodiment of the present disclosure;

[0009] Figure 2A is a schematic diagram of an exemplary task environment according to an embodiment;

[0010] Figure 2B yes Figure 2A A schematic diagram of an exemplary navigation map of the mission environment shown;

[0011] Figure 3 is a schematic diagram of a method for predicting a series of single-step behaviors according to an embodiment of the present disclosure;

[0012] Figure 4 It is a schematic diagram of the three-step action propagation;

[0013] Figure 5 is a schematic diagram of a navigation agent according to an embodiment of the present disclosure;

[0014] Figure 6 is a schematic structural diagram of a device for predicting a series of single-step behaviors according to an embodiment of the present disclosure; and

[0015] Figure 7 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the present disclosure. In addition, it should be noted that for ease of description, only components related to the relevant disclosure are shown in the accompanying drawings.

[0017] It should also be noted that some embodiments of the present disclosure and some features of the present disclosure can be combined with each other on a non-conflicting basis. The features of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] Figure 1 It is a flowchart of a method for navigating a robot in a task environment according to an embodiment of the present disclosure. The method disclosed in the embodiment of the present disclosure can be performed by a device for navigating a robot in a task environment, which can be implemented in software and / or hardware and can be integrated on any electronic device (such as a server or terminal) with computing capabilities.

[0019] like Figure 1As shown, the method for navigating a robot in a task environment disclosed in an embodiment of the present disclosure may include step S101 and step S102. Step S101 receives a navigation map of the task environment, a natural language instruction, and an initial position of the robot in the navigation map through a pre-trained sequence prediction model; step S102 predicts a series of single-step behaviors that the robot can perform in sequence through the pre-trained sequence prediction model to navigate the robot from the initial position to the destination. In the following, these two steps will be described in detail.

[0020] Step S101:

[0021] The robot may be placed in an environment and needs to navigate throughout the environment. The environment in which the robot needs to navigate is called a mission environment. The mission environment may be an environment where GPS positioning is not possible, an indoor space, etc. An exemplary mission environment is Figure 2A Shown in.

[0022] When the robot is traveling along a corridor, the robot only needs to know that it is moving along a space with the correct semantics, and the space does not need to have certain geometric specifications (such as path width and curvature). Therefore, the robot's navigation throughout the task environment can be achieved by representing the task environment as a topological graph. Nodes in the topological graph can refer to semantically meaningful locations (such as rooms and corridors), and edges can indicate connectivity. The topological map can be used as a navigation graph for navigating the robot throughout the task environment. Figure 2A An exemplary navigation diagram corresponding to the task environment is shown in Figure 2B Shown in.

[0023] In some embodiments, the navigation graph is encoded using undirected edges together with node coordinates. Undirected edges can be shown in the navigation graph as non-directional or bidirectional edges. Figure 2B In the figure, undirected edges are shown as bidirectional edges.

[0024] Presenting true geometric information in the navigation graph enables one to adaptively interpret environmental knowledge based on the actual online navigation progress. This can produce a more compact and directional representation of the route by filtering out unnecessary information for a particular navigation process.

[0025] In the exemplary scenario, as Figure 2A As shown, when the robot is located in office-3 and needs to navigate to office-0, various instruction sequences of natural language can be used to guide the robot to move to office-0 along the expected route. Figure 2AThe expected route is shown in . Various instruction sequences may include "Turn left after leaving the room. Move forward, turn left again and go along the corridor. Turn left at the end and enter the second room on the left", or "Leave the room. Turn left and reach the end of the corridor. Turn left and move forward until the end. Turn left. Move forward and reach the second room on the left". Although including the same navigation semantics (i.e., "leave", "turn left", "move", "turn left", etc.), these two instruction sequences rely on different segmentation of the route and the ordering of key behaviors. The expected route of the navigating robot from office-3 to office-0 is also shown in . Figure 2B Shown in. Figure 2B The node sequence 4-3-5-… shown in is equivalent to Figure 2A The expected route is shown.

[0026] In some embodiments, a navigation graph including nodes, coordinates of nodes, and edges between nodes, natural language instructions, and an initial position or initial node of the robot are input into a pre-trained sequence prediction model for the sequence prediction model to generate a series of single-step actions based thereon, and the robot can perform a series of single-step actions to navigate the robot from the initial position (e.g., office-3) to the destination (e.g., office-0).

[0027] Step S102:

[0028] The sequence prediction model can be a neural network model pre-trained using annotated sample navigation graphs and corresponding natural language sample instructions.

[0029] In some implementations, a sequence prediction model sequentially predicts a series of single-step actions that a robot can perform.

[0030] Figure 3 A schematic diagram of a method for predicting a series of single-step behaviors according to an embodiment of the present disclosure is shown. Figure 3 As shown in Figure 1, the method for predicting a sequence of single-step behaviors takes three inputs: (1) a navigation graph G with undirected edges E, nodes N, and node coordinates X; (2) natural navigation instructions I; and (3) an initial position S∈N. The sequence prediction model can generate a sequence of behaviors u 0:T-1 :=(u0、…、u T-1 ), the robot can execute the behavior sequence u 0:T-1 , to navigate from s to the goal state g as described in the natural navigation instruction I.

[0031] During the training of the model, given a training sample The goal is to infer a sequence of actions that reflect the instructions for considering a new navigation query by solving

[0032]

[0033] Since the goal is high-level behavior planning, the target state described by the instruction may only specify the target location, but not the desired direction of travel. Therefore, without losing navigation capabilities, the embodiments of the present disclosure maintain a simple behavior set B: = {b e (i.e., "leave"), b r (i.e., "turn right and move"), b l (i.e., "turn left and move"), and b f (i.e., “move forward”)}. The proposed solution is different from assigning separate behaviors to the same move in each different situation (such as “go straight at the T-junction” and “go straight in the corridor”). This compact set of behaviors helps focus learning on matching instructions to navigation moves rather than to specific environments, effectively improving generalizability.

[0034] Given an action sequence u 0:T-1 :=(u0,…,u T-1 ), the robot can take action u at time t t And from node n t Migrate to n t+1 The expected whole navigation starts from n0=s and ends at the target state n T =g. In some embodiments, assuming that the robot always moves in the moving direction, each action u can be calculated by comparing the moving direction of the robot before and after the movement. t Classification as u t =b∈B. Assume that at time t, the robot is at node n t Then, by calculating the direction of travel and The cross product of action u t Classification

[0035]

[0036] where n t+1 Can always be different from n t-1 In particular, at t = 0, assume that the robot is at room node s and has only one valid action, "leave". Thus, from n t to n t+1 The valid transitions of can be represented as tuples <n t ,u t , n t+1 >, where u t is inferred from equation (2) above. Special behavior b can also be encoded s (i.e., “Stop”), which can be taken by the robot at any time t ≥ T to indicate the termination of navigation.

[0037] exist Figure 3 The sequence prediction model proposed in is constructed into two parts: a gated recurrent unit (GRU) component, which takes the hidden state h t Tracking navigation progress in the form of; and a navigation agent that selects the next action u t At each step, the GRU component takes the previous action u t-1 As input and update the hidden state h t . Navigation agent combined with navigation progress h t , navigation graph G and instruction I to select the next action u t Then, the robot will t and the transition model defined by graph G, from n t Move to n t+1 The same process is repeated until the navigation agent selects an action b that indicates reaching the goal. s (i.e., “halt”). In other words, the proposed model solves a series of single-step action prediction problems, each of which depends on the solution in the previous steps.

[0038] In some embodiments, for each step during prediction, an adaptive context is generated by adapting the navigation map to the current prediction process corresponding to the step, and a step behavior for the current step is predicted based at least on the generated adaptive context and the natural language instruction. By adapting the navigation map to the actual prediction process corresponding to the current step, it follows the following observation when following the navigation instruction: humans usually search for relevant information in a local field of view, rather than always paying equal attention to the entire map. By adapting the navigation map to the actual prediction process corresponding to the current step, and predicting the step behavior for the current step based on the adapted context, the challenge of a flexible correspondence between instruction semantics and navigation planning is solved using limited expansion of new and large maps.

[0039] In some embodiments, knowledge base adaptation may be achieved through so-called d-step action propagation.Other methods for achieving knowledge base adaptation may also be employed.

[0040] As an example, Figure 4 shows the method for generating adaptive context 3-step action propagation. is a subset of the complete navigation graph G that only includes the 0:t And efficient navigation moves within bounded antecedent horizons.

[0041] Connectivity information G of graph G:<E,N,X> can be represented as a set of tuples {<n,b,n’> i}, each tuple represents a valid navigation behavior for moving from node n of type b to node n' of type b. As described in the previous section, for directed edges<n,n’> The valid behavior types b depend on the possible previous positions n of the robot before reaching n. prev Therefore, the transfer<n,b,n’> Alternatively, it can be expressed in the format of "previous-current-next" <n prev , n, n'>, from which b can be inferred according to the above equation (2). In order to adapt to the knowledge base G, search for effective actions that can be taken in the next d steps. In other words, the simulation from the immediate history <n t-1 , n t >Continue the robot movement and record any valid node transfer and its behavior type. In the following, this process is called d-step action propagation. This process is implemented as follows: <n t-1 , n t >As the initial element, perform a bounded breadth-first search for directed edges in G. Whenever a directed edge is popped from the queue <n prev , n>, collect non-n prev For each n',<n,n’> Add to the queue and form a tuple <n prev , n, n'>. The tuple is then converted to graph format<n,b,n’> , where the behavior type b is determined by the coordinates x(n prev ), x(n) and x(n'). n' and the current node n can be collected t All valid transitions whose distance between them is within d. This produces the adaptive context at time step t See Algorithm 1 for an overview of the d-step action propagation algorithm.

[0042]

[0043] Figure 5 Schematic diagram of a navigation agent according to an embodiment of the present disclosure. The construction of the navigation agent mainly includes:

[0044] Context and Directive Embedding: Navigation Context (or G in a static context) and instruction I. The transfer tuple in<n,b,n’> Each of is encoded as a vector of length 2|N|+|B|, where |N| and |B| refer to the number of nodes and the number of valid behaviors in the graph G, respectively. Finally, it is encoded to a size of In the matrix, L t Indicates included in The instruction I is encoded into a size of 1 using the token embedding from the BERT pre-trained model. where M represents the maximum number of tokens in an instruction.

[0045] Feature extraction: Feature extraction is performed on both the context and the instruction embedding. In some embodiments, a multi-layer bidirectional gated recurrent unit (GRU) is used to generate a Contextual features Where H represents the hidden size of the GRU. In some embodiments, the same GRU is used to generate a size of Instruction features

[0046] Contextual Instruction Attention: Correspondence between navigation context and instructions is now found via an attention mechanism. In some embodiments, a one-way attention is used where only context features relate to instruction features. It is worth noting that in the setting of adaptive context, the attention mechanism is not only similar to the way people search for paths on a map, but also similar to the fact that people mainly pay attention to the nearby environment when deciding the next move. This is especially true when instructions are based on local environments rather than global landmarks.

[0047] for Each line of Calculate the pair The attention distribution of w ti :

[0048]

[0049] in represents a trainable parameter. Each transfer feature is then calculated by the following equation The attention vector R ti :

[0050]

[0051] Aggregate all R ti , and obtain the size The attention matrix R t .

[0052] Progress-aware context: This section combines the navigation context Instruction attention feature R t , and the current hidden state h from the GRU component at time t t .

[0053] First, the combination and R t To compute mixed context and instruction features:

[0054]

[0055] in represents the trainable parameters that reduce the feature dimension to H. Then, note that the hidden state h t to C t To capture the contextual features related to the current navigation progress. t The calculation is as follows:

[0056]

[0057]

[0058] Among them, W1, is a trainable parameter. Then, the process-aware context is calculated as

[0059] Behavior prediction: the final process perceives the context S t and the hidden state h t are combined to generate a policy at time t. By t With h t Concatenate and feed into a fully connected layer to compute raw action probability features

[0060]

[0061] in represents a trainable parameter. The result is a preference vector for each navigation behavior b∈B and a specific stop action b indicating task termination. s .

[0062] To generate the action u t , apply the softmax function:

[0063]

[0064] u t =argmax(softmax(o t )) (9)

[0065] In some embodiments, the input of the masking function includes the entire navigation graph G and the navigation trajectory n up to the current step t. 0:t This function generates a The same zero vector, with -∞ replacing the invalid behavior. To determine whether a behavior b is valid, check whether there are n that satisfy the following condition t Neighbor node n':

[0066] n′≠nt-1 and b=b(n t-1 , n t ,n′)by Eq.(2) (10)

[0067] In some embodiments, when n t =nt-1(or u t- =bs), since navigation has terminated, the stop action is enforced at time t. It is worth noting that the effective action space at each step t is determined not only by the position nt, but also by the historical position nt-1. This setting removes the requirement of binding behavioral semantics to positions, thereby achieving compact knowledge representation and flexible inference of behavioral semantics.

[0068] Further references Figure 6 As an implementation of a method for predicting a series of single-step behaviors, an embodiment of the present disclosure provides a device for predicting a series of single-step behaviors, and the implementation of the device corresponds to the various method implementations described above, and the device can be specifically applied to various electronic devices.

[0069] like Figure 6 As shown, the device 600 for predicting a series of single-step behaviors of the present embodiment includes a receiving unit 601 and a prediction unit 602. The receiving unit 601 is configured to receive a navigation map of a task environment, a natural language instruction of a robot, and an initial position in the navigation map, wherein the navigation map includes nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between positions. The prediction unit 602 is configured to sequentially predict a series of single-step behaviors that the robot can perform to navigate the robot from the initial position to the destination.

[0070] In some embodiments, the prediction unit further includes an adaptive context generation subunit and a prediction subunit. The adaptive context generation subunit is configured to: for each single step during the prediction, generate an adaptive context by adapting the navigation graph to the current prediction process corresponding to the single step. The prediction subunit is configured to predict a single step behavior for the single step based at least on the generated adaptive context and the natural language instruction.

[0071] In some embodiments, the adaptive context generation subunit is further configured to: search in the navigation graph for valid node transfers between the current node corresponding to the single step and neighbor nodes other than the previous node of the current node; predict the behavior of the valid node transfer based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighbor nodes other than the previous node of the current node; use the neighbor node as the new current node, and repeat the search and prediction steps until the distance between the node serving as the new current node and the current node of the current single step is within a preset value; and convert all the found valid node transfers and their predicted behaviors into a graphical format to generate an adaptive context for each single step.

[0072] In some implementations, the behavior of valid node transitions is predicted from a behavior set consisting of: leave, turn right and move, turn left and move, and move forward.

[0073] In some embodiments, the adaptive context generation subunit is further configured to: determine the direction of travel of the robot at the current node by subtracting the coordinates of the previous node from the coordinates of the current node; determine the direction of travel of the robot at the neighboring node by subtracting the coordinates of the current node from the coordinates of the neighboring node; calculate the cross product of the direction of travel of the robot at the current node and the direction of travel of the robot at the neighboring node; and predict the single-step behavior of the effective node transfer based on the calculated cross product.

[0074] In some embodiments, the prediction subunit is further configured to predict a single-step behavior for a single step based on the generated adaptive context, the natural language instruction, and the current hidden state updated by a gated recurrent unit (GRU), wherein the GRU takes a previous single-step behavior of a previous single step as input and updates to obtain the current hidden state.

[0075] In some embodiments, the device for predicting a series of unidirectional behaviors also includes a navigation map creation unit, which is configured to: create a topological map of the task environment, wherein the locations in the task environment serve as nodes of the topological map and the connectivity between the locations serves as edges of the topological map; and determine the created topological map as a navigation map of the task environment.

[0076] The device 600 corresponds to the steps in the aforementioned method implementation. Therefore, the operations, features and technical effects that can be achieved in the above-mentioned method for predicting a series of single-step behaviors are also applicable to the device 600 and the units included therein, and their detailed description will be omitted.

[0077] According to an embodiment of the present disclosure, an electronic device and a readable storage medium are provided.

[0078] like Figure 7As shown, a block diagram of an electronic device for predicting a series of single-step behaviors according to an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as portable computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the embodiments of the present disclosure herein and / or claimed.

[0079] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces (including high-speed interfaces and low-speed interfaces) for connecting various components. The various components are connected to each other using different buses and can be installed on a universal motherboard or in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device connected to the interface). In other embodiments, as needed, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected, and these devices provide some necessary operations, such as a server array, a group of blade servers, or a multi-processor system. Figure 7 In, as an example, one processor 701 is used.

[0080] The memory 702 is a non-transitory computer-readable storage medium provided in the embodiments of the present disclosure. The memory stores instructions executable by at least one processor, so that at least one processor executes the method for predicting a series of single-step behaviors provided by the embodiments of the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions for causing a computer to execute the method for predicting a series of single-step behaviors provided in the embodiments of the present disclosure.

[0081] The memory 702 as a non-transitory computer-readable storage medium may be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules corresponding to the method for predicting a series of single-step behaviors in the embodiments of the present disclosure (e.g., Figure 6 The processor 701 executes the non-temporary software programs, instructions and modules stored in the memory 702 to perform various functional applications and data processing of the server, that is, to implement the method for predicting a series of single-step behaviors in the aforementioned method implementation.

[0082] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function, and the data storage area may store data created by using an electronic device for predicting a series of single-step behaviors. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely disposed relative to the processor 701, and these remote memories may be connected to the electronic device to predict a series of single-step behaviors through a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The electronic device for predicting a series of single-step behaviors may further include an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703, and the output device 704 may be connected via a bus 705 or other methods. Figure 7 In, as an example, the connection via bus 705 is used.

[0084] The input device 703 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device for predicting a series of single-step behaviors. The input device 703 is, for example, a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0085] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor. The programmable processor can be a special purpose or general purpose programmable processor, and can receive data and instructions from a memory system, at least one input device, and at least one output device, and transmit data and instructions to the memory system, at least one input device, and at least one output device.

[0086] These computer programs (also referred to as programs, software, software disclosures, or code) include machine instructions for a programmable processor and may be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, and a keyboard and pointing device such as a mouse or trackball, wherein the display device is used to display information to the user, and the user can use the keyboard and pointing device to provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form (including sound input, voice input, or tactile input) can be used to receive input from the user.

[0088] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or in a computing system that includes a middleware component (e.g., an application server), or in a computing system that includes a front-end component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or in a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any digital data communication form or medium (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0089] A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0090] The above description only provides an explanation of the preferred embodiments of the present disclosure and the technical principles used herein. Those skilled in the art should understand that the scope of the invention of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features. Without departing from the concept of the present disclosure, the scope of the present invention also covers other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features, such as but not limited to, the technical solution formed by exchanging the above-mentioned features with the technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for navigating a robot in a task environment, comprising: Receiving a navigation graph of the task environment, a natural language instruction, and an initial position of the robot in the navigation graph through a pre-trained sequence prediction model, wherein the navigation graph includes nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between the positions; and Predicting, by means of the pre-trained sequence prediction model, a series of single-step actions that can be performed by the robot in sequence, so as to navigate the robot from the initial position to the destination; The sequential prediction includes: For each single step during the forecast period, generating an adaptive context by adapting the navigation graph to a current prediction process corresponding to the single step; and predicting a single-step behavior for the single step based at least on the generated adaptive context and the natural language instruction; Wherein, for each single step during the prediction period, generating the adaptive context includes: In the navigation graph, searching for valid node transfers between a current node corresponding to the single step and neighbor nodes other than a previous node of the current node; Predicting a behavior of valid node transfer based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node of the current node; Adopting the neighbor node as a new current node, and repeating the steps of searching and predicting until the distance between the node serving as the new current node and the current node of the current single step is within a preset value; and The searched valid node transitions and the predicted valid node transition behaviors are all converted into a graph format to generate the adaptive context for each single step.

2. The method according to claim 1, wherein: The behavior of the valid node transition is predicted according to a behavior set consisting of: leaving, turning right and moving, turning left and moving, and moving forward.

3. The method according to claim 1, wherein: Based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node, the behavior of predicting the effective node transfer includes: determining a direction of travel of the robot at the current node by subtracting the coordinates of the previous node from the coordinates of the current node; Determine the travel direction of the robot at the neighbor node by subtracting the coordinates of the current node from the coordinates of the neighbor node; Calculating the cross product of the travel direction of the robot at the current node and the travel direction of the robot at the neighboring node; and Based on the calculated cross product, the single-step behavior of the valid node transition is predicted.

4. The method according to claim 1, wherein: Based at least on the generated adaptive context and the natural language instruction, predicting a single-step behavior for the single-step includes: A single-step behavior for the single-step is predicted based on the generated adaptation context, the natural language instruction, and a current hidden state updated by a gated recurrent unit, wherein the gated recurrent unit takes a previous single-step behavior of a previous single-step as input and updates to obtain the current hidden state.

5. The method according to claim 1, wherein: The method further comprises: Creating a topological graph of the task environment, wherein locations in the task environment are nodes of the topological graph, and connectivity between the locations are edges of the topological graph; and The created topology map is determined as the navigation map of the task environment.

6. An electronic device comprising: at least one processor; as well as A memory storing executable instructions for causing the at least one processor to perform operations, the operations comprising: Receiving a navigation graph of a task environment, a natural language instruction, and an initial position of a robot in the navigation graph through a pre-trained sequence prediction model, wherein the navigation graph includes nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between the positions; and Predicting, by means of the pre-trained sequence prediction model, a series of single-step actions that can be performed by the robot in sequence, so as to navigate the robot from the initial position to the destination; The sequential prediction includes: For each single step during the forecast period, generating an adaptive context by adapting the navigation graph to a current prediction process corresponding to the single step; and predicting a single-step behavior for the single step based at least on the generated adaptive context and the natural language instruction; Wherein, for each single step during the prediction period, generating the adaptive context includes: In the navigation graph, searching for valid node transfers between a current node corresponding to the single step and neighbor nodes other than a previous node of the current node; Predicting a behavior of valid node transfer based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node of the current node; Adopting the neighbor node as a new current node, and repeating the steps of searching and predicting until the distance between the node serving as the new current node and the current node of the current single step is within a preset value; and The searched valid node transitions and the predicted valid node transition behaviors are all converted into a graph format to generate the adaptive context for each single step.

7. The device according to claim 6, wherein: The behavior of the valid node transition is predicted according to a behavior set consisting of: leaving, turning right and moving, turning left and moving, and moving forward.

8. The device according to claim 6, wherein: Based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node, the behavior of predicting the effective node transfer includes: determining a direction of travel of the robot at the current node by subtracting the coordinates of the previous node from the coordinates of the current node; Determine the travel direction of the robot at the neighbor node by subtracting the coordinates of the current node from the coordinates of the neighbor node; Calculating the cross product of the travel direction of the robot at the current node and the travel direction of the robot at the neighboring node; and Based on the calculated cross product, the single-step behavior of the valid node transition is predicted.

9. The device according to claim 6, wherein: Based at least on the generated adaptive context and the natural language instruction, predicting a single-step behavior for the single-step includes: A single-step behavior for the single-step is predicted based on the generated adaptation context, the natural language instruction, and a current hidden state updated by a gated recurrent unit, wherein the gated recurrent unit takes a previous single-step behavior of a previous single-step as input and updates to obtain the current hidden state.

10. The device according to claim 6, wherein: The operations also include: Creating a topological graph of the task environment, wherein locations in the task environment are nodes of the topological graph, and connectivity between the locations are edges of the topological graph; and The created topology map is determined as the navigation map of the task environment.

11. A non-transitory computer-readable storage medium having stored therein an executable computer program for causing a processor to perform operations, the operations comprising: Receiving a navigation graph of a task environment, a natural language instruction, and an initial position of a robot in the navigation graph through a pre-trained sequence prediction model, wherein the navigation graph includes nodes indicating positions in the task environment, coordinates of the nodes, and edges indicating connectivity between the positions; and Predicting, by means of the pre-trained sequence prediction model, a series of single-step actions that can be performed by the robot in sequence, so as to navigate the robot from the initial position to the destination; The sequential prediction includes: For each single step during the forecast period, generating an adaptive context by adapting the navigation graph to a current prediction process corresponding to the single step; and predicting a single-step behavior for the single step based at least on the generated adaptive context and the natural language instruction; Wherein, for each single step during the prediction period, generating the adaptive context includes: In the navigation graph, searching for valid node transfers between a current node corresponding to the single step and neighbor nodes other than a previous node of the current node; Predicting a behavior of valid node transfer based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node of the current node; Adopting the neighbor node as a new current node, and repeating the steps of searching and predicting until the distance between the node serving as the new current node and the current node of the current single step is within a preset value; and The searched valid node transitions and the predicted valid node transition behaviors are all converted into a graph format to generate the adaptive context for each single step.

12. The medium according to claim 11, wherein The behavior of the valid node transition is predicted according to a behavior set consisting of: leaving, turning right and moving, turning left and moving, and moving forward.

13. The medium according to claim 11, wherein Based on the coordinates of the current node, the coordinates of the previous node of the current node, and the coordinates of the neighboring nodes except the previous node, the behavior of predicting the effective node transfer includes: determining a direction of travel of the robot at the current node by subtracting the coordinates of the previous node from the coordinates of the current node; Determine the travel direction of the robot at the neighbor node by subtracting the coordinates of the current node from the coordinates of the neighbor node; Calculating the cross product of the travel direction of the robot at the current node and the travel direction of the robot at the neighboring node; and Based on the calculated cross product, the single-step behavior of the valid node transition is predicted.

14. The medium according to claim 11, wherein Based at least on the generated adaptive context and the natural language instruction, predicting a single-step behavior for the single-step includes: A single-step behavior for the single-step is predicted based on the generated adaptation context, the natural language instruction, and a current hidden state updated by a gated recurrent unit, wherein the gated recurrent unit takes a previous single-step behavior of a previous single-step as input and updates to obtain the current hidden state.

Images