System and method for generating navigation commands for robots

Abstract

To provide a system for generating maneuvering commands for mobile robots.SOLUTION: One embodiment of the present invention acquires a first input or a second input, executes a pre-trained model, and generates a first output or a second output. The first input comprises a first semantic map representation of the environment and at least one first natural language instruction describing a first path for the robot. The first output comprises a first output semantic map representation of the environment, and the first output semantic map representation includes a visual representation of the first path for the robot to navigate within the environment. The second input comprises a second semantic map representation of the environment, and the second semantic map representation includes a visual representation of a second path for the robot. The second output comprises a second output semantic map representation of the environment and at least one second natural language instruction describing a second path for maneuvering the robot within the environment.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of robotic navigation, and more particularly to systems and methods for generating navigation instructions based on a semantic map representation of an environment. [Background technology]

[0002] In recent years, the use of autonomous and semi-autonomous systems has grown exponentially. In particular, autonomous and semi-autonomous systems, such as mobile robots, are used in various sectors (e.g., manufacturing, military, transportation, healthcare, education, services, etc.) to increase efficiency and productivity. Generally, mobile robots are configured to move between multiple locations to perform one or more tasks. For example, mobile robots are configured to receive instructions and / or commands to maneuver as needed.

[0003] Traditionally, mobile robots have been configured to receive navigation instructions via control devices such as joysticks, keyboards, switches, and user interfaces (e.g., graphical interfaces). However, these instructions are non-verbal in nature. Such instructions require a person (e.g., a user) to constantly monitor the mobile robot and physically control the mobile robot (e.g., using the user's hands via the control device). However, this approach has several drawbacks. For example, it may be undesirable to have a user constantly control the mobile robot when the user's hands are full.

[0004] More recently, mobile robots have been configured to receive simple verbal commands, such as "move forward," "turn left," "turn right," and "go to the container." While this approach can be preferable to physical control, it also has several challenges. One such challenge is that each utterance of a verbal command needs to be mapped to a physical action of the mobile robot. For example, "move forward" would need to be mapped to the physical action of the mobile robot moving forward by one step. Because of this one-to-one mapping between verbal commands and corresponding physical actions, this approach cannot support more natural instructions that may be composed of multiple steps, such as "move around the table and stop by the farthest chair."

[0005] Thus, there exists an unmet need for improved systems and methods for enhancing human-robot interaction and automatically generating navigation instructions for mobile robots. Specifically, there exists a demand for automatically generating navigation instructions from natural, complex instructions. Furthermore, there exists an unmet need for seamlessly adapting the navigation capabilities of a mobile robot to a known environment with new environments (e.g., new physical or virtual spaces) or new spatial entities (e.g., new objects). Summary of the Invention

[0006] Systems and methods for generating navigation instructions are disclosed herein. In one aspect, a computer-implemented method is provided. The method includes acquiring a first input or a second input in a processor. In response to receiving the first input, the method includes executing, via the processor, a pre-trained model to generate a first output. In response to receiving the second input, the method includes executing, via the processor, the pre-trained model to generate a second output. The first input comprises a first semantic map representation of an environment and at least one first natural language instruction describing a first path for the robot. The first output comprises a first output semantic map representation of the environment. The first output semantic map representation comprises a visual representation of a first path for the robot to navigate the robot within the environment. The second input comprises a second semantic map representation of the environment. The second semantic map representation comprises a visual representation of a second path for the robot. The second output comprises a second output semantic map representation of the environment and at least one second natural language instruction describing a second path for maneuvering the robot within the environment.

[0007] In some variations, the first semantic map representation is a top-down view of the environment and includes a plurality of spatial landmarks in the environment as a plurality of color-coded bounding boxes on the first semantic map representation. In some variations, the second semantic map representation is a top-down view of the environment and includes a plurality of spatial landmarks in the environment as a plurality of color-coded bounding boxes on the second semantic map representation and a visual representation of a second path as a series of points connected by lines on the second semantic map representation. The series of points includes a start point and an end point.

[0008] In some variations, at least one of the first semantic map representation and the second semantic map representation is created manually. In some variations, at least one of the first semantic map representation and the second semantic map representation is created automatically. In some variations, the pre-trained model is a multimodal text generation model.

[0009] In some variations, executing the pre-trained model in response to receiving the first input further includes extracting, via an encoder, one or more spatial regions of the environment from the first semantic map representation, extracting, via the encoder, one or more actions to be implemented by the robot from at least one of the at least one first natural language instruction and the at least one second natural language instruction, embedding, using the encoder, the one or more spatial regions as first vectors, and embedding, using the encoder, the one or more actions as second vectors. Executing the pre-trained model may further include encoding, based on the first and second vectors, sequential path information for the robot to navigate the robot through the environment via a neural network. Executing the pre-trained model may further include executing a decoder to generate a first output semantic map representation using the sequential path information.

[0010] In some variations, executing the pre-trained model in response to receiving the second input further includes heuristically determining one or more actions to be implemented by the robot based at least in part on a visual representation of a second path for the robot. In some variations, executing the pre-trained model in response to receiving the second input further includes extracting, via an encoder, one or more spatial regions of the environment from the second semantic map representation, embedding, using the encoder, the one or more spatial regions as third vectors, and embedding, using the encoder, one or more actions as fourth vectors. Executing the pre-trained model may further include encoding, via a neural network, sequential path information for the robot to maneuver in the second environment based on the third and fourth vectors. Executing the pre-trained model may further include executing a decoder to generate a second output semantic map representation and at least one second natural language instruction describing the second path using the sequential path information.

[0011] In some variations, the computer-implemented method may further comprise acquiring a first panoramic image of the environment. In response to receiving a first input, the method may comprise executing a pre-trained model to generate a first output based at least in part on the first panoramic image. In response to receiving a second input, the method may comprise executing the pre-trained model to generate a second output based at least in part on the first panoramic image. The method may further comprise training the pre-trained model using a plurality of pairs of training data. Each of the plurality of pairs may include image data and a corresponding natural language description of the image data.

[0012] In another aspect, a computer-implemented method is provided. The computer-implemented method comprises obtaining an input semantic map of an environment. The input semantic map includes a visual representation of a path for the robot to navigate the environment. The method may further include extracting one or more spatial regions of the environment from the input semantic map, heuristically determining one or more actions to be implemented by the robot based at least in part on the visual representation of the path to navigate the robot within the environment, embedding the one or more spatial regions to produce a region embedding, embedding the one or more actions to produce an action embedding, encoding sequential path information for the robot using the region embedding and the action embedding, and generating natural language instructions describing the path using the sequential path information to navigate the robot within the environment.

[0013] In yet another aspect, a computer-implemented method is provided that includes obtaining an input semantic map of an environment, obtaining natural language instructions to navigate a robot within the environment using the input semantic map, extracting one or more spatial regions of the environment from the input semantic map, extracting one or more actions to be implemented by the robot from the natural language instructions to navigate the robot within the environment, embedding the one or more spatial regions to produce a region embedding, embedding the one or more actions to produce an action embedding, encoding sequential path information for the robot using the region embedding and the action embedding, and generating an output semantic map of the environment using the sequential path information, wherein the output semantic map includes a visual representation of a path for the robot to navigate the robot within the environment.

[0014] Systems and methods, by way of non-limiting example, will now be described with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an example system for generating navigation instructions for one or more mobile robots. [Figure 2a] Figure 2a shows exemplary types of inputs that may be provided to the system of Figure 1. Figure 2a also shows exemplary types of outputs that may be produced by the system of Figure 1. [Figure 2b] Figure 2b illustrates another exemplary type of input provided by the system of Figure 1. Figure 2b also illustrates another exemplary type of output produced by the system of Figure 1. [Figure 3] FIG. 3 is a flow diagram illustrating an overview of an exemplary method for generating navigation instructions for a mobile robot. [Figure 4] FIG. 4 illustrates generating an exemplary first type output from an exemplary first type input. [Figure 5] FIG. 5 illustrates generating an exemplary second type output from an exemplary second type input. [Figure 6] FIG. 6 is a flow diagram of a method illustrating one or more operations for causing a pre-trained model to generate a first type of output from a first type of input. [Figure 7] FIG. 7 is a flow diagram of a method illustrating one or more acts for causing a pre-trained model to generate a second type of output from a second type of input. [Figure 8] Figure 8 is an illustration of augmenting the input to a pre-trained model with a panoramic image. [Figure 9] FIG. 9 illustrates an example architecture of a pre-trained model configured to receive augmented input and generate navigation instructions accordingly. [Figure 10] Figure 10 shows the performance of the pre-trained model compared to other navigation instruction generation methods. DETAILED DESCRIPTION OF THE INVENTION

[0016] As used herein, the term "robot" generally refers to an autonomous and / or semi-autonomous system configured to move from one location to another. In other words, as used herein, the term "robot" refers to a system configured to be mobile and is used interchangeably with the term "mobile robot."

[0017] As used herein, the term "environment" generally refers to a physical space and / or a virtual space. For example, the term "environment" may refer to a space in which a mobile robot will be operated. In some variations, the term "environment" may refer to any suitable physical space, such as, for example, an indoor space, an outdoor space, a combination thereof, and / or the like. Additionally or alternatively, the term "environment" may refer to any suitable virtual space, such as, for example, a computer-simulated environment.

[0018] As used herein, the term "semantic map representation" generally refers to an RGB image of a top-down view of an environment. In some variations, the "semantic map representation" of an environment may associate spatial entities within the environment (e.g., spatial landmarks, objects within the environment, combinations thereof, and / or the like) with spatial concepts. For example, the "semantic map representation" may include a topographical map and / or a metric map of the environment, and spatial entities within the environment may be symbolically represented in the "semantic map representation." In some variations, the "semantic map representation" may include metadata (e.g., meta-information) that represents one or more properties of the spatial entities (e.g., coordinates, size, visual properties, combinations thereof, and / or the like).

[0019] As used herein, the term “navigation instructions” may include instructions that enable a mobile robot to navigate within an environment. These instructions may be in any suitable format (e.g., images, audio, video, text, combinations thereof, and / or the like). For example, in some variations, “navigation instructions” may be a semantic map representation of the environment that indicates a navigation path for the mobile robot. In other words, in such variations, “navigation instructions” may simply be a semantic map representation of the environment that includes a visual representation of a navigation path for the mobile robot. In such variations where “navigation instructions” are primarily visual instructions (e.g., a visual representation of a navigation path), “navigation instructions” may be directly translated into actions for the mobile robot. Thus, in these variations, the mobile robot can identify a navigation path and perform actions to navigate by processing the semantic map representation of the environment. Additionally or alternatively, “navigation instructions” may be (spoken) natural language instructions that describe a navigation path for the mobile robot. In these variations, “navigation instructions” may include a semantic map representation of the environment (without a visual representation of the navigation path) and natural language instructions that describe a navigation path for the mobile robot. In these examples, the mobile robot may identify a navigation path by processing the semantic map representation and by interpreting the natural language instructions.

[0020] Disclosed herein are improved systems and methods for generating navigation instructions for a mobile robot. Specifically, the systems and methods disclosed herein can translate natural language instructions describing a navigation path into a visual representation of the navigation path, and vice versa. As discussed above, existing approaches for piloting a mobile robot have several challenges and drawbacks. One such existing approach includes piloting a mobile robot via control devices such as joysticks, keyboards, switches, and user interfaces (e.g., graphical interfaces). This approach may not be feasible or desirable in situations where a user (e.g., a human operator) does not have consistent access to the control devices. Therefore, in such situations, spoken language may be a more natural modality of communication between the user and the mobile robot.

[0021] Another existing approach involves controlling a mobile robot with simple linguistic commands. While controlling a mobile robot via simple linguistic commands may overcome some of the challenges of previous approaches, this approach leaves much to be desired. For example, given the one-to-one mapping between linguistic commands and mobile robot actions, the user may have to be very precise in their linguistic commands. Furthermore, this approach cannot support complex instructions that may be more natural to the user. Additionally, creating a comprehensive list of linguistic commands for performing various tasks and mapping each of these linguistic commands to a robot action can be difficult and time-consuming.

[0022] In recent years, several approaches have begun to use end-to-end visual language navigation (VLN) to navigate mobile robots. In these approaches, a model is trained to process panoramic images of the environment and to interpret natural language instructions (e.g., from a user). The model can be trained to predict the next navigation point or a set of robot actions from the panoramic images and natural language instructions. In other words, the trained model finds a navigation path by processing the panoramic images. Although this approach may be more preferable than previous approaches, there are still several challenges associated with this approach. First, to train a VLN model, the natural language instructions must be manually annotated. For example, panoramic images of the environment along the navigation path may need to be acquired. Annotators may need to manually navigate between two points on the navigation path and then (manually) write instructions for navigating this path. Therefore, annotating navigation instructions for training a VLN model can be time-consuming and expensive to scale up. Second, using panoramic images can be resource-intensive. This is because predicting the next set of navigation points or robot actions using panoramic images may require multiple images of different points in the environment. Third, panoramic images may contain many details that are irrelevant to the navigation task, and processing these irrelevant details may increase the computational time for generating navigation instructions. Fourth, this approach cannot be extended to new environments or to known environments with new objects. In other words, the trained model cannot be extended to environments that include spatial entities (e.g., landmarks, objects, etc.) that were not visible during model training. For these reasons, the performance of this approach was not high enough for real-world applications.

[0023] The technology disclosed herein enables the operationalization of end-to-end visual language navigation (VLN) by improving its performance. Unlike VLN approaches that use panoramic images, the technology disclosed herein uses a semantic map representation of the environment to enable a mobile robot to navigate within its environment. For example, the technology disclosed herein may use the semantic map representation to enable interpretation of the environment and to enable spatial reasoning for the mobile robot. The semantic map representation reduces the computation time for generating navigation instructions. Furthermore, a single semantic map representation may be sufficient to predict navigation points or a set of robot actions.

[0024] In some variations, by training a model with a semantic map representation of an environment, the model can be extended to other unseen environments. For example, a model can be trained to generate natural language instructions from a semantic map representation of a first environment. When the trained model receives as input a semantic map representation of a second unseen environment, the trained model can generate natural language instructions for maneuvering in the second environment. These natural language instructions can then be used to train a VLN model, thereby enabling the operation of the VLN. In a similar manner, a model can be trained to generate a visual representation of a navigation path from natural language instructions. The trained model can be extended to other unseen environments with different navigation paths.

[0025] In some variations, the technology disclosed herein can be configured to execute a pre-trained model to automatically generate (e.g., spoken) natural language instructions from a semantic map representation of an environment. Additionally or alternatively, the technology disclosed herein can be configured to execute the same pre-trained model to automatically generate (e.g., spoken) natural language instructions from a semantic map representation of an environment, the semantic map representation including a visual representation of a path for a mobile robot. In other words, the technology disclosed herein can execute the same pre-trained model to a) generate natural language instructions from a semantic map representation of an environment, the semantic map representation including a visual representation of a navigation path for a mobile robot, and b) generate (e.g., spoken) a semantic map representation of an environment including a visual representation of a navigation path from a natural language instruction.

[0026] Thus, the technology disclosed herein can enable a user to guide a mobile robot with (e.g., spoken) natural language commands. Furthermore, the technology disclosed herein can automatically generate navigation instructions that can guide a human user who may not be able to consistently control (e.g., physically or visually control) a mobile robot. Additionally, the technology disclosed herein can be used as training data for a VLN model, thereby enabling operationalization of the VLN. The technology disclosed herein can also facilitate improved human-robot interaction. For example, a mobile robot can be configured to disambiguate ambiguous commands, improving its understanding of physical space. This allows for seamless adaptation of a pre-trained model to environments containing previously unseen spatial entities (e.g., unseen during training) without the need for additional training data. In some variations, the technology disclosed herein can be configured to encourage a mobile robot to explain its actions, when this may be a desirable property.

[0027] FIG. 1 shows an example system 100 for generating navigation instructions for one or more mobile robots 112. The system 100 includes one or more controllers 102 configured to obtain one or more inputs. The controllers 102 may be configured to transform the inputs into one or more outputs. In other words, the controllers 102 may be configured to generate one or more outputs from the one or more inputs. The controllers 102 may be communicatively coupled to one or more mobile robots 112 via a network (e.g., the Internet, a local area network (LAN), a wide area network (WAN), and / or the like). The one or more outputs may be transmitted to the mobile robots 112 to maneuver the mobile robots 112 within an environment.

[0028] In some variations, the controller 102 may include one or more servers and / or one or more processors operating on a cloud platform (e.g., Microsoft Azure®, Amazon® Web Services, IBM® Cloud Computing, etc.). The servers and / or processors may be any suitable processing device configured to run and / or execute a set of instructions or code and may include one or more data processors, image processors, graphics processing units, digital signal processors, and / or central processing units. The servers and / or processors may be, for example, general-purpose processors, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and / or the like.

[0029] In some variations, the controller 102 may include a processor (e.g., a CPU). The processor may be any suitable processing device configured to run and / or execute a set of instructions or code and may include one or more data processors, image processors, graphics processing units, physics processing units, digital signal processors, and / or central processing units. The processor may be, for example, a general-purpose processor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or the like. The processor may be configured to run and / or execute application processes and / or other modules, processes and / or functions associated with the system and / or its associated network. The underlying device technologies may be provided in a variety of component types (e.g., MOSFET technologies such as complementary metal oxide semiconductor (CMOS), bipolar technologies such as emitter-coupled logic (ECL), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and / or the like.

[0030] In some variations, the controller 102 may be configured to obtain at least two types of input. A first type of input may include a semantic map representation of the environment and natural language instructions describing a path through the environment (e.g., a navigation path for the mobile robot 112). A second type of input may include a semantic map representation of the environment including a visual representation of a path through the environment (e.g., a navigation path for the mobile robot 112).

[0031] For example, FIG. 2a illustrates an exemplary first type of input 230a acquired at the controller 102. The first type of input 230a includes a semantic map representation 232a of an environment. The semantic map representation 232a is a top-down view of the environment and includes spatial entities (e.g., spatial landmarks, objects, etc.) as color-coded bounding boxes, e.g., bounding boxes 234a, 234b, 234c, 234d, 234e, 234f, etc. Reference numbers 234a-234f have been assigned to the bounding boxes in FIG. 2a for illustrative purposes only. It should be readily understood that other bounding boxes in FIG. 2a (e.g., bounding boxes without reference numbers) may also represent spatial entities within the environment. In some variations, at least some spatial entities (e.g., spatial landmarks, objects, etc.) within the environment may be identified and labeled on the semantic map representation 232a. For example, a sofa in the environment is color-coded with bounding box 234e and labeled as "sofa" on the semantic map representation 232a. In a similar manner, a chair in the environment is color-coded with bounding box 234f and labeled as "chair" on the semantic map representation 232a. As can be seen in Figure 2a, the first type of input 230a also includes natural language instructions 236 that describe a navigation path for the mobile robot 112.

[0032] FIG. 2b illustrates an exemplary second type of input 230b acquired at the controller 102. The second type of input 230b includes a semantic representation 232b of an environment. Similar to FIG. 2a, the semantic map representation 232b is a top-down view of the environment and includes spatial entities (e.g., spatial landmarks, objects, etc.) as color-coded bounding boxes, e.g., bounding boxes 234a′, 234b′, 234c′, 234d′, 234e′, 234f′, etc. Reference numbers 234a′-234f′ have been assigned to the bounding boxes in FIG. 2b for illustrative purposes only. It should be readily understood that other bounding boxes in FIG. 2b (e.g., bounding boxes without reference numbers) may also represent spatial entities in the environment. In some variations, at least some spatial entities (e.g., spatial landmarks, objects, etc.) in the environment may be identified and labeled on the semantic map representation 232b. As seen in FIG. 2b, the second type of input 230b includes a visual representation 238 of a navigation path for the mobile robot 112 within the semantic map representation 232b. The visual representation 238 of the navigation path may be represented as a discrete sequence of coordinates. In some variations, the visual representation 238 of the navigation path may be a series of points connected by lines. The series of points may include a start point and a stop point, as shown in FIG. 2b.

[0033] In some variations, the semantic map representation (e.g., 232a and 232b) may be symbolically represented as a list of spatial entities, e.g., objects, etc. For example, the semantic map representation may define objects in an environment with their corresponding coordinates (e.g., chair (x1, y1, w1, d1), table (x2, y2, w2, d2), etc., where x1, x2, y1, y2 represent the coordinates of the object's center, w1 and w2 represent width, and d1 and d2 represent depth). In some variations, the semantic map representation may include meta-information (e.g., metadata), such as data representing one or more properties of the spatial entities, such as coordinates, size, visual properties, etc. The semantic map representations (e.g., 232a and 232b) provided as input to the controller 102 may be created manually (e.g., written by a person), automatically (e.g., as projections from captures of 3D scenes), or semi-automatically (e.g., first automatically generated and then manually edited by a person). Spatial entities may be detected and / or labeled either manually or automatically.

[0034] 1 , the controller 102 may be configured to automatically generate instructions for maneuvering the mobile robot 112 from one or more inputs. For example, the controller 102 may be configured to implement a pre-trained model 104. The pre-trained model 104 may be trained with a pair of image data and corresponding natural language description data. The image data may include any suitable type of image of one or more environments. The natural language description data may comprise descriptions that describe one or more features in the corresponding image data.

[0035] In some variations, the pre-trained model may be any suitable pre-trained model trained with pairs of image data and corresponding natural language descriptions (e.g., captions), as described above. For example, the pre-trained model 104 may be a multimodal pre-trained model. For example, the pre-trained model may be a multimodal text generation model, such as a model using BLIP as described in Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, In International Conference on Machine Learning, pp. 12888-12900, PMLR. For example, a BLIP model may be trained with pairs of images and captions.

[0036] In some variations, the pre-trained model 104 may include an encoder 106 (e.g., a text encoder, an image encoder, a combination thereof, and / or the like), a neural network 108 such as, for example, a long short-term memory (LSTM) network, and a decoder 110. The controller 102 (e.g., a processor of the controller) may include instructions and / or software code for executing the pre-trained model 104, the encoder 106, the neural network 108, and / or the decoder 110. In some variations, the pre-trained model 104, the encoder 106, the neural network 108, and / or the decoder 110 may further include instructions and / or software code for performing one or more of the following: (1) obtaining one or more inputs, (2) extracting spatial regions of the environment from the inputs, (3) extracting or heuristically determining actions to be implemented by the mobile robot 112, (4) embedding the extracted spatial regions and actions, (5) encoding the embedded regions and actions, and (6) generating one or more outputs.

[0037] The pre-trained model 104 is configured to generate one or more outputs from one or more inputs. For example, if the input is a first type of input, such as the input 230a shown in FIG. 2a, the pre-defined model 104 may be configured to generate a first type of output from the first type of input. The first type of output may be a semantic map representation of the environment including a visual representation of a path described in the natural language instructions of FIG. 2a. For example, the first type of output may be the semantic map representation 230b as shown in FIG. 2b. As another example, if the input is a second type of input, such as the input 230b shown in FIG. 2b, the pre-defined model may be configured to generate a second type of output from the second type of input. The second type of output may be a semantic map representation of the environment with natural language instructions describing the visual representation of the path in FIG. 2b. For example, the second type of output may be 230a including a semantic map representation and natural language instructions as shown in FIG. 2a. In other words, when the input is, for example, 230a in Figure 2a, the pre-trained model 104 generates an output, for example, 230b in Figure 2b. Additionally or alternatively, when the input is, for example, 230b in Figure 2b, the pre-trained model 104 generates an output, for example, 230a in Figure 2a.

[0038] The output from the controller 102 is transmitted to the mobile robot 112. The mobile robot 112 can be configured to navigate within an environment based on one or more outputs received from the controller 102. The output includes a semantic map representation of the environment. Because the semantic map representation includes annotated representations of spatial entities within the environment, it allows the mobile robot to avoid the task of object detection (i.e., the task of recognizing spatial entities within the environment). The mobile robot 112 can be configured to identify a navigation path based on the semantic map representation by interpreting instructions received from the controller 102 without the need to identify and / or recognize objects.

[0039] In some variations, the mobile robot 112 may be configured to disambiguate ambiguities in the output. For example, consider an example output comprising a semantic map representation of an indoor environment and the natural language command "move around the table." As previously mentioned, the semantic map representation may include representations of spatial entities, such as landmarks (e.g., walls, doors, etc.), objects (e.g., furniture, tables, chairs, etc.), combinations thereof, and / or the like. These spatial entities may be annotated with labels (e.g., as seen in FIG. 2a). Such annotations may allow the mobile robot 112 to separate the tasks of visual object detection and interpretation of navigation instructions. This may then enable the mobile robot to resolve missing or uncertain information on the semantic map representation with a person in the loop. For example, consider the natural language command "move around the table." If "table" is a labeled, known object on the semantic map representation, the command interpretation model implemented by the mobile robot 112 may find a navigation path without having to visually recognize the object. However, if "table" is not labeled on the semantic map representation, the mobile robot 112 can be configured to ask a clarifying question to resolve the ambiguity.

[0040] FIG. 3 is a flow diagram 300 illustrating an overview of an example method for generating navigation instructions for a mobile robot (e.g., structurally and / or functionally similar to mobile robot 112 of FIG. 1 ). In some variations, method 300 may be performed by one or more controllers, such as controller 102 of FIG. 1 . At 302, the method may include obtaining one or more inputs. The inputs may be a first type of input, such as 230a in FIG. 2a , and / or a second type of input, such as 230b in FIG. 2b . The first type of input 230a may include a semantic map representation of an environment 232a and natural language instructions 236 describing a path for maneuvering the mobile robot within the environment. The second type of input 230b may include a semantic map representation of an environment 232b including a visual representation 238 of a path for maneuvering the mobile robot within the environment.

[0041] As previously mentioned, the semantic map representation (e.g., 232a in FIG. 2a or 232b in FIG. 2b) may be a top-down view of the environment. For example, the semantic map representation M s may be a top-down view of the scene s. N} (assuming there are N spatial entities in scene s) is a semantic map representation M s The visual representation of the path P = {p1,...p k} is represented as a series of points connected by white lines. As can be seen in Figure 2b, the starting point p1 and the ending point p k is the other intermediate point p jIt is a different color, where 1 < j < k. In some variations, the semantic map representation can be obtained from a virtual environment. For example, a virtual environment (e.g., Habitat) can include a three-dimensional representation that defines the positions of spatial entities within the environment. The semantic map representation can be extracted from this three-dimensional representation of the environment. Additionally or alternatively, the semantic map representation can be obtained by any suitable mobile robot using Simultaneous Localization and Mapping (SLAM). Additionally or alternatively, the semantic map representation can be created manually by a person.

[0042] If the input is of the first type of input, such as 230a in FIG. 2a, etc., at 304, the method includes executing a pre-trained model (e.g., structurally and / or functionally similar to the pre-trained model 104 of FIG. 1). Executing a pre-trained model when the input is of the first type of input can include performing one or more operations at "A" as further described in FIG. 6 below. At 306, the method includes generating an output of the first type, such as 230b in FIG. 2b, etc. FIG. 4 shows generating an exemplary output of the first type from an exemplary input of the first type. As seen in FIG. 4, the input 230a includes a semantic map representation 232a of the environment and a natural language instruction 234 that describes a navigation path for a mobile robot. Performing one or more operations within "A" generates the output 230b. The output 230b includes a semantic map representation 232b of the environment having a visual representation 238 of the navigation path described in the natural language instruction 234.

[0043] Referring back to FIG. 3 , if the input is a second type input, such as 230b in FIG. 2b , then at 304 the method includes executing the pre-trained model. When the input is a second type input, executing the pre-trained model may include performing one or more operations in “B” as further described below in FIG. 7 . At 308, the method includes generating a second type output, such as 230a in FIG. 2a . FIG. 5 illustrates generating an exemplary second type output from an exemplary second type input. As seen in FIG. 5 , input 230b includes a semantic map representation 232b of the environment. Semantic map representation 232b includes a visual representation 238 of a navigation path for the mobile robot. Executing one or more operations in “B” generates output 230a. Output 230a includes the semantic map representation 232a of the environment. Output 230a also includes natural language instructions 234 describing the navigation path in visual representation 238.

[0044] 6 is a flow diagram 600 of a method illustrating one or more operations in “A” that cause a pre-trained model (e.g., structurally and / or functionally similar to pre-trained model 104 of FIG. 1 ) to generate a first type of output from a first type of input. More specifically, given a first type of input (e.g., 230a) comprising a semantic map representation (e.g., 232a) and a natural language instruction (e.g., 236), one or more operations in method 600 may be configured to perform the task of generating a visual representation (e.g., 238) of a path described in the natural language instruction (e.g., 236).

[0045] In some variations, the method may include encoding at least a portion of the first type of input (e.g., a semantic map representation within the first type of input). The encoding may be performed by an encoder that may be structurally and / or functionally similar to encoder 106 of FIG. 1. The encoder may be an image encoder and / or a text encoder. The method may further include resizing the semantic map representation (e.g., 232a) by nearest sampling to 384x384 and feeding the resized semantic map representation to a vision neural network having a patch size of 16. At 612, the method may include extracting spatial regions (e.g., conference rooms, hallways, etc.) from the semantic map representation via an encoder. For example, the encoder may represent (e.g., encode) names of the extracted spatial regions as a sequence of character strings for each navigation point. At 614, the method may include extracting actions (e.g., left, right, straight, stop, etc.) to be performed by the mobile robot from the natural language instruction. At 616, the method includes embedding the extracted spatial regions via an encoder to produce a region embedding. The region embedding of each point may be obtained by applying an average pooling operation to the word embedding. At 618, the method includes embedding the extracted actions via an encoder to produce an action embedding. For example, an embedding layer may be applied to distinct action values ​​(e.g., left, right, straight, stop, etc.) to obtain an action embedding in the same dimension as the region embedding. In some variations, the spatial region may be embedded as a first vector and the action may be embedded as a second vector. By way of example, each navigation point may be associated with an extracted spatial region (e.g., "living room," "hallway," etc.). Such associations may be represented as a sequence of words. Each word in the sequence of words may be embedded. This may produce a vector. An average may be calculated over the embeddings of all words in the sequence to obtain the region embedding.

[0046] At 620, the method may include encoding, via a neural network (e.g., structurally and / or functionally similar to neural network 108 of FIG. 1 ), sequential path information for the mobile robot based on the action embedding and the region embedding. For example, the method may include adding together the region embedding and the action embedding at each point and using a three-layer LSTM to embed the sequential path information along the navigation path. At 622, the method includes generating, via a decoder (e.g., structurally and / or functionally similar to decoder 110 of FIG. 1 ), a semantic map representation of the environment including a visual representation of the path described in the natural language instructions at the input.

[0047] 7 is a method flowchart 700 illustrating one or more operations in "B" that cause a pre-trained model (e.g., structurally and / or functionally similar to pre-trained model 104 of FIG. 1) to generate a second type of output from a second type of input. More specifically, given a second type of input (e.g., 230b) comprising a semantic map representation (e.g., 232b) that includes a visual representation (e.g., 238) of a route, one or more operations in method 700 may be configured to perform the task of generating natural language instructions (e.g., 236) that describe the route.

[0048] In some variations, the method may include encoding at least a portion of the second type of input. The encoding may be performed by an encoder that may be structurally and / or functionally similar to encoder 106 of FIG. 1 . The encoder may be an image encoder and / or a text encoder. Method 700 may further include resizing the semantic map representation (e.g., 232b) by nearest sampling to 384x384 and feeding the resized semantic map representation to a vision neural network with a patch size of 16. At 712, the method may include extracting spatial regions (e.g., conference room, hallway, etc.) from the semantic map representation. For example, the encoder may represent the extracted spatial regions as a sequence of character strings for each navigation point. For example, the encoder may encode the name of the spatial region for each navigation point, such as "hallway" or "living room." At 714, the method may include heuristically determining an action (e.g., left, right, straight, stop, etc.) to be performed by the mobile robot from the visual representation of the navigation path. For example, the method may include identifying a shape of the navigation path from the visual representation (e.g., 238). The method may include heuristically determining an action based on an analysis of the shape of each navigation point on the navigation path.

[0049] At 716, the method includes embedding, via an encoder, the extracted spatial region to produce a region embedding. The region embedding of each point may be obtained by applying an average pooling operation to the word embedding. For example, each navigation point may be associated with the extracted spatial region as a sequence of words. Each word in the sequence of words may be embedded to produce a vector. An average may be calculated over all word sequences to produce the region embedding. At 718, the method includes embedding, via an encoder, the heuristically determined action to produce an action embedding. For example, an embedding layer may be applied to distinct action values ​​(e.g., left, right, straight, stop, etc.) to obtain an action embedding in the same dimension as the region embedding. In some variations, the spatial region may be embedded as a first vector and the action may be embedded as a second vector.

[0050] At 720, the method may include encoding, via a neural network (e.g., structurally and / or functionally similar to neural network 108 of FIG. 1 ), sequential path information for the mobile robot based on the action embedding and the region embedding. For example, the method may include adding together the region embedding and the action embedding at each point and using a three-layer LSTM to embed the sequential path information along the navigation path. At 722, the method includes generating, via a decoder (e.g., structurally and / or functionally similar to decoder 110 of FIG. 1 ), natural language instructions describing the navigation path for the mobile robot.

[0051] In some variations, the method 700 may be framed as an image captioning task where the input image is a semantic map representation (e.g., 232b) with a visual representation of a route (e.g., 238) and the output text is natural language instructions (e.g., 236) describing the route.

[0052] Augmenting the input data Panoramic images In some variations, input to a controller (e.g., controller 102 of FIG. 1 ) may be augmented by providing one or more panoramic images to a pre-trained model (e.g., pre-trained model 104 of FIG. 1 ). For example, in some scenarios, panoramic images may provide additional visual information that may not be shown in the semantic map representation. Specifically, some natural language instructions may include visual properties of spatial entities (e.g., color, shape, etc.). Some of these visual properties may not be annotated and / or represented in the semantic map representation. In such scenarios, the input may be augmented with panoramic images. The pre-trained model may process these panoramic images to identify visual properties in the natural language instructions. The pre-trained model may then be configured to learn these identified visual properties and apply them to subsequent input data.

[0053] To extract properties / features from the panoramic image, an image encoder may be initialized. For example, the pre-trained model described herein may include an image encoder that receives image data as input and an encoder 106 that receives text data. In some variations, the image encoder may be different from the encoder 106, which may be a text encoder. Because panoramas are photorealistic images as in the pre-training data, the parameters of the image encoder may be frozen when training on the panoramic image during tuning of the pre-trained model. The extracted features may be embedded to produce a visual embedding. In some variations, in addition to the image encoder, an additional multilayer perceptron with two linear layers may be applied to increase the flexibility of the visual embedding. The panorama may be treated as an individual frame, and the average value of the visual embedding may be used to represent the panoramic information of the navigation route. This panoramic information may be added to the sequential route information encoded by a neural network (e.g., structurally and / or functionally similar to the neural network 108 of FIG. 1). The panoramic information along with the sequential route information may be provided as output to a decoder (e.g., structurally and / or functionally similar to decoder 110 of FIG. 1 ) to output navigation instructions (e.g., natural language instructions or a semantic map representation with a visual representation of the route).

[0054] FIG. 8 is an illustration of augmenting input to a pre-trained model with a panoramic image. In this example, the input includes a semantic representation of an environment (not shown in FIG. 8) with natural language instructions 836. As seen in FIG. 8, the natural language instructions include visual properties of spatial entities, such as a "red carpet" and a "white and gold table." Thus, a panoramic image 842 of the environment can be provided as input to the pre-trained model, thereby augmenting the input to the pre-trained model. Spatial regions 844 are extracted from the semantic map representation, and actions 846 are extracted from the natural language instructions 836. The extracted regions 844 and actions 846 are embedded to produce region embeddings and action embeddings. Features extracted from the panoramic image 842 are embedded to produce a visual embedding. Sequential path information based on the region embeddings and action embeddings and panoramic information based on the visual embedding are provided to a decoder. The decoder outputs a semantic map representation 832 of the environment with a visual representation 838 of a navigation path superimposed on the semantic map representation.

[0055] Multimodal Alignment with Contrastive Loss In some modifications, multimodal alignment with contrastive loss can be applied to extend a pre-trained model (e.g., pre-trained model 104 of FIG. 1). Contrastive learning is an effective method used in self-supervised learning of visual representations and multimodal learning. The effectiveness of introducing contrastive training for navigation instruction generation tasks can be investigated as an auxiliary loss. In some modifications, the positive example P + (C gt ,I gt ) can be defined as a pair of combined input embeddings (e.g., region embeddings, action embeddings, embeddings of images in semantic map representations) and instruction embeddings (e.g., embeddings of words in natural language instructions). - (C gt,I rnd ) can be composed of a pair of input embeddings and randomly sampled instruction embeddings. For example, a positive example is an input embedding C gt and the corresponding matching instruction embedding I gt The negative examples may include ground truth data with pairs of input embeddings C gt Randomly generated instruction embedding for I rnd The multimodal input matrix E input and the text instruction matrix E text is the expected compatibility matrix C between the input and the label pred can then be multiplied to obtain C pred The cross entropy loss for the ground truth correspondence C gt It can be calculated as follows.

[0056] Expanding and Grounding with Prompts In some variations, a decoder (e.g., structurally and / or functionally similar to decoder 110 of FIG. 1 ) may be used in conjunction with a Large Language Model (LLM) to generate navigation instructions. For example, the decoder may also receive LLM prompts as input. LLM prompts may be generated from templates describing spatial entities and objects in the environment, such as “start at the dark yellow point near the sofa cushions in the living room.” A pre-trained model (e.g., pre-trained model 104) may be adjusted by prompting and providing prompt templates to the decoder during inference (e.g., while the decoder is generating navigation instructions). The LLM may provide additional textual descriptions to enhance the process of generating navigation instructions. Prompting may be useful in two ways. First, it can aid visual-linguistic grounding by describing nearby landmarks and areas in the prompt templates. In other words, prompting may provide correct / accurate descriptions of spatial entities. Second, at inference time, the generated instructions are autoregressively conditioned on prompt templates, resulting in a more controllable generation of navigation instructions.

[0057] 9 illustrates an example architecture of a pre-trained model (e.g., pre-trained model 104) configured to receive the augmented input described herein and thereby generate navigation instructions. Semantic map encoder 906a may be configured to encode a semantic map representation of the environment (e.g., by resizing the semantic map representation and feeding it to a vision neural network). Region encoder 906c may be configured to extract spatial regions from the semantic map representation and embed the extracted regions. Action embedding 906d may be configured to extract actions from natural language instructions and / or heuristically determine actions from a visual representation of a path in the semantic map. Additionally, action embedding 906d may embed the extracted and / or determined actions. In some variations, panorama encoder 906b may extract visual features from a panoramic image. The weights of panorama encoder 906b may be fixed.

[0058] In some variations, the semantic map encoder 906a, the panorama encoder 906b, the region encoder 906c, and the action embedding 906d may collectively perform the operations of the encoder 106 of Figure 1. In other words, the encoder 106 of Figure 1 may include the semantic map encoder 906a, the panorama encoder 906b, the region encoder 906c, and the action embedding 906d.

[0059] Neural network 908a may encode action embeddings within the sequential path information, and neural network 908b may encode region embeddings within the sequential path information. Neural networks 908a and 908b may share weights. Neural networks 908a and 908b may collectively perform the operations of neural network 108 of FIG. 1. In other words, neural network 108 of FIG. 1 may include neural networks 908a and 908b. Average pooling 909 may treat panoramas as individual frames and apply an average value of the visual embeddings from panorama encoder 906b. Based on this, average pooling 909 may represent panoramic information of the navigation path.

[0060] The panoramic information from average pooling 909 and the sequential path information from neural networks 908a and 908b may be provided to a decoder 910 (e.g., structurally and / or functionally similar to decoder 110 of FIG. 1). In some variations, an LLM may be used in conjunction with decoder 910 to augment the output. In some variations, multimodal alignment with contrastive loss may be applied to decoder 910 to augment the output. Decoder 910 may generate navigation instructions (e.g., natural language instructions or a visual representation of the path superimposed on a semantic map representation).

[0061] Training Data As mentioned above, the training data may include pairs of image data and corresponding natural language descriptions of the image data. In some variations, semantic map representations, spatial regions, and mobile robot actions may be extracted from a habitat simulation environment (e.g., https: / / aihabitat.org / ). As mentioned above, in some variations, each type of spatial entity on the semantic map representation may be represented by a unique color. Visual representations of navigation paths and human annotations from a R2R dataset (e.g., from “Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf (Umlauted English Letter u), Ian Reid, Stephen Gould, and Anton van den Hengel, 2018, Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)”) may be employed to obtain training data. Panoramic images in RGB can be obtained from the Matterport3D simulator (e.g., “Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang, 2017, Matterport3d: Learning from rgb-d data in indoor environments, arXiv preprint arXiv:1709.06158”) at individual navigation points.

[0062] Exemplary Training and Exemplary Results As an example, experiments were conducted to generate instructions from the semantic map representation using a pre-trained model (e.g., pre-trained model 104 of FIG. 1) that encodes the semantic map representation, actions (derived from a visual representation of the path shape), spatial domain information (e.g., room type), and panoramic images. The combined inputs were encoded into vectors (e.g., via encoder 106 and neural network 108 of FIG. 1) and used by a decoder (e.g., decoder 110 of FIG. 1) initialized with the pre-trained values ​​to generate natural language instructions.

[0063] As can be seen in Figure 10, using the unsupervised evaluation metric SPICE (i.e., Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, 2016, SPICE: Semantic propositional image caption evaluation, in ECCV), model performance is shown to be comparable to end-to-end generative methods. Advantages of the techniques described herein include the presence of an intermediate representation (e.g., semantic map representation) that simplifies portability to new domains and the introduction of new, unfamiliar objects by enabling interactions between users and mobile robots that can be reasoned based on the semantic map representation.

[0064] In another example, a pre-trained model (e.g., pre-trained model 104 in FIG. 1) is trained using the train split of the R2R dataset discussed above. The pre-trained model is validated on both the seen and unseen sets. The pre-trained model was used for experiments. Baselines were set up with different combinations of input: 1) top-down semantic map representation, 2) semantic map representation + regions and actions, and 3) semantic map representation + panoramic images. Additionally, contrastive loss and prompting were also experimented.

[0065] The performance of nine system variants was evaluated based on five selected system variants with automatic metrics and human target scores. SPICE (i.e., Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, 2016, SPICE: Semantic propositional image caption evaluation, in ECCV) was used for the automatic evaluation metrics, and three follow Zhao et al. (2021).

[0066] Table 1 below shows the automatic (SPICE) and human evaluation results with inputs of different modalities in seen and unseen environments. ** and * indicates statistically significant differences from baseline (p<0.01) and (p<0.05).

[0067] [Table 1]

[0068] As expected, the model performs better in the seen setting than in the unseen setting. For both settings, using region and action information with prompts can improve the model's performance. Adding panoramic images can further improve performance in the seen environment.

[0069] Thus, incorporating more information in different modalities may improve the performance for generating navigation instructions. Despite using a single image of a summarized semantic map representation as visual input, many of the described input variants that do not use panoramic images perform poorly in unseen / seen settings, as seen in Speaker Fol. 17.0 / 18.7 (Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell, 2018, Speaker follower models for vision-and-language navigation, Advances in Neural Information Processing Systems, 31) and EnvDrop 18.1 / 20.2 (Hao Tan, Licheng Yu, and Mohit Bansal, 2019, Learning to navigate unseen environments: Back translation with environmental dropout, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human It performs comparably to an LSTM-based end-to-end approach that uses only panoramic images with SPICE scores (Language Technologies, Volume 1 neural network).

Claims

1. obtaining, at a processor, a first input or a second input; In response to receiving the first input, executing a pre-trained model via the processor to generate a first output; In response to receiving the second input, executing the pre-trained model via the processor to generate a second output. Equipped with the first input comprises a first semantic map representation of an environment and at least one first natural language instruction describing a first path for the robot; the first output comprises a first output semantic map representation of the environment, the first output semantic map representation including a visual representation of the first path for the robot to maneuver the robot within the environment; the second input comprises a second semantic map representation of the environment, the second semantic map representation including a visual representation of a second path for the robot; The computer-implemented method, wherein the second output comprises a second output semantic map representation of the environment and at least one second natural language instruction describing the second path for maneuvering the robot within the environment.

2. 2. The computer-implemented method of claim 1, wherein the first semantic map representation is a top-down view of the environment and includes a plurality of spatial landmarks in the environment as a plurality of color-coded bounding boxes on the first semantic map representation.

3. the second semantic map representation is a top-down view of the environment; a plurality of spatial landmarks in the environment as a plurality of color-coded bounding boxes on the second semantic map representation; the visual representation of the second path as a series of points connected by lines on the second semantic map representation; The computer-implemented method of claim 1 , wherein the series of points includes a start point and an end point.

4. The computer-implemented method of claim 1 , wherein at least one of the first semantic map representation and the second semantic map representation is created manually.

5. The computer-implemented method of claim 1 , wherein at least one of the first semantic map representation and the second semantic map representation is created automatically.

6. The computer-implemented method of claim 1 , wherein the pre-trained model is a multimodal text generation model.

7. Executing the pre-trained model in response to receiving the first input includes: extracting, via an encoder, one or more spatial regions of the environment from the first semantic map representation; extracting, via the encoder, one or more actions to be implemented by the robot from at least one of the at least one first natural language instruction and the at least one second natural language instruction; Embedding, using the encoder, the one or more spatial regions as a first vector; embedding, using the encoder, the one or more actions as a second vector; and The computer-implemented method of claim 1 , further comprising:

8. Executing the pre-trained model includes: encoding, via a neural network, sequential path information for the robot for maneuvering the robot within the environment based on the first vector and the second vector; The computer-implemented method of claim 7 further comprising:

9. 9. The computer-implemented method of claim 8, wherein executing the pre-trained model further comprises executing a decoder to generate the first output semantic map representation using the sequential path information.

10. 2. The computer-implemented method of claim 1, wherein executing the pre-trained model in response to receiving the second input further comprises heuristically determining one or more actions to be implemented by the robot based at least in part on the visual representation of the second path for the robot.

11. Executing the pre-trained model in response to receiving the second input includes: extracting, via an encoder, one or more spatial regions of the environment from the second semantic map representation; embedding, using the encoder, the one or more spatial regions as a third vector; and embedding, using the encoder, the one or more actions as a fourth vector; and The computer-implemented method of claim 10 further comprising:

12. Executing the pre-trained model includes: encoding, via a neural network, sequential path information for the robot to navigate in a second environment based on the third vector and the fourth vector. The computer-implemented method of claim 11 further comprising:

13. 13. The computer-implemented method of claim 12, wherein executing the pre-trained model further comprises executing a decoder to use the sequential path information to generate the second output semantic map representation and the at least one second natural language instruction describing the second path.

14. acquiring a first panoramic image of the environment; responsive to receiving the first input, executing the pre-trained model to generate the first output based at least in part on the first panoramic image; responsive to receiving the second input, executing the pre-trained model to generate the second output based at least in part on the first panoramic image; The computer-implemented method of claim 1 further comprising:

15. training the pre-trained model using a plurality of pairs of training data; Furthermore, Each of the plurality of pairs is Image data; a corresponding natural language description of said image data; The computer-implemented method of claim 1 , comprising:

16. obtaining an input semantic map of an environment, the input semantic map including a visual representation of a path for the robot to navigate within the environment; extracting one or more spatial regions of the environment from the input semantic map; heuristically determining, based at least in part on the visual representation of the path, one or more actions to be implemented by the robot to maneuver the robot within the environment; Embedding the one or more spatial regions to produce a region embedding; Embedding the one or more actions to produce an action embedding; encoding sequential path information for the robot using the region embedding and the action embedding; generating natural language instructions describing the path using the sequential path information to maneuver the robot within the environment; and A computer-implemented method comprising:

17. obtaining an input semantic map of the environment; obtaining natural language commands and using the input semantic map to navigate a robot within the environment; extracting one or more spatial regions of the environment from the input semantic map; extracting from the natural language instructions one or more actions to be implemented by the robot to navigate the robot within the environment; Embedding the one or more spatial regions to produce a region embedding; Embedding the one or more actions to produce an action embedding; encoding sequential path information for the robot using the region embedding and the action embedding; generating an output semantic map of the environment using the sequential path information, the output semantic map including a visual representation of a path for the robot to navigate the robot within the environment; A computer-implemented method comprising:

Images