System and method for generating navigation commands for robots

Abstract

To provide a system for generating maneuvering commands for mobile robots.SOLUTION: One embodiment of the present invention acquires a first input or a second input, executes a pre-trained model, and generates a first output or a second output. The first input comprises a first semantic map representation of the environment and at least one first natural language instruction describing a first path for the robot. The first output comprises a first output semantic map representation of the environment, and the first output semantic map representation includes a visual representation of the first path for the robot to navigate within the environment. The second input comprises a second semantic map representation of the environment, and the second semantic map representation includes a visual representation of a second path for the robot. The second output comprises a second output semantic map representation of the environment and at least one second natural language instruction describing a second path for maneuvering the robot within the environment.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This disclosure relates, in general, to the field of robot navigation. More specifically, this disclosure relates to a system and method for generating navigation commands based on a semantic map representation of the environment. [Background technology]

[0002] In recent years, the use of autonomous and semi-autonomous systems has increased exponentially. In particular, autonomous and semi-autonomous systems such as mobile robots are being used in various sectors (e.g., manufacturing, military, transportation, medical, education, services, etc.) to increase efficiency and productivity. Generally, mobile robots are configured to move between multiple locations, thereby performing one or more tasks. For example, mobile robots are configured to receive instructions and / or commands to be steered as needed.

[0003] Traditionally, mobile robots have been configured to receive navigation commands via control devices such as joysticks, keyboards, switches, and user interfaces (e.g., graphical interfaces). However, these commands are inherently non-verbal. Such commands require a person (e.g., a user) to constantly monitor and physically control the mobile robot (e.g., using the user's hands via a control device). However, this approach has several drawbacks. For example, it may be undesirable to have a user constantly control a mobile robot when their hands are occupied.

[0004] More recently, mobile robots have been configured to receive simple linguistic commands, such as "move forward," "turn left," "turn right," and "go to the container." While this approach can be preferable to physical control, it also has several challenges. One such challenge is that each utterance of a linguistic command needs to be mapped to a physical action of the mobile robot. For example, "move forward" would need to be mapped to a physical action of the mobile robot that moves forward in one step. Because of this one-to-one mapping between linguistic commands and corresponding physical actions, this approach cannot support more natural commands that can be constructed in multiple steps, such as "move around the table and stop next to the furthest chair."

[0005] Therefore, there is an unmet need for improved systems and methods to enhance human-robot interaction and automatically generate navigation commands for mobile robots. Specifically, there is a need to automatically generate navigation commands from natural, complex commands. Furthermore, there is an unmet need to seamlessly adapt the navigation capabilities of mobile robots to known environments that have new environments (e.g., new physical or virtual spaces) or new spatial entities (e.g., new objects). [Overview of the project]

[0006] A system and method for generating navigation instructions are disclosed herein. In one embodiment, a computer implementation method is provided. The method includes taking a first input or a second input in a processor. In response to receiving a first input, the method includes running a pre-trained model via the processor and generating a first output. In response to receiving a second input, the method includes running a pre-trained model via the processor and generating a second output. The first input comprises a first semantic map representation of the environment and at least one first natural language instruction describing a first path for the robot. The first output comprises a first output semantic map representation of the environment. The first output semantic map representation includes a visual representation of a first path for the robot to maneuver within the environment. The second input comprises a second semantic map representation of the environment. The second semantic map representation includes a visual representation of a second path for the robot. The second output comprises a second output semantic map representation of the environment and at least one second natural language instruction describing a second path for maneuvering the robot within the environment.

[0007] In some variations, the first semantic map representation is a top-down view of the environment and includes multiple spatial landmarks within the environment as multiple color-coded bounding boxes on the first semantic map representation. In some variations, the second semantic map representation is a top-down view of the environment and includes multiple spatial landmarks within the environment as multiple color-coded bounding boxes on the second semantic map representation, and a visual representation of a second path as a series of points connected by lines on the second semantic map representation. The series of points includes a start point and an end point.

[0008] In some modification forms, at least one of the first and second semantic map representations is created manually. In some modification forms, at least one of the first and second semantic map representations is created automatically. In some modification forms, the pre-trained model is a multimodal text generation model.

[0009] In some modified forms, running a pre-trained model in response to receiving a first input further includes: extracting one or more spatial regions of the environment from a first semantic map representation via an encoder; extracting one or more actions to be implemented by the robot from at least one of at least one first natural language instruction and at least one second natural language instruction via an encoder; embedding one or more spatial regions as a first vector using an encoder; and embedding one or more actions as a second vector using an encoder. Running the pre-trained model may further include encoding sequential path information for the robot to maneuver within the environment via a neural network based on the first and second vectors. Running the pre-trained model may further include running a decoder to generate a first output semantic map representation using the sequential path information.

[0010] In some modified forms, running a pre-trained model in response to receiving a second input further includes heuristically determining one or more actions to be implemented by the robot, at least in part, based on a visual representation of a second path for the robot. In some modified forms, running a pre-trained model in response to receiving a second input further includes extracting one or more spatial regions of the environment from a second semantic map representation via an encoder, embedding one or more spatial regions as a third vector using the encoder, and embedding one or more actions as a fourth vector using the encoder. Running a pre-trained model may further include encoding sequential path information for the robot to maneuver in the second environment via a neural network, based on the third and fourth vectors. Running a pre-trained model may further include running a decoder to generate a second output semantic map representation and at least one second natural language instruction describing the second path using the sequential path information.

[0011] In some modified forms, the computer implementation method may further comprise acquiring a first panoramic image of an environment. In response to receiving a first input, the method may comprise running a pre-trained model and generating a first output based at least partially on the first panoramic image. In response to receiving a second input, the method may comprise running a pre-trained model and generating a second output based at least partially on the first panoramic image. The method may further comprise training the pre-trained model using a plurality of pairs of training data, each of which may comprise image data and a corresponding natural language description of the image data.

[0012] In another embodiment, a computer implementation method is provided. The computer implementation method comprises obtaining an input semantic map of an environment. The input semantic map includes a visual representation of a robot's path for maneuvering the robot within the environment. The method may further include extracting one or more spatial regions of the environment from the input semantic map; heuristically determining one or more actions to be implemented by the robot, at least in part on the visual representation of the path, to maneuver the robot within the environment; embedding one or more spatial regions to produce region embeddings; embedding one or more actions to produce action embeddings; encoding sequential path information for the robot using the region embeddings and action embeddings; and generating natural language instructions describing the path using the sequential path information to maneuver the robot within the environment.

[0013] In yet another embodiment, a computer implementation method is provided. The computer implementation method includes: obtaining an input semantic map of an environment; obtaining natural language instructions to steer a robot within the environment using the input semantic map; extracting one or more spatial regions of the environment from the input semantic map; extracting one or more actions to be implemented by the robot from the natural language instructions to steer the robot within the environment; embedding one or more spatial regions to produce region embeddings; embedding one or more actions to produce action embeddings; encoding sequential path information for the robot using the region embeddings and action embeddings; generating an output semantic map of the environment using the sequential path information, the output semantic map including a visual representation of a path for the robot to steer the robot within the environment.

[0014] Non-limiting examples of systems and methods will be described here with reference to the attached drawings. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 shows an exemplary system for generating navigation instructions for one or more mobile robots. [Figure 2a] Figure 2a shows an exemplary type of input provided to the system of Figure 1. Figure 2a also shows an exemplary type of output generated by the system of Figure 1. [Figure 2b] Figure 2b shows another exemplary type of input provided by the system of Figure 1. Figure 2b also shows another exemplary type of output generated by the system of Figure 1. [Figure 3] Figure 3 is a flowchart showing an overview of an exemplary method for generating navigation instructions for a mobile robot. [Figure 4] Figure 4 shows generating an exemplary first type of output from an exemplary first type of input. [Figure 5] Figure 5 shows generating an exemplary second type of output from an exemplary second type of input. [Figure 6] Figure 6 is a flowchart of a method showing one or more operations for causing a pre-trained model to generate a first type of output from a first type of input. [Figure 7] Figure 7 is a flowchart of a method showing one or more operations for causing a pre-trained model to generate a second type of output from a second type of input. [Figure 8] Figure 8 is an illustration of expanding the input to a pre-trained model with a panoramic image. [Figure 9] Figure 9 shows an exemplary architecture of a pre-trained model configured to receive the expanded input and thereby generate navigation instructions. [Figure 10] Figure 10 shows the performance of a pre-trained model compared to other navigation instruction generation methods. **DETAILED DESCRIPTION OF THE INVENTION**

[0016] In this specification, the term “robot” generally refers to an autonomous and / or semi-autonomous system configured to move from one location to another. In other words, in this specification, the term “robot” refers to a system configured to be mobile and is used synonymously with the term “mobile robot.”

[0017] In this specification, the term “environment” generally refers to physical and / or virtual space. For example, the term “environment” may refer to the space in which a mobile robot will be operated. In some modified forms, the term “environment” may refer to any suitable physical space, such as an indoor space, an outdoor space, a combination thereof, and / or similar. Additionally or alternatively, the term “environment” may refer to any suitable virtual space, such as a computer-simulated environment.

[0018] In this specification, the term “semantic map representation” generally refers to an RGB image of an environment viewed from above. In some variations, a “semantic map representation” of an environment may associate spatial entities within the environment (e.g., spatial landmarks, objects within the environment, combinations thereof, and / or similar) with spatial concepts. For example, a “semantic map representation” may include a topographic map and / or metric map of the environment, and spatial entities within the environment may be represented symbolically in the “semantic map representation.” In some variations, a “semantic map representation” may include metadata (e.g., meta-information) representing one or more properties of a spatial entity (e.g., coordinates, size, visual properties, combinations thereof, and / or similar).

[0019] In this specification, the term “navigation instructions” may include instructions that enable a mobile robot to maneuver within an environment. These instructions may be in any suitable format (e.g., images, audio, video, text, a combination thereof, and / or similar). For example, in some modifications, “navigation instructions” may also be a semantic map representation of the environment indicating a navigation path for the mobile robot. In other words, in such modifications, “navigation instructions” may simply be a semantic map representation of the environment including a visual representation of the navigation path for the mobile robot. In such modifications, where “navigation instructions” are primarily visual instructions (e.g., a visual representation of the navigation path), “navigation instructions” may be directly translated into actions for the mobile robot. Thus, in these modifications, the mobile robot can perform actions to maneuver by identifying the navigation path and processing the semantic map representation of the environment. Additionally or alternatively, “navigation instructions” may also be (spoken) natural language instructions describing a navigation path for the mobile robot. In these modifications, “navigation instructions” may include a semantic map representation of the environment (without a visual representation of the navigation path) and natural language instructions describing a navigation path for the mobile robot. In these examples, the mobile robot may identify a navigation path by processing semantic map representations and interpreting natural language instructions.

[0020] Disclosed herein are improved systems and methods for generating navigation instructions for mobile robots. Specifically, the systems and methods disclosed herein can translate natural language instructions describing a navigation path into a visual representation of the navigation path, and vice versa. As previously stated, existing methods for controlling mobile robots have several challenges and drawbacks. One such existing method involves controlling the mobile robot via control devices such as joysticks, keyboards, switches, and user interfaces (e.g., graphical interfaces). This method may be impractical or undesirable in situations where the user (e.g., a human operator) does not have consistent access to the control devices. In such situations, spoken language can therefore be a more natural modality of communication between the user and the mobile robot.

[0021] Another existing approach involves controlling a mobile robot with simple linguistic commands. While controlling a mobile robot via simple linguistic commands may overcome some of the challenges of previous approaches, this approach still has many undesirable aspects. For example, given a one-to-one mapping between linguistic commands and mobile robot actions, the user may need to be extremely precise in the linguistic commands. Furthermore, this approach cannot support complex commands that might be more natural to the user. In addition, creating a comprehensive list of linguistic commands for performing various tasks and mapping each of these linguistic commands to a robot action can be difficult and time-consuming.

[0022] In the early modern period, several methods began to use terminal-to-terminal visual language navigation (VLN) to control mobile robots. In these methods, a model is trained to process panoramic images of the environment and interpret natural language instructions (e.g., from a user). The model can be trained to predict the next navigation point or a set of robot actions from the panoramic images and natural language instructions. In other words, the trained model finds the navigation path by processing the panoramic images. While this method is sometimes preferable to previous methods, several challenges associated with it still exist. Firstly, in order to train a VLN model, natural language instructions must be manually annotated. For example, a panoramic image of the environment along the navigation path may need to be acquired. The annotator may have to manually steer between two points on the navigation path and then (manually) write instructions to steer along this path. Thus, annotating navigation instructions for training a VLN model can be time-consuming and costly to scale up. Secondly, using panoramic images can be resource-intensive. This is because predicting the next navigation point or set of robot actions using panoramic images may require multiple images of different points in the environment. Thirdly, panoramic images may contain numerous details irrelevant to the navigation task, and processing these irrelevant details can increase the computation time required to generate navigation instructions. Fourthly, this method cannot be extended to known environments with new environments or new objects. In other words, the trained model cannot be extended to environments containing spatial entities (e.g., landmarks, objects, etc.) that were not visible during the model's training. For these reasons, the performance of this method has not been high enough for real-world applications.

[0023] The technology disclosed herein enables the operationalization of terminal-to-terminal visual language navigation (VLN) by enhancing its performance. Unlike VLN methods that use panoramic images, the technology disclosed herein uses a semantic map representation of the environment to enable a mobile robot to navigate within that environment. For example, the technology disclosed herein may use a semantic map representation to enable interpretation of the environment and to enable spatial reasoning for the mobile robot. The semantic map representation reduces the computation time required to generate navigation instructions. Furthermore, a single semantic map representation may be sufficient to predict a navigation point or a set of robot actions.

[0024] In several forms of modification, a model can be extended to other invisible environments by training it on semantic map representations of environments. For example, a model can be trained to generate natural language instructions from a semantic map representation of a first environment. When the trained model receives a semantic map representation of a second invisible environment as input, it can generate natural language instructions for navigating the second environment. These natural language instructions can then be used to train a VLN model, thereby enabling the operationalization of VLNs. In a similar manner, a model can be trained to generate visual representations of navigation paths from natural language instructions. The trained model can then be extended to other invisible environments with different navigation paths.

[0025] In some modified forms, the techniques disclosed herein may be configured to run a pre-trained model to automatically generate natural language instructions (e.g., spoken) from a semantic map representation of an environment. Additionally or alternatively, the techniques disclosed herein may be configured to run the same pre-trained model to automatically generate a semantic map representation of an environment, including a visual representation of a path for a mobile robot, from natural language instructions (e.g., spoken). In other words, the techniques disclosed herein can run the same pre-trained model to do a) generate natural language instructions from a semantic map representation of an environment, where the semantic map representation includes a visual representation of a navigation path for a mobile robot, and b) generate a semantic map representation of an environment, including a visual representation of a navigation path, from natural language instructions.

[0026] Therefore, the technology disclosed herein can enable a user to guide a mobile robot with natural language commands (e.g., spoken). Furthermore, the technology disclosed herein can automatically generate navigation commands that can guide a human user who may not be able to consistently control the mobile robot (e.g., physically or visually). In addition, the technology disclosed herein can be used as training data for VLN models, thereby enabling the operationalization of VLNs. The technology disclosed herein can also facilitate improved human-robot interaction. For example, a mobile robot can be configured to clarify ambiguous commands, thereby improving its understanding of physical space. This allows for the seamless adaptation of a pre-trained model to an environment that includes spatial entities that were not previously visible (e.g., not visible during training) without the need for additional training data. In some modified forms, the technology disclosed herein can be configured to facilitate a mobile robot describing its actions, where this may be a desirable property.

[0027] Figure 1 shows an exemplary system 100 for generating navigation commands for one or more mobile robots 112. The system 100 includes one or more controllers 102 configured to take one or more inputs. The controllers 102 may be configured to transform their inputs into one or more outputs. In other words, the controllers 102 may be configured to generate one or more outputs from one or more inputs. The controllers 102 may be communicably connected to one or more mobile robots 112 over a network (e.g., the Internet, a local area network (LAN), a wide area network (WAN), and / or similar). One or more outputs may be transmitted to the mobile robots 112 to steer them within the environment.

[0028] In some forms of modification, the controller 102 may include one or more servers and / or one or more processors operating on a cloud platform (e.g., Microsoft Azure®, Amazon® Web Services, IBM® Cloud Computing, etc.). The servers and / or processors may be any suitable processing devices configured to run and / or execute a set of instructions or code, and may include one or more data processors, image processors, graphics processing units, digital signal processors, and / or central processing units. The servers and / or processors may be, for example, general-purpose processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and / or similar.

[0029] In some forms of modification, the controller 102 may include a processor (e.g., a CPU). The processor may be any suitable processing device configured to run and / or execute a set of instructions or code, and may include one or more data processors, image processors, graphics processing units, physical property processing units, digital signal processors, and / or central processing units. The processor may be, for example, a general-purpose processor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or similar. The processor may be configured to run and / or execute application processes and / or other modules, system-related processes and / or functions, and / or related networks. Fundamental device technologies can be provided in various component types, such as MOSFET technologies like complementary metal-oxide-semiconductor (CMOS), bipolar technologies like emitter-coupled logic (ECL), polymer technologies (e.g., silicon-conjugated polymers and metal-conjugated polymer-metal structures), mixed analog and digital, and / or similar.

[0030] In some modified forms, the controller 102 may be configured to take at least two types of input. The first type of input may include a semantic map representation of the environment and natural language instructions describing paths within the environment (e.g., navigation paths for the mobile robot 112). The second type of input may include a semantic map representation of the environment that includes a visual representation of the paths within the environment (e.g., navigation paths for the mobile robot 112).

[0031] For example, Figure 2a shows an exemplary first type of input 230a acquired by controller 102. The first type of input 230a includes a semantic map representation 232a of the environment. The semantic map representation 232a is an overhead view of the environment and includes spatial entities (e.g., spatial landmarks, objects, etc.) as color-coded bounding boxes, e.g., bounding boxes 234a, 234b, 234c, 234d, 234e, 234f, etc. Reference numbers 234a-234f are assigned to the bounding boxes in Figure 2a for illustrative purposes only. It should be readily apparent that other bounding boxes in Figure 2a (e.g., bounding boxes without reference numbers) may also represent spatial entities in the environment. In some modified forms, at least some spatial entities in the environment (e.g., spatial landmarks, objects, etc.) may be identified and labeled on the semantic map representation 232a. For example, a sofa in the environment is color-coded in bounding box 234e and labeled as “sofa” on semantic map representation 232a. In a similar manner, a chair in the environment is color-coded in bounding box 234f and labeled as “chair” on semantic map representation 232a. As seen in Figure 2a, the first type of input 230a also includes a natural language instruction 236 that describes a navigation path for the mobile robot 112.

[0032] Figure 2b shows an exemplary second type of input 230b acquired by controller 102. The second type of input 230b includes a semantic representation 232b of the environment. Similar to Figure 2a, the semantic map representation 232b is a top-down view of the environment and includes spatial entities (e.g., spatial landmarks, objects, etc.) as color-coded bounding boxes, e.g., bounding boxes 234a', 234b', 234c', 234d', 234e', 234f', etc. Reference numbers 234a' to 234f' are assigned to the bounding boxes in Figure 2b for illustrative purposes only. It should be readily apparent that other bounding boxes in Figure 2b (e.g., bounding boxes without reference numbers) may also represent spatial entities in the environment. In some modified forms, at least some spatial entities in the environment (e.g., spatial landmarks, objects, etc.) may be identified and labeled on the semantic map representation 232b. As seen in Figure 2b, the second type of input 230b includes a visual representation 238 of a navigation path for the mobile robot 112 within the semantic map representation 232b. The visual representation 238 of the navigation path may be represented as coordinates of a sequence of individual points. In some modified forms, the visual representation 238 of the navigation path may be a series of points connected by lines. The series of points may include a start point and a stop point, as shown in Figure 2b.

[0033] In some forms of modification, semantic map representations (e.g., 232a and 232b) can be symbolically represented as a list of spatial entities, such as objects. For example, a semantic map representation could define objects in an environment by the corresponding coordinates of an object (e.g., a chair (x1, y1, w1, d1), a table (x2, y2, w2, d2), etc., where x1, x2, y1, and y2 represent the coordinates of the object's center, w1 and w2 represent the width, and d1 and d2 represent the depth). In some forms of modification, a semantic map representation may include metadata, such as data representing one or more properties of a spatial entity, such as coordinates, size, and visual properties. The semantic map representations (e.g., 232a and 232b) provided as input to controller 102 can be created manually (e.g., written by a person), automatically (e.g., as a projection from a 3D scene capture), or semi-automatically (e.g., first automatically generated and then manually edited by a person). Spatial entities can be detected and / or labeled either manually or automatically.

[0034] Referring back to Figure 1, the controller 102 may be configured to automatically generate commands for controlling the mobile robot 112 from one or more inputs. For example, the controller 102 may be configured to implement a pre-trained model 104. The pre-trained model 104 may be trained on pairs of image data and corresponding natural language description data. The image data may include images of any suitable type of one or more environments. The natural language description data may consist of descriptions that describe one or more features in the corresponding image data.

[0035] In some forms of modification, the pre-trained model may be any suitable pre-trained model trained on pairs of image data and corresponding natural language descriptions (e.g., captions) as described above. For example, the pre-trained model 104 may be a multimodal pre-trained model. For example, the pre-trained model may be a multimodal text generation model, such as a model using BLIP as described in Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language under-standing and generation, International Conference on Machine Learning, pages 12888-12900, PMLR. For example, a BLIP model may be trained on pairs of images and captions.

[0036] In some modified forms, the pre-trained model 104 may include an encoder 106 (e.g., a text encoder, an image encoder, a combination thereof, and / or similar), a neural network 108 (e.g., a long short-term memory (LSTM) network), and a decoder 110. The controller 102 (e.g., the controller's processor) may include instructions and / or software code for executing the pre-trained model 104, encoder 106, neural network 108, and / or decoder 110. In some modified forms, the pre-trained model 104, encoder 106, neural network 108, and / or decoder 110 may further include instructions and / or software code to perform one or more of the following: (1) take one or more inputs; (2) extract a spatial region of the environment from the input; (3) extract or heuristically determine an action to be implemented by the mobile robot 112; (4) embed the extracted spatial region and the action; (5) encode the embedded region and the action; and (6) generate one or more outputs.

[0037] A pre-trained model 104 is configured to produce one or more outputs from one or more inputs. For example, if the input is a first type of input, such as input 230a shown in Figure 2a, the pre-defined model 104 may be configured to produce a first type of output from a first type of input. The first type of output may be a semantic map representation of an environment that includes a visual representation of a path described by the natural language instruction in Figure 2a. For example, the first type of output may be a semantic map representation 230b as shown in Figure 2b. As another example, if the input is a second type of input, such as input 230b shown in Figure 2b, the pre-defined model may be configured to produce a second type of output from a second type of input. The second type of output may be a semantic map representation of an environment that has a natural language instruction describing a visual representation of a path in Figure 2b. For example, the second type of output may be 230a including a semantic map representation and a natural language instruction as shown in Figure 2a. In other words, when the input is, for example, 230a in Figure 2a, the pre-trained model 104 produces an output such as, for example, 230b in Figure 2b. Additionally or alternatively, when the input is, for example, 230b in Figure 2b, the pre-trained model 104 produces an output such as, for example, 230a in Figure 2a.

[0038] Output from controller 102 is transmitted to mobile robot 112. Mobile robot 112 may be configured to navigate within the environment based on one or more outputs received from controller 102. Its outputs include a semantic map representation of the environment. Since the semantic map representation includes annotated representations of spatial entities within the environment, it allows the mobile robot to avoid the task of object detection (i.e., the task of recognizing spatial entities within the environment). Mobile robot 112 may be configured to identify a navigation path based on the semantic map representation by interpreting commands received from controller 102 without needing to identify and / or recognize objects.

[0039] In some forms of modification, the mobile robot 112 may be configured to clarify ambiguity in the output. For example, consider an exemplary output comprising a semantic map representation of an indoor environment and a natural language command "move around the table." As previously mentioned, the semantic map representation may include representations of spatial entities, such as landmarks (e.g., walls, doors, etc.), objects (e.g., furniture, tables, chairs, etc.), combinations thereof, and / or similar. These spatial entities may be annotated with labels (e.g., as seen in Figure 2a). Such annotations may allow the mobile robot 112 to separate the task of visual object detection from the interpretation of navigation commands. This may allow the mobile robot to resolve missing or uncertain information on the semantic map representation that has a person in the loop. For example, consider the natural language command "move around the table." If "table" is a labeled, known object on the semantic map representation, the command interpretation model implemented by the mobile robot 112 may find a navigation path without needing to visually recognize the object. However, if the “table” is not labeled on the semantic map representation, the mobile robot 112 may be configured to ask clarifying questions to resolve the ambiguity.

[0040] Figure 3 is a flowchart 300 showing an example of a method for generating navigation instructions for a mobile robot (for example, structurally and / or functionally similar to the mobile robot 112 in Figure 1). In some modified forms, method 300 may be performed by one or more controllers, for example, controller 102 in Figure 1. In 302, method may include taking one or more inputs. The inputs may be a first type of input, for example, 230a in Figure 2a, and / or a second type of input, for example, 230b in Figure 2b. The first type of input 230a may include a semantic map representation of the environment 232a and natural language instructions 236 describing a path for maneuvering the mobile robot within the environment. The second type of input 230b may include a semantic map representation of the environment 232b, which includes a visual representation 238 of the path for maneuvering the mobile robot within the environment.

[0041] As mentioned above, the semantic map representation (for example, 232a in Figure 2a or 232b in Figure 2b) can also be a top-down view of the environment. For example, semantic map representation M s This could be a view looking down from above scene s. Spatial entity O = {O1,…O N (Assuming there are N-space entities in scene s) is a semantic map representation M s In this context, it is represented as a color-coded rectangular shape. Visual representation of the path P={p1,…p k The} is represented as a series of points connected by white lines. As seen in Figure 2b, the starting point p1 and the ending point p k is the other midpoint p jIt is a different color, where 1 < j < k. In some variations, the semantic map representation can be obtained from a virtual environment. For example, a virtual environment (e.g., Habitat) can include a three-dimensional representation that defines the positions of spatial entities within the environment. The semantic map representation can be extracted from this three-dimensional representation of the environment. Additionally or alternatively, the semantic map representation can be obtained by any suitable mobile robot using Simultaneous Localization and Mapping (SLAM). Additionally or alternatively, the semantic map representation can be created manually by a person.

[0042] If the input is of the first type of input, such as 230a in FIG. 2a, etc., at 304, the method includes executing a pre-trained model (e.g., structurally and / or functionally similar to the pre-trained model 104 of FIG. 1). Executing the pre-trained model when the input is of the first type of input can include performing one or more operations in "A" as further described in FIG. 6 below. At 306, the method includes generating an output of the first type, such as 230b in FIG. 2b, etc. FIG. 4 shows generating an exemplary output of the first type from an exemplary input of the first type. As seen in FIG. 4, the input 230a includes a semantic map representation 232a of the environment and natural language instructions 234 that describe a navigation path for a mobile robot. Performing one or more operations within "A" generates the output 230b. The output 230b includes a semantic map representation 232b of the environment having a visual representation 238 of the navigation path described in the natural language instructions 234.

[0043] Referring back to Figure 3, if the input is a second type of input, such as 230b in Figure 2b, then in 304, the method includes running a pre-trained model. Running a pre-trained model when the input is a second type of input may include performing one or more actions within “B” as further described below in Figure 7. In 308, the method includes generating a second type of output, such as 230a in Figure 2a. Figure 5 shows generating an exemplary second type of output from an exemplary second type of input. As seen in Figure 5, the input 230b includes a semantic map representation of the environment 232b. The semantic map representation 232b includes a visual representation 238 of a navigation path for a mobile robot. Performing one or more actions within “B” generates output 230a. Output 230a includes a semantic map representation of the environment 232a. Output 230a also includes a natural language instruction 234 that describes the navigation path in the visual representation 238.

[0044] Figure 6 is a flowchart 600 of a method illustrating one or more operations within "A" that cause a pre-trained model (e.g., structurally and / or functionally similar to the pre-trained model 104 in Figure 1) to generate a first type output from a first type input. More specifically, given a first type input (e.g., 230a) comprising a semantic map representation (e.g., 232a) and a natural language instruction (e.g., 236), one or more operations in method 600 may be configured to perform the task of generating a visual representation (e.g., 238) of the path described in the natural language instruction (e.g., 236).

[0045] In some modified forms, the method may include encoding at least a portion of a first type of input (e.g., a semantic map representation within the first type of input). Encoding may be performed by an encoder that may be structurally and / or functionally similar to encoder 106 in Figure 1. The encoder may be an image encoder and / or a text encoder. The method may further include resizing a semantic map representation (e.g., 232a) by sampling to the nearest 384x384 and feeding the resized semantic map representation to a vision neural network having a patch size 16. In 612, the method may include extracting spatial regions (e.g., conference room, corridor, etc.) from the semantic map representation via an encoder. For example, the encoder may represent (e.g., encode) the names of the extracted spatial regions as a sequence of strings for each navigation point. In 614, the method may include extracting actions (e.g., left, right, straight, stop, etc.) from natural language instructions that will be performed by a mobile robot. In 616, the method includes embedding the extracted spatial regions via an encoder to produce a region embedding. The region embedding for each point may be obtained by applying an average pooling operation to the word embeddings. In 618, the method includes embedding the extracted actions via an encoder to produce an action embedding. For example, the embedding layer may be applied to individual action values ​​(e.g., left, right, straight, stop, etc.) to obtain action embeddings in the same dimension as the region embeddings. In some modified forms, the spatial regions may be embedded as a first vector and the actions as a second vector. As an example, each navigation point may be associated with an extracted spatial region (e.g., "living room", "corridor", etc.). Such associations may be represented as a sequence of words. Each word in the sequence of words may be embedded. This may produce a vector. The average may be calculated over the embeddings of all words in the sequence to obtain a region embedding.

[0046] In 620, the method may include encoding sequential path information for a mobile robot based on action embeddings and region embeddings via a neural network (e.g., structurally and / or functionally similar to neural network 108 in Figure 1). For example, the method may include adding both region embeddings and action embeddings at each point and using a three-layer LSTM to embed sequential path information along the navigation path. In 622, the method includes generating a semantic map representation of an environment, including a visual representation of a path described in natural language instructions at the input, via a decoder (e.g., structurally and / or functionally similar to decoder 110 in Figure 1).

[0047] Figure 7 is a flowchart 700 of a method illustrating one or more operations within “B” that cause a pre-trained model (e.g., structurally and / or functionally similar to the pre-trained model 104 in Figure 1) to generate a second type output from a second type input. More specifically, given a second type input (e.g., 230b) comprising a semantic map representation (e.g., 232b) including a visual representation of a path (e.g., 238), one or more operations in method 700 may be configured to perform the task of generating a natural language instruction (e.g., 236) that describes the path.

[0048] In some modified forms, the method may include encoding at least a portion of the second type of input. Encoding may be performed by an encoder that may be structurally and / or functionally similar to encoder 106 in Figure 1. The encoder may be an image encoder and / or a text encoder. Method 700 may further include resizing a semantic map representation (e.g., 232b) by sampling to the nearest 384x384 and feeding the resized semantic map representation to a vision neural network having a patch size of 16. In 712, the method may include extracting spatial regions (e.g., conference room, corridor, etc.) from the semantic map representation. For example, the encoder may represent the extracted spatial regions as a sequence of strings for each navigation point. For example, the encoder may encode the name of the spatial region for each navigation point, e.g., "corridor" or "living room," etc. In 714, the method may include heuristically determining from a visual representation of the navigation path what action (e.g., left, right, straight, stop, etc.) will be performed by the mobile robot. For example, the method may include identifying the shape of a navigation path from a visual representation (e.g., 238). The method may also include heuristically determining an action based on an analysis of the shape of each navigation point on the navigation path.

[0049] In 716, the method includes embedding the extracted spatial regions via an encoder to produce a region embedding. The region embedding for each point may be obtained by applying an average pooling operation to the word embedding. For example, each navigation point may be associated with an extracted spatial region as a sequence of words. Each word in the sequence of words may be embedded to produce a vector. The average may be calculated over the entire sequence of words to produce a region embedding. In 718, the method includes embedding heuristically determined actions via an encoder to produce an action embedding. For example, the embedding layer may be applied to individual action values ​​(e.g., left, right, straight, stop, etc.) to obtain action embeddings in the same dimension as the region embedding. In some modified forms, the spatial regions may be embedded as a first vector and the actions may be embedded as a second vector.

[0050] In 720, the method may include encoding sequential path information for a mobile robot based on action embeddings and region embeddings via a neural network (for example, structurally and / or functionally similar to neural network 108 in Figure 1). For example, the method may include adding both region embeddings and action embeddings at each point and using a three-layer LSTM to embed the sequential path information along the navigation path. In 722, the method includes generating natural language instructions describing the navigation path for the mobile robot via a decoder (for example, structurally and / or functionally similar to decoder 110 in Figure 1).

[0051] In some modified forms, method 700 can be constructed as an image caption task in which the input image is a semantic map representation (e.g., 232b) having a visual representation of a path (e.g., 238), and the output text is a natural language instruction (e.g., 236) describing the path.

[0052] Expanding the input data Panoramic image In some forms of modification, the input to a controller (e.g., controller 102 in Figure 1) can be augmented by providing one or more panoramic images to a pre-trained model (e.g., pre-trained model 104 in Figure 1). For example, in some scenarios, panoramic images can provide additional visual information that may not be shown in the semantic map representation. Specifically, some natural language instructions may include visual properties of spatial entities (e.g., color, shape, etc.). Some of these visual properties cannot be annotated and / or represented in the semantic map representation. In such scenarios, the input can be augmented with panoramic images. The pre-trained model can process these panoramic images to identify the visual properties in the natural language instructions. The pre-trained model can then be configured to learn these identified visual properties and apply them to subsequent input data.

[0053] To extract properties / features from a panoramic image, an image encoder may be initialized. For example, a pre-trained model described herein may include an image encoder that receives image data as input and an encoder 106 that receives text data. In some modifications, this image encoder may differ from encoder 106, which may be a text encoder. Since panoramas are photorealistic images, as in the pre-training data, the parameters of the image encoder may be frozen during training on panoramic images while tuning the pre-trained model. The extracted features may be embedded to produce visual embeddings. In some modifications, in addition to the image encoder, an additional multilayer perceptron with two linear layers may be applied to increase the flexibility of the visual embeddings. Panoramas may be treated as individual frames, and the average of the visual embeddings may be used to represent panoramic information of a navigation path. This panoramic information may be added to sequential path information encoded by a neural network (e.g., structurally and / or functionally similar to neural network 108 in Figure 1). Panorama information, along with sequential route information, may be provided as output to a decoder (for example, structurally and / or functionally similar to decoder 110 in Figure 1) to output navigation commands (for example, natural language commands or semantic map representations having a visual representation of the route).

[0054] FIG. 8 is an illustration of augmenting the input to a pre-trained model with a panoramic image. In this example, the input includes a semantic representation of the environment (not shown in FIG. 8) with a natural language instruction 836. As seen in FIG. 8, the natural language instruction includes visual properties of spatial entities such as "red carpet" and "white and gold table". Thus, the panoramic image 842 of the environment can be provided as an input to the pre-trained model, thereby augmenting the input to the pre-trained model. The spatial region 844 is extracted from the semantic map representation, and the action 846 is extracted from the natural language instruction 836. The extracted region 844 and action 846 are embedded to produce region embeddings and action embeddings. The features extracted from the panoramic image 842 are embedded to produce visual embeddings. Sequential path information based on the region embeddings and action embeddings and panoramic information based on the visual embeddings are provided to the decoder. The decoder outputs a semantic map representation 832 of the environment with a visual representation 838 of the navigation path superimposed on the semantic map representation.

[0055] Multi-modal Alignment with Contrastive Loss In some variations, multi-modal alignment with contrastive loss can be applied to augment a pre-trained model (e.g., the pre-trained model 104 of FIG. 1). Contrastive learning is an effective method used in self-supervised learning of visual representation learning and multi-modal learning. The effectiveness of introducing contrastive training for the navigation instruction generation task can be investigated as an auxiliary loss. In some variations, a positive example P + (C gt ,I gt ) can be defined as a pair of combined input embeddings (e.g., region embeddings, action embeddings, embeddings of images of semantic map representations) and instruction embeddings (e.g., embeddings of words in natural language instructions). A negative example P - (C gt,I rnd A can consist of a pair of input embeddings and randomly sampled instruction embeddings. For example, a positive example is input embedding C gt and the corresponding matching instruction embedding I gt It may include ground truth data with pairs of . A negative example is input embedding C gt Randomly generated instruction embedding I rnd This may include: Multimodal input matrix E input and text instruction matrix E text This is the predicted compatible matrix C between the input and the label. pred To obtain, it can be multiplied, then C pred The cross-entropy loss related to is the ground truth correspondence C gt It can be calculated as follows.

[0056] Prompt-based extension and grounding In some modified forms, a decoder (e.g., structurally and / or functionally similar to decoder 110 in Figure 1) may be used in conjunction with a Large Language Model (LLM) to generate navigation instructions. For example, a decoder may also take an LLM prompt as input. An LLM prompt may be generated from a template that describes spatial entities and spatial objects in an environment, such as "start from the dark yellow point near the living room sofa cushion." A pre-trained model (e.g., pre-trained model 104) may be tuned during inference (e.g., while the decoder is generating navigation instructions) by prompting and supplying prompt templates to the decoder. The LLM may provide additional textual descriptions to extend the process of generating navigation instructions. Prompting can be useful in two ways. Firstly, it can help with visual language grounding by describing neighboring landmarks and regions in the prompt template. In other words, prompting can provide a correct / accurate description of spatial entities. Secondly, during inference time, the generated instructions are autoregressively conditional on the prompt template, resulting in more controllable generation of navigation instructions.

[0057] Figure 9 shows an exemplary architecture of a pre-trained model (e.g., pre-trained model 104) configured to receive the augmented inputs described herein and thereby generate navigation instructions. The semantic map encoder 906a may be configured to encode a semantic map representation of the environment (e.g., by resizing the semantic map representation and feeding it to a vision neural network). The region encoder 906c may be configured to extract spatial regions from the semantic map representation and embed the extracted regions. The action embedding 906d may be configured to extract actions from natural language instructions and / or heuristically determine actions from visual representations of paths in the semantic map. Additionally, the action embedding 906d may embed the extracted and / or determined actions. In some modified forms, the panoramic encoder 906b may extract visual features from a panoramic image. The weights of the panoramic encoder 906b may be fixed.

[0058] In several variations, the semantic map encoder 906a, panoramic encoder 906b, region encoder 906c, and action embedding 906d can collectively perform the operation of the encoder 106 in Figure 1. In other words, the encoder 106 in Figure 1 may include the semantic map encoder 906a, the panoramic encoder 906b, the region encoder 906c, and the action embedding 906d.

[0059] Neural network 908a can encode action embeddings within sequential path information, and neural network 908b can encode region embeddings within sequential path information. Neural networks 908a and 908b can share weights. Neural networks 908a and 908b can collectively perform the operation of neural network 108 in Figure 1. In other words, neural network 108 in Figure 1 may include neural networks 908a and 908b. Average pooling 909 can treat the panorama as individual frames and apply the average value of the visual embeddings from panorama encoder 906b. Based on this, average pooling 909 can represent the panoramic information of the navigation path.

[0060] Panorama information from mean pooling 909 and sequential path information from neural networks 908a and 908b can be provided to decoder 910 (for example, structurally and / or functionally similar to decoder 110 in Figure 1). In some modifications, LLM can be used in conjunction with decoder 910 to extend the output. In some modifications, multimodal alignment with contrasting loss can be applied to decoder 910 to extend the output. Decoder 910 can generate navigation instructions (for example, natural language instructions or visual representations of paths superimposed on semantic map representations).

[0061] Training data As mentioned above, training data may include pairs of image data and their corresponding natural language descriptions. In some modified forms, semantic map representations, spatial domains, and mobile robot actions may be extracted from a habitat simulation environment (e.g., https: / / aihabitat.org / ). As mentioned above, in some modified forms, each type of spatial entity on the semantic map representation may be represented by a unique color. Visual representations of navigation paths and human annotations from R2R datasets (e.g., "Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel, 2018, Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)") may be used to obtain training data. Panoramic images within RGB can be obtained at individual navigation points from the Matterport3D simulator (for example, "Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang, 2017, Matterport3d: Learning from rgb-d data in indoor environments, arXiv preprint arXiv:1709.06158").

[0062] Exemplary training and exemplary results As an example, the experiment was conducted to generate instructions from a semantic map representation using a pre-trained model (e.g., pre-trained model 104 in Figure 1) that encodes a semantic map representation, an action (derived from a visual representation of the path shape), spatial domain information (e.g., room type), and a panoramic image. The combined input was encoded into a vector (e.g., via encoder 106 and neural network 108 in Figure 1) and used by a decoder (e.g., decoder 110 in Figure 1) initialized with pre-trained values ​​to generate natural language instructions.

[0063] As can be seen in Figure 10, using the unsupervised evaluation metric SPICE (i.e., Peter Anderson, Basura Fernando, Mark John-son, and Stephen Gould, 2016, Spice: Semantic propositional image caption evaluation, in ECCV), it is shown that the model performance is equivalent to the terminal generation method. The advantages of the techniques described herein include portability to new domains and the existence of intermediate representations (e.g., semantic map representations) that facilitate the introduction of new, unknown objects by enabling user-mobile robot interactions, which can be argued based on semantic map representations.

[0064] In another example, a pre-trained model (e.g., pre-trained model 104 in Figure 1) is trained using the train splitting of the R2R dataset discussed above. The pre-trained model is enabled for both visible and invisible sets. The pre-trained model was used for the experiment. The baseline was set up with different combinations of inputs: 1) semantic map representation viewed from above, 2) semantic map representation + region and action, and 3) semantic map representation + panoramic image. In addition, contrast loss and prompting were also experimented with.

[0065] The performance of nine system variants was evaluated based on five selected system variants with automated metrics and human target scores. SPICE (i.e., Peter Anderson, Basura Fernando, Mark John-son, and Stephen Gould, 2016, Spice: Semantic propositional image caption evaluation, at ECCV) was used for automated evaluation metrics, and three followed Zhao et al. (2021).

[0066] Table 1 below shows the results of automated (SPICE) and human evaluation with different modality inputs in visible and invisible environments. ** and * The baselines (p<0.01) and (p<0.05) show statistically significant differences.

[0067] [Table 1]

[0068] As expected, the model performs better in visible settings than in invisible settings. Using area and action information with prompts in both settings can improve the model's performance. Adding panoramic images can further enhance performance in visible environments.

[0069] Therefore, incorporating more information across different modalities can improve the performance of generating navigation instructions. Despite using a single image of the summarized semantic map representation as visual input, many described input transformation forms that do not use panoramic images are found in Speaker Fol. 17.0 / 18.7 (Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell, 2018, Speaker follower models for vision-and-language navigation, Advances in Neural Information Processing Systems, 31) and EnvDrop 18.1 / 20.2 (Hao Tan, Licheng Yu, and Mohit Bansal, 2019, Learning to navigate unseen environments: Back translation with environmental dropout, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human This method is equivalent to an LSTM-based interterminal approach that uses only panoramic images with SPICE scores (from Language Technologies, Volume 1 neural network).

Claims

1. The processor acquires either the first or second input, In response to receiving the first input, the pre-trained model is executed via the processor to generate a first output, In response to receiving the second input, the pre-trained model is executed via the processor to generate a second output. Equipped with, The first input comprises a first semantic map representation of the environment and at least one first natural language instruction describing a first path for maneuvering the robot within the environment. The first output comprises a first output semantic map representation of the environment, the first output semantic map representation includes a first navigation command which is a visual representation of the first path translated into an action of the robot. The second input comprises a second semantic map representation of the environment, the second semantic map representation comprising a second navigation command which is a visual representation of a second path translated into an action of the robot. A computer implementation method wherein the second output includes at least one second natural language instruction describing the second semantic map representation and the second path.

2. The computer implementation method according to claim 1, wherein the first semantic map representation is a top-down view of the environment and includes a plurality of spatial landmarks in the environment as a plurality of color-coded bounding boxes on the first semantic map representation.

3. The second semantic map representation is a view looking down from above the environment, Multiple spatial landmarks in the environment as multiple color-coded bounding boxes on the second semantic map representation, The visual representation of the second path as a series of points connected by lines on the second semantic map representation, The computer implementation method according to claim 1, wherein the series of points includes a start point and an end point.

4. The computer implementation method according to claim 1, wherein at least one of the first semantic map representation and the second semantic map representation is created manually.

5. The computer implementation method according to claim 1, wherein at least one of the first semantic map representation and the second semantic map representation is automatically generated.

6. The computer implementation method according to claim 1, wherein the pre-trained model is a multimodal text generation model.

7. In response to receiving the first input, running the pre-trained model is: Extracting one or more spatial regions of the environment from the first semantic map representation via an encoder, Extracting, via the encoder, one or more actions to be implemented by the robot from at least one of the at least one first natural language instruction and at least one second natural language instruction, The first vector is to embed one or more spatial regions using the encoder, The one or more actions are embedded as a second vector using the encoder. The computer implementation method according to claim 1, further comprising:

8. Running the aforementioned pre-trained model is Encoding sequential path information for the robot to operate the robot within the environment, based on the first vector and the second vector, via a neural network. The computer implementation method according to claim 7, further comprising:

9. The computer implementation method according to claim 8, further comprising running the pre-trained model to run a decoder to generate the first output semantic map representation using the sequential path information.

10. The computer implementation method according to claim 1, further comprising running the pre-trained model in response to receiving the second input, heuristically determining one or more actions to be implemented by the robot, based at least in part on the visual representation of the second path for the robot.

11. In response to receiving the second input, running the pre-trained model is: Extracting one or more spatial regions of the environment from the second semantic map representation via an encoder, The third vector is to embed one or more of the aforementioned spatial regions using the encoder, The fourth vector is to embed one or more of the aforementioned actions using the encoder. The computer implementation method according to claim 10, further comprising:

12. Running the aforementioned pre-trained model is Encoding sequential path information for the robot to be operated in a second environment via a neural network based on the third vector and the fourth vector. The computer implementation method according to claim 11, further comprising:

13. The computer implementation method according to claim 12, further comprising running the pre-trained model to run a decoder to generate the second semantic map representation and the at least one second natural language instruction describing the second path using the sequential path information.

14. To acquire a first panoramic image of the aforementioned environment, In response to receiving the first input, the pre-trained model is executed to generate the first output based at least partially on the first panoramic image, In response to receiving the second input, the pre-trained model is executed to generate the second output based at least partially on the first panoramic image. The computer implementation method according to claim 1, further comprising:

15. The aforementioned pre-trained model is trained using multiple pairs of training data. Furthermore, Each of the aforementioned pairs is Image data and, The corresponding natural language description of the aforementioned image data and The computer implementation method according to claim 1, including the method described in claim 1.

16. The processor acquires the second input, Extracting one or more spatial regions of the environment from the second semantic map representation, The robot is to be operated within the environment by determining one or more actions to be implemented by the robot, at least in part, based on the visual representation of the second path. The embedding of one or more spatial regions and the creation of a region embedding, Embedding one or more of the aforementioned actions and creating an action embedding, The sequential path information for the robot is encoded using the region embedding and the action embedding. The second natural language instruction describing the second path is generated using the sequential path information, and the robot is operated within the environment. The computer implementation method according to claim 1, comprising:

17. The processor acquires the first input, Extracting one or more spatial regions of the environment from the first semantic map representation, Extracting one or more actions to be implemented by the robot from the first natural language command, and controlling the robot within the environment, The embedding of one or more spatial regions and the creation of a region embedding, Embedding one or more of the aforementioned actions and creating an action embedding, The sequential path information for the robot is encoded using the region embedding and the action embedding. Using the sequential path information, generate the first output semantic map representation of the environment. The computer implementation method according to claim 1, comprising:

18. The computer implementation method according to claim 1, wherein the second output includes a second output semantic map representation of the environment, and the second output semantic map representation includes a second navigation instruction.

Images