TRAINING OF MOBILE ROBOT PASSABILITY DETECTION WITH SIMULATED DATA

Abstract

A system and method for training a machine learning model are disclosed, designed to determine the traverseability of a real-world environment by a mobile robot based on an image of that environment. The method advantageously generates high-quality synthetic training data for training a traverse detection model to distinguish between traversable and impassable areas in captured images of a real-world environment. The synthetic training data is generated by simulating a virtual robot in a virtual environment. Once the traverse detection model is trained, it can be applied to the mobile robot to predict the traverseability of a real-world environment.

Description

[0001] The devices and methods disclosed in this document relate to mobile robots and, in particular, training models for detecting passability using simulated data.

[0002] Unless otherwise stated herein, the materials described in this section are not recognized as prior art by their inclusion in this section.

[0003] Mobile robots take many forms and have many functions, such as cleaning robots, autonomous vehicles, unmanned aerial vehicles (UAVs), delivery robots, telepresence robots, and so on. A key task for a mobile robot is to identify areas of its environment that can be safely traversed or passed through. One way to detect passable space is for the mobile robot to move at low speed and detect obstacles by bumping into them. However, there are some hazards that the mobile robot should not come into contact with, such as animal feces. Another way to detect passable space is to use LiDAR sensors to detect the positions of obstacles from a distance. However, using LiDAR does not allow the mobile robot to detect all types of hazards or to distinguish between different types of hazards, such as...to distinguish between a wall and a puddle.

[0004] To address some of these challenges, previous work has suggested that a mobile robot could integrate an image-based machine learning model that receives images of its environment and predicts the traversability of areas captured in the image. However, training modern machine learning models requires a large amount of correctly labeled training data. Labeling real-world images can be tedious and error-prone. Since manually labeling images is costly, previous work has proposed automatically labeling images based on experience. Specifically, as soon as an obstacle or bumper sensor on the mobile robot detects a collision, images immediately preceding the collision are labeled as impassable.However, this leads to the problem of assigning images to a future collision event, which can be an error-prone process due to miscalculations of the robot's odometry or external events, such as pets that change the robot's trajectory.

[0005] Therefore, a method for training a machine learning model is needed to predict the drivability or passability of areas of an environment captured in an image, which does not require large amounts of manually labeled training data. SUMMARY

[0006] A method for training a machine learning model is disclosed, which is configured to determine the traverseability of a real-world environment by a mobile robot based on an image of the real-world environment. The method includes generating a virtual environment using a variety of three-dimensional models. The method further includes generating a synthetic image of the virtual environment. The method further includes determining a labeling mask for the synthetic image based on a simulation of a virtual robot in the virtual environment, wherein the labeling mask specifies the traverseability of respective areas of the virtual environment that are captured or recorded in the synthetic image. The method further includes training the machine learning model based on the synthetic image and the labeling mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above-mentioned designs and other features of the system and the procedures are explained in the following description in conjunction with the attached drawings. Fig. 1 summarizes components and processes of a mobile robot system. Fig. Figure 2A shows an example embodiment of the mobile robot. Fig. Figure 2B shows an example embodiment of the computer system. Fig. Figure 3 shows a flowchart for a procedure for training a machine learning model that is set up to determine the traversability of a real environment by a mobile robot based on an image of the real environment. Fig. Figure 4 shows different types of impassable robot configurations. Fig. Figure 5 shows several example configurations of a virtual robot. Fig. Figure 6 shows a flowchart for a procedure to determine a passability label for a specific pixel in a synthetic image. Fig. Section 7 summarizes a ray tracing process using a pinhole camera model. DETAILED DESCRIPTION

[0008] For a better understanding of the principles of the disclosure, reference is now made to the embodiments illustrated in the drawings and described in the following written patent specification. It is understood that this is not intended to limit the scope of the disclosure. Furthermore, it is understood that the present disclosure includes all changes and modifications of the illustrated embodiments as well as further applications of the principles of the disclosure, such as would normally occur to a person skilled in the art in the field to which this disclosure belongs. Overview

[0009] With reference to Fig. Section 1 summarizes the components and processes of a mobile robot system 10. The mobile robot system 10 comprises at least one mobile robot 120 configured to perform a task in an environment. Advantageously, the mobile robot system 10 is configured to utilize a passability detection model 20, which is configured to determine the passability of a real-world environment based on images of that environment. The passability detection model 20 is any model from the field of machine learning, including neural networks, support vector machines, Gaussian mixture models, etc.

[0010] In general, the mobile robot 120 comprises a controller 122 configured to actuate or control one or more sensors 126 and one or more actuators 128 to navigate autonomously in an environment to perform a task. In some embodiments, the mobile robot 120 may include a cleaning robot, such as a vacuuming robot or a mopping robot, configured to navigate in the environment to clean a floor area. In other embodiments, the mobile robot 120 may include an autonomous road vehicle, an unmanned aerial vehicle (UAV), a delivery robot, or a telepresence robot. However, it should be understood by a person skilled in the art that the systems and methods described herein may be applicable to a wide variety of mobile robots that navigate autonomously in an environment to perform a task.

[0011] While the mobile robot 120 is operating to perform tasks in its environment, the controller 122 actuates and controls the sensors 126 to capture images of the environment and other sensor data. This data is used to determine the positions of walls, objects, or other obstacles in the environment for mapping, navigation, motion planning, and trajectory optimization tasks. The mobile robot 120 advantageously utilizes the passability detection model 20 to process the captured images and determine which parts of the environment captured in the image are passable. Passability detection is typically performed within the mobile robot by the controller 122, but it could also be performed on a remote server, with input images and output results transmitted via a network connection.Based on passability information as well as mapping data or other sensor information, the controller 122 actuates the actuators 128 to navigate in the environment and perform tasks in the environment.

[0012] With continued reference to Fig. The mobile robot system 10 further comprises a computer system 150, which could be physically located in the robot (e.g., the controller 122), near the robot (e.g., a local PC in the same building), or remotely (e.g., in the cloud). The computer system 150 advantageously comprises program instructions corresponding to a simulator 40, which are executed to generate synthetic training data for training the passability detection model 20. Furthermore, the computer system 150 comprises program instructions corresponding to a trainer 60, which are executed to train the passability detection model 20 to distinguish between passable and impassable areas in images taken from a real-world environment.

[0013] For the purpose of generating synthetic images, the simulator 40 of the computer system 140 uses a variety of models. The models used by the simulator 40 include 3D models 42 (e.g., triangular meshes) of virtual objects that can be combined to generate virtual scenes 44. The models used by the simulator 40 also include a robot model 46 that simulates not only the spatial size and shape of the mobile robot 120, but also the mechanics by which the mobile robot 120 moves through an environment and performs tasks. Finally, the models used by the simulator 40 include sensor models 48, which include at least one camera model (e.g., a pinhole camera) that simulates how the sensors 126 of the mobile robot measure sensor data.

[0014] The computer system 150 executes a scene generator 50 of the simulator 40 to randomly or procedurally generate unique virtual environments, referred to herein as virtual scenes 44. For example, a household scene could be generated by randomly selecting a room layout and randomly arranging furniture, lamps, and other household items in the various rooms. The generated virtual scenes 44 are stored in memory. The computer system 150 places a virtual robot, based on the robot model 46, at various passable positions or locations within the virtual scene 44 and, using the camera model, generates synthetic images captured from the perspective of the virtual robot.

[0015] Finally, the computer system 150 executes an image labeler 50 of the simulator 40 to determine ground-truth traversability labels for the synthetic images, e.g., in the form of a labeling mask. Specifically, the computer system 150 automatically calculates which areas in a synthetic image correspond to traversable or impassable areas. As used here, "impassable" areas in an environment, broadly defined, include obstacles (e.g., low furniture) that essentially prevent passage by the mobile robot, as well as hazards (e.g., liquids, animal excrement) and unstable terrain (e.g., stairs, sand) that do not necessarily prevent passage by the mobile robot but should nevertheless be avoided. The image labeler 50 includes collision checking for obstacle localization and forward dynamics (e.g.,(a discrete-time solver of Newton's equations of motion) for stability testing. Since the computer system 150 has complete knowledge of the simulated environment, the synthetic images are labeled "perfectly," provided the simulation is realistic. Building on these innovations, generating training data for passability detection becomes practical.

[0016] Once a sufficient corpus of training data 62 has been generated (i.e., synthetic images with ground-truth passability labels), the computer system 150 executes the trainer 60 to train the passability detection model 20 to predict passable and impassable areas of an environment based on images of the environment. Specifically, the computer system 150 trains the passability detection model 20 on the generated training data 62 using an optimizer 64. The optimizer 64 is implemented with any machine learning algorithm for fitting a model to data, including gradient descent, stochastic gradient descent, and Newton's method.In some embodiments, the synthetic training data 62 can be augmented with real training data, which includes real images that have been labelled manually or using additional sensor systems, such as LIDAR or bumper sensors.

[0017] The passability detection model 20 is then used by the mobile robot to detect passable areas for safe navigation. Specifically, when the mobile robot 120 is operated to perform tasks in its environment, the controller 122 executes the trained passability detection model 20 to generate passability labels based on images of the real environment. The controller 122 generates operating commands to actuate the actuators 128, at least partially, based on these passability labels.

[0018] In some embodiments, the computer system 150 also executes another trainer (not shown), the trainer 60, to train the controller 122 to generate actuator commands using the simulator 40 and reinforcement learning algorithms. Specifically, the input state to the controller 122 includes an image labeled with a passability, and the output action includes commands to move the robot. The computer system 150 trains the controller 122 to generate the commands using a simulated robot moving in a simulated environment, and the controller 122 learns to maximize the expected future reward. Mobile robot

[0019] Fig. Figure 2A shows an embodiment of the mobile robot 120. In the illustrated embodiment, the mobile robot 120 includes, for example, the controller 122, the memory 124, one or more sensors 126, one or more actuators 128, and at least one network communication module 130. It is understood that the illustrated embodiment of the mobile robot 120 is only an embodiment and is merely representative of various types, arrangements, or configurations of mobile robots that navigate autonomously in an environment to perform a task.

[0020] The controller 122 is configured to execute instructions to operate the mobile robot 120 in order to enable the features, functions, characteristics, and / or the like described herein. For this purpose, the controller 122 is functionally connected to the memory 124, the one or more sensors 126, and the one or more actuators 128. The controller 122 generally has one or more processors that can operate in parallel or otherwise coordinate with one another. A person skilled in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information.Therefore, the control 122 can include a system with a central processing unit, graphics processing units, multiple processing units, a dedicated circuit for achieving functionality, programmable logic, or other processing systems.

[0021] Memory 124 is configured to store data and program instructions which, when executed by the controller 122, enable the mobile robot 120 to perform various operations described herein. Memory 124 can be any type of device capable of storing information accessible to the controller 122, such as a memory card, ROM, RAM, hard disks, floppy disks, flash memory, or any other computer-readable medium that serves as a data storage device, as known to the average person skilled in the art. The controller 122 is configured to execute program instructions of an operating sequence 132 stored in memory 124 to navigate the environment to perform a task, such as cleaning a floor area in the environment.Operational Sequence 132 uses the passability detection model 20 to assist in navigating the environment to perform the task as mentioned above.

[0022] The one or more sensors 126 can comprise a variety of different sensors, such as cameras, structured light sensors, LiDAR sensors, radar sensors, SONAR sensors, and the like. The sensors 126 include at least one or more cameras configured to capture a variety of images of the environment as the mobile robot 120 navigates through it. The camera(s) generate image frames of the environment, each of which has a two-dimensional arrangement of pixels. Each pixel has corresponding photometric information (color, intensity, and / or brightness). In some embodiments, the camera(s) is / are configured to generate RGB-D images in which each pixel has corresponding photometric and geometric information (depth and / or distance).In such embodiments, the camera(s) can take the form of an RGB camera operating in conjunction with a LiDAR or IR sensor, in particular a LiDAR or IR camera configured to provide both photometric and geometric information. The LiDAR or IR camera can be separate from the RGB camera or directly integrated into it. Alternatively or additionally, the camera can have two RGB cameras configured to capture stereoscopic images from which depth and / or distance information can be derived. Based on RGB-D images captured while the mobile robot 120 navigates its environment, the mobile robot 120 can implement visual and / or visual-inertial odometry methods such as SLAM (Simultaneous Localization and Mapping) techniques.

[0023] In some embodiments, the sensors 126 comprise a light sensor (e.g., LiDAR or another time-of-flight or structured light sensor) configured to emit measurement light (e.g., laser) and receive the measurement light after it has been reflected in the environment. In time-of-flight-based embodiments, the controller 122 is configured to calculate time-of-flight and / or return times for the measurement light. In structured light-based embodiments, the controller 122 applies an algorithm to extract a 3D profile of surfaces onto which the structured light is projected (e.g., based on a stripe pattern generated on a surface).

[0024] In some embodiments, the sensors 126 comprise sensors configured to measure one or more accelerations, rotational speeds, and / or orientations of the mobile robot 120. In one embodiment, the sensors 126 comprise one or more accelerometers configured to measure linear accelerations of the mobile robot 120 along one or more axes (e.g., roll, pitch, and yaw axes), or one or more gyroscopes configured to measure rotational speeds of the mobile robot 120 along one or more axes (e.g., roll, pitch, and yaw axes), and / or an inertial measurement unit configured to measure all of the above quantities.

[0025] The one or more actuators 128 comprise at least motors of a locomotion system that, for example, drive a set of wheels to cause the mobile robot 120 to move through its environment to perform the task. The actuators 128 may similarly include brakes or propellers to assist locomotion. Furthermore, the actuators 128 comprise a variety of motors, joints, and the like that are actuated to perform tasks in the environment. In some embodiments, the actuators 128 include a vacuum suction system configured to vacuum a floor surface while the mobile robot 120 navigates its environment. Mobile robots 120 that perform other tasks in the environment may, of course, comprise different types of actuators 128 suitable for other tasks.

[0026] The Network Communications Module 130 may include one or more transceivers, modems, processors, memory, oscillators, antennas, or other hardware typically contained in a communications module to enable communication with various other devices, including at least the Computer System 150. In particular, the Network Communications Module 130 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and / or a Wi-Fi router (not shown). Furthermore, the Network Communications Module 130 may include a Bluetooth® module (not shown). Finally, the Network Communications Module 130 may include one or more cellular modems configured to communicate with or connect to wireless telephone networks.

[0027] The mobile robot 120 may also include a suitable battery or other power source (not shown) configured to power the various components within the mobile robot 120. In one embodiment, the battery of the mobile robot 120 is a rechargeable battery configured to be charged when the mobile robot 120 is connected to a base station configured for use with the mobile robot 120. computer system

[0028] Fig. Figure 2B shows an embodiment of the computer system 150. The computer system 150 comprises one or more computers 152 and one or more storage devices 162 (e.g., databases). Each computer 152 includes, for example, a processor 154, memory 156, a user interface 158, and a network communication module 160. It is understood that the illustrated embodiment of the computers 152 is only one embodiment of a computer 152 and is merely representative of various types or arrangements / configurations of a personal computer, server, or other data processing system that is operational in the manner described herein.

[0029] The processor 154 is configured to execute instructions to operate the computer 152 in order to enable the features, functionalities, characteristics, and / or the like described herein. For this purpose, the processor 154 is functionally connected to the memory 156, the user interface 158, and the network communication module 160. The processor 154 generally comprises one or more processors that can operate in parallel or otherwise in coordination with one another. An average person will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Accordingly, the processor 154 may comprise a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuitry for achieving functionality, programmable logic, or other processing systems.

[0030] The storage device 162 is configured to store the training data 62 used to train the passability detection model 20. The storage device 162 can be any type of long-term non-volatile storage device capable of storing information accessible to the processor 154, such as hard disks, solid-state drives, or any other computer-readable storage medium known to the average person. Likewise, the memory 156 is configured to store program instructions which, when executed by the processor 154, enable the computer 152 to perform various operations described herein, including the simulator 40 for generating synthetic training data and the trainer 60 for training the passability detection model 20.The memory 156 can be any type of device or combination of devices that can store information which the processor 154 can access, such as memory cards, ROM, RAM, hard disks, floppy disks, flash memory or various other computer-readable media known to the average professional.

[0031] The computer 152 can be operated locally or remotely by an administrator. To enable local operation, the computer 152 can include the user interface 158. In at least one embodiment, the user interface 158 can suitably include an LCD display screen or the like, a mouse or other pointing device, a keyboard or other keypad, speakers, and a microphone, as is known to a person skilled in the art. Alternatively, in some embodiments, an administrator can operate the computer 152 remotely from another computing device that communicates with it via the network communication module 160 and has an analog user interface.

[0032] The network communication module 160 provides an interface that allows communication with any of the various devices, including at least the mobile robot 120. In particular, the network communication module 160 can include a LAN (Local Area Network) port that allows communication with any of the various local computers located in the same or a nearby facility. Generally, the computer 152 communicates with remote computers over the Internet through a separate modem and / or router connection of the local network. Alternatively, the network communication module 160 can also include a WAN (Wide Area Network) port that allows communication over the Internet. In one embodiment, the network communication module 160 is equipped with a Wi-Fi transceiver or other wireless communication device.Therefore, it follows that communication with the computer 152 can take place via wired or wireless communication. Communication can be carried out using any of the various known communication protocols. Method for training and deploying a passability detection model for a mobile robot

[0033] The following describes various methods and processes for training and deploying a passability detection model for use by a mobile robot. In these descriptions, statements that a method, processor, and / or system performs a task or function refer to a controller or processor (e.g., processor 154 of computer 152 or controller 122 of mobile robot 120) that executes programmed instructions stored in non-volatile, computer-readable memory media (e.g., memory 156 of computer 152 or memory 124 of mobile robot 120) that are functionally connected to the controller or processor to manipulate data or actuate one or more components in computer 152 or mobile robot 120 to perform the task or function.Furthermore, the steps of the procedures can be carried out in any possible chronological order, regardless of the order shown in the figures or the order in which the steps are described.

[0034] Fig. Figure 3 shows a flowchart for Procedure 200 for training a machine learning model configured to determine the traversability of a real-world environment by a mobile robot based on an image of that environment. Procedure 200 advantageously generates high-quality synthetic training data for training a machine learning model to distinguish between traversable and impassable areas in images captured of a real-world environment. The synthetic training data is generated by simulating a virtual robot in a virtual environment.

[0035] Method 200 begins by generating a three-dimensional virtual scene (Block 210). In particular, the processor 152 of the computer system 150 generates at least one unique virtual environment using a variety of three-dimensional models of virtual objects and other environment geometry, such as floors and walls. In one embodiment, the processor 152 generates the geometry of the virtual environment randomly or procedurally using primitive shapes or a dataset of 3D models and / or 3D polygon meshes (i.e., the 3D models 42), such as the Zillow Indoor Dataset or ShapeNet. In another embodiment, the processor 152 generates an environment layout randomly or procedurally to create a virtual environment.In one embodiment, the processor 152 randomly or procedurally selects virtual objects from a plurality of virtual object models and randomly or procedurally determines the positions of the virtual objects within the virtual environment. In one example, the processor 152 creates a household scene by randomly selecting a room layout from a plurality of predefined room layouts and randomly or procedurally arranging furniture, lamps, and other household items in the various rooms. In another example, the processor 152 creates a city scene by randomly or procedurally generating a street layout and randomly or procedurally arranging buildings, pedestrians, and vehicles in the virtual scene. The processor 152 stores the one or more unique virtual environments in the memory 156 or in the storage devices 162.

[0036] In at least some embodiments, the virtual objects within the generated virtual environments are labeled or marked with relevant semantic information. In particular, some virtual objects can represent hazards in the environment, such as animal droppings or a puddle of water. In such cases, these virtual objects are labeled as hazards. As explained in more detail below, at least in some embodiments, hazards can be treated differently than other obstacles in the virtual environment.

[0037] Procedure 200 continues with the generation of a synthetic image of the virtual environment (Block 220). In particular, Processor 152 generates a synthetic image of the virtual environment. To this end, Processor 152 first defines a configuration of a virtual robot within the virtual environment. It is understood, however, that only certain configurations are valid for the virtual robot within the virtual environment. In particular, Processor 152 must verify that the defined configuration is passable by the virtual robot, e.g., using Robot Model 46. Once a passable configuration for the virtual robot is defined, Processor 152 renders a synthetic image of the virtual environment from the perspective of the virtual robot with the configuration, using a virtual camera of the virtual robot and a corresponding camera model (i.e.,one of the sensor models 48).

[0038] It should be noted that here, the "configuration" of a real or virtual robot refers to a specification of the position of each part of the robot or each point on the robot in a physical or virtual 3D space. For example, if the robot is essentially a rigid body, the robot's configuration defines its 3D position and orientation within its environment. However, if the robot is a non-rigid body, its configuration may include multiple positions, angles, or orientations of several parts of the robot to fully define its spatial state.For example, if the robot includes a chassis with wheels and a robot arm mounted on it with rigid connections and actuable joints, the configuration of the robot could be specified as a 3D position and orientation of the chassis with wheels and by the angle of each actuated joint of the robot arm or the position and orientation of each rigid connection of the robot arm.

[0039] In at least some embodiments, in order to identify a valid, passable configuration of the virtual robot within the virtual environment, the processor 152 randomly and iteratively selects candidate configurations of the virtual robot within the virtual environment. For each candidate configuration, the processor 152 checks whether the candidate configuration of the virtual robot is passable within the virtual environment until a valid, passable configuration is identified. The processor 152 selects a specific candidate configuration in response to a determination that the candidate configuration is passable.

[0040] In some embodiments, processor 152 determines candidate configurations of the virtual robot across the entire configuration space before the labeling process. In other words, processor 152 determines candidate configurations of the virtual robot for all possible positions in the virtual environment. In this case, processor 152 computes the entire configuration space once for each unique virtual environment and selects candidate configurations from the configuration space.

[0041] The processor 152 selects candidate configurations of the virtual robot in the virtual environment either randomly or uniformly according to a procedure. In one embodiment, the processor 152 determines the candidate configurations uniformly randomly in a bounded configuration space. In another embodiment, the processor 152 determines the candidate configurations randomly, but with a preference for regions where the current set of samples is sparse, e.g., as in Rapidly Exploring Random Trees (RRT). In yet another embodiment, the processor 152 determines the candidate configurations procedurally on a d-dimensional grid, where d is the dimension of the configuration space and the grid spacing is predetermined. In a further embodiment, the processor 152 determines the candidate configurations procedurally by moving the robot along boundaries of the virtual environment.

[0042] To identify a valid passable configuration of the virtual robot within the virtual environment, processor 152 evaluates whether the candidate configuration will cause the virtual robot to collide, come into contact with a dangerous substance, or enter an unstable state. Fig. Figure 4 shows different types of impassable robot configurations. In the illustration, the virtual robot 300 is depicted as a black cylinder. On the left, the virtual robot 300 is shown in collision with an obstacle 310. In the middle, the virtual robot 300 is shown in contact with a hazard 320. On the right, the virtual robot 300 is shown unstable on a surface 330.

[0043] To verify whether a candidate configuration of the virtual robot collides with an obstacle, the processor 152 determines whether the virtual robot with the candidate configuration intersects an obstacle in the virtual environment. In at least one embodiment, the processor 152 uses a 3D mesh-based collision checker, such as Open Dynamics Engine or NVIDIA Omniverse, to verify whether the robot collides with obstacles. An obstacle can include any virtual object in the virtual environment. It should be noted that a "virtual object," as used here, can refer to any part or section of the virtual environment, including virtual floors / terrain, virtual walls, virtual ceilings, and virtual objects placed in the virtual environment, such as furniture, trees, toys, etc.In response to the fact that the virtual robot is in collision with an obstacle (i.e., intersecting it), processor 152 determines that the candidate configuration is impassable.

[0044] Fig. Figure 5 shows several example configurations of a virtual robot 400. In the diagram, configurations of the virtual robot 400 are shown as triangles, and the course or direction of the virtual robot 400 is indicated by an arrow (for readability, only a subset is labeled with the reference symbol 400). An obstacle 410 is indicated by the hatched area. Since the depicted virtual robot 400 is flat and triangular, the robot's configuration consists of its position and orientation. Here, configurations are randomly sampled in the virtual environment. Partially hatched virtual robots 400 collide with the obstacle 410 in the center of the virtual environment. As can be seen, both position and orientation are important; for example, the two candidate configurations 420 in the lower left have the same position, but one orientation is in collision and the other is not.Similarly, if the robot had 400 arms or propellers, the positions of the arms / propellers would also affect the passability.

[0045] To check whether a candidate configuration of the virtual robot is in contact with a hazard, the processor 152 first determines whether the virtual robot with the candidate configuration intersects with a virtual hazard (i.e., a virtual object labeled as a hazard) in the virtual environment, similarly to what is described above. In at least one embodiment, the processor 152 uses a 3D mesh-based collision checker, such as Open Dynamics Engine or NVIDIA Omniverse, to check whether the robot is in contact with a hazardous substance. However, some hazards can be represented in the virtual environment as a flat, two-dimensional element. For example, a puddle of water can be represented by an area of ​​a virtual surface labeled as a puddle of water, or by a two-dimensional virtual object on the virtual surface.In such cases, the processor 152 also determines whether the virtual robot with the candidate configuration is located above a two-dimensional element corresponding to a virtual hazard (i.e., a virtual object labeled as a hazard). For example, in one embodiment, the processor 152 projects the model of the virtual robot onto the virtual floor or terrain and determines whether the two-dimensional element corresponding to a hazard intersects the projection of the virtual robot. As a result of the virtual robot colliding with (i.e., intersecting) or otherwise being in contact with a hazard (e.g., being directly above it), the processor 152 determines that the candidate configuration is impassable.

[0046] Finally, to verify the stability of a candidate virtual robot configuration, the processor 152 determines whether the virtual robot's center of mass is located above a virtual floor or in open space. In some embodiments, the processor 152 determines that the candidate configuration is impassable in response to the finding that the center of mass is located in open space. Similarly, in some embodiments, the processor 152 checks whether several relevant parts of the robot (e.g., positions corresponding to the virtual robot's wheels) are located above a virtual floor or in free space.Furthermore, or alternatively, to verify the stability of a candidate virtual robot configuration, Processor 152 simulates the virtual robot moving forward a predetermined number of time steps through the virtual environment under the influence of gravity. If, based on the simulation, the virtual robot falls or tips over by more than a predetermined threshold during this simulation, Processor 152 determines that the candidate configuration is impassable.

[0047] In some embodiments, the processor 152 additionally determines candidate velocities and / or accelerations of the virtual robot in a similar manner to how it generates the candidate configurations. It should be noted that the speed and / or acceleration of the virtual robot are derivatives of the virtual robot's configuration. In some embodiments, the processor 152 determines, based on the candidate velocities and / or accelerations, whether the virtual robot will get stuck in a small gap in a virtual floor.

[0048] With continued reference to Fig. 3. After identifying a passable configuration of the virtual robot, the processor 152 renders a synthetic image of the virtual environment from the perspective of the virtual robot with the passable configuration, using a virtual camera of the virtual robot and a corresponding camera model (i.e., one of the sensor models 48). In some embodiments, the processor 152 renders the synthetic image using one or more known graphics and computer simulation APIs or SDKs, such as OpenGL, NVIDIA Omniverse, and NVIDIA Isaac.

[0049] It is important to note that the synthetic image must be photorealistic to close the "sim-to-real gap." Specifically, any systematic difference between simulated and real images would lead to unpredictable output from the passability detection model 20, and the detection accuracy would be lower for real images than for synthetic ones. In one embodiment, the processor 152 applies noise to the synthetic image to minimize the "sim-to-real gap" and generate more diverse training data. Another challenge is capturing the diversity of everyday life in the simulation. It should be noted that using a large dataset, such as the Zillow Indoor Dataset and ShapeNet, when generating the virtual environments helps to provide diverse synthetic images in the training dataset 62.

[0050] With renewed reference to Fig. 3. Method 200 is continued by determining a labeling mask for the synthetic image based on a simulation of a virtual robot in the virtual environment (Block 230). In particular, the processor 152 determines a labeling mask for the synthetic image based on a simulation of the virtual robot in the virtual environment. The labeling mask indicates and / or quantifies the passability of respective areas of the virtual environment that are captured in respective parts / sections of the synthetic image. In at least some embodiments, the labeling mask has the form of a two-dimensional arrangement of passability label values ​​that have the same dimensions as the synthetic image to which it corresponds.In this way, each pixel in the synthetic images can be assigned a corresponding passability label, which indicates and / or quantifies whether a corresponding position within the virtual environment can be traversed by the virtual robot. For the sake of simplicity, it can be assumed, for example, that a passability label value in row i, column j of the label mask corresponds to the pixel in row i, column j in the original synthetic image.

[0051] Accordingly, in some embodiments, the processor 152 determines a passability label for each pixel in the synthetic image to determine the labeling mask. This label indicates whether the virtual robot can traverse a corresponding position within the virtual environment. The processor 152 then constructs the labeling mask from this passability label for each pixel in the synthetic image. Furthermore, in some embodiments, the processor 152 determines the labeling mask more efficiently by checking passability for sets of pixels rather than each individual pixel. For example, a 2x2 block of pixels, or a set of pixels belonging to an object, could be checked collectively as a single set of pixels according to an object segmentation of the image. The center of gravity of the set could then be used for ray tracing.

[0052] Fig. Figure 6 shows a flowchart for a procedure 500 for determining a passability label for a given pixel in a synthetic image. The procedure 500 begins by tracing a ray from a given pixel of the virtual camera to a virtual object that is first encountered in the virtual scene (block 510). Specifically, for each pixel of the synthetic image and / or for each pixel of the label mask, the processor 152 traces a given ray to a position in the scene. Specifically, the processor 152 identifies a given position within the virtual environment that corresponds to the given pixel in the synthetic image by tracing a given ray from the virtual camera used to generate the synthetic image. The given position is a position that coincides with the given ray.

[0053] Fig. Figure 7 summarizes the ray tracking process using a pinhole camera model. In the illustration, a first ray 610 is tracked by a first pixel 612 of a virtual camera until it encounters a virtual floor 642 of a virtual environment 640. In contrast, a second ray 620 and a third ray 630 are tracked by a second pixel 622 and a third pixel 632 of the virtual camera, respectively, until they encounter an obstacle 644. In the illustrated example, the processor 152 tracks the ray using a pinhole camera model. However, it should be noted that a camera model other than a pinhole camera model can also be used. In particular, in some embodiments, the processor 152 tracks the ray using a camera model that includes a lens positioned between the camera sensor and the environment to be detected, and the ray passing through the lens is simulated.

[0054] In at least some embodiments, the processor 152 determines the respective position corresponding to a given pixel in the synthetic image as the position at which the respective ray first intersects or crosses a virtual object. In the examples of Fig. 7 is the respective position corresponding to pixel 612, the point where beam 610 intersects the virtual floor 642. Similarly, the respective position corresponding to pixel 622 is the point where beam 620 intersects the virtual obstacle 644. Likewise, the respective position corresponding to pixel 632 is the point where beam 630 intersects the virtual obstacle 644. For embodiments where the passability detection model 20 is applied to a ground-based mobile robot 120 (e.g., a robotic vacuum cleaner), this position, which corresponds to the first virtual object encountered, is generally the most relevant.

[0055] In some embodiments, however, the processor 152 determines the respective position only within a predetermined maximum distance from the virtual camera. Specifically, if the respective beam does not intersect a virtual object in the virtual environment within the predetermined maximum distance, the processor 152 determines the respective position as the position at the predetermined maximum distance along the respective beam. Conversely, if the respective beam intersects a virtual object in the virtual environment within the predetermined maximum distance, the processor 152 determines the respective position as the position at which the respective beam first intersects the virtual object. This approach can be applicable to embodiments in which the passability detection model 20 is to be applied to a mobile robot 120 that is in the air (e.g., a UAV).

[0056] Furthermore, in some embodiments, the processor 152 determines whether the beam extends infinitely far into space without collision. In one embodiment, the processor 152 checks for intersections at a maximum distance greater than the size of the virtual environment. In another embodiment, the processor 152 checks each polygon in the virtual environment for intersections with the beam. In such cases, where the beam does not collide with any geometry of the virtual environment, the processor 152 can immediately determine passability. In particular, if the passability detection model 20 is to be applied to a mobile robot 120 that is in the air (e.g., a UAV), the processor 152 then determines that the pixel is passable. Conversely, if the passability detection model 20 is to be applied to a mobile robot 120 that is ground-based (e.g., a UAV), the processor 152 determines that the pixel is passable.(a robot vacuum cleaner), the processor 152 then determines that the pixel is not passable.

[0057] Procedure 500 continues by checking whether the first encountered virtual object is a floor (block 520). If the first encountered virtual object is not a floor, procedure 500 continues by labeling the pixel as impassable (block 530). Specifically, processor 152 determines the respective passability label for each pixel as "impassable" in response to the fact that the respective ray intersects a virtual object in the virtual environment that is not a virtual floor of the environment. In other words, positions in the virtual environment corresponding to virtual objects that are not floors are labeled as "impassable." Such a labeling process is suitable, for example, for embodiments in which the passability detection model 20 is applied to a ground-based mobile robot 120 (e.g., a robotic vacuum cleaner).

[0058] In some embodiments, for example, those in which the passability detection model 20 is to be applied to a mobile robot that is in the air (e.g., a UAV), the ground-to-non-ground distinction can be modified or removed. In particular, in some embodiments, the processor 152 determines the respective passability label for each pixel as "not passable" in response to the fact that the respective beam intersects any part of the virtual environment within the specified maximum distance from the virtual camera. In other words, for a flying mobile robot 120, positions that intersect any part of the virtual environment are labeled as "not passable".

[0059] Conversely, if the first virtual object encountered is a floor, then procedure 500 continues by scanning robot configurations at the position where the beam intersected the floor (block 540). Specifically, in response to the fact that the respective beam intersects a virtual floor of the virtual environment, processor 152 determines a multitude of example configurations of the virtual robot at the corresponding respective position. As explained above, a “configuration” of a real robot or a virtual robot refers to a specification of the position of each part of the robot or each point on the robot in a physical or virtual 3D space.Therefore, using a rigid virtual robot as an example, the multitude of example configurations can be easily characterized by a multitude of different orientations of the virtual robot at its respective position within the virtual environment. Referring again to the representation in... Fig. In Figure 5, the two overlapping virtual robot configurations 420 are representative of two example configurations of the virtual robot 400 at a specific position. Similarly, using a non-rigid virtual robot as an example, the multitude of example configurations can be further characterized by a multitude of different angles, positions, or orientations of different parts of the virtual robot at the respective position within the virtual environment.

[0060] In some embodiments, processor 152 determines possible configurations of the virtual robot across the entire configuration space prior to the labeling process. In other words, processor 152 determines configurations of the virtual robot for all possible positions in the virtual environment, and a subset of these configurations at a given position is used as example configurations during the labeling of a specific pixel of the labeling mask and / or the synthetic image.

[0061] The processor 152 samples the configuration space of the virtual robot in the virtual environment either randomly or uniformly according to a specific procedure. In one embodiment, the processor 152 determines the example configurations uniformly randomly in a bounded configuration space. In another embodiment, the processor 152 determines the example configurations randomly, but with a preference for regions where the current set of examples is sparse, e.g., as in Rapidly Exploring Random Trees (RRT). In yet another embodiment, the processor 152 determines the example configurations procedurally on a d-dimensional grid, where d is the dimension of the configuration space and the grid spacing is predetermined.

[0062] Procedure 500 continues with labeling the pixel depending on the proportion of passable robot configurations (Block 550). Specifically, processor 152 determines for each example configuration from the plurality of example configurations whether the virtual robot can pass the corresponding position with that particular example configuration. The passability of each example configuration is determined in a similar manner to how it was described above regarding the placement of the virtual robot in the virtual scene for capturing the synthetic image.

[0063] To check whether a sample configuration of the virtual robot collides with an obstacle, the processor 152 determines whether the virtual robot with the sample configuration intersects an obstacle in the virtual environment. In at least one embodiment, the processor 152 uses a 3D mesh-based collision checker, such as Open Dynamics Engine or NVIDIA Omniverse, to check whether the robot collides with obstacles. An obstacle can include any virtual object in the virtual environment. It should be noted that a "virtual object," as used here, can refer to any part of the virtual environment, including virtual floors / terrain, virtual walls, virtual ceilings, and virtual objects placed in the virtual environment, such as furniture, trees, toys, etc. In response to the virtual robot being in collision with an obstacle (i.e.,(i.e., intersects with it), processor 152 determines that the example configuration is not passable.

[0064] To check whether a sample configuration of the virtual robot is in contact with a hazard, the processor 152 first determines whether the virtual robot with the sample configuration intersects with a virtual hazard (i.e., a virtual object labeled as a hazard) in the virtual environment, similarly as explained above. In at least one embodiment, the processor 152 uses a 3D mesh-based collision checker, such as Open Dynamics Engine or NVIDIA Omniverse, to check whether the robot is in contact with a hazardous substance. However, some hazards may be represented in the virtual environment as a flat, two-dimensional element. For example, a puddle of water may be represented by an area of ​​a virtual surface labeled as a puddle, or by a two-dimensional virtual object on the virtual surface.In such cases, the processor 152 also determines whether the virtual robot with the example configuration is located above a two-dimensional element corresponding to a virtual hazard (i.e., a virtual object labeled as a hazard). For example, in one embodiment, the processor 152 projects the model of the virtual robot onto the virtual floor or terrain and determines whether the two-dimensional element corresponding to a hazard intersects the projection of the virtual robot. As a response to the virtual robot being in collision with a hazard (i.e., intersecting it) or otherwise in contact with a hazard (i.e., directly above it), the processor 152 determines that the example configuration is impassable.

[0065] Finally, to verify the stability of a sample configuration of the virtual robot, the processor 152 determines whether the center of mass of the virtual robot with the sample configuration lies above a virtual floor or an open space. In some embodiments, the processor 152 determines that the sample configuration is impassable in response to the finding that the center of mass lies above an open space. In some embodiments, the processor 152 similarly checks whether several relevant parts of the robot (e.g., positions corresponding to the wheels of the virtual robot) are located above a virtual floor or an open space.Furthermore, or alternatively, to verify the stability of a sample virtual robot configuration, Processor 152 simulates the virtual robot moving forward a predetermined number of time steps through the virtual environment under the influence of gravity. If, based on the simulation, the virtual robot falls or tips over by more than a predetermined threshold during this simulation, Processor 152 determines that the sample configuration is impassable.

[0066] In some embodiments, the processor 152 additionally determines example speeds and / or accelerations of the virtual robot in a similar manner to generating the example configurations. It should be understood that the speed and / or acceleration of the virtual robot are derivatives of the virtual robot's configuration. In some embodiments, the processor 152 determines, based on the example speeds and / or accelerations, whether the virtual robot will get stuck in a small gap in a virtual floor.

[0067] Once the passability of each example configuration at a given position has been determined, the processor 152 determines a passability label for the corresponding pixel of the label mask and / or synthetic image based on the passability of each example configuration.

[0068] In some embodiments, the processor 152 determines the respective passability label as the ratio of (i) example configurations with which the robot can pass the respective position, and (ii) example configurations with which the robot cannot pass the respective position. In other words, the passability label for each pixel of the label mask and / or synthetic image is determined according to the proportion of configurations that are passable, collide with obstacles, come into contact with hazards (e.g., a puddle of water or animal droppings), and are unstable (e.g., tipping over). The processor 152 divides the number of passable configurations by the total number of example configurations. The processor 152 records this ratio in the corresponding pixel of the label mask and / or synthetic image from which the beam originated.

[0069] Alternatively, in some embodiments, the processor 152 determines the respective passability label as passable in response to a determination that the corresponding respective position within the virtual environment is passable by the virtual robot using at least one example configuration. In other words, the processor 152 determines the label mask as a binary mask, where a value of "1" or "true" indicates that a passable configuration was found at the corresponding respective position, and a value of "0" or "false" indicates that no passable configuration was found at the corresponding respective position.

[0070] Finally, procedure 500 continues with a check to see if every pixel has been labeled (block 560). If there are still pixels of the labeling mask and / or the synthetic image that have not yet been labeled, procedure 500 returns to block 510 and the process is repeated for the next pixel. Otherwise, if all pixels of the labeling mask and / or the synthetic image have been labeled, procedure 500 is complete.

[0071] With renewed reference to Fig.3. In some embodiments, the processor 152 generates several labeling masks that correspond to different restrictions of traversability or types of impassability (e.g., obstacles, hazards, instability, or uneven ground). In one embodiment, the processor 152 determines a first labeling mask for the synthetic image that indicates whether corresponding positions / locations within the virtual environment are impassable due to a first restriction of traversability (e.g., obstacles). In another embodiment, the processor 152 determines a second labeling mask for the synthetic image that indicates whether corresponding positions within the virtual environment are impassable due to a second restriction of traversability (e.g., hazards).In one embodiment, the processor 152 determines a third labeling mask for the synthetic image, indicating whether corresponding positions within the virtual environment are impassable due to a third traversability restriction (e.g., instability). In another embodiment, the processor 152 determines a fourth labeling mask for the synthetic image, indicating whether corresponding positions within the virtual environment are impassable due to a fourth traversability restriction (e.g., non-ground). Each different labeling mask would be determined in the same process described above, but the labeling values ​​in each labeling mask would only quantify the traversability type corresponding to the mask type.

[0072] Once one or more labeling masks have been determined, the processor 152 stores the labeling mask(s) in association with the corresponding synthetic image on the storage devices 162, i.e., as training data 62. The procedure 200 continues with a check to see if sufficient training data is available (block 240). For this purpose, in one embodiment, the processor 152 determines whether a threshold number of training samples is available, or otherwise evaluates the training data set against a metric. If sufficient training data is not yet available, the procedure 200 returns to block 220 to generate a new synthetic image. Alternatively, once a threshold number of synthetic images has been generated using a specific virtual scene, the procedure 200 instead returns to block 210 to generate a new virtual scene within which new synthetic images are generated.

[0073] Once sufficient training data is available, the procedure 200 continues with training the machine learning model based on the synthetic image and the labeling mask (block 250). Specifically, the processor 152 trains the passability detection model 20 based on the synthetic images and the corresponding labeling masks in the captured or collected training data 62. Specifically, once a sufficient corpus of training data 62 has been generated (i.e., synthetic images with associated ground-truth passability labeling masks), the processor 152 executes the optimizer 64 of the trainer 60 to train the passability detection model 20 to predict passable and impassable areas of an environment based on images of the environment.For this purpose, the processor 152 can use any machine learning algorithm to fit a model to data, including gradient descent, stochastic gradient descent, and Newton's method. In one embodiment, the processor 152 trains the passability detection model 20 using augmented learning.

[0074] In some embodiments, the synthetic training data 62 can be augmented with real-world training data comprising real-world images that have been labelled manually or using additional detection systems, such as LiDAR or bumper sensors. Real-world training data can be generated by multiple human labelers and / or remotely controlled robots and aggregated via a network connection. The robot(s) used to label real-world images need not be of the same type as those used for passability detection; that is, they could have additional, more expensive sensors for detecting passability, and they could have stronger, more expensive shells or enclosures to prevent damage from hazards.

[0075] As explained above, in some embodiments, multiple labeling masks are generated according to different restrictions of passability or types of impassability (e.g., obstacles, hazards, instability, or non-ground). Likewise, in some embodiments, the processor 152 trains multiple passability detection models 20 according to different restrictions of passability or types of impassability. In one embodiment, the processor 152 trains a first passability detection model 20 to predict passable and impassable areas of an environment based on images of the environment, based on a first restriction of passability (e.g., obstacles). In another embodiment, the processor 152 trains a second passability detection model 20 to predict passable areas and areas impassable due to a second restriction of passability (e.g., obstacles).In one embodiment, the processor 152 trains a third passability detection model 20 to predict passable areas and areas impassable due to a third passability restriction (e.g., instability) based on images of the environment. In another embodiment, the processor 152 trains a fourth passability detection model 20 to predict passable areas and areas impassable due to a fourth passability restriction (e.g., non-ground) based on images of the environment.

[0076] Once the passability detection model 20 is trained, it can be deployed on the mobile robot. The controller 122 operates and controls a camera and / or other sensors 122 to capture real-world images of the environment. Next, the controller 122 executes the passability detection model 20 to predict passable and impassable areas of the environment based on the captured images, i.e., to generate a predicted labeling mask. Based on the predicted passable and impassable areas, the controller 122 executes the operations 132 to generate operating commands for actuating the actuators 128 to navigate the environment in a manner that avoids the impassable areas.Alternatively, in some embodiments, the passability detection model 20 is deployed and executed on behalf of the mobile robot 120 on a remote cloud server.

[0077] In some embodiments, in addition to training the passability detection model 20, the processor 152 trains a further machine learning model of the operational sequences 132 to generate the instructions for actuating the actuators 128 in order to navigate the environment in such a way as to avoid the impassable areas. In particular, the further machine learning model is configured to generate operational instructions based on a captured image of the real environment and an associated labeling mask generated by the passability detection model 20.

[0078] In one embodiment, the processor 152 trains a further machine learning model of the operations 132 using the simulator 40 and augmented learning algorithms. For this purpose, the processor 152 provides as input to the further machine learning model a synthetic image of a virtual environment with an associated traversability label mask. The processor 152 executes the further machine learning model to generate operating instructions as output. The processor 152 simulates operation of the virtual robot using the generated operating instructions. There is a reward function that defines how the machine learning model of the operations 132 should behave. In one example, a reward of 1 is received if the robot moves forward 1 meter, and a reward of -10 is received if the robot encounters an obstacle.Initially, the operational machine learning model 132 causes the virtual robot to move randomly in the simulation. Over time, however, the operational machine learning model 132 learns to maximize the expected future reward, as occurs in the field of reinforcement learning.

[0079] Embodiments within the scope of this disclosure may also include non-volatile computer-readable storage media or machine-readable media on which computer-executable instructions (also referred to as program instructions) or data structures are stored or are stored. Such non-volatile computer-readable storage media or machine-readable media may be any available media accessible to a general-purpose or specialized computer. By way of example, and not as a limitation, such non-volatile computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to receive, transmit, or store desired program code means in the form of computer-executable instructions or data structures.Combinations of the above-mentioned elements should also fall within the scope of non-volatile computer-readable storage media or machine-readable media.

[0080] Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions.

[0081] Computer-executable instructions also include program modules that are executed by computers in standalone or networked environments. In general, program modules comprise routines, programs, objects, components, and data structures, etc., that perform specific tasks or implement / realize certain abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for performing steps of the procedures disclosed herein. The specific sequence of such executable instructions or associated data structures represents examples of corresponding actions for implementing / realizing the functions described in such steps.

[0082] Although the disclosure has been illustrated and described in detail in the drawings and the preceding description, it should nevertheless be regarded as illustrative and not limiting. It is understood that only preferred embodiments have been presented and that all changes, modifications, and further applications that are within the scope of the disclosure are to be protected.

Claims

[1] Method for training a machine learning model set up to determine the traversability of a real environment by a mobile robot based on an image of the real environment, comprising the method: Creating a virtual environment using a variety of three-dimensional models; Generating a synthetic image of the virtual environment; Determining a labeling mask for the synthetic image based on a simulation of a virtual robot in the virtual environment, wherein the labeling mask specifies the traversability of respective areas of the virtual environment captured in the synthetic image; and Training the machine learning model based on the synthetic image and the labeling mask. [2] The method of claim 1, wherein the creation of the virtual environment further comprises: Determining a spatial layout for the virtual environment, wherein the spatial layout defines virtual walls and virtual floors of the virtual environment; and Determining the positions of a large number of virtual objects located within the virtual environment. [3] Method according to claim 1, wherein the generation of the synthetic image further comprises: Defining a configuration of a virtual robot within the virtual environment; and Generating the synthetic image of the virtual environment from the perspective of a virtual camera of the virtual robot with the configuration. [4] Method according to claim 3, wherein defining the configuration of the virtual robot further comprises: random selection of a candidate configuration of the virtual robot within the virtual environment; Verify that the candidate configuration of the virtual robot is passable within the virtual environment; and Defining the configuration as the candidate configuration in response to the fact that the candidate configuration is passable [5] Method according to claim 4, wherein checking whether the candidate configuration of the virtual robot is passable further comprises: Determine whether the virtual robot with the candidate configuration collides with a virtual obstacle in the virtual environment. [6] Method according to claim 4, wherein the check to see if the candidate configuration of the virtual robot is passable further comprises: Determine whether the virtual robot with the candidate configuration (i) collides with a virtual hazard in the virtual environment or (ii) is directly above a virtual hazard in the virtual environment. [7] Method according to claim 4, wherein checking whether the candidate configuration of the virtual robot is passable further comprises: Determine whether the virtual robot with the candidate configuration is stably supported by a virtual floor in the virtual environment. [8] Method according to claim 1, wherein determining the labeling mask further comprises: Determining a passability label for each pixel in the synthetic image, indicating whether the virtual robot can traverse a given position within the virtual environment; and Forming the labeling mask from the respective passability label for each pixel in the synthetic image. [9] Method according to claim 8, wherein determining the respective passability label for each respective pixel in the synthetic image further comprises: Identifying the corresponding respective position within the virtual environment by tracking a respective ray from a virtual camera used to generate the synthetic image, wherein the corresponding respective position within the virtual environment is a position that coincides with the respective ray. [10] Method according to claim 9, wherein the respective position within the virtual environment is one of (i) a position along the respective beam at a predetermined maximum distance from the virtual camera and (ii) a position at which the respective beam first intersects the virtual environment, which is smaller than the predetermined maximum distance from the virtual camera. [11] Method according to claim 9, wherein the respective position within the virtual environment is a position at which the respective ray first intersects the virtual environment. [12] Method according to claim 9, wherein determining the respective passability label for each respective pixel in the synthetic image further comprises: Determining the respective passability label as impassable in response to the fact that the respective ray intersects with a virtual object in the virtual environment that is not a virtual floor of the virtual environment at the respective position. [13] Method according to claim 9, wherein determining the respective passability label for each respective pixel in the synthetic image in response to the respective ray intersecting with a virtual floor of the virtual environment at the corresponding respective position further comprises: Determining a variety of example configurations of the virtual robot at the corresponding respective position within the virtual environment; and Determine for each of the many example configurations whether the virtual robot can pass through the corresponding position with that particular example configuration. [14] Method according to claim 13, wherein determining the respective passability label for each respective pixel in the synthetic image further comprises: Determining the respective passability label as a ratio of (i) example configurations from the multitude of example configurations with which the robot can pass the respective position, and (ii) example configurations from the multitude of example configurations with which the robot cannot pass the respective position. [15] Method according to claim 13, wherein determining the respective passability label for each respective pixel in the synthetic image further comprises: Determining the respective passability label as passable in response to determining that the corresponding respective position within the virtual environment is passable by the virtual robot using at least one example configuration from the multitude of example configurations. [16] The method of claim 13, wherein determining whether the virtual robot can pass the respective position with the respective example configuration further comprises: Determine whether the virtual robot with the example configuration collides with a virtual obstacle in the virtual environment. [17] The method of claim 13, wherein determining whether the virtual robot can pass the respective position with the respective example configuration further comprises: Determine whether the virtual robot with the example configuration (i) collides with a virtual hazard in the virtual environment or (ii) is directly above a virtual hazard in the virtual environment. [18] The method of claim 13, wherein determining whether the virtual robot can pass the respective position with the respective example configuration further comprises: Determine whether the virtual robot, with the respective example configuration, is stably supported by a virtual floor in the virtual environment. [19] Method according to claim 1, wherein determining the labeling mask further comprises: Determining a first labeling mask for the synthetic image, wherein the first labeling mask indicates whether corresponding positions within the virtual environment are impassable due to a first restriction of traversability; and Determining a second labeling mask for the synthetic image, wherein the second labeling mask indicates whether corresponding positions within the virtual environment are not passable due to a second restriction of traversability. [20] Method according to claim 1, further comprising: Training a second machine learning model using the trained first machine learning model, wherein the second machine learning model is set up to generate operating commands for the mobile robot based on a captured image of the real environment and a labeling mask generated by the first machine learning model based on the image.

Images