Navigation mobile robot

By using image data in the navigation mobile robot system to generate intersection points pointing to vector and terrain estimates, transmit waypoint commands, and simulate environmental rotation using virtual rectangle and scene camera technology, the autonomous navigation and scene of navigation mobile robots in complex environments is solved. Rotating problems improve the operator's control experience and navigation flexibility.

CN118915722APending Publication Date: 2024-11-08BOSTON DYNAMICS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410944785.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-08-07
Filing Date
2019-10-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing navigation mobile robot systems are difficult to navigate autonomously in complex environments, especially when obstacles exist, and it is difficult for operators to intuitively control the robot to rotate between multiple scenes.

Method used

By receiving image data on the data processing hardware of the operator device, a graphical user interface is generated, a point point point point point to the estimation of vectors and terrain is determined, a waypoint command is transmitted to enable the robot to navigate to the target position, and simulate the environment without changing the robot's physical position Rotate, using virtual rectangles and scene camera technology to rotate the preview scene.

Benefits of technology

It realizes the robot's autonomous navigation and multi-scene rotation in complex environments, improves the operator's intuitive control experience, and enhances the flexibility and accuracy of navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915722A_ABST
    Figure CN118915722A_ABST
Patent Text Reader

Abstract

A computer-implemented method, system, and legged robot are provided, the method comprising: receiving, by data processing hardware of an operator device, image data from at least one image sensor, the image data corresponding to an environment of the robot; and instructing by the data processing hardware to display a graphical user interface via a screen of the operator device, the graphical user interface configured to: display a scene based on the image data; receiving a rotation input to rotate a field of view of the environment; and displaying the preview scene by rotating the field of view without requiring physical movement of the body of the robot.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with application date of October 23, 2019, application number 201980100845.9, and invention name “Navigation Mobile Robot”. Technical Field

[0002] The present disclosure relates to navigating mobile robots. Background Art

[0003] As technology advances, various types of robotic devices are being created to perform a variety of functions that can assist users. Robotic devices can be used for applications involving material handling, transportation, welding, assembly, and distribution, among others. Over time, the way these robotic systems operate has become more intelligent, efficient, and intuitive. As robotic systems become more common in many aspects of modern life, people expect robotic systems to be easy and intuitive to operate and navigate. Summary of the invention

[0004] One aspect of the present disclosure provides a method for controlling a robot. The method includes receiving image data from at least one image sensor at data processing hardware of an operator device. The image data corresponds to an environment surrounding the robot. The method also includes executing, by the data processing hardware, a graphical user interface (GUI) for display on a screen of the operator device. The GUI is configured to display a scene of the environment surrounding the robot based on the image data, and receive an input indication indicating a pixel position within the scene of the environment surrounding the robot. The method also includes determining, by the data processing hardware, a pointing vector V based on the selection of the pixel position. The pointing vector represents a direction of travel for navigating the robot in the environment. The method also includes transmitting a waypoint command W to the robot by the data processing hardware. When the robot receives the waypoint command W, the robot is navigated to a target position T. The target position is based on the intersection between the pointing vector V and a terrain estimate G of the robot. The terrain estimate G may include a ground plane estimate. For simplicity, the present disclosure refers to "terrain estimate" as "ground plane estimate".

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the scene of the environment around the robot includes any of the following: a front scene of the environment based on image data, the image data being captured by a left front camera and a right front camera disposed on the robot; a left scene of the environment based on image data, the image data being captured by a left camera disposed on the robot; a right scene of the environment based on image data, the image data being captured by a right camera disposed on the mobile robot; a rear scene of the environment based on image data, the rear scene being captured by a rear camera disposed on the robot; or a top-down scene of the robot based on image data, the image data being captured by a payload camera, a left front camera, a right front camera, a left camera, a right camera, and a rear camera.

[0006] In some examples, the method further includes determining, by the data processing hardware, whether the intersection between the pointing vector and the terrain estimate is in front of the robot, and identifying, by the data processing hardware, the intersection as the target location when the intersection between the pointing vector and the terrain estimate is in front of the robot. In these examples, the method may also include, when the intersection between the pointing vector and the terrain estimate is behind the robot: reflecting, by the data processing hardware, the intersection to determine the reflected intersection in front of the robot; and identifying, by the data processing hardware, the reflected intersection as the target location. Additionally or alternatively, in these examples, the method may also include, before identifying the intersection as the target location: determining, by the data processing hardware, that a first distance between the intersection between the pointing vector and the terrain estimate and the current location of the robot does not satisfy a threshold distance; and moving, by the data processing hardware, the intersection to an intermediate location closer to the robot to form a moving intersection, wherein a second distance between the current location of the robot and the moving intersection satisfies the threshold distance.

[0007] The robot may be configured to determine whether the target location is located behind an obstacle. Here, the robot is configured to autonomously navigate to the target location when the target location is not located behind an obstacle. However, when the target location is located behind an obstacle, the robot is configured to determine whether a maneuver around the obstacle is feasible, and when the maneuver around the obstacle is feasible, autonomously navigate the robot around the obstacle and reach the target location. In addition, when the target location is located behind an obstacle and when the maneuver around the obstacle is not feasible, the robot may also be configured to prevent the robot from autonomously navigating to the target location.

[0008] In some implementations, the at least one image sensor includes one or more fisheye cameras and one or more infrared cameras. The at least one image sensor may be disposed on the robot, and the operator device may communicate with the image sensor via a network.

[0009] In some examples, the graphical user interface is further configured to: receive a rotation input to rotate the field of view of the environment around the robot in a direction away from the current scene displayed in the graphical user interface; and display the preview scene by rotating the field of view of the environment around the robot in a direction away from the current scene. The graphical user interface can be configured to display the preview scene without requiring physical movement of the robot, and / or the graphical user interface can be configured to receive the rotation input in response to receiving an input indication indicating selection of a rotation graphic displayed in the graphical user interface. In other examples, the field of view of the environment around the robot is rotated in a direction away from the current scene, simulating the robot performing a turning maneuver in a direction away from the current scene and toward the preview scene.

[0010] Another aspect of the present disclosure provides a system for operating a robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving image data from at least one image sensor. The image data corresponds to an environment surrounding the robot. The operations also include executing a graphical user interface (GUI) for display on a screen of an operator device. The GUI is configured to display a scene of the environment surrounding the robot based on the image data, and receive an input indication indicating a pixel position within the scene of the environment surrounding the robot. The operations also include determining a pointing vector V based on the selection of the pixel position. The pointing vector represents a direction of travel for navigating the robot in the environment. The operations also include transmitting a waypoint command W to the robot. When the robot receives the waypoint command W, the robot is caused to navigate to a target position T. The target position is based on the intersection between the pointing vector V and the terrain estimate G of the robot.

[0011] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the scene of the environment around the robot includes any of the following: a front scene of the environment based on image data, the image data being captured by a left front camera and a right front camera disposed on the robot; a left scene of the environment based on image data, the image data being captured by a left camera disposed on the robot; a right scene of the environment based on image data, the image data being captured by a right camera disposed on the mobile robot; a rear scene of the environment based on image data, the rear scene being captured by a rear camera disposed on the robot; or a top-down scene of the robot based on image data, the image data being captured by a payload camera, a left front camera, a right front camera, a left camera, a right camera, and a rear camera.

[0012] In some examples, the operations also include determining whether the intersection between the pointing vector and the terrain estimate is in front of the robot, and when the intersection between the pointing vector and the terrain estimate is in front of the robot, identifying the intersection as the target location. In these examples, the operations may also include, when the intersection between the pointing vector and the terrain estimate is behind the robot: reflecting the intersection to determine the reflected intersection in front of the robot; and identifying the reflected intersection as the target location. Additionally or alternatively, in these examples, the operations may also include, before identifying the intersection as the target location: determining that a first distance between the intersection between the pointing vector and the terrain estimate and the current location of the robot does not satisfy a threshold distance; and moving the intersection to an intermediate position closer to the robot to form a moving intersection, wherein a second distance between the current location of the robot and the moving intersection satisfies the threshold distance.

[0013] The robot may be configured to determine whether the target location is located behind an obstacle. Here, the robot is configured to autonomously navigate to the target location when the target location is not located behind an obstacle. However, when the target location is located behind an obstacle, the robot is configured to determine whether a maneuver around the obstacle is feasible, and when the maneuver around the obstacle is feasible, autonomously navigate the robot around the obstacle and reach the target location. In addition, when the target location is located behind an obstacle and when the maneuver around the obstacle is not feasible, the robot may also be configured to prevent the robot from autonomously navigating to the target location. The robot may include a quadruped robot.

[0014] In some implementations, the at least one image sensor includes one or more fisheye cameras and one or more infrared cameras. The at least one image sensor may be disposed on the robot, and the operator device may communicate with the image sensor via a network.

[0015] In some examples, the graphical user interface is further configured to: receive a rotation input to rotate the field of view of the environment around the robot in a direction away from the current scene displayed in the graphical user interface; and display the preview scene by rotating the field of view of the environment around the robot in a direction away from the current scene. The graphical user interface can be configured to display the preview scene without requiring physical movement of the robot, and / or the graphical user interface can be configured to receive the rotation input in response to receiving an input indication indicating selection of a rotation graphic displayed in the graphical user interface. In other examples, the field of view of the environment around the robot is rotated in a direction away from the current scene, simulating the robot performing a turning maneuver in a direction away from the current scene and toward the preview scene.

[0016] Another aspect of the present disclosure provides a method for navigating a mobile robot. The method includes receiving, at data processing hardware, a pixel position in a scene of a robot environment captured by a camera on the mobile robot, and determining, by the data processing hardware, a pointing vector based on the pixel position and calibration information of the camera. The pixel position being received is selected by an operator of the mobile robot. The method also includes combining, by the data processing hardware, the pointing vector with a terrain estimate of the mobile robot to form an intersection. The intersection includes a physical location in the robot environment. The method also includes transmitting, by the data processing hardware, the physical location to the mobile robot so that the mobile robot navigates to the physical location in the robot environment.

[0017] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the scene is any one of the following: i) a front scene of the robot environment captured by a left front camera and a right front camera on the mobile robot, ii) a left scene of the robot environment captured by a left camera on the mobile robot, iii) a right scene of the robot environment captured by a right camera on the mobile robot, iv) a rear scene of the robot environment captured by a rear camera on the mobile robot, and v) a top-down scene of the robot environment captured by an imaginary camera positioned above the mobile robot.

[0018] In some examples of the method, combining the pointing vector with the terrain estimate to form an intersection point includes determining, by data processing hardware, whether a distance between the mobile robot and the intersection point is greater than a distance threshold of the mobile robot. These examples of the method also include moving, by the data processing hardware, the intersection point closer to the mobile robot to form a mobile intersection point when the distance is greater than the distance threshold of the mobile robot, such that a shortened distance between the mobile robot and the mobile intersection point is less than or equal to the distance threshold.

[0019] In other examples of the method, combining the pointing vector with the terrain estimate to form an intersection includes determining by data processing hardware whether the intersection is behind the mobile robot. These examples of the method also include reflecting by data processing hardware the intersection in front of the mobile robot to form a reflected intersection when the intersection is behind the mobile robot. Here, the reflected intersection includes another physical location in the robot environment for the mobile robot to navigate to. Optionally, some other examples of the method also include determining by data processing hardware whether the distance between the mobile robot and the reflected intersection is greater than a distance threshold of the mobile robot. These other examples of the method also include moving the reflected intersection closer to the mobile robot to form a mobile intersection by data processing hardware when the distance is greater than the distance threshold of the mobile robot, so that the shortened distance between the mobile robot and the mobile intersection is less than or equal to the distance threshold of the mobile robot.

[0020] In some examples of the method, the scene of the robot environment includes a first scene of the robot environment captured by a first camera on the mobile robot, and at least one second scene of the robot environment that is different from the first scene of the robot environment. The at least one second scene of the robot environment is captured by at least one second camera on the mobile robot that is different from the first camera on the mobile robot. In these examples, the method also includes displaying the first scene of the robot environment and the at least one second scene of the robot environment by the data processing hardware. The method also includes changing the direction of navigation of the mobile robot by the data processing hardware in response to the operator changing between the first scene of the robot environment and the at least one second scene of the robot environment.

[0021] In other examples of the method, the scene is a first-person view of the robot environment, and the method further includes displaying, by the data processing hardware, the scene and a top-down scene of the robot environment captured by a hypothetical camera positioned above the mobile robot. The method further includes mirroring, by the data processing hardware, a pixel position selection in another of the displayed scenes in response to a pixel position selection in one of the displayed scenes.

[0022] In some examples of the method, transmitting the physical location in the robot environment includes transmitting a positioning command with the physical location to the mobile robot. Optionally, the camera can have a fisheye camera and two infrared cameras. In addition, the mobile robot can have four legs.

[0023] Another aspect of the present disclosure provides a system for navigating a mobile robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including: receiving pixel positions in a scene of a robot environment captured by a camera on the mobile robot, and determining a pointing vector based on the pixel positions and calibration information of the camera. An operator of the mobile robot selects the pixel position. The operations performed by the data processing hardware also include combining the pointing vector with a terrain estimate of the mobile robot to form an intersection. The intersection includes a physical location in the robot environment. The operations performed by the data processing hardware also include transmitting the physical location to the mobile robot so that the mobile robot navigates to the physical location in the robot environment.

[0024] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the scene is any one of the following: i) a front scene of the robot environment captured by a left front camera and a right front camera on the mobile robot, ii) a left scene of the robot environment captured by a left camera on the mobile robot, iii) a right scene of the robot environment captured by a right camera on the mobile robot, iv) a rear scene of the robot environment captured by a rear camera on the mobile robot, and v) a top-down scene of the robot environment captured by an imaginary camera positioned above the mobile robot.

[0025] In some examples of the system, the operation of combining the pointing vector with the terrain estimate to form the intersection point includes determining whether a distance between the mobile robot and the intersection point is greater than a distance threshold of the mobile robot. The operations also include moving the intersection point closer to the mobile robot to form a mobile intersection point when the distance is greater than the distance threshold of the mobile robot, so that a shortened distance between the mobile robot and the mobile intersection point is less than or equal to the distance threshold.

[0026] In other examples of the system, the operation of combining the pointing vector with the terrain estimate to form an intersection includes determining whether the intersection is behind the mobile robot. These operations also include reflecting the intersection in front of the mobile robot to form a reflected intersection when the intersection is behind the mobile robot. Here, the reflected intersection includes another physical location in the robot environment for the mobile robot to navigate to. Optionally, in some other examples of the system, these operations also include determining whether the distance between the mobile robot and the reflected intersection is greater than a distance threshold of the mobile robot. These operations also include moving the reflected intersection closer to the mobile robot by data processing hardware to form a mobile intersection when the distance is greater than the distance threshold of the mobile robot, so that the shortened distance between the mobile robot and the mobile intersection is less than or equal to the distance threshold of the mobile robot.

[0027] In some examples of the system, the scene of the robot environment includes a first scene of the robot environment captured by a first camera on the mobile robot, and at least one second scene of the robot environment that is different from the first scene of the robot environment. The at least one second scene of the robot environment is captured by at least one second camera on the mobile robot that is different from the first camera on the mobile robot. In these examples of the system, the operations also include displaying the first scene of the robot environment and the at least one second scene of the robot environment. The operations also include changing the direction of navigation of the mobile robot in response to an operator changing between the first scene of the robot environment and the at least one second scene of the robot environment.

[0028] In other examples of the system, the scene is a first-person view of the robot environment, and the operations further include displaying the scene and a top-down scene of the robot environment captured by an imaginary camera positioned above the mobile robot. The operations further include, in response to a pixel position selection in one of the displayed scenes, mirroring the pixel position selection in another of the displayed scenes.

[0029] In some examples of the system, the operation of transmitting the physical location in the robot environment includes transmitting a positioning command with the physical location to the mobile robot. Optionally, the camera can have a fisheye camera and two infrared cameras. In addition, the mobile robot can have four legs.

[0030] Another aspect of the present disclosure provides a method for "pre-rotating" a mobile robot. The method includes receiving, at data processing hardware, a real image of a front view of a robot environment captured by one or more real cameras disposed on the mobile robot. The mobile robot is in a forward orientation and faces in a forward direction. The method also includes determining, by the data processing hardware, a preview angle to which the mobile robot will rotate when it rotates from the forward orientation to a rotated orientation, wherein the mobile robot faces in a right direction or a left direction. The method also includes determining, by the data processing hardware, for each point on a virtual rectangle projected by the imaginary scene camera, whether the corresponding point corresponds to a pixel in the real image, and displaying, by the data processing hardware, a preview scene including the corresponding pixels. The preview scene being displayed is a view of the robot environment, the preview scene being rotated by a preview angle in a right direction or a left direction, while the mobile robot is not actually rotated from the forward orientation to the rotated orientation.

[0031] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the method further includes coloring the corresponding point on the virtual rectangle with the image color from the corresponding pixel by the data processing hardware when the corresponding point on the virtual rectangle corresponds to a pixel in the real image. The method further includes coloring the corresponding point black by the data processing hardware when the corresponding point on the virtual rectangle does not correspond to any pixel in the real image. The method further includes rendering the preview scene based on the color in the virtual rectangle by the data processing hardware.

[0032] In other examples of the method, determining the preview angle includes receiving, by the data processing hardware, a virtual velocity corresponding to a user input provided by an operator for controlling the mobile robot to turn from a forward orientation to a turned orientation. These examples of the method also include integrating, by the data processing hardware, the virtual velocity over time to generate the preview angle.

[0033] Some examples of the method also include initiating a countdown timer by the data processing hardware in response to a user input provided by an operator for controlling the mobile robot to rotate from the forward orientation to the rotated orientation. These examples of the method also include continuously determining, by the data processing hardware, a preview angle by which the view of the robot environment is to be rotated based on the user input provided by the operator before the countdown timer expires. These examples of the method also include transmitting, by the data processing hardware, a rotation angle equal to the preview angle to the mobile robot when the countdown timer expires, so that the mobile robot rotates the rotation angle from the forward orientation to the rotated orientation.

[0034] In other examples of the method, a real image of a front view of the robot environment is captured by a left front camera and a right front camera provided on the mobile robot. Optionally, the camera may have a fisheye camera and two infrared cameras. In addition, the mobile robot may have four legs.

[0035] Another aspect of the present disclosure provides a system for "pre-rotating" a mobile robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations that include receiving a real image of a front view of a robot environment captured by one or more real cameras disposed on the mobile robot, wherein the mobile robot is in a forward orientation and faces a forward direction. The operations performed by the data processing hardware also include determining a preview angle that the mobile robot will rotate to when it rotates from a forward orientation to a rotated orientation, wherein the mobile robot faces a right direction or a left direction. The operations performed by the data processing hardware also include, for each point on a virtual rectangle projected by an imaginary scene camera, determining whether the corresponding point corresponds to a pixel in the real image, and displaying a preview scene including the corresponding pixel. The preview scene is a view of the robot environment that is rotated by a preview angle in a right direction or a left direction, while the mobile robot is not actually rotated from a forward orientation to a rotated orientation.

[0036] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes coloring the corresponding point with an image color from the corresponding pixel when the corresponding point on the virtual rectangle corresponds to a pixel in the real image. The operation further includes coloring the corresponding point black by the data processing hardware when the corresponding point on the virtual rectangle does not correspond to any pixel in the real image. The operation also renders the hardware preview scene based on the color in the virtual rectangle.

[0037] In other examples of the system, the operation of determining the preview angle includes receiving a virtual velocity corresponding to a user input provided by an operator to control the mobile robot to turn from a forward orientation to a turned orientation. These examples of the system also include integrating the virtual velocity over time to generate the preview angle.

[0038] In some examples of the system, the operations also include starting a countdown timer in response to an operator providing a user input for controlling the mobile robot to rotate from the forward orientation to the rotated orientation. In these examples of the system, the operations also include continuously determining a preview angle of rotation of the view of the robot environment based on the user input provided by the operator before the countdown timer expires. In these examples of the system, the operations also include transmitting a rotation angle equal to the preview angle to the mobile robot when the countdown timer expires, so that the mobile robot rotates the rotation angle from the forward orientation to the rotated orientation.

[0039] In other examples of the system, a real image of the front view of the robot environment is captured by a left front camera and a right front camera set on the mobile robot. Optionally, the camera can have a fisheye camera and two infrared cameras. In addition, the mobile robot can have four legs.

[0040] One aspect of the present disclosure provides a computer-implemented method, comprising: receiving, by data processing hardware of an operator device, image data from at least one image sensor, the image data corresponding to an environment of a robot; and instructing, by the data processing hardware, to display a graphical user interface via a screen of the operator device, the graphical user interface being configured to: display a scene based on the image data; receive a rotation input to rotate a field of view of the environment; and display a preview scene by rotating the field of view without requiring physical movement of a body of the robot.

[0041] One aspect of the present disclosure provides a system, comprising: data processing hardware of an operator device, in communication with a robot; and memory hardware, in communication with the data processing hardware, the memory hardware storing instructions, wherein the instructions are executed by the data processing hardware so that the data processing hardware: receives image data from at least one image sensor, the image data corresponding to an environment of the robot; and instructs display of a graphical user interface via a screen of the operator device, the graphical user interface being configured to: display a scene based on the image data; receive a rotation input to rotate a field of view of the environment; and display a preview scene by rotating the field of view without requiring physical movement of a body of the robot.

[0042] One aspect of the present disclosure provides a computer-implemented method, comprising: receiving, by data processing hardware, first image data from a first image sensor located at a first portion of a body of a robot; receiving, by the data processing hardware, second image data from a second image sensor located at a second portion of the body of the robot; instructing, by the data processing hardware, to display the first image data and the second image data via a user interface of a computing device; receiving, by the data processing hardware, input via the computing device; and instructing, by the data processing hardware, the robot to navigate to a waypoint based on the input.

[0043] One aspect of the present disclosure provides a system comprising: data processing hardware; and a memory in communication with the data processing hardware, the memory storing instructions, wherein execution of the instructions by the data processing hardware causes the data processing hardware to: receive first image data from a first image sensor located at a first portion of a body of a robot; receive second image data from a second image sensor located at a second portion of the body of the robot; instruct display of the first image data and the second image data via a user interface of a computing device; receive input via the computing device; and instruct the robot to navigate to a waypoint based on the input.

[0044] One aspect of the present disclosure provides a legged robot, comprising: a body; a first image sensor and a second image sensor, wherein the first image sensor is located at a first portion of the body, and wherein the second image sensor is located at a second portion of the body; data processing hardware; and a memory, in communication with the data processing hardware, the memory storing instructions, wherein execution of the instructions by the data processing hardware causes the data processing hardware to: receive first image data from the first image sensor; receive second image data from the second image sensor; provide the first image data and the second image data for display via a user interface of a computing device; and instruct the legged robot to navigate to a waypoint based on input received via the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a block diagram of an example system for navigating a robot.

[0046] Figure 2 is a schematic diagram of an example mobile device that an operator can use to navigate a robot.

[0047] Figures 3A-3E is a schematic diagram of an example of a robot that can be operated using the systems and methods described herein.

[0048] Figure 4A-4B is a block diagram of an example of a touch to go application for navigating a robot.

[0049] Figures 5A-5H is a diagram of an example method for a touch-and-go application to determine waypoints.

[0050] Figures 6A-6C is a screenshot of an operator using an example of the Touch and Go application to navigate a robot, where the robot avoids obstacles.

[0051] Figures 7A-7C is a screenshot of an operator using an example of the Touch and Go app to navigate a robot, where the robot goes around a corner.

[0052] Figures 8A-8D is a screenshot of an operator using an example of the Touch and Go application to change the robot's direction of travel by changing the scene.

[0053] Figures 9A-9D is a screenshot of an operator using an example of the Touch and Go application to change the robot's direction of travel by selecting a scene from multiple scenes.

[0054] Figures 10A-10B is a screenshot of an operator using an example of the Touch and Go app to navigate the robot from a top-down scenario.

[0055] Fig.11A and11B is a screenshot of an operator using an example of the Touch and Go app to navigate the robot using a split screen.

[0056] Fig. 12A and 12B is a screenshot of an operator using an example of the Touch and Go app to “pre-rotate” the robot.

[0057] Figures 13A-13E is a block diagram of an example method for a touch-and-go application of a "pre-rotation" robot.

[0058] Figures 14A-14H is a combination of the example scene and the example robot environment displayed on the graphical user interface during the pre-rotation process.

[0059] Fig.15 is a flow chart of an example arrangement of operations of a method for navigating a robot.

[0060] Fig.16 is a flow chart of an example arrangement of operations for a method of pre-rotating a robot.

[0061] Fig.17 is a flow chart of an example arrangement of operations for a method of controlling a robot.

[0062] Fig.18 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0063] Like reference symbols in the various drawings indicate like elements.

[0064] Specific implementation method

[0065] Figure 1 A system 100 for navigating a robot 300 is shown in which an operator 10 uses a mobile device 200 (also referred to herein as an operator device 200) to navigate the robot 300 within a robotic environment. The robotic environment generally refers to a spatial region associated with some type of terrain in which the robot 300 can travel. For example, the terrain may include urban terrain with buildings, streets, sidewalks, parks, alleys, stairs, ramps, tunnels, etc.; rural terrain with roads, fields, hills, mountains, caves, etc.; underground terrain with caves, tunnels, etc.; or any combination thereof.

[0066] The robot 300 collects information about the robot environment around the robot 300 and information associated with operating in the robot environment. The robot 300 sends some or all of this information to the mobile device 200, including an image of the robot environment captured by the robot 300. The mobile device 200 then displays the image to the operator 10 so that the operator 10 can observe the robot environment in which the robot 300 is operating, which in this example is the interior of a warehouse. In some examples, the operator 10 observes the robot environment from the perspective of the robot 300. This “first-person” experience may be beneficial to certain applications of the system 100. The mobile device 200 displays the image in the user interface 221 ( Figure 2 ) displays the robot environment 301( Figure 2 ) image, and the operator 10 may provide navigation input (e.g., a waypoint command W) indicating a location within the selected image that corresponds to a physical location in the robot environment 301 to which the operator 10 wishes the robot 300 to navigate (e.g., a target location T). In some examples, the navigation input (e.g., a waypoint command W) indicating a location within the selected image is sent from the mobile device 200 to the robot 300 as a point in world space corresponding to a starting point from which a scene of the robot environment 301 is rendered, and a direction in world space corresponding to the location within the image selected by the operator 10. For example, the mobile device 200 may execute a method for selecting a location within the image on the screen 220 ( Figure 2 ) displays an image of the robot environment on a graphical user interface (GUI) 221, and the operator 10 can provide navigation input by touching a location within the image to navigate the robot 300 to the target location T. In additional examples, the operator 10 provides navigation input via a physical button, a mouse, a joystick, or any other input mechanism. As used herein, navigation input refers to an input indication of selecting a location with an image corresponding to the target location T (e.g., a physical location).

[0067] In response to receiving the navigation input, the mobile device 200 sends a positioning command to the robot 300, which instructs the robot 300 to navigate to the target location in the robot environment. When navigating to the target location in response to receiving the positioning command, the robot 300 may need to avoid obstacles (e.g., pillars) or go around corners. Taking into account the constraints of the robot environment, the robot 300 can plan how to navigate to the target location. In some scenarios, the robot 300 determines that it cannot navigate to the target location selected by the operator 10, in which case the mobile device 200 can notify the operator 10 (e.g., by displaying a notification on the GUI and / or providing an audible alarm).

[0068] As the robot 300 moves toward the target location, the characteristics of the robot environment relative to the robot 300 may change (e.g., change from flat terrain to hilly terrain). Accordingly, the robot 300 may continuously or at regular intervals send updated environmental information to the mobile device 200, including updated image data reflecting changes in the robot environment. The mobile device 200 may display the updated image data to allow the operator 10 to observe the progress of the robot 300 as it travels through the robot environment toward the target location.

[0069] In some implementations, the mobile device 200 is configured to display a destination image showing the robot 300 at the target location upon reaching the target location. The operator 10 can then provide new navigation input in the destination image corresponding to a new target location in the robot environment to which the operator 10 wants the robot 300 to navigate. The foregoing operations are repeated until the operator 10 completes navigation of the robot 300 (e.g., the robot 300 has reached the final destination in the robot environment).

[0070] When navigating the robot 300, it may be necessary to turn the robot 300 in the robot environment and change from a current orientation to a target orientation. The operator 10 clicks / touches a direction in the image that corresponds to the target direction that the operator 10 wants the robot 300 to turn from the current orientation (e.g., left or right) in the robot environment. In response, the mobile device 200 displays an updated image of the view that is turned in the direction corresponding to the target direction. This gives the operator 10 the impression that the robot 300 is turning from the current orientation to the target orientation. The updated image of the view displayed in the direction corresponding to the target direction can occur without actually turning / maneuvering the robot from the current orientation, thereby allowing the operator 10 to observe the robot environment in the updated image from the perspective of the target orientation.

[0071] In fact, the robot 300 does not immediately perform any turning maneuvers, so that the actual orientation of the robot 300 remains fixed to the current orientation in the robot environment. The mobile device 200 continues to rotate the view until the operator 10 stops clicking / touching the direction and stops at the target orientation. The rotation angle separates the current orientation and the target orientation.

[0072] Thereafter, the mobile device 200 may send a rotation command to the robot 300 instructing the robot 300 to perform a turning maneuver in the robot environment based on the rotation angle set by the operator 10. Pre-rotating the robot 300's view of the robot environment and displaying the view on the mobile device 200 provides immediate feedback to the operator 10, allowing the operator 10 to see the end result of rotating the robot 300 before the robot 300 performs the turning maneuver. This "stick and carrot" approach may be beneficial in applications of the system 100 where there is a latency or delay from the time the operator 10 issues a command to the time the robot 300 executes the command.

[0073] Figure 2 An example of a mobile device 200 used by an operator 10 to navigate a robot 300 around a robotic environment is shown. The mobile device 200 includes a housing 210, as well as a display 220 and a plurality of controls 230 located around the display 220. As shown, the controls 230 are located at the front and top edge of the housing 210 and include buttons 230a, a joystick 230b, a directional d-pad 230c, and shoulder buttons 230d. The housing 210 is shaped to be held in both hands of the operator 10 so that the thumb of the operator 10 is positioned to operate the controls located at the front of the mobile device 200 (e.g., joystick 230b), and the index finger of the operator 10 is positioned to operate the controls located at the top edge of the mobile device 200 (e.g., shoulder buttons 230d). The mobile device 200 may also include one or more additional controls located at the back of the mobile device 200 in locations operated by other fingers of the operator 10.

[0074] A display (eg, screen) 220 displays an image or scene 222 of the robot's environment to the operator 10 based on image data captured by the robot 300. In some examples, the display 220 presents the scene 222 as a video feed.

[0075] The mobile device 200 also includes a processing device 240, a storage device 250 (e.g., a memory device), and a communication interface 260 inside the housing 210. In the example shown, the processing device 240 executes a GUI 221 for display on the screen 220 to present a scene 222 of the robot environment for the operator 10 to view. The mobile device 200 may include Figure 2240 may include one or more processors that execute computer-executable instructions and associated memory (e.g., RAM and / or ROM) that stores computer-executable instructions. In implementations where processing device 240 includes more than one processor, the processors may execute in a distributed or separate manner. Processing device 240 may execute operating system 242 and touch-and-go application 400 ( Figure 4A ) for navigating the robot 300 in the robot environment. In some examples, the processing device 240 executes other applications 244, which may be local or non-local to the mobile device 200, such as a web browser, all of which may be implemented as computer-readable instructions.

[0076] The storage device 250 may include one or more computer-readable media (e.g., a hard drive, a solid-state memory drive, and / or a flash drive). The storage device 250 may store any suitable data used by the operating system 242, other applications 244, and the touch-and-go application 400. The communication interface 260 includes one or more devices configured to communicate with the robot 300 in the robot environment. In some implementations, the communication interface 260 is configured to communicate with the robot 300 over a network. The communication interface 260 may include one or more transceivers for wired or wireless communication. Examples of the communication interface 260 may include, but are not limited to, a transceiver configured to communicate using the IEEE 802.11 wireless standard, an Ethernet port, a wireless transmitter, and a universal serial bus (USB) port.

[0077] refer to Figure 3A and 3B , the robot 300 includes a body 310 and a movement system 320 coupled to the body 310 for moving the robot 300 within the robot environment. The robot 300 also includes a control system 330, which is communicatively coupled to the sensor system 340. The control system 330 determines how the robot 300 moves within the robot environment based on data collected by the sensor system 340. The robot 300 also includes electrical components 350, one or more power sources 360, and one or more communication links 370 for implementing various operations of the robot 300.

[0078] The robot 300 also includes data processing hardware 312 and memory hardware 314 housed within the body 310. The data processing hardware 312 can operate as one or more general purpose processors or special purpose processors (e.g., digital signal processors, application specific integrated circuits, etc.). The data processing hardware 312 can be configured to execute computer readable program instructions 316 that are stored in the memory hardware 314 and executable to provide the operation of the robot 300 described herein. For example, the program instructions 316 can be executed to provide the operation of the control system 330, wherein the control system 330 can be configured to cause activation and deactivation of the mobile system 320 and the electrical components 116. The data processing hardware 312 can operate and enable the robot 300 to perform various functions, including the functions described herein. Although the robot 300 implements the data processing hardware 312 and memory hardware 314 in the example shown, the data processing hardware 312 and / or the memory can reside on a remote system 190 ( Figure 4B ). Similarly, data processing hardware 312 and / or memory may be distributed among robot 300 and remote system 190 ( Figure 4B ).

[0079] Memory hardware 314 may exist as various types of storage media. For example, memory hardware 314 may include or take the form of one or more computer-readable storage media that can be read or accessed by data processing hardware 312. One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be integrated with data processing hardware 312 in whole or in part. In some implementations, memory hardware 314 may be implemented using a single physical device (e.g., an optical, magnetic, organic, or other memory or disk storage unit), while in other implementations, memory hardware 314 may be implemented using two or more physical devices that may communicate via wired or wireless communications. In addition, in addition to computer-readable program instructions 316, memory hardware 314 may include additional data such as diagnostic data.

[0080] The mobile system 320 represents the hardware of the robot 300 that enables the robot 300 to operate and perform physical functions. As a few examples, the mobile system 320 may include (multiple) actuators, (multiple) extendable legs ("legs"), (multiple) arms, (multiple) wheels, and / or other mechanical components. The mobile system 320 may depend on the design of the robot 300. In addition, the mobile system 320 may be based on the functions and / or tasks that the robot 300 is configured to perform. Therefore, different mobile systems may be used by the robot 300 depending on the operation and function of the robot 300. In some examples, the robot 300 may be configured to add and / or remove components of the mobile system 320, which may involve assistance from a user and / or another robotic device. For example, the robot 300 may initially be configured with four legs and operate as a quadruped, but may be changed by the user or the robot 300 to remove two of the four legs to operate as a biped. Other examples of the mobile system 320 may be included.

[0081] exist Figure 3A In the example shown, the mobile system 320 includes four legs 322 coupled to the body 310. Each of the four legs 322 includes an upper portion 324 and a lower portion 326 separated by a leg joint 328. The lower portion 326 of each leg 322 terminates in a foot 329. The foot 329 of each leg is optional, and the end of the lower portion 326 of one or more legs 322 can be coupled to a wheel. The robot 300 has a vertical gravity axis V along the direction of gravity. g and the center of mass CM, which is the point where the sum of the weighted relative positions of the distributed masses of the robot 300 is zero. z Vertical to the gravity axis V g The robot 300 also has a vertical gravity axis V g The legs 322 may be moved relative to the body 310 to change the posture P of the robot 300 (i.e., a combination of the positioning of the robot's CM and the posture or orientation of the robot 300).

[0082] In some implementations, the robot 300 also includes one or more appendages, such as an articulated arm disposed on the body 310 and configured to move relative to the body 310. The articulated arm can have five or more degrees of freedom. In addition, the articulated arm can be interchangeably referred to as a manipulator arm or simply an appendage. In some examples, the articulated arm includes two parts that can rotate relative to each other and relative to the body. In other examples, the articulated arm can include more or fewer parts without departing from the scope of the present disclosure. The first part can be separated from the second part by an articulated arm joint. An end effector, interchangeably referred to as a manipulator head, can be coupled to the distal end of the second part of the articulated arm and can include one or more actuators for grasping / grasping an object.

[0083] The control system 330 serves as a link between parts of the robot 300, such as a link between the movement system 320 and / or the sensor system 340. In some cases, the control system 330 serves as an interface between the robot 300 and another computing device, such as the mobile device 200 ( Figure 2 ). Therefore, the control system 330 can be used as an interface between the robot 300 and the operator 10 using the mobile device 200. Figure 3C An example of a control system 330 is shown, which includes a traversal system 380 and a perception system 390. The traversal system 380 is responsible for coordinating and / or moving the robot 300 around the robot environment. The perception system 390 receives image data or sensor data 342 from the sensor system 340 (sensing system / camera) and generates one or more maps 394 indicating obstacles in the surrounding robot environment.

[0084] As the robot 300 moves around the robot environment, the traversal system 380 may analyze the terrain, plan the movement trajectory of the robot 300, and / or instruct the robot 300 to perform various movements. The traversal system 380 may use various systems of the robot 300 to attempt to successfully traverse the robot environment while avoiding collisions and / or damage to the robot 300 or the robot environment.

[0085] In some implementations, such as Figure 3C As shown, the traversal system 380 includes at least one controller 382, ​​a path generator 384, a pace planner / locator 386, and a body planner 388. The traversal system 380 is configured to communicate with the sensor system 340 and the perception system 390.

[0086] The controller 382 is configured to control the movement of the robot 300 to traverse around the robot environment based on input or feedback from the robot 300's systems (e.g., the traversal system 380, the perception system 390, etc.) and input from the operator 10 through the mobile device 200. This can include movement between postures and / or behaviors of the robot 300. For example, the controller 382 controls different footstep patterns, leg patterns, body movement patterns, or vision system sensing patterns. Here, the controller 382 is configured to communicate with the data processing hardware 312 and / or memory hardware 314 of the robot 300 to execute movement provided from the robot 300's systems (e.g., the traversal system 380, the perception system 390, etc.) or from the operator 10.

[0087] In some examples, the controller 382 includes multiple controllers 382, ​​each of which has a fixed cadence. A fixed cadence refers to a fixed timing of the steps or swing phases of the legs 322. For example, the controller 382 instructs the robot 300 to move the legs 322 (e.g., take a step) at a specific frequency (e.g., one step every 250 milliseconds, 350 milliseconds, etc.). Using multiple controllers 382, ​​each of which has a fixed cadence, the robot 300 can experience variable timing by switching between controllers 382. In some implementations, the robot 300 continuously switches / selects fixed cadence controllers 382 (e.g., reselects controllers 382 every three milliseconds) as the robot 300 traverses the robot environment.

[0088] The traversal system 380 may select a controller 382 based on a step plan of the controller 382. The traversal system 380 may generate a step plan for each controller 382 based on a cadence (i.e., a movement pace programmed for a given controller) and at least one steering command to move the robot 300 around the robot environment. If there are no obstacles or collision areas in the terrain around the robot environment based on the cadence of the controller 382, ​​then the step plan refers to a map (referred to as an unconstrained map) of unconstrained foot placements corresponding to where the robot 300 will step (i.e., where the distal end of the foot 329 or leg 322 is placed). Given the step plan, the traversal system 380 compares the step plan to a constraint map 394 generated by the perception system 390. Based on the comparison, the traversal system 380 selects a controller 382 whose step plan violates the least amount of constraints identified within the constraint map 394 (i.e., the step plan that requires the least amount of deviation to achieve the constraint map 394). By selecting the controller 382 with the smallest deviation, the traversal system 380 can ensure that the robot 300 requires the fewest adjustments to avoid obstacles or collision areas.

[0089] In some implementations, the traversal system 380 scores each controller 382 based on the relationship between the step plan and the constraint map 394, and selects the controller 382 with the best score. For example, the traversal system 380 selects the controller 382 with the lowest score, which corresponds to the controller 382 with the step plan that achieves the minimum deviation from the constraint map 394. In some examples, in addition to the relationship between the step plan and the constraint map 394, the score corresponds to a cost function based on the soft constraints or conditions of the controller 382. The traversal system 380 can also be configured with conditions that exclude specific controllers 382 before scoring and / or selection. Some examples of conditions include whether the desired contact with the terrain can be achieved based on the position of the swing foot, or whether a specific step height is required during the traversal. In some configurations, the cost function applies weights to different conditions, where the deviation between the step plan and the constraint map is the most important condition.

[0090] In some examples, the memory hardware 314 stores the movements or movement constraints of the traversal system 380 locally on the robot 300. In other examples, these movements or constraints are stored and / or accessed remotely by the traversal system 380 (see below). Figure 4B describe).

[0091] Continue to see Figure 3C , the path generator 384 is configured to determine horizontal movement of the robot 300. For example, horizontal movement refers to translation and / or deflection of the robot 300. The path generator 384 determines obstacles within the robot environment around the robot 300 based on the sensor data 342. The path generator 384 transmits the obstacles to the step planner / locator 386 so that the step planner / locator 386 can identify the footstep placement of the leg 322 of the robot 300 (e.g., the location of the distal end / terminal end of the leg 322 or the location of the foot 329). The step planner / locator 386 uses input from the perception system 390 to generate the footstep position (i.e., the location where the robot 300 should step).

[0092] The main body planner 388 determines the height and pitch of the robot 300 based on inputs such as the constraint map 394 (or terrain map) and the footstep positions from the footstep planner / localizer 386, and the constraint set. In some implementations, to determine the height and pitch of the robot 300, the main body planner 388 is a quadratic programming trajectory planner that is configured to determine the optimal trajectory of the robot 300 that satisfies the constraint set based on the inputs.

[0093] In some examples, the perception system 390 includes a map generator 392 that generates a map based on sensor data 342 of the terrain surrounding the robot environment. Based on the generated map, the perception system 390 generates a constraint map 394 for the robot 300 (e.g., for execution by the traversal system 380 of the robot 300). The perception system 390 typically transmits the constraint map 394 or a portion of the constraint map 394 (e.g., collision / non-collision areas) to the traversal system 380 (e.g., to the traversal system 380's step planner / localizer 386). The step planner / localizer 386 refers to the robot 300's trajectory planner (also known as motion planning), which is configured to plan a motion trajectory that avoids collisions when traversing the robot environment.

[0094] In some implementations, the map generator 392 generates an incomplete map and is configured to fill in gaps in the sensor data 342 based on the properties of the sensor data 342. In other words, the map generator 392 of the perception system 390 infers details about the obstacle terrain (also referred to as inferred terrain) from the sensor data 342 collected for the visible terrain. Based on the sensor data 342, the map generator 392 is programmed to assume that the missing sensor data corresponds to smooth terrain or flat terrain. When the sensor data 342 indicates that a near object is adjacent to a far object, the map generator 392 assumes that this near-far contrast occurs due to occlusion by the sensor system 340 within the robot environment. When the map generator 392 assumes that occlusion occurs, the map generator 392 fills in the gaps in the sensor data 342 by mapping the gaps to flat terrain. Conversely, when the sensor data 342 does not indicate a near-far contrast, the map generator 392 assumes that the missing sensor data 342 is due to poor vision of the sensor system 340 and maps the missing sensor data 342 to smooth terrain.

[0095] The sensor system 340 includes one or more sensors, such as force sensors, proximity sensors, motion sensors, load sensors, positioning sensors, touch sensors, depth sensors, ultrasonic ranging sensors, infrared sensors, etc. The sensor system 340 can provide sensor data to the data processing hardware 312 to allow the robot 300 to interact appropriately with the robot environment and monitor the operation of the system of the robot 300. The sensor data can be used for the control system 330 and / or computing system of the robot 300 to evaluate various factors of activation and deactivation of the movement system 320 and the electrical component 350.

[0096] The sensor system 340 may provide information indicative of the robot environment of the robot 300 to the control system 330 and / or computing system (e.g., an operator 10 using the touch-and-go application 400 running on the mobile device 200 to navigate the robot 300) for use in determining the operation of the robot 300. For example, the sensor system 340 may capture data corresponding to the topography of the environment or the location of nearby objects, which may assist in environmental recognition and navigation, etc. The sensor system 340 may monitor the robot environment in real time and detect obstacles, terrain elements, weather conditions, temperature, and / or other parameters of the environment for the robot 300.

[0097] Additionally, the robot 300 may include other sensor(s) configured to receive information indicative of a state of the robot 300, including sensor(s) that may monitor the state of various components of the robot 300. The sensor(s) may measure the activity of the system of the robot 300 and receive information based on the operation of various features of the robot 300, such as the operation of extendable legs, arms, or other mechanical and / or electrical features of the robot 300. The sensor data provided by the sensor system 340 may enable the control system 330 of the robot 300 to determine errors in operation and monitor the overall functionality of the components of the robot 300.

[0098] For example, the control system 330 may use the sensor data to determine the stability of the robot 300 during operation, as well as measurements related to power levels, communication activity, components requiring maintenance, and other information. As an example configuration, the robot 300 may include (multiple) gyroscopes, (multiple) accelerometers, and / or other possible sensors to provide sensor data related to the operating state of the robot 300. In addition, the sensor system 340 may also monitor the current state of the function that the robot 300 may currently be operating, such as gait. In addition, the sensor system 340 may measure the distance between a given robot leg of the robot and the center of mass of the robot. Other example uses of the sensor system 340 may also exist.

[0099] exist Figure 3A and Figure 3D In the example shown, the robot 300 includes five cameras: a left front camera 344a; a right front camera 344b; a left camera 344c; a right camera 344d; and a rear camera 344e. In other examples, the robot 300 can have more than five cameras (e.g., seven) or less than five cameras (e.g., three). For example, the robot 300 can further include one or more payload cameras 344 mounted on the robot. In one example, the payload camera includes a ring camera. Each camera 344 has a corresponding field of view, referred to as a "view", which defines the sensing range or area of ​​the camera 344. The cameras 344 are positioned around the body 310.

[0100] Special References Figure 3D The main body 310 has an X-axis A extending in the left-right direction. X and a Y axis A extending longitudinally in the front-to-back direction Y The left front camera 344a and the Y axis A Y The left front camera 344a is angled and positioned on the body 310 so that the left front camera 344a points to the right front direction and captures the right front view 349b. For example, the left front camera 344a is positioned at the left front portion of the body 310. The right front camera 344b is aligned with the Y axis A. Y The left front camera 344a is at an angle and is located on the subject 310 so that the right front camera 344b points in the left front direction and captures the left front view 349a. For example, the left front camera 344a is located at the left front portion of the subject 310. With this arrangement of the left front camera 344a and the right front camera 344b, if a ray R1 is drawn from the right front camera 344b to the left front view 349a, and a second ray R2 is drawn from the left front camera 344a to the right front view 349b, the resulting rays R1 and R2 cross each other as shown. Therefore, it may be convenient to refer to this arrangement as a "crossed camera" arrangement.

[0101] The left camera 344c and the X-axis A X The left camera 344c is aligned and located on the main body 310 so that the left camera 344c points to the left direction and captures the left view 349c. For example, the left camera 344c is located on the left side of the main body 310. The right camera 344d is aligned with the X-axis A. X The right camera 344d is aligned and located on the body 310 so that the right camera 344d points to the right direction and captures the right view 349d. For example, the right camera 344d is located on the right side of the body 310. The rear camera 344e is aligned with the Y axis A Y Aligned and located on the main body 310, so that the rear camera 344e points to the rear direction and captures the rear view 349e. For example, the rear camera 344e is located at the rear of the main body 310.

[0102] In some examples, the robot 300 combines or "stitches" the left front view 349a and the right front view 349b together to form a front view 349f that is larger than the separate left front view 349a and the right front view 349b. With the wider front view 349f, the robot 300 can have a clearer understanding of what is in front of it, such as obstacles and other restrictions, which is particularly advantageous because the robot 300 moves in the forward direction most of the time. As shown, portions of the left front view 349a and the right front view 349b can overlap, which may be useful when stitching views. In another example, the robot 300 combines the left front view 349a, the right front view 349b, the left view 349c, the right view 349d, and the rear view 349e together to form a surrounding view that approximates a 360-degree view around the robot 300.

[0103] exist Figure 3A In the example shown, each camera 344 includes three individual cameras. For ease of reference, each arrangement of three cameras is referred to as a "camera triplet." Other examples of camera 344 may include more (e.g., four) or fewer (e.g., two) individual cameras. Example cameras that make up camera triplet 344 include, but are not limited to, stereo cameras, scanning light detection and ranging (LIDAR) sensors, scanning laser detection and ranging (LADAR) sensors, and infrared cameras. As an example, camera triplet 344 includes a fisheye camera 346 and two infrared cameras 348, 348a-b. Fisheye camera 346 has a wide axis and a narrow axis, and its layout is different for different camera triplets.

[0104] refer to Figure 3A and Figure 3E For the left, right and rear camera triplets 344c-e, the layout of the fisheye cameras 346c-e has a horizontal direction with respect to the X-axis A X and Y axis A Y The wide axis is roughly aligned with the Z axis A and has a vertical direction with the Z axis A z Roughly aligned narrow axis. Figure 3E As shown, this layout produces a rectangular-like field of view (i.e., left view 349c, right view 349d, and rear view 349e), wherein the horizontal field of view (V H ) is larger than the vertical field of view (V V ).

[0105] For the left front camera triplet 344a and the right front camera triplet 344b, the layout of the fisheye cameras 346a, 346b has a narrow axis in the horizontal direction and a wide axis in the vertical direction. Figure 3E As shown, this layout produces a rectangular-like field of view (i.e., left front view 349a and right front view 349b), wherein the horizontal field of view (V H ) is smaller than the vertical field of view (V V ). The front view 349f (ie, the result of combining the left front view 349a and the right front view 349b) resembles a square.

[0106] Infrared camera 348( Figure 3A ) detects infrared light. Each infrared camera 348 includes a projector that emits a pattern of infrared light onto a surface that may not have a pattern. The emitted pattern reflects off the surface of the object and is detected by the infrared camera 348. In this way, the infrared camera 348 can "see" texture on an object that does not have a texture.

[0107] When the field of view is surveyed with the sensor (e.g., with the left front view 349a of the right front camera 344b), the sensor system 340 generates sensor data 342 corresponding to the field of view. In some examples, the sensor data 342 includes image data corresponding to a three-dimensional volumetric point cloud generated by a three-dimensional volumetric image sensor. Additionally or alternatively, as the robot 300 maneuvers around the robot environment, the sensor system 340 collects pose data of the robot 300, which pose data includes inertial measurement data (e.g., measured by an inertial measurement unit). In some examples, the pose data includes kinematic data and / or orientation data about the robot 300. Using the sensor data 342, the perception system 390 can generate a map, such as a constraint map 394 of the terrain around the robot environment 10.

[0108] Sensor data 342 (such as image data, pose data, inertial data, kinematic data, etc.) collected by the sensor system 340 can be communicated to the traversal system 380 (e.g., data processing hardware 312 and memory hardware 314) of the robot 300. In some examples, the sensor system 340 collects and stores the sensor data 342 (e.g., in Figure 4B In another example, the sensor system 340 collects the sensor data 342 in real time and processes the sensor data 342 without storing the raw (i.e., unprocessed) sensor data 342. In yet another example, the traversal system 380 and / or the remote resource 192 ( Figure 4B ) stores both processed sensor data 342 and raw sensor data 342.

[0109] exist Figure 3B In the example robot architecture of FIG. 3 , data processing hardware 312 and memory hardware 314 are located centrally in the robot 300 so that various systems of the robot 300 can share the use of the hardware (e.g., control system 330, sensor system 340, traversal system 380, and perception system 390). In some configurations, one or more systems of the robot 300 (e.g., control system 330, sensor system 340, traversal system 380, and / or perception system 390) have their own dedicated hardware (e.g., in communication with data processing hardware 312 and memory hardware 314).

[0110] refer to Figure 3B, the robot 300 may include one or more power sources 360 configured to provide power to various components / systems of the robot 300. Among possible power sources, the robot 300 may include a hydraulic system, an electrical system, a battery, and / or other types of power sources. As an example, the robot 300 may include one or more batteries configured to provide power to the components / systems via a wired and / or wireless connection. In an example, the components of the mobile system 320 and the electrical components 350 may each be connected to a different power source, or may be powered by the same power source. The components of the robot 300 may also be connected to multiple power sources.

[0111] In an example configuration, any type of power source may be used to provide power to the robot 300, such as a gasoline engine. In addition, the power source(s) 360 may be charged using various types of charging, such as a wired connection to an external power source, wireless charging, combustion, or other examples. Other configurations are also possible. In addition, the robot 300 may include a hydraulic system configured to provide power to the mobile system 320 using fluid power. For example, the components of the robot 300 may be operated based on hydraulic fluid transmitted to various hydraulic motors and hydraulic cylinders through the hydraulic system. The hydraulic system of the robot 300 may transmit a large amount of power through small tubes, flexible hoses, or other links between the components of the robot 300. Other power sources may be included within the robot 300.

[0112] The electrical components 350 may include, for example, various components capable of processing, transferring, providing an electric charge or an electrical signal. In possible examples, the electrical components 350 may include wires, circuits, and / or wireless communication transmitters and receivers to enable the operation of the robot 300. The electrical components 350 may interact with the mobile system 320 to enable the robot 300 to perform various operations. For example, the electrical components 350 may be configured to provide power from (multiple) power sources 360 to components of the mobile system 320. In addition, the robot 300 may include an electric motor. Other examples of electrical components 350 may also exist.

[0113] In some implementations, the robot 300 may also include communication link(s) 370 that are configured to send and / or receive information. The communication link(s) 370 may transmit data indicating the status of various components / systems of the robot 300. For example, information read in by the sensor system 340 may be transmitted to a separate computing device (e.g., Figure 2Other diagnostic information indicating the integrity or health of the data processing hardware 312, the memory hardware 314, the control system 330, the movement system 320, the power supply(s) 360, and / or the electrical components 350 may be transmitted to an external device (e.g., a computer used by the operator 10 to navigate the robot 300) via the communication link(s) 370. Figure 2 mobile device 200).

[0114] In some implementations, the robot 300 receives information processed by the data processing hardware 312 at the communication link(s) 370. For example, the received information may indicate data accessible by the data processing hardware 312 during execution of the program instructions 316. Additionally, the received information may change aspects of the control system 330 that may affect the behavior of the movement system 320 or the electrical components 350. In some cases, the received information indicates a query requesting a specific piece of information (e.g., an operational status of one or more components / systems of the robot 300), and the data processing hardware 312 may then transmit the specific piece of information back to the communication link(s) 370.

[0115] Examples of the communication link(s) 370 include a communication link for connecting the robot 300 to an external device (e.g., a communication link used by the operator 10 to navigate the robot 300). Figure 2 The robot 300 may include one or more ports to connect the communication link(s) 370 to an external device (e.g., a wireless device 200) without any physical cables. Some example wireless connections may utilize a cellular connection, such as CDMA, EVDO, GSM / GPRS, 4G telecommunications (such as WiMAX or LTE), or 5G telecommunications. Alternatively or additionally, the wireless connection may utilize a Wi-Fi connection to transmit data to a wireless local area network (WLAN). In some implementations, the wireless connection may also communicate via an infrared link, Bluetooth, or a near field communication (NFC) device. In addition to or in lieu of a wireless connection, other examples of the communication link(s) 370 may include a wired connection. In these examples, the robot 300 may include one or more ports to connect the communication link(s) 370 to an external device (e.g., a Figure 2 mobile device 200).

[0116] Please note, Figures 3A-3E The robot 300 shown in the figure is for illustration purposes and may include more or fewer components without departing from the scope of the present disclosure. The various components of the robot 300 may be connected in any manner, including wired or wireless connections, etc. In addition, in some examples, the components of the robot 300 may be located on multiple different physical entities rather than on a single physical entity. Other example configurations of the robot 300 may also exist.

[0117] References Figures 3A-3E The robot 300 described and its examples represent example robotic devices configured to perform the operations described herein. In addition, the robot 300 can be configured to operate autonomously, semi-autonomously, and / or using directions provided by an operator 10 (e.g., via a mobile device 200), and can exist in various forms, such as a humanoid robot or a quadruped robot (e.g., a dog), among other examples. In addition, the robot 300 can also be referred to as a robotic device, a mobile robot, or a robot, among other names.

[0118] refer to Figure 4A , the touch-and-go application 400 executing on the mobile device 200 (e.g., executing on the processing device 240 (e.g., the data processing hardware 240 of the mobile device 200)) determines a target location T (e.g., a physical location) in the robot environment or a "waypoint" to which the operator 10 wants the robot 300 to navigate. From the sensor system 340 ( Figure 3B and 3C ), the touch and go application 400 receives the image data 342, camera position / location 404, and terrain estimate 406 of the robot 300. One or more cameras on the robot 300, such as those described above with reference to Figure 3D The fisheye camera 346 depicted may capture image data 342. The image data 342 corresponds to one or more fields of view (e.g., Figure 3D The mobile device 200 executes the GUI 221 to render the image data 342 as an image or scene 222 on the display 220 so that the operator 10 sees a view of the robot environment from the perspective of the robot 300. The operator 10 provides navigation input (e.g., input indication) pointing to a location in the scene 222 that corresponds to a physical or target location T in the robot environment to which the operator 10 wishes the robot 300 to navigate. For ease of reference, the location within the scene 222 that corresponds to the target location may be referred to as a “pixel location” 224.

[0119] From the pixel position 224 and the inputs 342, 404, 406, the touch-and-go application 400 determines a waypoint W (described in more detail below) and provides the waypoint W to the traversal system 380 ( Figure 3C ). For example, the touch and go application 400 transmits a positioning command with a waypoint to the robot 300. Here, the positioning command with a waypoint may correspond to a waypoint command. The traversal system 380 then uses the waypoint W together with the information from the sensor system 340 and the perception system 390 ( Figure 3C ) input to control and navigate the robot 300 to waypoint W, as shown in the reference above Figure 3C as described.

[0120] refer to Figure 4B, the system 100 may include network components. For example, the traversal system 380 communicates with the remote system 190 via the network 180. The remote system 190 may be a server or cloud-based environment that includes remote resources 192, such as remote data processing hardware 194 and remote memory hardware 196. In some implementations, movement or traversal constraints are stored and / or processed on the remote system 190 using the remote resources 192. These constraints are communicated to the traversal system 380 of the robot 300 via the network 180. In yet another example, different outputs associated with the traversal system 380 are remotely (e.g., via the remote system 190) and locally (e.g., via the memory hardware 314 ( Figure 3B The waypoint W determined by the touch-and-go application 400 may be transmitted to the robot 300 via the network 180 (eg, as a waypoint command).

[0121] refer to Figures 5A-5C The touch-and-go application 400 uses the horizontal and vertical positions of the pixel positions 224 to determine a navigation direction input by the operator 10 for maneuvering the robot 300 ( Figure 5A ). The horizontal position and the vertical position may be associated with the horizontal and vertical coordinates of the pixel position 224 within the scene 222, respectively. The navigation direction input provided by the operator 10 may be represented by a pointing vector V ( Figure 5B and 5C ). Thus, the touch and go application can determine a pointing vector V based on the selection of the pixel position 224 within the scene 222, wherein the pointing vector V represents a direction of travel for navigating the robot 300 in the environment 301. The pointing vector V can be further based on calibration information of at least one camera 344 that captures image data 242 corresponding to the scene 222, such as the camera position 404. The touch and go application 400 combines the pointing vector V with the ground plane estimate G of the robot 300 to find the pointing vector V and the ground plane estimate G ( Figure 5D ) between the intersection points I( Figure 5D ). When the intersection I between the pointing vector V and the ground plane estimate G is in front of the robot 300, as Figure 5D As shown, the touch-and-go application 400 sends / transmits the location of the intersection I as a waypoint (e.g., a target location) to the robot 300, so that the robot 300 navigates to the waypoint (e.g., autonomously). In this example, when the distance between the current location of the robot 300 and the intersection meets (e.g., is less than or equal to) a threshold distance (e.g., four meters), the touch-and-go application 400 can identify the intersection I as the target location. As used herein, a waypoint sent to the robot 300 by the application 400 corresponds to a waypoint command W, which instructs the robot 300 to navigate to the target location associated with the waypoint.

[0122] Alternatively, rather than sending an actual “waypoint” based on the intersection point I, the application 400 executing on the mobile device 200 sends a waypoint command W based on a selection of a pixel location 224 within the scene 222, the waypoint command W including a point in world space corresponding to the origin of at least one camera 344 capturing image data 242 corresponding to the scene 222 and a pointing vector V in world space. In this case, upon receiving a waypoint command W that includes only a point in world space and a pointing vector V, the robot 300 responsively travels in the direction of the pointing vector V until a distance threshold is reached or the robot 300 detects an obstacle that intersects the pointing vector V. Advantageously, the mobile device 200 does not have to determine a terrain estimate (e.g., a ground plane estimate G) used to calculate the actual “waypoint coordinates” sent to the robot 300, but rather allows the robot 300 to continuously recalculate a terrain estimate (e.g., a ground plane estimate) while navigating in the direction of the pointing vector V as the robot 300 approaches the target location T.

[0123] In some cases, the operator 10 selects a pixel position 224 corresponding to a target position in the robot environment 301 that exceeds (e.g., does not meet) a threshold distance or specified range (e.g., an autonomous navigation range of four meters) that the robot 300 is allowed, intended, or otherwise designed to navigate autonomously. The touch-and-go application 400 instructs the robot 300 to begin moving toward the target position and stop at an intermediate position before the target position (e.g., four meters from the starting position of the robot 300), rather than returning an error message requiring the operator 10 to select a new pixel position. Here, the distance between the position of the robot 300 when the pixel position 224 (i.e., the navigation direction input) is received and the intermediate position can correspond to the threshold distance that the robot 300 is allowed to navigate autonomously. Therefore, by enabling the robot 300 to move in a direction toward the target position without requiring the operator 10 to select another pixel position that meets the threshold distance, the user experience of the operator 10 controlling the robot 300 is improved.

[0124] refer to Figure 4B , 5E 5F, when the first distance D1 between the intersection point I between the pointing vector V and the ground plane estimate G in front of the robot 300 and the robot 300 does not meet (e.g., exceed) a threshold distance (e.g., four meters), the touch-and-go application 400 sends a waypoint command W to the robot 300 ( Figure 4B ), so that the robot 300 moves to the intersection I and moves the robot 300 to the intersection I M ( Fig. 5F ) between the shortened distance D2 ( Fig. 5F ). Fig. 5FIt is shown that the shortened distance D2 is equal to a second limit, which is equal to or less than a threshold distance (e.g., equal to four meters), which is less than the first limit ( Figure 5E In other words, when the distance between the intersection point I and the robot 300 does not meet the threshold distance, the application 400 may transfer or move the intersection point I closer to the robot 400 so that the resulting moved intersection point I M The touch and go application 400 moves the intersection I M The position of is sent as a waypoint command W to the robot 300, and then the robot 300 navigates to the waypoint by itself.

[0125] The operator 10 may select a pixel position 224 that corresponds to a position above the horizon in the robot environment, i.e., a position that is not on the ground. Rather than returning a warning or error message, the touch-and-go application 400 interprets the operator's 10 selection as a desire to navigate the robot 300 to a distant location, but the operator 10 accidentally selected a position above the horizon. Based on this interpretation, the touch-and-go application 400 instructs the robot 300 to move in the direction intended by the operator 10 without the operator 10 having to pick another position.

[0126] refer to Figure 4B , 5G 5H, when the pixel position 224 selected by the operator 10 is above the horizon, the intersection I between the pointing vector V representing the direction in which the operator 10 wants the robot 300 to navigate and the ground plane estimate G of the robot 300 is located behind the robot 300 ( Figure 5G ).in this case, Figure 5H The touch-and-go application 400 reflects the intersection I, so that the reflection intersection I R In front of the robot 300. Here, the distance between the robot 300 and the intersection point I behind the robot 300 is equal to the distance between the reflection intersection point I in front of the robot 300. R and the distance between the robot 300. In this way, the robot 300 is not instructed to move in a direction opposite to the expected direction. The touch and go application 400 will reflect the intersection I R Move closer to the robot 300 or farther away from the robot 300 so that the robot 300 is closer to the reflection intersection point (I R )( Figure 5H ) satisfies the threshold distance (eg, less than or equal to 4 meters). The touch and go application 400 reflects the intersection I R The position of is sent as a waypoint command to the robot 300, so that the robot 300 navigates to the intersection point I R Associated waypoints.

[0127] refer to Figures 6A-6C and Figures 7A-7CIn some cases, the operator 10 wants the robot 300 to navigate to a location where an obstacle O ( Fig. 6A ) or around the corner C ( Fig. 7A ) (e.g., target location T). For example, Fig. 6A The operator 10 is shown providing navigation input by inputting a pixel position 224 behind the obstacle O to cause the robot 300 to maneuver to the target position T behind the obstacle O. Similarly, Fig. 7A The operator 10 is shown providing navigation input by inputting the pixel position 224 around the corner C to cause the robot 300 to maneuver to the target position T, which includes a corridor or general area around the corner C. In these cases, the traversal system 380 with input from the sensor system 340 and the perception system 390 controls the robot 300 so that the robot 300 avoids the obstacle O by navigating around the obstacle O and maneuvering to the target position T ( Figure 6B and 6C ). That is, the traversal system 380 is configured to override the navigation input to allow the robot 300 to traverse around obstacles when feasible. In the event that traversal around obstacles is not feasible, the traversal system 380 may stop performing navigation maneuvers to the target location. In these cases, the traversal system 380 may notify the application 400 that navigation to the target location T is not feasible, and the application 400 may output a message (visual, tactile, and / or audio) notifying the operator 10 that the navigation input is not feasible.

[0128] Similarly, if Fig. 7A and 7B As shown, through input from the sensor system 340 and the perception system 390, the traversal system 380 controls the robot 300 to approach the corner ( Figure 7B ), then maneuver around corner C (e.g., relative to Figure 7C10) to reach a target location in the robot environment associated with the pixel location 224 input by the operator 10. Advantageously, the operator 10 can provide the pixel location 224 as a high-level command to reach the target location T, and the robot 300 is tasked with locating a side path to reach the target location T. In the example shown, the robot 300 bypasses the wall until it finds an opening around corner C to pass through to reach the target location. This obstacle avoidance functionality alleviates the operator 10 from having to provide multiple intermediate navigation inputs to navigate the robot 300 around each obstacle (e.g., obstacle O or corner C) in order to maneuver the robot 300 to the target location. Some implementations may rely on the control / traversal techniques disclosed in U.S. application Ser. No. 62 / 744,954, filed on Oct. 12, 2018, and entitled “Terrain and Constraint Planning for Footstep Plan,” and U.S. application Ser. No. 16 / 382,390, filed on Apr. 12, 2019, and entitled “Robotically Negotiating Stairs,” the entire contents of which are incorporated herein.

[0129] In some implementations, the Touch and Go application 400 allows the operator 10 to change the direction in which the robot 300 moves by changing the scene. Figure 3D and 3E As described above, each camera 344, 344a-e on the robot 300 surveys or captures a portion of the robot's environment within its corresponding field of view 349, 349a-f (or simply referred to as a "view") and provides image data 342 corresponding to the field of view. The touch-and-go application 400 renders the image data 342 into an image or scene so that the operator 10 sees the changes in the scene. For example, Fig. 8A and 8B A GUI 221 is shown presenting a front scene 222, 222a rendered from a front view 349f captured by the left and right front cameras 344a, 344b of the robot 300. The front scene 222a presents the robot environment in front of the robot 300 to the operator 10. Figure 8C and 8D A GUI 221 is shown presenting a rear scene 222, 222b rendered from a rear view 349e captured by a rear camera 344e of the robot 300. The rear scene 222b is displayed to the operator 10 so that the operator 10 can see what is behind the robot 300.

[0130] refer to Figure 8B and 8C, the application 400 may present a scene changer 226 graphical element (e.g., a button) on the GUI 221 to allow the operator 10 to change between different scenes and change the direction in which the robot 300 moves, as described in more detail below. In the example shown, the scene changer 226 graphical element includes four directional arrows that the operator 10 can select to switch between different scenes. For example, the operator 10 can provide an input indication indicating the selection of the forward directional arrow of the scene changer 226 to view the front scene 222a ( Figure 8B ), and then switch to observe the rear scene 222b ( Figure 8C ). The operator 10 may further switch between left and right scenes (not shown) by providing input indications indicating selection of left and right directional arrows of the scene changer 226, respectively. In these cases, based on the input indications received by the scene changer 226, the GUI 221 may present the left scene 222 or the right scene 222 for the operator 10 to observe before the touch-and-go application 400 sends a corresponding turn command to cause the robot to perform a turn maneuver associated with the direction change. Although the scene changer 226 only provides four options for changing scenes (e.g., front, back, left, and right), the touch-and-go application 400 may implement more or less than four options for changing scenes.

[0131] Figures 8A-8D An example of the GUI 221 is provided when the operator 10 uses the touch-and-go application 400 to navigate the robot 300 in the robot environment. Fig. 8A In FIG. 4 , the operator 10 is observing the front scene 222a and provides navigation input by selecting a pixel position 224 corresponding to a target position T to which the robot 300 is to navigate. In response to receiving the navigation input, the touch-and-go application 400 instructs (e.g., sends a waypoint command W) the robot 300 to move forward (relative to the position T) toward the determined target position T. Fig. 8A and 8B ), as referenced above Figures 5A-5F The robot 300 then navigates to the target position T associated with the pixel position 224 selected by the operator 10. When the robot 300 navigates to the target position T in the front scene 222a, Figure 8B The operator 10 is shown providing another navigation input by selecting another pixel position 224 corresponding to a new target position to which the robot 300 is to navigate. Thus, the operator 10 can continue to move the robot 300 in a forward direction by selecting a new pixel position 224 as the robot 300 traverses forward.

[0132] Figure 8CThe operator 10 is shown selecting the rear direction arrow of the scene changer 226 graphic element to view the rear scene 222b and cause the robot 300 to reverse the travel direction in the rear direction. Although the GUI 221 presents the rear scene 222b in response to the input indication of selecting the rear direction arrow of the scene changer 226, Fig.8D The operator 10 is shown providing navigation input by selecting a pixel position 224 corresponding to a target location T to which the robot 300 is to navigate. In response to receiving the navigation input associated with the pixel position 224 in the back scene 222b, the touch-and-go application 400 instructs (e.g., sends a waypoint command W) the robot 300 to move in a rear direction (e.g., backward) toward the target location T. In some examples, rather than moving backward, the robot 300 first performs a turning maneuver to turn substantially 180 degrees in response to receiving the waypoint command W to face the rear direction 222b as the new forward direction, and then moves forward toward the determined target location. The robot 300 may decide whether to traverse in the rear direction or first perform a turning maneuver to traverse in the forward direction based on one or more factors. For example, non-limiting factors may include the distance to the target location T, the feasibility of the robot 300 performing the turning maneuver, familiarity with the robot's environment, camera positioning 404 ( Figure 4A ) or user preference. In some configurations, the robot 300 is not configured to move in the rear direction, thereby requiring the robot 300 to first perform a turning maneuver so that it can move to the target position T by moving in the forward direction.

[0133] Figures 9A-9D An example of a GUI 221 of a touch-and-go application 400 is provided that renders a simultaneous display of a front scene 222a, a rear scene 222b, a right scene 222c, and a left scene 222d. Here, the operator 10 need not use a scene changer 226 to change scenes, but can provide navigation input to a target location T by selecting a pixel position 224 in a corresponding one of the available scenes 222, 222ad that present the target location T. For example, Fig.9A The operator is shown selecting a pixel position 224 from the foreground scene 222a corresponding to a target position T to which the robot 300 is to navigate. In response to receiving the navigation input, the touch-and-go application 400 instructs (e.g., sends a waypoint command W) the robot 300 to move forward (relative to the target position T) toward the determined target position T. Fig.9A and 9B ), as referenced above Figures 5A-5F described.

[0134] Fig. 9CThe operator is shown selecting a new pixel position 224 from the right scene 222 c that corresponds to a new target position T to which the robot 300 is to navigate. In response to receiving the navigation input associated with the pixel position 224 in the right scene 222 c, the Touch & Go application 400 instructs (e.g., sends a waypoint command W) the robot 300 to perform a turn maneuver to the right and then move forward toward the determined new target position T. Fig.9D GUI 221 is shown that renders a simultaneous display of scenes 222a-d as the robot 300 moves toward a new target position T after performing a right turn maneuver. That is, Fig. 9C The view of the robot environment depicted in each of scenes 222a-d is rotated 90 degrees so that Fig.9D The previous scene 222a corresponds to Fig. 9C Right scene 222c, Fig.9D The right scene 222c corresponds to Fig. 9C After scene 222b, Fig.9D The latter scene 222b corresponds to Fig. 9C The left scene 222d, and Fig.9D The left scene 222d corresponds to Fig. 9C Pre-scene 222a.

[0135] Fig. 10A and 10B An example of a GUI 221 of a touch-and-go application 400 is provided that renders a display of a top-down scene 222e of a robot 300. The application 400 may construct the top-down scene 222e using image data 342 captured from any combination of cameras 344, 344a-e on the robot 300 within their respective fields of view 349, 349a-f (or simply “views”). The image data 342 used to construct the top-down scene 222e may further be captured from a payload camera 344 mounted on the robot 300. In order to navigate the robot 300 to a target location T, Fig. 10A The operator 10 is shown providing navigation input by selecting a pixel position 224 within the top-down scene 222e that is associated with a target position T. Based on the pixel position 224, the touch-and-go application 400 is able to determine the target position T for navigating the robot 10. Thus, in response to receiving the navigation input, Fig. 10B The touch-and-go application 400 is shown to instruct (eg, send a waypoint command W) the robot 300 to move forward (relative to the direction of the determined target position T). Fig. 10A and 10B ), as referenced above Figures 5A-5FA graphic depicting the robot 300 may be rendered on the top-down scene 222e to display the current location of the robot 300 in the robot environment. For example, the graphic of the robot 300 may move to depict the instantaneous location of the robot 300 when traversing to the target location T.

[0136] In another example, Fig.11A and 11B The GUI 221 of the touch-and-go application 400 is displayed, which is configured to display a split screen 229 that simultaneously depicts a top-down scene 222e ( Fig. 10A and 10B ) and the user-selected scene 228. The orientation of the scenes 222e, 228 can be displayed on opposite sides of the split screen 229 without departing from the scope of the present disclosure. In some examples, the application 400 allows the operator 10 to customize the position of the scenes 222e, 228 rendered on the GUI 221 and / or adjust the window size of any one of the scenes 222e, 228. In the example shown, the user-selected scene 228 corresponds to the front scene 222a. In other examples, the user-selected scene 228 corresponds to any one of the rear scene 222b, the right scene 222c, or the left scene 222d. The GUI 221 can present a button (not shown) for allowing the operator 10 to select which scene 222a-d is rendered in the GUI 221 as the user-selected scene 228. Figures 8A-10B The GUI 221 of any of the above may similarly present one or more buttons for allowing the operator 10 to select between various different GUI configurations. For example, the operator 10 may prefer to use the top-down scene 222e to provide high-level navigation input, and then switch to the GUI 221 to render four scenes 222a-d simultaneously when fine-grained navigation input is needed.

[0137] To navigate the robot 300 to the target position T, Fig.11A The operator 10 is shown providing navigation input by selecting a pixel position 224 in a top-down scene 222e associated with a target location T or a pixel position 224 in a user-selected scene 228 associated with a target location T. Although the pixel position 224 in each of the scenes 222e, 228 is different, the touch-and-go application 400 is able to determine the same target location T for navigating the robot 300 regardless of which scene 222e, 228 the operator 10 uses to provide navigation input. In other implementations, the application 400 allows the operator 10 to provide navigation input to only one of the scenes 222e, 228. Based on the pixel position 224, the touch-and-go application 400 is able to determine the target location T for navigating the robot 10. Thus, in response to receiving the navigation input, Fig. 11BThe touch-and-go application 400 is shown to instruct (eg, send a waypoint command W) the robot 300 to move forward (relative to the direction of the determined target position T). Fig.11A and 11B ), as referenced above Figures 5A-5F A graphic depicting the robot 300 may be rendered on the top-down scene 222e to display the current location of the robot 300 in the robot environment. For example, the graphic of the robot 300 may move to depict the instantaneous location of the robot 300 when traversing to the target location T.

[0138] The user-selected scene 228 and the top-down scene 222e are synchronized so that the operator 10 simultaneously and in real time observes the robot environment relative to the robot environment in the first-person view 228 and the bottom-up scene 222e. In addition, selecting a pixel location 224 of a target location T in one of the scenes 222e, 228 may cause the other scene 222e, 228 to mirror the selection of a corresponding pixel location 224 of the target location T. Advantageously, with the split screen 229, the operator 10 has the option of using a particular view / scene to navigate the robot 300 appropriate for a given task, situation, and / or environment.

[0139] refer to Fig. 12A and 12B In some implementations, the Touch and Go application 400 is configured to allow the operator 10 to preview the scene 222, 222P of the robot environment. Fig. 12B ) without physically instructing the robot 300 to move. For example, when in the GUI 221 ( Fig. 12A ) presents the current scene 222, 222C on the GUI 221, the GUI 221 also presents the first and second rotating graphics 272, 274, when the first and second rotating graphics 272, 274 are selected by the operator 10, so that the GUI 221 presents the preview scene 222P by rotating the field of view in one of a first direction away from the current scene 222C or an opposite second direction ( Fig. 12B ). Alternatively or additionally, the operator 10 may use controls 230 on the mobile device 200, such as a joystick 230b or a directional pad 230c, to perform the functions of the first and second rotational graphics 272, 274. In this way, the GUI 221 may receive a rotation input that causes the field of view to rotate away from the current scene 222C when either of the rotational graphics 272, 274 and / or user input to the joystick 230b or the directional pad 230c is selected.

[0140] For example, the GUI 221 may receive a rotation input indicating a selection of the first rotation graphic 272 to rotate the field of view in a first direction (e.g., to the right) away from the current scene 222C to communicate a rightward turning maneuver of the robot 300 without the robot 300 physically moving or performing the turning maneuver. In some examples, the selection of the first rotation graphic 272 also causes the touch-and-go application 400 to command the robot 300 to perform a turning maneuver in the first direction. In these examples, the GUI 221 may responsively rotate the field of view in the first direction away from the current scene 222C to provide the operator 10 with immediate feedback that the robot 300 is performing the turning maneuver before the robot 300 actually performs the turning maneuver due to a lag (e.g., a communication lag between the mobile device 200 and the robot 300, and a lag caused by the time to physically control the robot 300 to perform the turning). In other examples, the selection of the first rotation graphic 272 only causes the field of view to rotate in the first direction away from the current scene 222C to allow the operator 10 to observe the preview scene 222D without physical movement of the robot 300. For example, the operator 10 can rotate the field of view in a first direction to observe the scene to the right of the robot 300 to assist the operator 10 in making navigation decisions before instructing the robot 300 to maneuver. In some examples, the amount by which the field of view rotates is proportional to the time that the first rotating graphic 272 is in contact with the operator 10 (e.g., by touching). For example, the field of view can rotate in the first direction until the operator 10 releases contact with the first rotating graphic 272. In another example, the field of view rotates by one increment each time the first rotating graphic 272 is touched and released. The application 400 can enhance (e.g., change the color of the first rotating graphic 272) to indicate when the graphic 272 is selected.

[0141] Similarly, the GUI 221 may receive a rotation input indicating a selection of the second rotation graphic 274 to rotate the field of view in a second direction (e.g., left) away from the current scene 222C to communicate a left turn maneuver of the robot 300 without the robot 300 physically moving or performing a turn maneuver. In some examples, the selection of the second rotation graphic 274 also causes the touch-and-go application 400 to command the robot 300 to perform a turn maneuver in the second direction. In these examples, the GUI 221 may responsively rotate the field of view in the second direction away from the current scene 222C to provide the operator 10 with immediate feedback that the robot 300 is performing a turn maneuver before the robot 300 actually performs the turn maneuver due to a lag (e.g., a communication lag between the mobile device 200 and the robot 300, and a lag caused by the time to physically control the robot 300 to perform the turn). In other examples, the selection of the second rotation graphic 274 only causes the field of view to rotate in the second direction away from the current scene 222C to allow the operator 10 to observe the preview scene 222D without physical movement of the robot 300. For example, the operator 10 can rotate the field of view in the second direction to observe the scene to the left of the robot 300 to assist the operator 10 in making navigation decisions before instructing the robot 300 to maneuver. In some examples, the amount by which the field of view rotates is proportional to the time that the second rotating graphic 274 is in contact with the operator 10 (e.g., by touching). For example, the field of view can rotate in the second direction until the operator 10 releases contact with the second rotating graphic 274. In another example, the field of view rotates by one increment each time the second rotating graphic 274 is touched and released. The application 400 can enhance (e.g., change the color of the second rotating graphic 274) to indicate when the graphic 274 is selected.

[0142] Regardless of whether the field of view is rotated in a first direction away from the current scene 222C or in a second direction away from the current scene 222C, the image data 342 rendered for display on the GUI 221 does not simply translate / slide in the first direction or the second direction. Instead, the touch-and-go application 400 is configured to compile and distort the image data 342 captured by the various cameras 394 on the robot 300 so that the rotation of the field of view in the first direction or the second direction simulates the robot 300 actually turning in the corresponding one of the first direction or the second direction. The availability of the image data 342 can constrain how far the preview scene 222P can be rotated away from the current scene 222C in either direction. Therefore, when the image data 342 becomes unavailable, the application 400 can mask the peripheral devices of the preview scene 222P for which the image data 342 is unavailable.

[0143] refer to Fig.13A, using a virtual rectangle 520 to render a scene 500 (e.g., current scene 222C) from a scene camera 510. The scene camera 510 includes an imaginary camera located near the center of the robot 300 and pointing forward toward the virtual rectangle 520. The scene camera 510 can utilize one or more real cameras 344 of the robot 300 to capture image data 342 for rendering the scene 500. The virtual rectangle 520 is located at a distance D in front of the scene camera 510, and the distance D is called the "stitching distance". The scene camera 510 has a programmable adjustable field of view (FOV). For example, the angle of the FOV (called the viewing angle) can be increased or decreased so that the scene with objects more or less falls within the FOV.

[0144] refer to Fig. 13B To render the scene 500 (e.g., the current scene 222C), the touch-and-go application 400 "colors" the points on the virtual rectangle 520 using the image colors of the corresponding pixels on one or more real cameras 530 (e.g., camera 344) on the robot 300. For the sake of explanation, the real camera 530 can be thought of as a projector that projects the real image 532 from a known location along a known direction into space. For example, referring to Figure 3D , the real camera 530 may be the right front camera 344b, and the real image 532 may be the left front view 349a.

[0145] The touch and go application 400 uses the location of the virtual rectangle 520 relative to the robot 300 along with the intrinsic and extrinsic calibration information from the real camera 530 to color in the virtual rectangle 520. The location of the virtual rectangle 520 relative to the robot 300 is defined as being at the stitching distance in front of the robot 300, as described above with reference to Fig.13A The Touch & Go application 400 determines the position of the real camera 530 relative to the robot 300 based on the external calibration information. Based on the intrinsic calibration information, the Touch & Go application 400 determines that any point on the virtual rectangle 520 corresponds to a pixel in the real image 532 (represented as a square in the figure).

[0146] If the points on the virtual rectangle 520 correspond to pixels on the real camera, the touch-and-go application 400 colors these points (indicated as O in the figure) with the color from the real image 532. If the points on the virtual rectangle 520 do not correspond to pixels on the real camera 530, the touch-and-go application 400 colors these points in black or other colors (indicated as X in the figure). When the scene camera 510 is wide enough so that all points on the virtual rectangle 520 correspond to pixels on the real camera 530 and these points are colored with the color from the real image 532, the operator 10 can recognize the boundary of the real image 532 from the rendered scene, as shown in FIG. Fig. 13B shown.

[0147] refer to Fig. 13C When the operator 10 wants the robot 300 to rotate from the current position (indicated by a circle in the figure) to the target position (indicated by a square in the figure), and the two are separated by an angle, the touch-and-go application 400 rotates the scene camera 510 and the virtual rectangle 520 to a preview angle PA that matches the angle. rotated and the rotated virtual rectangle 520 rotated The Touch and Go application 400 then proceeds to use the colors from the real image (not shown for clarity) to rotate the virtual rectangle 520. rotated The points on the image are shaded to render the preview scene 540 (eg, preview scene 222P).

[0148] In one implementation, the touch-and-go application 400 integrates a "virtual speed" over time that corresponds to the operator's 10 user input to command the robot 300 to rotate. For example, the virtual speed involves the operator 10 selecting and holding the first rotation graphic 272 ( Fig. 12B ) or the second rotating graphic 274 ( Fig. 12B ). By using the virtual velocity, the rotating scene camera 510 rotate and the rotated virtual rectangle 520 rotated The preview angle PA is rotated, and the rendered preview scene 540 responds to the command of the operator 10, thereby simulating that the robot 300 actually rotates the preview angle PA. That is, from the preview scene 540, the robot 300 appears to rotate as the operator 10 commands the rotation, and it appears that the robot 300 stops rotating when the operator 10 stops commanding the rotation. This instant or near-instant user feedback can overcome the challenges of remotely controlling a mobile robot, specifically, where there is a delay in communicating with the mobile robot and / or a delay in the mobile robot executing the command.

[0149] refer to Fig.13D , since the robot 300 is not rotated, the real cameras 530, 530ab on the robot 300 maintain the current field of view, and the real images 532, 532ab from the real cameras 530, 530ab maintain the current scene. When the scene camera (not shown in the figure for clarity) is rotated to an aggressive preview angle, the data from the real camera 530 is only sufficient to fill the rotated virtual rectangle 520 rotated A portion of, depicted as O in the figure. The rotated virtual rectangle 520 rotated The remaining portion of is filled with black (or other color), depicted as an X in the figure. The operator 10 sees it as a masked area 1340 (or other colored bar / area) on the far right side of the preview scene 540, as shown in FIG. Fig.13E shown.

[0150] Figures 14A-14H An example GUI 221 ( Fig.14A , 14C , 14E, 14G) and example robot environment 301 ( Fig. 14B , 14D , 14F, 14H). The operator 10 using the mobile device 200 provides user input to control the robot 300. The touch-and-go application 400 responds to the user input of the operator 10 by pre-rotating the robot 300 so that the operator sees the robot 300 rotating toward a target orientation in the robot environment from the perspective of the robot 300 before the robot 300 physically rotates.

[0151] For explanation purposes, reference is made to the start time T start and stop time T stop Describe the pre-rotation process. Start time T start corresponds to the time just before the operator starts controlling the robot, and the stop time T stop corresponds to the time just after the operator stops controlling the robot. In addition, the reference start time T start and stop time T stop The middle time T 1 and T 2 , so that the complete ordered time series is T start , T 1 , T 2 and T stop Intermediate time T 1 and T 2 This corresponds to when the touch-and-go application 400 is pre-rotating the robot 300 while the operator 10 is controlling the robot 300 .

[0152] The scene rendered in the GUI 221 displayed on the screen of the mobile device 200 provides the operator 10 with a view of the robot environment 301 from the perspective of the robot 300 . Fig.14A and 14B Depicts the start time T start The previous scene 550, Fig. 14C and 14D Depicts the time T 1 Preview scene 552, Fig.14E and 14F Depicts the time T 2 The next preview scene 554, and Figure 14G and 14H Depicted in T stopEach scene (550, 552, 554, 556) is rendered on the GUI 221 from a scene camera 560 that projects a virtual rectangle (562, 564, 566), as described above with reference to Figures 13A-13E For example, refer to Fig.14A and 14B To render the front scene 550 on the GUI 221, the touch-and-go application 400 colors the points on the virtual rectangle 562 that correspond to the pixels in the real images 570a, 570b with the image color of the corresponding pixel. For the points on the virtual rectangle 562 that do not correspond to any pixels in the real images 570a, 570b, the touch-and-go application 400 colors these points in black (or another color) (e.g., Fig.13E The real images 570a, 570b are captured by a real camera (not shown for clarity) on the robot 300. For example, referring to Figure 3D , the real cameras may be the left front camera 344a and the right front camera 344b.

[0153] Continue to refer Fig.14A and 14B , at the start time T start , the robot 300 is in the current orientation, or for simplicity, the robot 300 faces the forward direction. The real images 570a, 570b captured by the real camera on the robot 300 are the robot environment in the forward direction, and for ease of reference, are referred to as the "front view" in this example. The Touch and Go application 400 renders the front scene 550 for display on the GUI 221 by coloring in the virtual rectangle 562 with the image colors of the real image 570a and the real image 570b from the front view. For the operator 10 observing the GUI 221, it can be seen from the front scene 550 that the robot 300 faces the forward direction.

[0154] The operator 10 commands the robot 300 to turn right or left relative to the current orientation using the first rotation graphic 232 and the second rotation graphic 234 displayed on the GUI 221. For this example, the operator 10 presses the first rotation graphic 232 to provide a user input indication indicating a command to rotate the robot 300 right from the current orientation to the target orientation.

[0155] refer to Fig. 14C and 14D , during the intermediate time T when the operator 10 controls the robot 300 1, the touch and go application 400 determines the preview angle PA according to the virtual speed corresponding to the user input indication. In the example shown, the preview angle PA at the intermediate time T1 is shown as the angle between the current orientation (represented as a circle) and the intermediate orientation (represented as a square). The touch and go application 400 rotates the scene camera 560 by the preview angle PA, thereby generating a rotated virtual rectangle 564. However, the robot 300 remains facing the forward direction, and the real image 570a and the real image 570b are still front views.

[0156] At the preview angle PA that separates the rotated virtual rectangle 564 from the real image 570a and the real image 570b, Fig.14D It is shown that each point on the rotated virtual rectangle 564 finds a correspondence with a pixel in the real image 570a or the real image 570b. The touch-and-go application 400 renders the preview scene 552 for display on the GUI 221 by coloring in the rotated virtual rectangle 564 with the image colors of the real image 570a and the real image 570b from the front view. For the operator 10 observing the GUI 221, it can be seen from the preview scene 552 that the robot 300 is turning right in response to their command. However, in reality, the robot 300 is still facing the forward direction (i.e., the current orientation of the robot 300).

[0157] The operator 10 continues to press or hold the right arrow button 232 to command the robot 300 to continue turning right toward the target orientation. Fig.14E and 14F , during the intermediate time T when the operator 10 is controlling the robot 300 2 The touch-and-go application 400 determines the next preview angle PA according to the virtual speed corresponding to the user input 202. next In the example shown, the intermediate time T 2 Next preview angle PA next is shown as the angle between the current orientation (represented as a circle) and the next intermediate orientation (represented as a triangle). The Touch and Go application 400 rotates the scene camera 560 to the next preview angle PA next , thereby generating the next rotated virtual rectangle 566. However, the robot 300 remains facing the forward direction, and the real image 570a and the real image 570b are still front views.

[0158] At the next preview angle PA that separates the next rotated virtual rectangle 566 from the real image 570a and the real image 570b next Department, Fig.14FIt is shown that not every point on the next rotated virtual rectangle 566 finds a correspondence with a pixel in either the real image 570a or the real image 570b. Simply put, the scene camera 560 is at such a large or extreme angle relative to the real camera on the robot 300 that there is not enough information or image data from the real image 570a and the real image 570b to color all points on the next rotated virtual rectangle 566. The touch-and-go application 400 renders the next preview scene 554 for display on the GUI 221 by coloring the points on the next rotated virtual rectangle 566 that match the pixels found in the real image 570a and the real image 570b (indicated as O in the figure) with the image color of those corresponding pixels.

[0159] For those points on the next rotated virtual rectangle 566 that do not match any pixels found in real image 570a and real image 570b, the touch-and-go application 400 colors those points (indicated as Xs in the figure) black or another color. The mobile device 200 displays the next preview scene 554 to the operator 10, where the black area on the right side of the scene indicates the absence or non-existence of image data from real image 570a and real image 570b. For the operator 10, it can be seen from the next preview scene 554 that the robot 300 is turning to the right in response to their command. However, in reality, the robot 300 is still facing in the forward direction. At the intermediate time T 2 , which is different from the previous time T 1 , due to the next preview angle PA next Largely, the operator's 10 view of the robot's environment from the perspective of the robot 300 is incomplete or partial.

[0160] When the operator 10 sees that the robot 300 has reached the target position, the operator 10 releases the right arrow button 232 and stops providing the user input 202. stop , the touch and go application 400 determines the rotation angle AOR according to the virtual speed corresponding to the last one of the user inputs 202. Figure 14G and 14F Together with Figure 4A , the touch-and-go application 400 provides a rotation command R with a rotation angle AOR to the traversal system 380. The traversal system 380 in turn implements the rotation command R using input from the sensor system 340 and / or the perception system 390, and the robot 300 rotates from its current orientation to a target orientation (represented as a hexagon in the figure).

[0161] After rotating to the target orientation, the robot 300 now faces the right direction. The real camera on the robot 300 captures real images 572a, 572b of the robot environment 301 in the right direction, which for ease of reference are referred to as the "right view" in this example. The Touch and Go application 400 returns or otherwise resets the scene camera 560 and the virtual rectangle 562 to their non-rotated states so that they also face the right direction. It is convenient to say that the robot 300 has "caught up" with the pre-rotation process of the Touch and Go application 400 in terms of rotation.

[0162] With the real camera on the robot 300 and the scene camera 560 facing the same general direction, Fig.14H It is shown that each point on the virtual rectangle 562 has a corresponding pixel in the real image 572a or the real image 572b of the right view. Fig.14F In the example shown, scene camera 560 is rotated too far to the right of the real camera, so that there is not enough information from real image 570a and real image 570b of the front view to color all points on the next rotated virtual rectangle 566. The touch-and-go application 400 renders the right scene 556 for display on the GUI 221 by coloring in the virtual rectangle 562 with the image colors of real image 572a and real image 572b from the right view.

[0163] In the foregoing discussion of the pre-rotation process, the Touch & Go application 400 is described as providing the rotation angle (in the rotation command R) once the operator 10 stops controlling the robot 300 (i.e., they no longer provide user input). Other implementations of the Touch & Go application 400 provide the rotation angle to the robot 300 after receiving user input from the operator 10 for a set period of time (e.g., after a timer expires). For illustrative purposes, the Touch & Go application 400 provides the rotation angle every five milliseconds after receiving the user input. In this example, the operator 10 provides 15 milliseconds of user input, and the Touch & Go application 400 responds by providing a first rotation angle for 5 milliseconds, a second rotation angle for 10 milliseconds, and a third rotation angle for 15 milliseconds. With this implementation of the Touch & Go application 400, the robot 300 rotates incrementally while the Touch & Go application 400 continuously pre-rotates the robot 300. Advantageously, this implementation can reduce the amount of image data from the real image that is lost when rendering the preview scene while pre-rotating the robot 300.

[0164] Furthermore, the foregoing discussion of the pre-rotation process refers to discrete instances in time (eg, intermediate times T 1 and the intermediate time T 2) describes a touch and go application 400. Other implementations of the pre-rotation process may occur on a continuous basis. For example, the touch and go application 400 continuously pre-rotates the robot 300 in response to continuous user input provided by the operator 10. Pre-rotating in this manner provides immediate feedback, allowing the operator 10 to see the final results of their commands before they are actually sent to the robot 300.

[0165] Fig.15 is a flow chart of an example arrangement of operations of a method 600 for navigating a robot 300. At operation 602, the method 600 receives a pixel position in a scene of a robot environment captured by a camera on the robot 300. The received pixel position is selected by an operator of the robot 300. At operation 604, the method 600 determines a pointing vector based on the pixel position and calibration information of the camera. At operation 606, the method 600 combines the pointing vector with a ground plane estimate of the robot 300 to form an intersection point. Here, the intersection point includes a physical location in the robot environment. At operation 608, the method 600 transmits the physical location to the robot 300 so that the robot 300 navigates to the physical location in the robot environment.

[0166] Fig.16 7 is a flow chart of an example arrangement of operations of a method 700 for "pre-rotating" a robot 300. In operation 702, the method 700 receives a real image of a front view of a robot environment captured by one or more real cameras disposed on the robot 300, wherein the robot 300 is in a forward orientation and facing a forward direction. In operation 704, the method 700 determines a preview angle by which the robot 300 will rotate when rotating from the forward orientation to a rotated orientation, wherein the robot 300 faces a rightward direction or a leftward direction. In operation 706, the method 700 rotates the virtual scene camera by the preview angle. Here, the virtual scene camera is rotated relative to the forward orientation of the robot 300. In operation 708, for each point on a virtual rectangle projected by the virtual scene camera, the method 700 determines whether the corresponding point corresponds to a pixel in the real image. In operation 710, the method 700 displays a preview scene including the corresponding pixels. Here, the preview scene is a view of the robot environment that is rotated by the preview angle in the rightward direction or the leftward direction, while the mobile robot is not actually rotated from the forward orientation to the rotated orientation.

[0167] Fig.17 1 is a flow chart of an example arrangement of operations of a method 1700 for controlling a robot 300. At operation 1702, the method 1700 includes receiving image data 342 from at least one image sensor 344 (e.g., a camera 344) at the data processing hardware 240 of the operator device 200. The image data 342 corresponds to an environment 301 surrounding the robot 300.

[0168] At operation 1704, the method 1700 includes executing, by the data processing hardware 240, a graphical user interface (GUI) 221 for display on the screen 220 of the operator device 200. The GUI 221 is configured to display a scene 222 of the environment 301 surrounding the robot 300 based on the image data 342, and to receive an input indication indicating a selection of a pixel position 224 within the scene 222 of the environment 301 surrounding the robot 300.

[0169] At operation 1706, the method 1700 includes determining, by the data processing hardware 240, a pointing vector V based on the selection of the pixel position 224. The pointing vector represents a direction of travel for navigating the robot 300 in the environment 301. At operation 1708, the method 1700 includes transmitting, by the data processing hardware 240, a waypoint command W to the robot 300. The waypoint command W, when received by the robot 300, causes the robot 300 to navigate to a target position T. The target position is based on an intersection between the pointing vector V and a terrain estimate (e.g., a ground plane estimate) G of the robot 300.

[0170] In some implementations, the data processing hardware 240 determines a terrain estimate (e.g., a ground plane estimate) and sends a waypoint command W as an actual waypoint based on the intersection between the pointing vector V and the ground plane estimate G. In other implementations, rather than sending an actual “waypoint” based on the intersection I, the application 400 executing on the mobile device 200 sends a waypoint command W based on a selection of a pixel location 224 within the scene 222, the waypoint command W including a point in world space corresponding to the starting point of at least one camera 344 that captured image data 242 corresponding to the scene 222 and a pointing vector V in world space. In this case, upon receiving a waypoint command W that includes only a point in world space and a pointing vector V, the robot 300 responsively travels in the direction of the pointing vector V until a distance threshold is reached or the robot 300 detects an obstacle that intersects the pointing vector V. Advantageously, the data processing hardware 240 of the mobile device 200 does not have to determine a terrain estimate (e.g., a ground plane estimate G) for use in calculating the actual “waypoint coordinates” sent to the robot 300, but rather allows the robot 300 to continuously recalculate the terrain estimate (e.g., a ground plane estimate) as the robot 300 approaches the target location T while navigating in the direction of the pointing vector V. This can optimize the navigation of the robot 300 in situations where the terrain estimate (e.g., a ground plane estimate) calculated / determined on the mobile device 200 is invalid.

[0171] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0172] Fig.18 800 is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are merely exemplary and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0173] The computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of the components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 810 can process instructions for execution within the computing device 800, including instructions stored in the memory 820 or on the storage device 830 to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 880 coupled to the high-speed interface 840). In other implementations, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memories, can be used appropriately. In addition, multiple computing devices 800 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a group of blade servers, or a multi-processor system).

[0174] The memory 820 stores information non-temporarily within the computing device 800. The memory 820 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-temporary memory 820 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0175] The storage device 830 can provide mass storage for the computing device 800. In some implementations, the storage device 830 is a computer-readable medium. In various implementations, the storage device 830 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods (such as those described above). The information carrier is a computer or machine-readable medium, such as a memory 820, a storage device 830, or a memory on a processor 810.

[0176] The high-speed controller 840 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 860 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 840 is coupled to the memory 820, to the display 880 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 850 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 860 is coupled to the storage device 830 and the low-speed expansion port 890. The low-speed expansion port 890, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (such as a keyboard, a pointing device, a scanner) or (e.g., through a network adapter) to a network device (such as a switch or a router).

[0177] As shown, computing device 800 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 800a, or multiple implementations as a laptop computer 800b in a group of such servers 800a, or implemented as part of a rack server system 800c.

[0178] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0179] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0180] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by dedicated logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)). For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from or transmit data to, or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0181] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user, and an optional keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0182] Many implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method comprising: Image data is received by data processing hardware of the operator device from at least one image sensor, the image data corresponding to the robot's environment ; as well as The data processing hardware directs the display of a graphical user interface via a screen of an operator device, the graphical user interface being configured to: Based on image data display scene; receiving a rotation input to rotate a field of view of an environment; and The preview scene is displayed by rotating the field of view without requiring physical movement of the robot's body.

2. The computer-implemented method of claim 1, in, The scene is based on a virtual rectangle of image data representing the field of view.

3. The computer-implemented method of claim 2, in, Rotating the field of view includes adjusting the virtual rectangle of the image data.

4. The computer-implemented method of claim 2, further comprising: include: A first point of the virtual rectangle is tracked to a corresponding pixel of the at least one image sensor to fill the first point with a color based on the corresponding pixel, wherein the graphical user interface is configured to display a scene based on the filled first point.

5. The computer-implemented method of claim 4, in, The second point of the virtual rectangle is filled with a default color.

6. The computer-implemented method of claim 4, in, A first portion of the first point is tracked to a corresponding pixel of a first image sensor of the at least one image sensor, and a second portion of the first point is tracked to a corresponding pixel of a second image sensor of the at least one image sensor.

7. A computer-implemented method according to any one of claims 1 to 6, in: The scene represents a field of view of a main body of the robot facing a first direction; The preview scene represents the field of view of the robot's main body facing the second direction; The rotation input comprises a command for the robot to perform a turning maneuver to physically move a body of the robot from facing a first direction to facing a second direction; and The graphical user interface displays a preview scene before the main body of the robot is physically facing the second direction.

8. A computer-implemented method according to any one of claims 1 to 6, in, The graphical user interface is configured to receive a rotation input in response to receiving a selection of a rotation graphic displayed in the graphical user interface.

9. The computer-implemented method of claim 8, in, The rotation input causes the field of view to rotate in proportion to the selected size.

10. A computer-implemented method according to any one of claims 1 to 6, in, The rotating field of view simulates a robot performing a turning maneuver.

11. A computer-implemented method according to any one of claims 1 to 6, in, The scenarios include any of the following: A front scene based on image data captured by a left front camera and a right front camera disposed on the robot; a left scene based on image data captured by a left camera disposed on the robot; a right scene based on image data captured by a right camera disposed on the robot; A rear scene based on the image data, the rear scene being captured by a rear camera disposed on the robot; or A top-down scene of the robot based on image data captured by the payload camera, left front camera, right front camera, left camera, right camera, and rear camera.

12. A computer-implemented method according to any one of claims 1 to 6, in, The robot includes a quadruped robot.

13. The computer-implemented method of any one of claims 1 to 6, wherein: The at least one image sensor is disposed on the robot; and The operator device communicates with at least one image sensor via a network.

14. A system comprising: Data processing hardware of the operator device, communicating with the robot; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions, in, Execution of the instruction by the data processing hardware causes the data processing hardware to: receiving image data from at least one image sensor, the image data corresponding to an environment of the robot; and Instructing to display a graphical user interface via a screen of an operator device, the graphical user interface being configured to: Based on image data display scene; receiving a rotation input to rotate a field of view of an environment; and The preview scene is displayed by rotating the field of view without requiring physical movement of the robot's body.

15. The system according to claim 14, in, The scene of the environment is based on a virtual rectangle of image data representing the field of view.

16. The system according to claim 15, in, Rotating the field of view includes adjusting the virtual rectangle of the image data.

17. The system according to claim 15, in, Execution of the instruction by the data processing hardware also causes the data processing hardware to: A first point of the virtual rectangle is tracked to a corresponding pixel of the at least one image sensor to fill the first point with a color based on the corresponding pixel, wherein the graphical user interface is configured to display a scene based on the filled first point.

18. The system according to claim 17, in, The second point of the virtual rectangle is filled with a default color.

19. The system according to claim 17, in, A first portion of the first point is tracked to a corresponding pixel of a first image sensor of the at least one image sensor, and a second portion of the first point is tracked to a corresponding pixel of a second image sensor of the at least one image sensor.

20. The system according to any one of claims 14 to 19, in: The scene represents a field of view of a main body of the robot facing a first direction; The preview scene represents the field of view of the robot's main body facing the second direction; The rotation input comprises a command for the robot to perform a turning maneuver to physically move the body of the robot from facing a first direction to facing a second direction; and The graphical user interface displays a preview scene before the main body of the robot is physically facing the second direction.

21. The system according to any one of claims 14 to 19, in, The graphical user interface is configured to receive a rotation input in response to receiving a selection of a rotation graphic displayed in the graphical user interface.

22. The system of claim 21, in, The rotation input causes the field of view to rotate in proportion to the selected size.

23. The system according to any one of claims 14 to 19, in, The rotating field of view simulates a robot performing a turning maneuver.

24. The system according to any one of claims 14 to 19, in, The scenarios include any of the following: A front scene based on image data captured by a left front camera and a right front camera disposed on the robot; a left scene based on image data captured by a left camera disposed on the robot; a right scene based on image data captured by a right camera disposed on the robot; A rear scene based on the image data, the rear scene being captured by a rear camera disposed on the robot; or A top-down scene of the robot based on image data captured by the payload camera, left front camera, right front camera, left camera, right camera, and rear camera.

25. The system according to any one of claims 14 to 19, in, The robot includes a quadruped robot.

26. The system according to any one of claims 14 to 19, in: The at least one image sensor is disposed on the robot; and The operator device communicates with at least one image sensor via a network.

27. A computer-implemented method, include: receiving, by data processing hardware, first image data from a first image sensor located at a first portion of a body of the robot; receiving, by the data processing hardware, second image data from a second image sensor located on a second portion of the body of the robot; instructing, by the data processing hardware, to display the first image data and the second image data via a user interface of the computing device; receiving input by data processing hardware via a computing device; and The robot is instructed to navigate to the waypoint based on the input by the data processing hardware.

28. The computer-implemented method of claim 27, in, Instructing to display the first image data and the second image data includes: Instructs to display the first image data and the second image data simultaneously.

29. The computer-implemented method of claim 27, in, Instructing to display the first image data and the second image data includes: Instructing to display the first image data; receiving scene change input; and Displaying of the second image data is instructed based on the scene change input.

30. The computer-implemented method of claim 27, in, Instructing to display the first image data and the second image data includes: Instructing to display the first image data in the first portion of the user interface; and Displaying the second image data in the second portion of the user interface is instructed.

31. The computer-implemented method of claim 27, further comprising: include: stitching the first image data and the second image data to obtain stitched image data, Wherein, instructing to display the first image data and the second image data comprises: Instructs to display stitched image data.

32. The computer-implemented method of claim 27, further comprising: include: receiving updated first image data from the first image sensor; receiving updated second image data from a second image sensor; and Display of the updated first image data and the updated second image data is indicated.

33. The computer-implemented method of claim 27, in, Instructing the robot to navigate to a waypoint involves: The robot is instructed to 1) change the direction of travel of the robot in the environment and navigate to the target location, or 2) perform a turning maneuver and navigate to the target location.

34. The computer-implemented method of claim 27, in, The first image sensor is located on the body of the robot between the right front leg of the robot and the right back leg of the robot, and wherein the second image sensor is located on the body of the robot between the left front leg of the robot and the left back leg of the robot.

35. The computer-implemented method of claim 27, in, The first portion of the robot's body includes a left side of the robot's body, and wherein the second portion of the robot's body includes a right side of the robot's body.

36. The computer-implemented method of claim 27, in, The first image sensor or the second image sensor is located at the front of the body of the robot between the left front leg of the robot and the right front leg of the robot.

37. The computer-implemented method of claim 27, in, At least one of the first image sensor or the second image sensor is rotatable.

38. The computer-implemented method of claim 27, in, A first group of image sensors is mounted horizontally along a length of a first portion of a body of the robot, wherein a second group of image sensors is mounted horizontally along a length of a second portion of a body of the robot, wherein the first group of image sensors includes a first image sensor, and wherein the second group of image sensors includes a second image sensor.

39. The computer-implemented method of claim 27, further comprising: include: receiving third image data from a third image sensor located at a third portion of the body of the robot; and Instructs to display the third image data.

40. The computer-implemented method of claim 27, in, The input indicates a target location.

41. The computer-implemented method of claim 27, in, The user interface includes a split screen.

42. The computer-implemented method of claim 27, in, The input received includes: Input is received via buttons, mouse, joystick, or touch screen.

43. The computer-implemented method of claim 27, in, An input is provided with respect to the first image data or the second image data.

44. The computer-implemented method of claim 27, in, Instructing to display the first image data and the second image data includes: The first image data and the second image data are instructed to be displayed in real time.

45. A system, include: Data processing hardware; and a memory in communication with the data processing hardware, the memory storing instructions, wherein execution of the instructions by the data processing hardware causes the data processing hardware to: receiving first image data from a first image sensor located at a first portion of a body of the robot; receiving second image data from a second image sensor located at a second portion of the body of the robot; instructing display of the first image data and the second image data via a user interface of the computing device; receiving input via a computing device; and Instructs the robot to navigate to a waypoint based on input.

46. ​​A legged robot, include: main body; a first image sensor and a second image sensor, wherein the first image sensor is located at a first portion of the body, and wherein the second image sensor is located at a second portion of the body; data processing hardware; and a memory in communication with the data processing hardware, the memory storing instructions, wherein execution of the instructions by the data processing hardware causes the data processing hardware to: receiving first image data from a first image sensor; receiving second image data from a second image sensor; providing the first image data and the second image data for display via a user interface of a computing device; and Based on input received via the computing device, the legged robot is instructed to navigate to a waypoint.

47. The legged robot according to claim 46, in, To provide the first image data and the second image data for display, execution of the instructions by the data processing hardware further causes the data processing hardware to: The first image data and the second image data are provided for simultaneous display.

48. The legged robot according to claim 46, in, The first image data is displayed via a user interface, and wherein the second image data is displayed via the user interface based on the received scene change input.

49. The legged robot according to claim 46, in, To provide the first image data and the second image data for display, execution of the instructions by the data processing hardware further causes the data processing hardware to: providing first image data for display in a first portion of a user interface; and Second image data is provided for display in a second portion of the user interface.

50. The legged robot according to claim 46, in, The first image data and the second image data are stitched together to obtain stitched image data, wherein the stitched image data is displayed via a user interface.

51. The legged robot according to claim 46, in, Execution of the instructions by the data processing hardware further causes the data processing hardware to: receiving updated first image data from the first image sensor; receiving updated second image data from a second image sensor; and The updated first image data and the updated second image data are provided for display.

52. The legged robot according to claim 46, in, To instruct the legged robot to navigate to a waypoint, execution of the instructions by the data processing hardware further causes the data processing hardware to: The legged robot is instructed to 1) change the direction of travel of the legged robot in the environment and navigate to a target location, or 2) perform a turning maneuver and navigate to a target location.

53. The legged robot according to claim 46, in, The first image sensor is located on the body between the right front leg of the legged robot and the right rear leg of the legged robot, and wherein the second image sensor is located on the body between the left front leg of the legged robot and the left rear leg of the legged robot.

54. The legged robot according to claim 46, in, The first portion of the body includes a left side of the body, and wherein the second portion of the body includes a right side of the body.

55. The legged robot according to claim 46, in, The first image sensor or the second image sensor is located at the front of the body between the left front leg of the legged robot and the right front leg of the legged robot.

56. The legged robot according to claim 46, in, At least one of the first image sensor or the second image sensor is rotatable.

57. The legged robot according to claim 46, in, A first group of image sensors is mounted horizontally along a length of a first portion of the body, wherein a second group of image sensors is mounted horizontally along a length of a second portion of the body, wherein the first group of image sensors includes a first image sensor, and wherein the second group of image sensors includes a second image sensor.

58. The legged robot according to claim 46, in, Execution of the instructions by the data processing hardware further causes the data processing hardware to: receiving third image data from a third image sensor located at a third portion of the subject; and The third image data is provided for display.

59. The legged robot according to claim 46, in, The input indicates a target location.

60. The legged robot according to claim 46, in, The user interface includes a split screen.

61. The legged robot according to claim 46, in, The input is received via buttons, a mouse, a joystick, or a touch screen.

62. The legged robot according to claim 46, in, The input is provided with respect to the first image data or the second image data.

63. The legged robot according to claim 46, in, To provide the first image data and the second image data for display, execution of the instructions by the data processing hardware further causes the data processing hardware to: The first image data and the second image data are provided for real-time display.