Mobile robotic system for automated asset inspection

Abstract

Methods and apparatus are provided for performing automated inspection of one or more assets in an environment using a movable robot. The method includes defining a region of interest of an asset in an environment including a robot within an image captured by a sensor of the robot, where the asset is associated with an asset identifier, configuring at least one parameter of a computer vision model based on the asset identifier, the image data within the region of interest is processed using a computer vision model to determine whether an alert should be generated, and an alert is output when it is determined that the alert should be generated.

Description

Background Art

[0001] A robot is typically a reprogrammable and versatile manipulator, usually designed to move materials, parts, tools, or specialized equipment through variable programmed motions to perform tasks. A robot can be a physically anchored manipulator (e.g., an industrial robot arm), a mobile robot that moves throughout an environment (e.g., using legs, wheels, or traction-based mechanisms), or some combination of manipulators and mobile robots. Robots are used in a variety of industries, including, for example, manufacturing, warehouse logistics, transportation, hazardous environments, exploration, and healthcare. Summary of the invention

[0002] In some embodiments, a method is provided. The method includes defining a region of interest within an image captured by a sensor of a robot, the region of interest including an asset in an environment of the robot, wherein the asset is associated with an asset identifier, configuring at least one parameter of a computer vision model based on the asset identifier, processing image data within the region of interest using the computer vision model to determine whether an alarm should be generated, and outputting an alarm when it is determined that an alarm should be generated.

[0003] In one aspect, defining the region of interest includes defining the region of interest using asset information stored in a data structure associated with the mission record. In one aspect, the data structure is associated with an action of capturing an image at a first waypoint indicated in the mission record, and an asset identifier is included in the data structure. In one aspect, the data structure includes the at least one parameter of the computer vision model.

[0004] In one aspect, the image captured by the sensor of the robot is a thermal image. In one aspect, at least one parameter of the computer vision model includes a temperature threshold. In one aspect, processing the image data within the region of interest using the computer vision model to determine whether an alarm should be generated includes: determining a temperature of the asset based on an analysis of the thermal image within the region of interest, comparing the determined temperature of the asset to the temperature threshold, and determining to generate an alarm when the determined temperature meets or exceeds the temperature threshold.

[0005] In one aspect, at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold. In one aspect, outputting the alert includes displaying a representation of the image annotated with an indication of the alert on a display. In one aspect, outputting the alert includes sending a message to a computing device via at least one network, the message including the alert. In one aspect, the method also includes storing the image and metadata indicating the asset identifier on at least one storage device.

[0006] In one aspect, the region of interest is a first region of interest that includes a first asset, the method further comprising defining a second region of interest within the image, the second region of interest including a second asset in the robot's environment, and processing image data within the region of interest using a computer vision model to determine whether an alarm should be generated comprises processing image data within the first region of interest using a computer vision model to determine a first result, processing image data within the second region of interest using the computer vision model to determine a second result, and determining whether the alarm should be generated based at least in part on the first result and the second result.

[0007] In one aspect, the method further comprises, for each of a plurality of images captured over time, and each of the images having a region of interest defined therein, processing image data within the region of interest of the image using a computer vision model to determine at least one quantity associated with the asset, and generating a trend analysis for the at least one quantity based on the at least one quantity associated with the asset determined for the plurality of images. In one aspect, the at least one quantity comprises one or more of temperature, pressure, vibration, and radiation quantity. In one aspect, the method further comprises providing an indication of the trend analysis for the at least one quantity on a user interface. In one aspect, the method further comprises generating an alert based at least in part on the trend analysis.

[0008] In some embodiments, a robot is provided. The robot includes a perception system, the perception system including an image sensor configured to capture an image and at least one computer processor. The at least one computer processor is configured to define a region of interest within an image captured by the image sensor, the region of interest including an asset in an environment of the robot, wherein the asset is associated with an asset identifier, configure at least one parameter of a computer vision model based on the asset identifier, process image data within the region of interest using the computer vision model to determine whether an alarm should be generated, and output the alarm when it is determined that an alarm should be generated.

[0009] In one aspect, defining the region of interest includes defining the region of interest using asset information stored in a data structure associated with the mission record. In one aspect, the data structure is associated with an action of capturing an image at a first waypoint indicated in the mission record, and an asset identifier is included in the data structure. In one aspect, the data structure includes the at least one parameter of the computer vision model.

[0010] In one aspect, the image captured by the image sensor of the robot is a thermal image. In one aspect, at least one parameter of the computer vision model includes a temperature threshold. In one aspect, processing the image data within the region of interest using the computer vision model to determine whether an alarm should be generated includes: determining a temperature of the asset based on an analysis of the thermal image within the region of interest, comparing the determined temperature of the asset to the temperature threshold, and determining to generate an alarm when the determined temperature meets or exceeds the temperature threshold.

[0011] In one aspect, at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold. In one aspect, outputting an alert includes displaying a representation of an image annotated with an indication of the alert on a display. In one aspect, outputting an alert includes sending a message to a computing device via at least one network, the message including the alert. In one aspect, at least one computer processor is further configured to store the image and metadata indicating an asset identifier on at least one storage device. In one aspect, the region of interest is a first region of interest including a first asset, and at least one computer processor is further configured to define a second region of interest within the image, the second region of interest including a second asset in the robot's environment, wherein processing image data within the region of interest using a computer vision model to determine whether an alert should be generated includes: processing image data within the first region of interest using a computer vision model to determine a first result, processing image data within the second region of interest using the computer vision model to determine a second result, and determining whether the alert should be generated based at least in part on the first result and the second result.

[0012] In one aspect, the at least one computer processor is further configured to: for each of a plurality of images captured over time, wherein a region of interest is defined in each of the images, process image data within the region of interest for the image using a computer vision model to determine at least one quantity associated with the asset, and based on the at least one quantity determined to be associated with the asset for the plurality of images, generate a trend analysis for the at least one quantity. In one aspect, the at least one quantity includes one or more of temperature, pressure, vibration, and radiation quantity. In one aspect, the at least one computer processor is further configured to provide an indication of the trend analysis for the at least one quantity on a user interface. In one aspect, the at least one computer processor is further configured to generate an alert based at least in part on the trend analysis.

[0013] In some embodiments, a non-transitory computer-readable medium encoded with a plurality of instructions is provided, the plurality of instructions when executed by at least one computer processor, performs a method. The method includes defining a region of interest within an image captured by a sensor of a robot, the region of interest including an asset in an environment of the robot, wherein the asset is associated with an asset identifier, configuring at least one parameter of a computer vision model based on the asset identifier, processing image data within the region of interest using the computer vision model to determine whether an alarm should be generated, and outputting an alarm when it is determined that an alarm should be generated.

[0014] In one aspect, defining the region of interest includes defining the region of interest using asset information stored in a data structure associated with the mission record. In one aspect, the data structure is associated with an action of capturing an image at a first waypoint indicated in the mission record, and an asset identifier is included in the data structure. In one aspect, the data structure includes the at least one parameter of the computer vision model.

[0015] In one aspect, the image captured by the sensor of the robot is a thermal image. In one aspect, at least one parameter of the computer vision model includes a temperature threshold. In one aspect, processing the image data within the region of interest using the computer vision model to determine whether an alarm should be generated includes: determining a temperature of the asset based on an analysis of the thermal image within the region of interest, comparing the determined temperature of the asset to the temperature threshold, and determining to generate an alarm when the determined temperature meets or exceeds the temperature threshold.

[0016] In one aspect, at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold. In one aspect, outputting an alert includes displaying a representation of an image annotated with an indication of the alert on a display. In one aspect, outputting an alert includes sending a message to a computing device via at least one network, the message including the alert. In one aspect, the method further includes storing the image and metadata indicating an asset identifier on at least one storage device. In one aspect, the region of interest is a first region of interest including a first asset, the method further includes defining a second region of interest within the image, the second region of interest including a second asset in the robot's environment, wherein processing image data within the region of interest using a computer vision model to determine whether an alert should be generated includes processing image data within the first region of interest using a computer vision model to determine a first result, processing image data within the second region of interest using the computer vision model to determine a second result, and determining whether the alert should be generated based at least in part on the first result and the second result.

[0017] In one aspect, the method further comprises, for each of a plurality of images captured over time, wherein each of the plurality of images has a region of interest defined therein, processing image data within the region of interest for the image using a computer vision model to determine at least one quantity associated with the asset, and generating a trend analysis for the at least one quantity based on the at least one quantity associated with the asset determined for the plurality of images. In one aspect, the at least one quantity comprises one or more of temperature, pressure, vibration, and radiation quantity. In one aspect, the method further comprises providing an indication of the trend analysis for the at least one quantity on a user interface. In one aspect, the method further comprises generating an alert based at least in part on the trend analysis.

[0018] In some embodiments, a method is provided. The method includes navigating a mobile robot to traverse a route through an environment, generating a mission record during navigation of the mobile robot along the route, the mission record including a plurality of waypoints and pairs of edges connecting the plurality of waypoints, receiving a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint of the plurality of waypoints, receiving a second input from the user via the user interface, the second input identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint, and associating a first asset identifier with the first action in the mission record, wherein the first asset identifier uniquely identifies the first asset in the environment.

[0019] In one aspect, the method further includes commanding the mobile robot to run a first task corresponding to the task record, automatically capturing second sensor data when the mobile robot reaches a first waypoint along the route, and automatically storing an asset identifier as metadata associated with the second sensor data on at least one storage device. In one aspect, the method further includes commanding the mobile robot to perform a second task corresponding to the task record, automatically capturing third sensor data when the mobile robot reaches a first waypoint along the route, and automatically storing an asset identifier as metadata associated with the third sensor data on at least one storage device. In one aspect, the method further includes displaying an indication of the second sensor data and the third sensor data on a user interface. In one aspect, the second sensor data and the third sensor data include second image data and third image data, respectively, and the method further includes processing each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, and displaying an indication of the second sensor data and the third sensor data includes displaying the second sensor data with an indication of the first output, and displaying the third sensor data with an indication of the second output.

[0020] In one aspect, the method further includes prompting the user on the user interface to provide a second input, the second input identifying the first asset within the first sensor data. In one aspect, the method also includes receiving a third input from the user via the user interface, the third input identifying a second asset within the first sensor data captured by the mobile robot at the first waypoint, and associating the second asset identifier with the first action in the mission record, wherein the second asset identifier uniquely identifies a second asset in the environment. In one aspect, the method also includes receiving a third input from the user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among a plurality of waypoints, receiving a fourth input from the user via the user interface, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint, and associating the first asset identifier with the second action in the mission record.

[0021] In some embodiments, a robot is provided. The robot includes a navigation system, a perception system, the navigation system configured to navigate a mobile robot to traverse a route through an environment, the perception system including an image sensor configured to capture images, and at least one computer processor. The at least one computer processor is configured to generate a mission record during navigation of the mobile robot along the route, the mission record including a plurality of waypoints and pairs of edges connecting the plurality of waypoints, receive a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint of the plurality of waypoints, receive a second input from the user via the user interface identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint, and associate a first asset identifier with the first action in the mission record, wherein the first asset identifier uniquely identifies the first asset in the environment.

[0022] In one aspect, at least one computer processor is further configured to command the mobile robot to run a first task corresponding to the task record, automatically capture second sensor data when the mobile robot reaches a first waypoint along the route, and automatically store an asset identifier as metadata associated with the second sensor data on at least one storage device. In one aspect, at least one computer processor is further configured to command the mobile robot to perform a second task corresponding to the task record, automatically capture third sensor data when the mobile robot reaches a first waypoint along the route, and automatically store an asset identifier as metadata associated with the third sensor data on at least one storage device. In one aspect, the at least one computer processor is further configured to display an indication of the second sensor data and the third sensor data on a user interface. In one aspect, the second sensor data and the third sensor data include second image data and third image data, respectively, and the at least one computer processor is further configured to process each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, and displaying the indication of the second sensor data and the third sensor data includes displaying the second sensor data with an indication of the first output, and displaying the third sensor data with an indication of the second output.

[0023] In one aspect, the at least one computer processor is further configured to prompt the user on the user interface to provide a second input, the second input identifying the first asset within the first sensor data. In one aspect, the at least one computer processor is further configured to receive a third input from the user via the user interface, the third input identifying a second asset within the first sensor data captured by the mobile robot at the first waypoint, and associating the second asset identifier with the first action in the mission record, wherein the second asset identifier uniquely identifies the second asset in the environment. In one aspect, the at least one computer processor is further configured to receive a third input from the user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among a plurality of waypoints, receive a fourth input from the user via the user interface, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint, and associating the first asset identifier with the second action in the mission record.

[0024] In some embodiments, a non-transitory computer-readable medium encoded with a plurality of instructions is provided that, when executed by at least one computer processor, performs a method. The method includes navigating a mobile robot to traverse a route through an environment, generating a mission record during navigation of the mobile robot along the route, the mission record comprising a plurality of waypoints and pairs of edges connecting the plurality of waypoints, receiving a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint of the plurality of waypoints, receiving a second input from the user via the user interface identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint, and associating a first asset identifier with the first action in the mission record, wherein the first asset identifier uniquely identifies the first asset in the environment.

[0025] In one aspect, the method further includes instructing the mobile robot to run a first task corresponding to the task record, automatically capturing second sensor data when the mobile robot reaches a first waypoint along the route, and automatically storing an asset identifier as metadata associated with the second sensor data on at least one storage device. In one aspect, the method further includes instructing the mobile robot to run a second task corresponding to the task record, automatically capturing third sensor data when the mobile robot reaches the first waypoint along the route, and automatically storing an asset identifier as metadata associated with the third sensor data on at least one storage device. In one aspect, the method further includes displaying an indication of the second sensor data and the third sensor data on a user interface. In one aspect, the second sensor data and the third sensor data include second image data and third image data, respectively, and the method further includes processing each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, and displaying an indication of the second sensor data and the third sensor data includes displaying the second sensor data with an indication of the first output, and displaying the third sensor data with an indication of the second output.

[0026] In one aspect, the method further includes prompting the user on the user interface to provide a second input, the second input identifying the first asset within the first sensor data. In one aspect, the method also includes receiving a third input from the user via the user interface, the third input identifying a second asset within the first sensor data captured by the mobile robot at the first waypoint, and associating the second asset identifier with the first action in the mission record, wherein the second asset identifier uniquely identifies a second asset in the environment. In one aspect, the method also includes receiving a third input from the user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among a plurality of waypoints, receiving a fourth input from the user via the user interface, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint, and associating the first asset identifier with the second action in the mission record.

[0027] In some embodiments, a method for monitoring physical assets in an environment over time using a mobile robot is provided. The method includes determining, within a region of interest defined within each of a plurality of images captured by the mobile robot over time, at least one quantity associated with a physical asset represented in the region of interest, wherein the determining is performed using a computer vision model configured based on asset information associated with the physical asset, generating a trend analysis for the at least one quantity based on the at least one quantity associated with the asset of the plurality of images, and outputting an indication of the trend analysis.

[0028] In one aspect, the at least one quantity comprises one or more of temperature, pressure, vibration, and radiation quantity. In one aspect, outputting an indication of the trend analysis comprises providing an indication of the trend analysis on a user interface. In one aspect, the method further comprises generating an alert based at least in part on the trend analysis.

[0029] The foregoing device and method embodiments can be implemented using any suitable combination of the aspects, features and actions described above or further described in detail below. These and other aspects, embodiments and features of the present teachings can be more fully understood from the following description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Various aspects and embodiments will be described with reference to the following drawings. It should be understood that the drawings are not necessarily drawn to scale. In the drawings, each identical or nearly identical component illustrated in the various figures is represented by the same number. For clarity, not every component may be labeled in every figure.

[0031] FIG1A illustrates an example of a robot configured to navigate in an environment along a route according to some embodiments;

[0032] FIG. 1B is a block diagram of components of a robot, such as the robot shown in FIG. 1A ;

[0033] 2 illustrates components of a navigation system for navigating a robot, such as the robot of FIG. 1A , in an environment according to some embodiments;

[0034] 3A shows a portion of a user interface provided on a robot controller to enable an operator to create actions during a task recording process according to some embodiments;

[0035] 3B shows a portion of a user interface provided on a robot controller to enable an operator to specify a region of interest on an isothermal rendering of the robot's environment according to some embodiments;

[0036] FIG3C shows a portion of a user interface provided on a robot controller to enable an operator to specify a region of interest on a color rendering of the robot's environment in accordance with some embodiments;

[0037] 4 is a flow diagram of a process for automatically monitoring assets in an environment using one or more images captured by a mobile robot, according to some embodiments;

[0038] 5 is a flow diagram of a process for generating a task record including asset information according to some embodiments;

[0039] FIG6 illustrates an image captured by a robot that has been annotated with alert information according to some embodiments;

[0040] 7A and 7B illustrate portions of a user interface configured to display historical data of images captured over time for a single asset in an environment, according to some embodiments;

[0041] 8 illustrates a portion of a user interface configured to display multiple images of an asset captured during different runs of a mission, according to some embodiments;

[0042] FIG. 9 illustrates historical trend analysis of one or more characteristics of monitored assets in an environment according to some embodiments; and

[0043] 10 is a block diagram of components of a robot upon which some embodiments may be implemented. DETAILED DESCRIPTION

[0044] Some robots are used to navigate an environment to perform various tasks or functions. These robots are often operated to perform a "task" by navigating the robot through the environment. Sometimes the task is recorded so that the robot can perform the task again at a later time. In some tasks, the robot navigates through the environment and interacts with the environment. The interaction sometimes takes the form of collecting data using one or more sensors.

[0045] Industrial facilities that include physical assets that require inspection and maintenance over several years of operation are examples of one type of environment in which an automated inspection system using such a robot may be useful. The infrastructure in industrial facilities often lacks instrumentation that allows remote monitoring of the health of physical assets, resulting in predictive maintenance of assets being an expensive manual process. To this end, some embodiments of the present disclosure are directed to an automated inspection platform integrated with a mobile robot to enable remote, repeated, and reliable inspection of physical assets in a facility, thereby reducing the reliance on manual inspections by humans for predictive maintenance of such assets.

[0046] An example of a robot 100 that can perform a task is described below in conjunction with FIGS. 1A-1B . To enable the robot 100 to run a task, the robot 100 can undergo an initial mapping process during which the robot 100 moves around the environment 10 (typically in response to commands input by a user to a tablet or other controller) to collect data about the environment 10 (e.g., via one or more sensors), and can generate a topological map 204 (an example of which is shown in FIG. 2 ) that defines waypoints 212 along the path of travel of the robot 100 and edges 214 between corresponding pairs representing the waypoints 212. Each waypoint 212 can be associated with sensor data, fiducials, and / or robot pose information, for example, at a specific time and place, and each edge 214 can topologically connect the waypoints 212.

[0047] In some existing systems, a given “mission record” may identify a sequence of actions to occur at specific waypoints 212 included on the topological map 204. For example, a mission record may indicate that the robot 100 will go to a first waypoint 212 and perform a first action, then go to a second waypoint 212 and perform a second action, etc. In some embodiments, such a mission record need not specify all of the waypoints 212 that the robot 100 will actually traverse when executing the mission, but may only specify those waypoints 212 at which specific actions are to be performed.

[0048] As explained in detail below, such a task record can be run by the task execution system 184 (shown in FIG. 1B ) of the robot 100. The task execution system 184 can communicate with other systems of the robot 100 as needed to successfully execute the task. For example, in some embodiments, the task execution system 184 can communicate with the navigation system 200 (also shown in FIG. 1B ) to request that the navigation system 200 use the topological map 204 and the task record to determine a navigation route 202, which includes various waypoints 212 of the topological map 204 identified in the task record, and any number of additional waypoints 212 of the topological map 204 located between the waypoints 212 identified in the task record. The determined navigation route 202 can also include edges 214 located between corresponding pairs of such waypoints 212. Enabling the robot to follow a navigation route 202 that includes all waypoints 212 identified in the task record can enable the task execution system 184 to perform corresponding actions in the task record when the robot 100 reaches those waypoints 212.

[0049] As described below with reference to FIG. 2 , the navigation system 200 may include a navigation generator 210 that may generate a navigation route 202 that includes specified waypoints 212 (e.g., waypoints identified in a mission record) and a route executor 220 that is configured to control the robot 100 to move along the identified navigation route 202, possibly rerouting the robot along an alternate path 206, for example, if necessary to avoid obstacles 20 that may not have existed when the mission was recorded.

[0050] Examples of actions that can be performed at waypoints 212 of a mission record include capturing sensor data for inspecting one or more characteristics of physical assets in an environment. For example, the sensor data can include image data (e.g., visual image data and / or thermal image data) that can be processed using one or more computer vision models to determine one or more characteristics of one or more physical assets (e.g., wear, temperature, radiation, sound, vibration, etc.).

[0051] As described above, in some existing systems, the mission record may identify a specific action to be taken by the robot 100 when it reaches a particular waypoint 212. For example, referring to the right hand side of FIG. 2, the mission record may specify that the robot 100 captures a first image when it reaches the first waypoint 212d, and captures a second image when it reaches the second waypoint 212e. In this case, when the mission execution system 184 determines that the robot 100 has reached the first waypoint 212d, the mission execution system 184 may command the robot's system to capture the first image. Similarly, when the mission execution system 184 determines that the robot 100 has reached the second waypoint 212e, the mission execution system 184 may command the same system (or a different system) of the robot to capture the second image. Repeatedly running a mission over days, months, and / or years may be used to implement an inspection system in which a data set including sensor data captured at a consistent location may be acquired. The sensor data in the data set may be used, for example, to perform long-term trend analysis of characteristics of physical assets in a facility and / or automatically detect anomalies regarding physical assets that may require manual intervention.

[0052] The inventors have recognized and appreciated that manually processing the amount of sensor data (e.g., gigabytes) that can be acquired using an inspection system (such as the inspection system described herein) can be a time-consuming and error-prone process. Therefore, some embodiments of the present disclosure relate to techniques for facilitating automated analysis of sensor data for physical asset inspection. For example, as described in more detail below, some embodiments provide a user interface that enables a user (e.g., an operator of a robot) to specify a region of interest (ROI) within an image captured by a sensor of the robot, wherein the ROI includes a physical asset of interest to be monitored. A unique asset identifier (asset ID) can be associated with the physical asset to distinguish the data of the physical asset from the data of another physical asset of interest, which data can be associated with a different asset ID. The asset IDs assigned to different monitored physical assets in a facility can be used to track characteristics of the same asset across multiple images in a data set and / or to distinguish multiple assets in a single image, among other uses. Processing images using asset IDs is described in more detail below, as one type of sensor data that can be acquired using an automated inspection system as described herein.

[0053] 1A , the robot 100 may include a body 110 having a motion-based structure, such as legs 120 a-120 d coupled to the body 110, which enables the robot 100 to move around the environment 10. In some embodiments, each leg 120 may be an articulated structure such that one or more joints J allow a member 122 of the leg 120 to move. For example, each leg 120 may include an upper member 122, 122 of the leg 120. u A hip joint J coupled to the body 110 H , and the upper member 122 of the leg 120 u The lower member 122 coupled to the leg 120 L Knee joint J K For impact detection, the hip joint J H It can be further decomposed into the frontal plane of the robot 100 (i.e., the X-axis A x and Z-axis A Z The hip joint J in the XZ plane extends in the direction of H The abduction-adduction rotation occurs in the sagittal plane of the robot 100 (ie, in the Y direction of the axis A Y and Z-axis A Z The hip joint J in the YZ plane extends in the direction of H 1A depicts a quadruped robot having four legs 120a-120d, it should be understood that the robot 100 may include any number of legs or motion-based structures (e.g., a bipedal or humanoid robot having two legs) that provide a means for traversing terrain within the environment 10.

[0054] To traverse terrain, each leg 120 may have a distal end 124 that contacts a surface 14 of the terrain (i.e., a traction surface). In other words, the distal end 124 of the leg 120 is the end of the leg 120 used by the robot 100 to pivot, securely position, or generally provide traction during movement of the robot 100. For example, the distal end 124 of the leg 120 may correspond to a "foot" of the robot 100. In some examples, although not shown, the distal end 124 of the leg 120 may include an ankle joint so that the distal end 124 may be relative to the lower member 122 of the leg 120. L Articulated.

[0055] In the illustrated example, the robot 100 includes an arm 126 that functions as a robotic manipulator. The arm 126 can be configured to move about multiple degrees of freedom in order to grasp (engage) elements in the environment 10 (e.g., objects within the environment 10). In some embodiments, the arm 126 can include one or more members 128, wherein the members 128 are coupled by joints J so that the arm 126 can pivot or rotate about the joints J. For example, in the case of having more than one member 128, the arm 126 can be configured to extend or retract. To illustrate the example, FIG. 1A depicts an arm 126 having three members 128, wherein the three members 128 correspond to the lower member 128. L 、Upper member 128 U and hand component 128 H (For example, hand member 128 H Also called end effector 128 H Here, the lower member 128L may surround one or more arm joints J located adjacent to the body 110. A (e.g., where the arm 126 is connected to the body 110 of the robot 100) to rotate or pivot. For example, FIG. 1A depicts a robot that can rotate or pivot about a first arm joint J A1 The arm 126 can be rotated by a yaw arm joint. The arm 126 can rotate about the vertical gravity axis (eg, shown as A) of the robot 100 by using the yaw arm joint. z ) Axial rotation of "360" degrees (or some fraction thereof). Lower member 128 L Can be around the second arm joint J A2 Pivot (e.g., when rotating). For example, the second arm joint J A2 (shown adjacent to the body 110 of the robot 100) allows the arm 126 to pitch to a certain angle (eg, raise or lower one or more members 128 of the arm 126). L At the third arm joint J A3 is coupled to the upper member 128 U , and the upper member 128 U At the fourth arm joint J A4 Coupled to the hand member 128 H In some examples, such as FIG. 1A , the hand member 128 H or end effector 128 H The end effector 128 may be a mechanical gripper including one or more movable jaws that are configured to perform different types of grasping of elements within the environment 10. In the example shown, the end effector 128 HThe invention comprises a fixed first jaw and a movable second jaw, which grasps an object by clamping the object between the jaws. The movable jaw can be configured to move relative to the fixed jaw so as to move between an open position of the gripper and a closed position of the gripper (e.g., tightly encircling the object).

[0056] In some embodiments, the arm 126 may include an additional joint J A , such as the fifth arm joint J A5 and / or sixth arm joint J A6 . Fifth joint J A5 Can be located on the upper member 128 U With hand component 128 H near the coupling and can be used to allow the hand member 128 H With respect to the lower member 128 U In other words, the fifth arm joint J A4 Can be used as a fourth arm joint similar to J A4 The torsion joint of the hand member 128H or the wrist joint of the arm 126 adjacent to the hand member 128H. For example, as a torsion joint, one member coupled at the joint J can move or rotate relative to another member coupled at the joint J (for example, the first member portion coupled at the torsion joint is fixed, while the second member portion coupled at the torsion joint rotates). Here, the fifth joint J A5 The arm 126 may also be configured to enable the hand member 128 to H Rotate in a rotating manner so that the hand member 128 H It is possible to yaw instead of pitch. For example, the fifth joint J A5 The arm 126 may be allowed to twist within a "180" degree range of motion so that the hand member 128 H The associated jaws may pitch, yaw, or some combination of the two. This may be useful for hooking portions of the arm 126 around an object or improving the grip of the hand member 128. H The way the object is grasped may be advantageous. A6 Can be connected with the fifth arm joint J A5 Functions similarly (e.g., as a torsion joint). For example, the sixth arm joint J A6 It is also possible to allow a portion of the arm member 128 (eg, the upper arm member 128 U ) rotates or twists within a "180" degree range of motion (eg, relative to another portion of the arm member 128 or another arm member 128). Here, the fifth arm joint J A5 and the sixth arm joint J A6The combination of the range of motion of can achieve "360" degree rotation. In some embodiments, the arm 126 can be connected to the robot 100 at a socket on the body 110 of the robot 100. In some configurations, the socket can be configured as a connector so that the arm 126 can be attached to the robot 100 or detached from the robot 100, depending on whether the arm 126 is needed for operation. In some examples, the first arm joint J A1 and the second arm joint J A2 It may be located at, adjacent to, or a portion of a socket that connects the arm 126 to the body 110 .

[0057] The robot 100 may have a vertical gravity axis (eg, shown as a Z-axis A) along the direction of gravity. Z ) and the center of mass CM, which is the point where the sum of the weighted relative positions of the masses distributed by the robot 100 is zero. The robot 100 may also have a vertical gravity axis A Z The robot 100 may be configured to define a particular posture or stance taken by the robot 100 by referring to a CM-based posture P of the robot 100 (i.e., a fixed reference frame with respect to gravity). The posture of the robot 100 may be defined by the orientation or angular position of the robot 100 in space. Movement of the legs 120 relative to the body 110 may modify the posture P of the robot 100 (i.e., a combination of the position of the robot's CM and the posture or orientation of the robot 100). Here, height (i.e., vertical distance) generally refers to the height along (e.g., parallel to) the z-direction (i.e., z-axis A). Z ) The sagittal plane of the robot 100 corresponds to the axis A in the y direction. Y and z-axis A Z In other words, the sagittal plane divides the robot 100 into left and right sides. Generally perpendicular to the sagittal plane, the ground plane (also called the transverse plane) passes through the x-axis A X and the y-axis A Y The ground plane refers to a support surface 14 where the distal ends 124 of the legs 120 of the robot 100 can generate traction to help the robot 100 move around the environment 10. Another anatomical plane of the robot 100 is a frontal plane that extends across the body 110 of the robot 100 (e.g., from the left side of the robot 100 with the first leg 120a to the right side of the robot 100 with the second leg 120b). The frontal plane passes through the x-axis Ax and the z-axis Ax. Z The XZ plane extends in the direction of the axis.

[0058] As a legged robot moves around an environment 10, the robot's legs 120 may undergo a gait cycle. Generally, a gait cycle begins when the leg 120 lands or contacts the support surface 14 and ends when the same leg 120 contacts the ground surface 14 again. The landing of the leg 120 may also be referred to as a "footfall," which defines the point or location at which the distal end 124 of the motion-based structure 120 lands in contact with the support surface 14. The gait cycle may be primarily divided into two phases, a swing phase and a stance phase. During the swing phase, the leg 120 may experience (i) lifting off from the support surface 14 (sometimes also referred to as toe-off and the transition between the stance phase and the swing phase), (ii) the knee joint J of the leg 120 may experience K 14, (iii) flexion of the knee joint JK of the leg 120, and (iv) landing (or stepping) back to the support surface 14. Here, the leg 120 in the swing phase is referred to as the swing leg 120. SW When swinging legs 120 SW Perform swing phase 120 SW , the other leg 120 performs a stance phase. The stance phase refers to the period of time when the distal end 124 (e.g., foot) of the leg 120 is located on the support surface 14. During the stance phase, the leg 120 may experience (i) initial support surface contact, which triggers the transition from the swing phase to the stance phase, (ii) a load response, in which the leg 120 inhibits support surface contact, (iii) an intermediate stance support, which is used when the contralateral leg (i.e., the swing leg 120) SW ) when lifted off and swung to the equilibrium position (approximately halfway through the swing phase), and (iv) the final stance support, which is used from the time when the robot's CM is above the leg 120 until the contralateral leg 120 lands on the support surface 114. Herein, the leg 120 in the stance phase is referred to as the stance leg 120. ST .

[0059] To maneuver around the environment 10 or perform tasks using the arm 126, the robot 100 may include a sensor system 130 having one or more sensors 132, 132a-132n. For example, FIG. 1A illustrates a first sensor 132, 132a mounted at the head of the robot 100, a second sensor 132, 132b mounted near the hip of the second leg 120b of the robot 100, a third sensor 132, 132c corresponding to one of the sensors 132 mounted on a side of the body 110 of the robot 100, a fourth sensor 132, 132d mounted near the hip of the fourth leg 120d of the robot 100, and an end effector 128 mounted on the arm 126 of the robot 100. HA fifth sensor 132, 132e is provided at or near the sensor 132. The sensor 132 may include a vision / image sensor, an inertial sensor (e.g., an inertial measurement unit (IMU)), a force sensor, and / or a motion sensor. Some examples of the sensor 132 include a camera, such as a stereo camera, a time-of-flight (TOF) sensor, a scanning light-detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor. In some embodiments, the respective sensor 132 may have a corresponding field of view F that defines a sensing range or area corresponding to the sensor 132. v For example, FIG. 1A depicts the field of view F of the robot 100. v Each sensor 132 may be pivotable and / or rotatable such that the sensor 132 may change its field of view F, for example, about one or more axes (eg, an x-axis, a y-axis, or a z-axis relative to a ground plane). V .

[0060] In some embodiments, the sensor system 130 may include sensors 132 coupled to the joints J. In some embodiments, these sensors 132 may be coupled to motors (e.g., sensors 132, 132a-b) that operate the joints J of the robot 100. Here, these sensors 132 may generate joint dynamics in the form of joint-based sensor data 134 (shown in FIG. 1B). The joint dynamics collected as the joint-based sensor data 134 may include joint angles (e.g., the upper member 122u relative to the lower member 122 L ), joint velocity (e.g., joint angular rate or joint angular acceleration), and / or joint torque (also referred to as joint force) experienced at joint J. Here, the joint-based sensor data 134 generated by one or more sensors 132 can be raw sensor data, data that is further processed to form different types of joint dynamics, or some combination of the two. For example, the sensor 132 can measure the joint position (or the position of the member 122 coupled at the joint J), and the system of the robot 100 can perform further processing to derive the rate and / or acceleration from the position data. In other examples, one or more sensors 132 can be configured to directly measure the rate and / or acceleration.

[0061] When the sensor 132 is used to survey the field of view F V When the sensor system 130 can also generate a corresponding field of view F VThe sensor system 130 may generate a field of view F using sensors 132 mounted on or near the body 110 of the robot 100, such as (one or more) sensors 132a, 132b. v The sensor system may additionally and / or alternatively utilize an end effector 128 mounted on the arm 126 H A sensor 132 at or near the sensor 132c, for example, sensor(s) 132c, is used to generate a field of view F v .

[0062] The one or more sensors 132 may capture sensor data 134 that defines a three-dimensional point cloud of a field within the environment 10 surrounding the robot 100. In some examples, the sensor data 134 may be image data corresponding to the three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132.

[0063] Additionally or alternatively, the sensor system 130 may collect posture data of the robot 100, including inertial measurement data (e.g., inertial measurement data measured by an IMU), as the robot 100 maneuvers around the environment 10. In some examples, the posture data may include kinematic data and / or orientation data about the robot 100, such as kinematic data and / or orientation data about a joint J or other portion of a leg 120 or arm 126 of the robot 100. Using the sensor data 134, various systems of the robot 100 may use the sensor data 134 to define a current state of the robot 100 (e.g., a current state of the kinematics of the robot 100) and / or a current state of the environment 10 around the robot 100.

[0064] As the sensor system 130 collects sensor data 134, the computing system 140 may store, process, and / or transmit the sensor data 134 to various systems of the robot 100 (e.g., the computing system 140, the control system 170, the perception system 180, and / or the navigation system 200). To perform computing tasks related to the sensor data 134, the computing system 140 of the robot 100 may include data processing hardware 142 and memory hardware 144. The data processing hardware 142 may be configured to execute instructions stored in the memory hardware 144 to perform computing tasks related to the activities (e.g., movement and / or movement-based activities) of the robot 100. In general, the computing system 140 refers to one or more instances of the data processing hardware 142 and / or the memory hardware 144.

[0065] 1A and 1B , in some embodiments, the computing system 140 may be a local system located on the robot 100. When located on the robot 100, the computing system 140 may be centralized (i.e., in a single location / domain on the robot 100, e.g., the body 110 of the robot 100), decentralized (i.e., located at various locations around the robot 100), or a hybrid combination of the two (e.g., with a majority of centralized hardware and a small amount of decentralized hardware). A decentralized computing system 140 may, for example, allow processing to occur at active locations (e.g., at the motors of the joints of the moving legs 120), while a centralized computing system 140 may, for example, allow a central processing hub to communicate with systems located at various locations on the robot 100 (e.g., communicating with the motors of the joints of the moving legs 120).

[0066] Additionally or alternatively, the computing system 140 may include computing resources located remotely from the robot 100. For example, the computing system 140 may communicate with a remote system 160 (e.g., a remote computer / server or a cloud-based environment) via a network 150. Much like the computing system 140, the remote system 160 may include remote computing resources, such as remote data processing hardware 162 and remote memory hardware 164. Here, sensor data 134 or other processed data (e.g., data processed locally by the computing system 140) may be stored in the remote system 160 and may be accessed by the computing system 140. In some embodiments, the computing system 140 may be configured to utilize the remote resources 162, 164 as an extension of the computing resources 142, 144, such that resources of the computing system 140 may reside on resources of the remote system 160.

[0067] In some embodiments, as shown in FIGS. 1A and 1B , the robot 100 may include a control system 170 and a perception system 180. The perception system 180 may be configured to receive sensor data 134 from the sensor system 130 and process the sensor data 134 to generate one or more perception maps 182. The perception system 180 may transmit such perception map(s) 182 to the control system 170 in order to perform controlled actions for the robot 100, such as moving the robot 100 around the environment 10. In some embodiments, by having the perception system 180 separate from the control system 170 but in communication with the control system 170, the processing of the control system 170 may be focused on controlling the robot 100, while the processing of the perception system 180 may be focused on interpreting the sensor data 134 collected by the sensor system 130. For example, these systems 170, 180 may run their processing in parallel to ensure accurate and smooth movement of the robot 100 in the environment 10.

[0068] In some embodiments, the control system 170 may include one or more controllers 172, a path generator 174, a step locator 176, and a body planner 178. The control system 170 may be configured to communicate with at least one sensor system 130 and any other systems of the robot 100 (e.g., the perception system 180 and / or the navigation system 200). The control system 170 may use the hardware 140 to perform operations and other functions. The controller 172 may be configured to control the movement of the robot 100 to traverse the environment 10 based on input or feedback from the systems of the robot 100 (e.g., the control system 170, the perception system 180, and / or the navigation system 200). This may include movement between postures and / or behaviors of the robot 100. For example, the controller 172 may control different footstep patterns, leg patterns, body movement patterns, or vision system sensing patterns.

[0069] In some embodiments, the controller 172 may include multiple controllers 172, where each controller 172 may be configured to operate the robot 100 at a fixed cadence. A fixed cadence refers to a fixed timing of the steps or swing phases of the legs 120. For example, a separate controller 172 may command the robot 100 to move the legs 120 (e.g., take a step) at a specific frequency (e.g., a step every 250 milliseconds, 350 milliseconds, etc.). With multiple controllers 172, where each controller 172 is configured to operate the robot 100 at a fixed cadence, the robot 100 may experience variable timing by switching between different controllers 172. In some embodiments, the robot 100 may continuously switch / select a fixed cadence controller 172 (e.g., reselecting a controller 170 every three milliseconds) as the robot 100 traverses the environment 10.

[0070] In some embodiments, the control system 170 may additionally or alternatively include one or more dedicated controllers 172 dedicated to a specific control purpose. For example, the control system 170 may include one or more stair controllers dedicated to planning and coordinating the movement of the robot to traverse a set of stairs. For example, the stair controller may ensure that the swing leg 120 SW The walking path of the embodiment of the present invention maintains the swing height to cross the riser and / or edge of the stairs. Other special controllers 172 may include a path generator 174, a step locator 176 and / or a body planner 178.

[0071] 1B , the path generator 174 may be configured to determine horizontal motion of the robot 100. As used herein, the term “horizontal motion” refers to translation (i.e., movement in the XY plane) and / or yaw (i.e., movement around the Z-axis A) of the robot 100. z10). The path generator 174 can determine obstacles within the environment 10 surrounding the robot 100 based on the sensor data 134. The path generator 174 can determine the trajectory of the robot's body 110 within some future time period (e.g., within the next second). However, such determination of the trajectory of the body 110 by the path generator 174 can occur more frequently, such as hundreds of times per second. In this way, in some embodiments, the path generator 174 can determine a new trajectory for the body 110 every few milliseconds, where each new trajectory is planned for a time period of about one second in the future.

[0072] The path generator 174 may communicate information about the currently planned trajectory and the identified obstacles to the step locator 176 so that the step locator 176 may identify foot drop points for the legs 120 of the robot 100 (e.g., where the distal ends 124 of the legs 120 of the robot 100 are placed). The step locator 176 may use input from the perception system 180 (e.g., the perception map(s) 182) to generate foot drop points (i.e., where the robot 100 should step). The body planner 178, much like the step locator 176, may receive input from the perception system 180, e.g., the perception map(s) 182. In general, the body planner 178 may be configured to adjust the dynamics (e.g., rotations, such as pitch or yaw and / or height of the CM) of the body 110 of the robot 100 to successfully move around the environment 10.

[0073] The perception system 180 can enable the robot 100 to move more accurately in terrain with various obstacles. When the sensors 132 collect sensor data 134 of the space around the robot 100 (i.e., the robot's environment 10), the perception system 180 can use the sensor data 134 to form one or more perception maps 182 of the environment 10. In some embodiments, the perception system 180 can also be configured to modify an existing perception map 182 (e.g., by projecting the sensor data 134 onto a pre-existing perception map) and / or remove information from the perception map 182.

[0074] In some embodiments, the one or more perception maps 182 generated by the perception system 180 may include a ground height map 182, 182a, a no-step map 182, 182b, and a body obstacle map 182, 182c. The ground height map 182a refers to a perception map 182 generated by the perception system 180 based on voxels from a voxel map. In some embodiments, the ground height map 182a may function such that at each XY position within a grid of the perception map 182 (e.g., designated as a cell of the ground height map 182a), the ground height map 182a specifies a height. In other words, the ground height map 182a may communicate that at a particular XY position in the horizontal plane, the robot 100 should step at a certain height.

[0075] The no-step map 182b generally refers to the perception map 182, which defines areas where the robot 100 is not allowed to take steps, so as to inform the robot 100 when the robot 100 can take steps at a specific horizontal position (i.e., a position in the XY plane). In some embodiments, much like the body obstacle map 182c and the ground height map 182a, the no-step map 182b can be divided into a grid of cells, where each cell represents a specific area in the environment 10 of the robot 100. For example, each cell can correspond to three square centimeters in the XY plane within the environment 10. When the perception system 180 generates the no-step map 182b, the perception system 180 can generate a Boolean map, where the Boolean map identifies no-step areas and step areas. The no-step area refers to an area of ​​one or more cells where obstacles exist, and the step area refers to an area of ​​one or more cells where the presence of obstacles is not sensed. The perception system 180 can further process the Boolean map so that the no-step map 182b includes a signed distance field. Here, the signed distance field of the no-step map 182b may include a distance to an obstacle boundary (eg, a distance to a boundary of the no-step region 244) and a vector “v” to the obstacle boundary (eg, defining the closest direction to the boundary of the no-step region 244).

[0076] The body obstacle map 182c can be used to determine whether the body 110 of the robot 100 overlaps with a location in the XY plane relative to the robot 100. In other words, the body obstacle map 182c can identify obstacles of the robot 100 to indicate whether the robot 100 is at risk of collision or potential damage with obstacles near or at the same location by overlapping at the location in the environment 10. As a map of obstacles of the body 110 of the robot 100, the system of the robot 100 (e.g., the control system 170) can use the body obstacle map 182c to identify the boundaries adjacent to or closest to the robot 100, and to identify the direction (e.g., the best direction) to move the robot 100 to avoid the obstacle. In some embodiments, much like other perception maps 182, the perception system 180 can generate the body obstacle map 182c based on a cell grid (e.g., a grid of the XY plane). Here, each cell within the body obstacle map 182c can include a distance to the obstacle and a vector pointing to the closest cell that is the obstacle (i.e., the boundary of the obstacle).

[0077] With further reference to FIG. 1B , the robot 100 may also include a navigation system 200 and a task execution system 184. The navigation system 200 may be a system of the robot 100 that navigates the robot 100 along a path referred to as a navigation route 202 in order to traverse the environment 100. The navigation system 200 may be configured to receive the navigation route 202 as an input or to generate the navigation route 202 (e.g., to generate the route in its entirety or some portion thereof). In order to generate the navigation route 202 and / or to guide the robot 100 along the navigation route 202, the navigation system 200 may be configured to operate in conjunction with the control system 170 and / or the perception system 180. For example, the navigation system 200 may receive a perception map 182 that may inform decisions performed by the navigation system 200 or otherwise influence some form of mapping performed by the navigation system 200 itself. The navigation system 200 may operate in conjunction with the control system 170 such that one or more controllers 172 and / or dedicated controllers 174 , 176 , 178 may control movement of components of the robot 100 (eg, legs 120 and / or arms 126 ) to navigate along the navigation route 202 .

[0078] The task execution system 184, described in further detail below, may be the system of the robot 100 responsible for executing recorded tasks. A recorded task may, for example, specify a sequence of one or more actions that the robot 100 is to perform at corresponding waypoints 212 defined on the topological map 204 (shown in FIG. 2 ).

[0079] As further shown in FIG. 1B , in some embodiments, the robot controller 188 may communicate wirelessly (or wired) with the robot 100 (via the network 150 or otherwise) and may allow an operator to control the robot 100. In some embodiments, the robot controller 188 may be a tablet computer with "soft" UI controls for the robot 100 presented via a touch screen of the tablet computer. In other embodiments, the robot controller 188 may take the form of a conventional video game controller, but may include a display screen, and may include various physical buttons and / or soft buttons that may be pressed or otherwise manipulated to control the robot 100.

[0080] In some embodiments, an operator may use the robot controller 188 to initiate a task recording process. During such a process, the operator may direct the movement of the robot 100 (e.g., via the robot controller 188) and command the robot 100 to take various "task actions" (e.g., obtain sensor readings, monitor video, etc.) along the desired path of the task. An example of a user interface for controlling the operation of the robot 100 presented on the robot controller 188 is shown in FIG. 3A, which is described in more detail below. When a task is recorded, the robot 100 may generate a topological map 204 (as shown in FIG. 2), which includes waypoints 212 at various locations along its path, and edges 214 between these waypoints 212. In some embodiments, for each task action that the operator commands the robot to perform, a new waypoint 212 may be added to the topological map 204 generated on the robot 100. In addition, for each such task action, data may be stored in the topological map 204 and / or the task record to associate the task action identified in the task record with the waypoint 212 of the topological map 204 that performs the task action. In some implementations, at the end of the mission recording process, the topology map 204 generated during the mission recording may be transmitted to the robot controller 188 and / or another computing device in communication with (eg, wirelessly coupled to) the robot and may be stored in association with the mission recording.

[0081] After the task recording process, the task record and the associated topology map 204 (if not already present on the robot 100 ) may be provided to the robot 100 , and the robot 100 may be commanded (eg, autonomously) to run the recorded task.

[0082] A detailed description of the route runner 220 of the navigation system 200 will now be provided with reference to FIG. 2 . As described above, the navigation route 202 executed by the route runner 220 may include a sequence of instructions for moving the robot 100 along a path that corresponds to a sequence of waypoints 212 defined on the topological map 204 (shown in FIG. 2 ). As the route runner 220 guides the robot 100 through the movement of following the navigation route 202, the route runner 220 may determine whether the navigation route 202 is blocked by an object. As described above, in some embodiments, the navigation route 202 may include one or more features of the topological map 204. For example, as previously described, such a topological map 204 may include waypoints 212 and edges 214, and the navigation route 202 may indicate that the robot 100 will travel along a path that includes a specific sequence of these waypoints 212. In some embodiments, the navigation route 202 may also include movement instructions that specify how the robot 100 moves from one waypoint 212 to another waypoint 212. Such movement instructions may, for example, take objects or other obstacles into account when recording waypoints 212 and edges 214 to the topological map 204 .

[0083] Since the environment 10 can change dynamically from the time when the waypoint 212 is recorded to the topological map 204, the route runner 220 can be configured to determine whether the navigation route 202 is blocked by an object that was not identified when the waypoint 212 was previously recorded on the topological map 204 being used by the navigation route 202. Such an object can be considered an "unforeseen obstacle" in the navigation route 202 because the initial drawing process that notified the navigation route 202 did not identify the object in the position of the blocking object. For example, this may occur when an object is moved or introduced into the drawn environment.

[0084] As shown in FIG. 2 , when an unforeseen obstacle blocks the navigation route 202, the route runner 220 may attempt to generate an alternative path 206 to another feature on the topological map 204 that avoids the unforeseen obstacle. The alternative path 206 may temporarily deviate from the navigation route 202, but then resume the navigation route 202 after the deviation. Unlike other methods of generating obstacle avoidance paths, the route runner 220 attempts to only temporarily deviate from the navigation route 202 to avoid the unforeseen obstacle so that the robot 100 can return to using the road features used for the navigation route 202 (e.g., similar to the topological features from the topological map 204). In this sense, successful obstacle avoidance by the route runner 220 occurs when the obstacle avoidance path (i) avoids the unforeseen obstacle and (ii) enables the robot 100 to resume a portion of the navigation route 202. This technique may be advantageous in merging back with the navigation route 202 after obstacle avoidance because the navigation route 202 may be important for the work or task performance of the robot 100 (or the operator of the robot 100). For example, the operator of the robot 100 may have assigned the robot 100 a job to perform inspection work at waypoint 212 of the navigation route 202. The navigation system 200 is intended to facilitate the success of the robot's 100 work or mission by generating an obstacle avoidance route that continues on the navigation route 202 after avoiding obstacles.

[0085] For illustration, FIG. 1A depicts a robot 100 traveling along a navigation route 202 that includes three waypoints 212a-212c. While moving along a first portion of the navigation route 202 (e.g., shown as a first side 214a) from a first waypoint 212a to a second waypoint 212b, the robot 100 encounters an unforeseen obstacle 20, which is depicted as a partial pallet of boxes. The unforeseen obstacle 20 prevents the robot 100 from completing the first portion of the navigation route 202 to the second waypoint 212b. Here, the "X" above the second waypoint 212b indicates that the robot 100 cannot successfully travel to the second waypoint 212b given the pallet of boxes. As shown, the navigation route 202 generally has a second portion (e.g., shown as a second side 214b) extending from the second waypoint 212b to the third waypoint 212c. However, due to the unforeseen object 20, the route runner 220 generates an alternate path 206 that directs the robot 100 to move to avoid the unforeseen obstacle 20 and travel to a third waypoint 212c of the navigation route 202 (e.g., from a point along the first portion of the navigation route 202). In this regard, the robot 100 may not be able to successfully navigate to one or more waypoints 212, such as the second waypoint 212b, but may resume a portion of the navigation route 202 after avoiding the obstacle 20. For example, the navigation route 202 may include additional waypoints 212 after the third waypoint 212c, and the alternate path 206 may enable the robot 100 to continue to those additional waypoints 212 after the navigation system 200 directs the robot 100 to the third waypoint 212c via the alternate path 206.

[0086] As shown in FIG. 2 , and as briefly noted above, the navigation system 200 may include a navigation generator 210 operating in conjunction with a route generator 220. The navigation generator 210 (also referred to as the generator 210) may be configured to construct a topological map 204 (e.g., during a mission recording process) and to generate a navigation route 202 based on the topological map 204. To generate the topological map 204, the navigation system 200, and more specifically, the generator 210, may record sensor data corresponding to locations within the environment 10 that the robot 100 has traversed or is traversing as waypoints 212. As described above, the waypoints 212 may include representations of what the robot 100 senses at a particular location within the environment 10 (e.g., according to its sensor system 120). The generator 210 may generate the waypoints 212, for example, based on image data 134 collected by the sensor system 130 of the robot 100. For example, the robot 100 may perform an initial mapping process in which the robot 100 moves through the environment 10. While moving through the environment 10, systems of the robot 100, such as the sensor system 130, may collect data (e.g., sensor data 134) as a means of understanding the environment 10. By gaining an understanding of the environment 10 in this manner, the robot 100 may later use the information gathered from the initial mapping process, or derivatives thereof, to move around the environment 10 (e.g., autonomously, semi-autonomously, or with the assistance of a user).

[0087] In some implementations, the navigation generator 210 may construct the topology map 204 by running at least one waypoint heuristic (e.g., a waypoint search algorithm) that triggers the navigation generator 210 to record a waypoint drop at a particular location in the topology map 204. For example, such a waypoint heuristic may be configured to detect a threshold feature detection within the image data 134 at the location of the robot 100, for example, when generating or updating the topology map 204. The navigation generator 210 (e.g., using the waypoint heuristic) may identify features within the environment 10 that serve as reliable visual sensor features that provide repeatability for the robot 100 to maneuver around the environment 10. For example, the waypoint heuristic of the generator 210 may be pre-programmed for feature recognition (e.g., programmed with stored features), or programmed to identify features where spatial clustering of the volumetric image data 134 (e.g., corners of a room or edges of a wall) occurs. In response to at least one waypoint heuristic triggering a waypoint drop, the navigation generator 210 may record a waypoint 212 on the topology map 204. The navigation generator 210 can repeat the waypoint identification process as the robot 100 drives through the field (e.g., the robot environment 10). For example, the operator of the robot 100 can manually drive the robot 100 through the field for the initial mapping process of the waypoints 212 used to establish the topological map 204.

[0088] When recording each waypoint 212, the generator 210 can associate waypoint edges 214 (also referred to as edges 214) with consecutive pairs of corresponding waypoints 212, so that the topological map 204 generated by the generator 210 includes waypoints 212 and edges 214 between pairs of these waypoints 212. The edge 214 can indicate how one waypoint 212 (e.g., the first waypoint 212a) is related to another waypoint 212 (e.g., the second waypoint 212b). For example, the edge 214 can represent the positional relationship between a pair of adjacent waypoints 212. In other words, the edge 214 can represent a connection or a specified path between two waypoints 212 (e.g., the edge 214a shown in Figure 2 can represent a connection between the first waypoint 212a and the second waypoint 212b).

[0089] In some embodiments, each edge 214 can therefore represent a path between a pair of waypoints 212 that it interconnects (e.g., a movement path of the robot 100). In addition, in some embodiments, each edge 214 can also reflect additional useful information. In particular, the route runner 220 of the navigation system 200 can be configured to recognize specific annotations on the edges 214 and control other systems of the robot 100 to take actions indicated by such annotations. For example, one or more edges 214 can be annotated to include movement instructions that inform the robot 100 how to move or navigate between their interconnected waypoints 212. Such movement instructions can, for example, identify a pose transformation of the robot 100 before the robot 100 moves along the edge 214 between the two waypoints 212. Therefore, the pose transformation can describe one or more positions and / or orientations that the robot 100 assumes to successfully navigate along the edge 214 between the two waypoints 212. In some embodiments, the edge 214 can be annotated to specify a complete three-dimensional pose transformation (e.g., six numbers). Some of these numbers represent estimates, such as dead reckoning pose estimates, vision-based estimates, or other estimates based on kinematic and / or inertial measurements of the robot 100 .

[0090] In some embodiments, one or more edges 214 may additionally or alternatively include annotations that further provide an indication / description of the environment 10. Some examples of annotations include descriptions or indications that the edge 214 is associated with or located on a certain feature of the environment 10. For example, an annotation for an edge 214 may specify that the edge 214 is located on a staircase or through a doorway. Such annotations may assist the robot 100 during maneuvers, particularly when visual information is lost or absent (e.g., due to the presence of a doorway). In some configurations, the edge annotations may additionally or alternatively identify one or more directional constraints (which may also be referred to as "pose constraints"). Such directional constraints may, for example, specify an alignment and / or orientation (e.g., pose) of the robot 100 to enable it to navigate on or through a particular environmental feature. For example, such an annotation may specify a particular alignment or pose that the robot 100 is to assume before going up or down stairs or down a narrow hallway that may restrict the robot 100 from turning.

[0091] In some implementations, the sensor data 134 may be associated with each waypoint 212 of the topological map 204. Such sensor data 134 may have been collected by the sensor system 130 of the robot 100 when the generator 210 recorded the corresponding waypoint 212 to the topological map 204. The sensor data 134 stored for each waypoint 212 may enable the robot 100 to locate by comparing the real-time sensor data 134 collected while the robot 100 traverses the environment 10 according to the topological map 204 (e.g., via the route 202) with the sensor data 134 stored for the waypoint 212 of the topological map 204. In some configurations, after the robot 100 moves along the edge 214 (e.g., with the goal of reaching the target waypoint 212), the robot 100 may locate by directly comparing the real-time sensor data 134 with the sensor data 134 associated with the expected target waypoint 212 of the topological map 204. In some embodiments, by storing raw or near-raw sensor data 134 (i.e., data with minimal processing) for waypoints 212 of the topological map 204, the robot 100 can use the real-time sensor data 134 to effectively locate itself as the robot 100 maneuvers within the mapped environment 10. In some examples, an iterative closest point (ICP) algorithm can be used to locate the robot 100 relative to a given waypoint 212.

[0092] By using the waypoints 212 and the edges 214 to generate the topological map 204, the topological map 204 can be locally consistent (e.g., spatially consistent within the domain due to adjacent waypoints), but does not need to be globally accurate and / or consistent. That is, as long as the geometric relationship (e.g., edge 214) between the adjacent waypoints 212 is roughly accurate, the topological map 204 does not require the accurate global metric positioning of any sensed object in the robot 100 and the environment 10. Therefore, the navigation route 202 derived or constructed using the topological map 202 does not require accurate global metric information. In addition, because the topological map 204 can be constructed based on the relationship between the waypoints 212 and the waypoints (e.g., edge 214), the topological map 204 can be considered as an abstract or high-level map, rather than a metric map. That is, in some embodiments, the topological map 204 may not have other metric data about the drawn environment 10 that is unrelated to the waypoints 212 or their corresponding edges 214. For example, in some embodiments, the drawing process (e.g., performed by the generator 210) that creates the topological map 204 may not store or record other metric data, and / or the drawing process may remove the recorded metric data to form the topological map 204 of the waypoints 212 and the edges 214. In either case, navigating using the topological map 204 can simplify the hardware required for navigation and / or the computing resources used during navigation. That is, when compared with navigation using metric positioning that usually requires expensive LIDAR sensors and / or expensive IMU sensors, topology-based navigation can utilize low-cost vision and / or low-cost inertial measurement unit (IMU) sensor operations. Metric-based navigation tends to require more computing resources than topology-based navigation because metric-based navigation usually performs positioning (e.g., utilizing waypoints 212) at a much higher frequency than topological navigation. For example, common navigation methods using simultaneous localization and mapping (SLAM, Simultaneous Localization and Mapping) of a global occupancy grid continuously perform robot positioning.

[0093] 2 , the navigation generator 210 may record a plurality of waypoints 212, 212a-212n on the topological map 204. From the plurality of recorded waypoints 212, the navigation generator 210 may select a certain number of the recorded waypoints 212 as a sequence of waypoints 212 that form a navigation route 202 for the robot 100. In some embodiments, an operator of the robot 100 may use the navigation generator 210 to select or construct a sequence of waypoints 212 to form the navigation route 202. In some embodiments, the navigation generator 210 may generate the navigation route 202 based on receiving a destination location and a starting location of the robot 100. For example, the navigation generator 210 may match the starting location with the nearest waypoint 212, and similarly match the destination location with the nearest waypoint 212. Then, the navigation generator 210 may select a certain number of waypoints 212 between these nearest waypoints 212 to generate the navigation route 202.

[0094] In some configurations, the navigation generator 210 may receive, for example, a mission record as input from the mission execution system 184, and possibly also receive an associated topology map 204, and in response, may generate a navigation route 202 that includes various waypoints 212 included in the mission record, as well as intermediate waypoints 212 and edges between pairs of waypoints 212. For example, for a mission to inspect different locations on a pipeline, the navigation generator 210 may receive a mission record that identifies waypoints 212 at which inspections are to occur and a topology map 204 generated during the recording process, and may generate a navigation route 202 that includes waypoints 212 that coincide with the identified inspection locations. In the example shown in FIG. 2 , the navigation generator 210 has generated a navigation route 202 having a sequence of waypoints 212 that include nine waypoints 212a-212i and their corresponding edges 214a-214h. 2 illustrates each waypoint 212 of the navigation route 202 with a double circle, rather than recorded waypoints 212 of portions of the navigation route 202 having only a single circle. As illustrated, the navigation generator 210 may then transmit the navigation route 202 to the route runner 220.

[0095] The route runner 220 may be configured to receive and run the navigation route 202. To run the navigation route 202, the route runner 220 may coordinate with other systems of the robot 100 to control motion-based structures (e.g., legs) of the robot 100 to drive the robot 100 through the sequence of waypoints 212 included in the navigation route 202. For example, the route runner 220 may transmit movement instructions to the control system 170 that are associated with an edge 214 connecting the waypoints 212 in the sequence of waypoints 212 of the navigation route 202. The control system 170 may then use such movement instructions to position the robot 100 (e.g., in an orientation) according to one or more posture transformations to successfully move the robot 100 along the edge 214 of the navigation route 202.

[0096] As the robot 100 travels along the navigation route 202, the route runner 220 may also determine whether the robot 100 is unable to execute a particular move instruction for a particular edge 214. For example, the robot 100 may be unable to execute the move instruction for the edge 214 because the robot 100 encounters an unforeseen obstacle 20 while moving to the waypoint 212 along the edge 214. Here, the route runner 220 may identify that the unforeseen obstacle 20 is blocking the path of the robot 100 (e.g., using real-time or near real-time sensor data 134), and may be configured to determine whether there is an alternate path 206 for the robot 100 to an untraveled waypoint 212, 212U in the sequence of the navigation route 202. An untraveled waypoint 212U refers to a waypoint 212 of the navigation route 202 that the robot 100 has not yet successfully traveled to. For example, if the robot 100 has traveled to three waypoints 212a-212c of the nine waypoints 212a-212i of the navigation route 202, the route runner 220 can try to find an alternative path 206 to one or the remaining six waypoints 212d-212i, if possible. In this sense, the alternative path 206 can be an obstacle avoidance path that avoids unforeseen obstacles 20, and can also be a path that allows the robot 100 to resume the navigation route 202 (e.g., toward a specific goal or work). This means that after the robot 100 travels along the alternative path 206 to the destination of the untraveled waypoint 212U, the route runner 220 can continue to run the navigation route 202 from the destination of the alternative path 206. This approach can enable the robot 100 to return to navigation using the sparse topology map 204.

[0097] For example, referring to FIG. 2 , if an unforeseen obstacle 20 blocks a portion of the third side 214c (e.g., blocks some portions of the third side 214c and the fourth waypoint 212d), the robot 100 has already traveled to three waypoints 212a-212c. In this case, the route runner 220 can generate an alternative path 206 to the fifth waypoint 212e that avoids the unforeseen obstacle 20, and the fifth waypoint 212e is the untraveled waypoint 212U. The robot 100 can then continue to traverse the sequence of waypoints 212 of the navigation route 202 from the fifth waypoint 212e. This means that the robot 100 will then follow the sequence of waypoints 212 of the navigation route 202 to travel to the untraveled portion (e.g., by using the movement instructions of the side 214 of the untraveled portion). In the example shown, the robot 100 will therefore travel from the fifth waypoint 212e to the sixth, seventh, eighth, and finally ninth waypoints 212, 212f-212i, thereby preventing detection of some other unforeseen objects 20. This means that, despite the presence of unforeseen objects 20 along the third side 214c, the robot 100 only missed a single waypoint during its movement path while running the navigation route 202, namely the fourth waypoint 212d.

[0098] As described above, some embodiments include a robot controller 188 that can be manipulated by an operator to control the operation of the robot 100. In the illustrated example shown in FIG3A, the robot controller 188 is a computing device (e.g., a tablet computer such as a Samsung Galaxy Tab, an Apple iPad, or a Microsoft Surface) that includes a touch screen configured to present a plurality of "soft" UI control elements. As illustrated in FIG3A, in some embodiments, the screen 300 may present a pair of joystick controllers 302, 304, a pair of slider controllers 306, 308, a pair of mode selection buttons 310, 312, and a camera view selector switch 314.

[0099] In some embodiments, the mode selection buttons 310, 312 may allow the operator to place the robot 100 in a non-ambulatory mode, e.g., "Stand", upon selection of the mode selection button 310, or to place the robot 100 in an ambulatory mode, e.g., "Walk", upon selection of the mode selection button 312. For example, in response to selection of the mode selection button 310, the robot controller 188 may cause a first pop-up menu to be presented that allows the operator to select from a number of operating modes that do not involve translational movement of the robot 100 (i.e., movement in the XY directions). Examples of such non-ambulatory modes include "Sit" and "Stand". Similarly, in response to selection of the mode selection button 312, the robot controller 188 may cause a second pop-up menu to be presented that allows the operator to select from a number of operating modes that do involve translational movement of the robot 100. Examples of such ambulatory modes include "Walk", "Crawl", and "Stairs".

[0100] In some embodiments, the functionality of one or both of the joystick controllers 302, 304 and / or slider controllers 306, 308 may depend on the currently selected operating mode (selected via the mode selection buttons 310, 312). For example, when a non-ambulatory mode (e.g., "Stand") is selected, the joystick controller 302 may control the pitch (i.e., rotation about the X-axis) and yaw (i.e., rotation about the Z-axis) of the body 110 of the robot 100. z The joystick controller 302 may alternatively control the translation (i.e., movement in the XY plane) of the body 110 of the robot 100 when the walk mode (e.g., walk) is selected. The slider controller 306 may control the height of the body 110 of the robot 100, for example, to make it stand taller or squat. The slider controller 308 may control the speed of the robot 100 when the walk mode (e.g., walk) is selected. In some embodiments, the camera selector switch 314 may control which of the robot's cameras is selected to have its output displayed on the screen 300, and the joystick controller 304 may control the pan direction of the selected camera.

[0101] In some embodiments, a create button 316 presented on the screen 300 may enable an operator of the robot controller 188 to select and invoke a process for creating a new action for the robot 100, e.g., when recording a task. For example, if the operator of the robot 100 wants the robot 100 to acquire images of a particular piece of equipment within a facility, the operator may select the create button 316 to select and invoke a process for defining where and how to acquire the images. In some embodiments, in response to selecting the create button 316, the robot controller 188 may present a list of actions that may be created for the robot 100, e.g., as a drop-down menu or pop-up menu.

[0102] 3A illustrates how the screen 300 may appear after the user has selected the create button 316 and has further selected an action to capture thermal images using a thermal camera mounted on the robot. As shown, in some embodiments, the name of the selected action may be presented in a status bar 318 on the screen 300. As also shown, the screen 300 may also present instructions 320 for implementing the selected action, and may present a first UI button 322 for specifying a position at which the robot 100 begins to perform the action, and a second UI button 324 for the robot 100 to stop performing the action. In the case of capturing images during a mission, the robot's starting and ending positions may be the same, so only one of the UI button 322 or the UI button 324 may be displayed on the screen 300.

[0103] In some embodiments, when creating an action, a user may interact with a user interface to specify additional information about an asset of interest in an image to be associated with the image. For example, after capturing an image, a user may interact with a user interface to define a region of interest (ROI) within the image that includes a particular asset of interest. The user may then interact with the user interface to specify asset information to be included in a data structure that may be associated with the ROI and stored as metadata with the captured image. The asset information may include an asset identifier or "asset ID" that uniquely identifies an asset in an environment so that data (e.g., the ROI and / or the entire image) that has been associated with the asset identifier may later be identified and compared between images, actions, and tasks, as described in more detail below. In this way, according to some embodiments, the use of an asset identifier to identify an asset in sensor data decouples the sensor data from the specific actions and tasks for which it was recorded.

[0104] In some embodiments, the asset identifier is implemented as an alphanumeric string. The asset information included in the data structure may also include, but is not limited to, device category information associated with the asset and the location in the environment in which the corresponding image was captured. In some embodiments, the asset information may also include information about the posture of the robot when the image was acquired or any other suitable information. Because the asset information is associated with the specific action recorded by the task (e.g., captured sensor data), each time the robot runs a task, the sensor data corresponding to the action is recorded, and the asset identifier specified in the asset information and defined by the user can be automatically associated with all or part of the recorded sensor data (e.g., ROI).

[0105] In some instances, multiple assets of interest may be present in a single image captured by the robot. In this case, a user may interact with a user interface to define multiple ROIs in the image, each of which may be associated with different asset information including a unique asset identifier as described above. In some embodiments, the same asset of interest may be included in multiple images captured at different angles and / or at different waypoints, for example, during mission recording. In such instances, the same asset identifier may be associated with all or part of each of the multiple images, which may improve the reliability of the inspection system, such as in the event that one or more of the captured images show false results that should be ignored.

[0106] In embodiments where multiple ROIs are defined for different assets (either within a single image or across multiple images), information within the multiple ROIs may be analyzed and compared to perform one or more exception operations. For example, a first temperature (or other measured quantity, such as radiation, vibration, sound, pressure, etc.) of a first asset defined within a first ROI may be determined, a second temperature (or other measured quantity) of a second asset defined within a second ROI may be determined, and an alarm may be generated based at least in part on a comparison of the first temperature and the second temperature (or other measured quantity). Such an exception operation facilitates analysis of the relative temperature (or one or more other measured quantities) of assets in an environment. Although a single, identical measured quantity (e.g., temperature) has been used in the examples provided above, it should be understood that more than one measured quantity (e.g., temperature and radiation) and / or different measured quantities may be determined in different ROIs for comparison and alarm generation according to the techniques described herein.

[0107] 3B and 3C illustrate example portions of a user interface (e.g., which may be displayed on the robot controller 188) to enable a user to define a region of interest within a captured image. As shown in FIG3B , the user interface may be configured to display a captured image and information associated with a measured quantity (e.g., temperature) within the image. In the example of FIG3B , an isothermal image is shown along with information about detected temperatures within the image and within a region of interest defined by a user. A user may interact with the user interface to specify a region of interest 330 that includes assets to be monitored. A user may also interact with the user interface to specify asset information, which may be used, for example, to generate an alarm. As shown in FIG3B , when the monitored quantity is temperature, the asset information may include minimum and / or maximum temperature thresholds.

[0108] In the example of FIG3C , a color image is shown along with information about detected temperatures within the image and within a region of interest defined by a user. The user may interact with the user interface to specify a region of interest 340 that includes assets to be monitored. Similar to the portion of the user interface shown in FIG3B , the user may also interact with the user interface shown in FIG3C to specify asset information that may be used, for example, to generate an alert.

[0109] In some embodiments, the ROI and asset information are provided by the user during the recording of the task. In other embodiments, at least some of the ROI and / or asset information is provided by the user after the task is completed. For example, all images captured during the task can be stored and later viewed by the user, and the user can define the ROI and / or provide corresponding asset information for the viewed images or other sensor data. Whenever the user defines or provides the ROI and asset information, they are stored in the task record together with the corresponding action. Importantly, once the ROI and asset information are incorporated into the task record by associating it with the action of the task, the definition of the ROI and the assignment of asset information during the running task can be automatically performed each time the task is run thereafter, and the asset information includes a unique asset identifier on the newly captured image, i.e., no further manual intervention is required. In this way, a consistent set of sensor data collected over time is recorded, which can be used to monitor assets in an environment (such as an industrial facility), assess trends in the performance and / or characteristics of assets, and / or detect abnormal behavior for which an alarm should be generated.

[0110] 4 is a flow chart of a process 400 for performing automated asset inspection using a mobile robot according to some embodiments. In action 410, a region of interest (ROI) is defined within an image captured by the mobile robot. As described above, the ROI can be defined based on user input provided via a user interface (e.g., screen 300) during recording of a task. However, it should be understood that the ROI in the image can alternatively be defined after the task is completed based on an evaluation of the images captured during recording of the task. In some embodiments, the image captured during recording of the task may include an asset of interest, but may not have an ROI defined for the image. In such an embodiment, the entire image can be considered to be the defined ROI.

[0111] Process 400 then proceeds to act 420, where an asset identifier (asset ID) is associated with the ROI defined in act 410. The asset ID can be included along with other asset information, examples of which include, but are not limited to, a device class associated with the asset, a computer vision model for processing the image, and one or more parameters that can be used to configure the computer vision model to analyze the image. For example, when the image is a thermal image, the one or more parameters may include one or more temperature thresholds for determining whether an alarm should be generated, as described in more detail below. When the sensor data (e.g., image) is configured to capture quantities other than temperature (examples of which include, but are not limited to, radiation, vibration, pressure, sound), the one or more parameters may include other relevant metrics for generating an alarm.

[0112] Then, process 400 proceeds to act 430, where one or more parameters of a computer vision model are configured based on the information associated with the asset ID. In some implementations, the sensor data captured during the execution of the task includes a thermal image of an asset in the environment. In such an implementation, the asset ID can be associated with one or more parameters that specify one or more temperature thresholds used by the computer vision model to determine whether the temperature of the asset represented in the captured thermal image is above or below the threshold. It should be appreciated that the one or more parameters used to configure the computer vision model in act 430 may depend on the type of sensor data to be analyzed and / or the type of computer vision model used. For example, if the sensor data is a color image, the one or more parameters may include a color value threshold, if the sensor data is video data, the one or more parameters may include movement threshold information, if the sensor data is audio data, the one or more parameters may include frequency threshold information, etc. In some embodiments, the information associated with the asset ID can be used to select a particular computer vision model from a plurality of computer vision models associated with the robot. In some embodiments, the information associated with the asset ID can be used to determine when the sensor data should be processed with the computer vision model, for example, during or after the task is run.

[0113] After one or more parameters of the computer vision model have been configured in action 430, process 400 proceeds to action 440, where the captured image data within the ROI is processed by the configured computer vision model to determine whether an alarm should be generated. Then, process 400 proceeds to action 450, where it is determined whether an alarm should be generated based on the output of the computer vision model. If it is determined in action 450 that an alarm should be generated (e.g., because the temperature of the asset is above a temperature threshold specified in the computer vision model), then process 400 proceeds to action 460, where an alarm is generated. Non-limiting examples of generating alarms are provided with reference to FIGS. 6A-9 described in more detail below. Regardless of whether it is determined in action 450 that an alarm should be generated, process 400 proceeds to action 470, where sensor data (e.g., image data) and an associated asset ID are stored (e.g., in a storage device on the robot) for future analysis. As described above, one of the advantages of assigning asset IDs to assets in a monitoring environment is that the same asset ID can be used across actions and tasks, thereby enabling the automated asset inspection system to reliably monitor assets over any desired time period.

[0114] FIG. 5 illustrates a process 500 for generating a task record including an asset identifier according to some embodiments. In action 510, a mobile robot (such as the robot 100 shown in FIG. 1A) is navigated to traverse a route through an environment (such as an industrial facility). Then, the process 500 proceeds to action 520, wherein, during the robot's navigation through the environment, a task record including waypoints and edges is generated. As described above, as the operator navigates the robot, the waypoints and edges of the task record can be generated in the background so that the operator is unaware that they are being added to the task record. The process 500 then proceeds to action 530, wherein a first user input is received commanding the mobile robot to perform an action of the recorded sensor data. Similar to the generation of waypoints and edges, the creation and subsequent association of an action with a task record may not be apparent to the operator of the robot, except for the operator providing user input via the robot controller to initiate the execution of the action. In some embodiments, when an action is created, a new waypoint is added to the task record at the location where the action is performed, so that the action is associated with the new waypoint. The process 500 then proceeds to action 540, wherein a second user input identifying an asset within the sensor data is received via a user interface. For example, as described herein, the recorded sensor data may be an image, and a user may interact with a user interface presented on a robot controller to define a ROI within the image and associate asset information with the defined ROI. Process 500 then proceeds to act 550, where the asset information, including a unique asset ID identifying the asset of interest, is associated with the action in the mission record, such that when the mission is rerun, the same asset information is associated with the sensor data recorded when the robot performed the action at its associated waypoint along the route.

[0115] As discussed above in conjunction with process 400 in FIG. 4 , in some embodiments, after analyzing the sensor data within the defined ROI by the computer vision model, it may be determined that an alert should be generated. The alert may be provided to the user in any suitable manner, including, but not limited to, displaying an image highlighting the anomaly, providing an alert to a cloud service, or sending an alert to another computing device (e.g., a removable device) via an electronic message (e.g., email, etc.) or via an application installed on the computing device. FIG. 6 illustrates an example of providing an alert, where a captured thermal image is annotated with an overlay of a defined ROI and an indication that the temperature of an asset within the ROI is above a threshold value, the threshold value being set by the computer vision model analyzing the image.

[0116] As described in conjunction with process 400 in FIG. 4 , regardless of whether an alert is generated for a particular asset identified in the recorded sensor data, in some embodiments, all recorded sensor data and associated metadata including asset information are stored (e.g., on a storage device associated with the robot), which enables time-based analysis of asset data recorded and processed in a monitored facility. FIG. 7A illustrates a portion of a user interface showing multiple images captured during different actions and / or tasks, wherein each of the multiple images is associated with the same asset via its unique asset ID. In the example shown in FIG. 7A , the user interface includes descriptive data 710 and thumbnail images 720 for each image, wherein an overlay of the ROI is displayed when, for example, an alert has been generated for the image. Such a user interface enables a user to view historical data corresponding to each time sensor data for an asset was captured, regardless of when the task and action for the particular sensor data capture occurred.

[0117] FIG7B illustrates another portion of the user interface in which one of the thumbnail images 720 is selected for viewing. As shown, the ROI defined in the image can be superimposed on the image, and an indication of whether an alarm is generated for the ROI can be displayed. Also shown in the user interface of FIG7B are characteristics of the asset derived from the computer vision model used to analyze the image, and the characteristics of the tracked asset are captured across a series of images. For example, as shown, the maximum temperature of the asset within the ROI across all images, the minimum temperature of the asset, and the average temperature of the asset are shown. This analysis enables long-term trend analysis of the characteristics of the assets in the environment, and can be used, for example, to adjust one or more parameters (e.g., thresholds) used by the computer vision model to generate alarms.

[0118] FIG8 illustrates a portion of a user interface in which images of the same view of an asset recorded across multiple runs of a mission are displayed. As shown, each image may be annotated with information regarding whether an alarm was generated based on analysis of the images using a computer vision model. FIG9 illustrates a portion of a user interface in which monitored asset characteristics (e.g., average temperature) are plotted as a function of time based on images captured during multiple runs of a mission. Associating each asset in a monitored environment with a unique asset identifier enables visualization of time-based trends, which provides insight into the operational status of the asset without the need to manually inspect the asset as is typically required with existing inspection systems. In some embodiments, the trend analysis may be provided separately from or in conjunction with an image displayed on the user interface on which one or more alarms are shown.

[0119] In some embodiments, trend analysis performed according to the techniques described herein can be used for preventive maintenance of one or more assets in an environment. For example, trend analysis can reveal information that may not be readily apparent when only thresholds are used to generate alerts. As an example, trend analysis can reveal that a particular asset or multiple assets have been heating up over time and should be repaired or replaced before the asset fails. As another example, trend analysis can reveal that an asset has been experiencing increased vibration over time, which can indicate that the asset needs to be serviced, for example, to prevent damage to the asset. Trend analysis generated according to the techniques described herein can be useful for other reasons not mentioned herein, and the embodiments are not limited in this respect.

[0120] FIG. 10 illustrates an example configuration of a robotic device (or “robot”) 1000 in accordance with some embodiments. The robotic device 1000 may correspond, for example, to the robot 100 described above. The robotic device 1000 represents an illustrative robotic device configured to perform any of the techniques described herein. The robotic device 1000 may be configured to operate autonomously, semi-autonomously, and / or using user-provided directions, and may exist in various forms, such as a humanoid robot, a bipedal, a quadrupedal, or other mobile robot, etc. In addition, the robotic device 1000 may also be referred to as a robotic system, a mobile robot, or a robot, among other names.

[0121] As shown in Figure 10, the robotic device 1000 may include a processor 1002, a data storage device 1004, a program instruction 1006, a controller 1008, a sensor 1010, a power supply 1012, a mechanical component 1014, and an electrical component 1016. The robotic device 1000 is shown for illustrative purposes and may include more or fewer components without departing from the scope of the present disclosure. The various components of the robotic device 1000 can be connected in any manner, including via electronic communication means, such as wired or wireless connections. In addition, in some examples, the components of the robotic device 1000 can be placed on multiple different physical entities rather than on a single physical entity.

[0122] The processor 1002 may operate as one or more general purpose processors or special purpose processors (e.g., digital signal processors, application specific integrated circuits, etc.). The processor 1002 may correspond, for example, to the data processing hardware 142 of the robot 100 described above. The processor 1002 may be configured to execute computer readable program instructions 1006, which are stored in the data storage 1004 and are executable to provide the operation of the robotic device 1000 described herein. For example, the program instructions 1006 may be executable to provide the operation of the controller 1008, wherein the controller 1008 may be configured to cause the activation and / or deactivation of the mechanical components 1014 and the electrical components 1016. The processor 1002 may operate and enable the robotic device 1000 to perform various functions, including the functions described herein.

[0123] The data storage device 1004 may exist as various types of storage media, such as memory. The data storage device 1004 may correspond, for example, to the memory hardware 144 of the robot 100 described above. The data storage device 1004 may include or take the form of one or more non-transitory computer-readable storage media that can be read or accessed by the processor 1002. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disk storage devices, which may be fully or partially integrated with the processor 1002. In some embodiments, the data storage device 1004 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other embodiments, the data storage device 1004 may be implemented using two or more physical devices that can communicate electronically (e.g., via wired or wireless communication). In addition, in addition to the computer-readable program instructions 1006, the data storage device 1004 may also include additional data, such as diagnostic data, etc.

[0124] The robotic device 1000 may include at least one controller 1008, which may be interfaced with the robotic device 1000 and may be integral with the robotic device, or separate from the robotic device 1000. The controller 1008 may serve as a link between parts of the robotic device 1000, such as a link between mechanical components 1014 and / or electrical components 1016. In some cases, the controller 1008 may serve as an interface between the robotic device 1000 and another computing device. In addition, the controller 1008 may serve as an interface between the robotic system 1000 and a user. The controller 1008 may include various components for communicating with the robotic device 1000, including one or more joysticks or buttons and other features. The controller 1008 may also perform other operations of the robotic device 1000. Other examples of controllers may also exist.

[0125] In addition, the robotic device 1000 may include one or more sensors 1010, such as image sensors, force sensors, proximity sensors, motion sensors, load sensors, position sensors, touch sensors, depth sensors, ultrasonic range sensors, and / or infrared sensors, or combinations thereof, etc. The sensors 1010 may correspond, for example, to the sensors 132 of the robot 100 described above. The sensors 1010 may provide sensor data to the processor 1002 to allow for proper interaction of the robotic system 1000 with the environment and to monitor the operation of the system of the robotic device 1000. The sensor data may be used by the controller 1008 and / or a computing system of the robotic device 1000 to evaluate various factors for activation and deactivation of the mechanical components 1014 and the electrical components 1016.

[0126] The sensor 1010 may provide information indicative of the environment of the robotic device for use by the controller 1008 and / or computing system in determining the operation of the robotic device 1000. For example, the sensor 1010 may capture data corresponding to the topography of the environment or the location of nearby objects, which may assist in environmental recognition and navigation, etc. In an example configuration, the robotic device 1000 may include a sensor system that may include a camera, a radar (RADAR), a LIDAR, a time-of-flight camera, a global positioning system (GPS) transceiver, and / or other sensors for capturing information of the environment of the robotic device 1000. The sensor 1010 may monitor the environment in real time and detect obstacles, terrain elements, weather conditions, temperature, and / or other parameters of the environment of the robotic device 1000.

[0127] Additionally, the robotic device 1000 may include other sensors 1010 configured to receive information indicative of a state of the robotic device 1000, including sensors 1010 that may monitor the state of various components of the robotic device 1000. The sensors 1010 may measure the activity of the system of the robotic device 1000 and receive information based on the operation of various features of the robotic device 1000, such as the operation of extendable legs, arms, or other mechanical and / or electrical features of the robotic device 1000. The sensor data provided by the sensors may enable a computing system of the robotic device 1000 to determine errors in operation and to monitor the overall functionality of components of the robotic device 1000.

[0128] For example, the computing system may use sensor data to determine the stability of the robotic device 1000 during operation and measurements related to power levels, communication activity, parts that need repair, and other information. As an example configuration, the robotic device 1000 may include a gyroscope, an accelerometer, and / or other possible sensors to provide sensor data related to the operating state of the robotic device. In addition, the sensor 1010 may also monitor the current state of the function that the robotic system 1000 may currently be operating, such as gait. In addition, the sensor 1010 may measure the distance between a given robotic leg of the robotic device and the center of mass of the robotic device. Other example uses of (one or more) sensors 1010 may also exist.

[0129] In addition, the robotic device 1000 may also include one or more power sources 1012, which are configured to provide power to various components of the robotic device 1000. Among possible power systems, the robotic device 1000 may include a hydraulic system, an electrical system, a battery, and / or other types of power systems. As an example, the robotic device 1000 may include one or more batteries, which are configured to provide power to the components via a wired and / or wireless connection. In an example, the components of the mechanical components 1014 and the electrical components 1016 can each be connected to a different power source or can be powered by the same power source. The components of the robotic system 1000 can also be connected to multiple power sources.

[0130] In an example configuration, any suitable type of power source can be used to power the robotic device 1000, such as gasoline and / or electric engines. In addition, (one or more) power sources 1012 can be charged using various types of charging, such as a wired connection to an external power source, wireless charging, combustion, or other examples. Other configurations are also possible. In addition, the robotic device 1000 may include a hydraulic system that is configured to use fluid power to provide power to the mechanical components 1014. For example, the components of the robotic device 1000 can be operated based on hydraulic fluid transmitted to various hydraulic motors and hydraulic cylinders throughout the hydraulic system. The hydraulic system of the robotic device 1000 can transmit a large amount of power through small tubes, flexible hoses, or other links between the components of the robotic device 1000. Other power sources may be included in the robotic device 1000.

[0131] The mechanical components 1014 may represent hardware of the robotic system 1000 that may enable the robotic device 1000 to operate and perform physical functions. As a few examples, the robotic device 1000 may include (one or more) actuators, (one or more) extendable legs ("legs"), (one or more) arms, (one or more) wheels, one or more structural bodies for housing computing systems or other components, and / or other mechanical components. The mechanical components 1014 may depend on the design of the robotic device 1000, and may also be based on the functions and / or work that the robotic device 1000 may be configured to perform. Thus, depending on the operation and function of the robotic device 1000, the robotic device 1000 may use different mechanical components 1014. In some examples, the robotic device 1000 may be configured to add and / or remove mechanical components 1014, which may involve assistance from a user and / or other robotic devices. For example, the robotic device 1000 may initially be configured with four legs, but may be changed by a user or the robotic device 1000 to remove two of the four legs to operate as a bipedal robot. Other examples of mechanical components 1014 may be included.

[0132] The electrical components 1016 may include, for example, various components capable of processing, transferring, providing an electric charge or an electrical signal. In possible examples, the electrical components 1016 may include wires, circuits, and / or wireless communication transmitters and receivers to enable the operation of the robotic device 1000. The electrical components 1016 may cooperate with the mechanical components 1014 to enable the robotic device 1000 to perform various operations. For example, the electrical components 1016 may be configured to provide power from the power source 1012 to the various mechanical components 1014. In addition, the robotic device 1000 may include an electric motor. Other examples of the electronic components 1016 may also exist.

[0133] In some embodiments, the robotic device 1000 may also include a communication link 1018 configured to send and / or receive information. The communication link 1018 may transmit data indicative of the status of various components of the robotic device 1000. For example, information read in by the sensor(s) 1010 may be transmitted to a separate device via the communication link(s) 1018. Other diagnostic information indicative of the integrity or health of the power source 1012, mechanical components 1014, electrical components 1018, processor 1002, data storage device 1004, and / or controller 1008 may be transmitted to an external communication device via the communication link 1018.

[0134] In some implementations, the robotic device 1000 may receive information processed by the processor 1002 at the communication link 1018. For example, the received information may indicate data accessible by the processor 1002 during execution of the program instructions 1006. Additionally, the received information may alter aspects of the controller 1008 that may affect the behavior of the mechanical component 1014 or the electrical component 1016. In some cases, the received information indicates a query requesting a particular piece of information (e.g., an operational status of one or more components of the robotic device 1000), and the processor 1002 may then transmit the particular piece of information back to the communication link 1018.

[0135] In some cases, the communication link 1018 comprises a wired connection. The robotic device 1000 may include one or more ports to connect the communication link 1018 to an external device. In addition to or as an alternative to a wired connection, the communication link 1018 may include a wireless connection. Some example wireless connections may utilize a cellular connection, such as CDMA, EVDO, GSM / GPRS, or 4G telecommunications, such as WiMAX or LTE. Alternatively or in addition, the wireless connection may utilize a Wi-Fi connection to transmit data to a wireless local area network (WLAN). In some embodiments, the wireless connection may also communicate via an infrared link, radio, Bluetooth, or a near field communication (NFC) device.

[0136] The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be run on any suitable processor or set of processors, whether provided in a single computer or distributed between multiple computers. It should be understood that any component or set of components that perform the above functions may generally be considered to be one or more controllers that control the above functions. One or more controllers may be implemented in a variety of ways, such as using dedicated hardware or using one or more processors programmed using microcode or software to perform the above functions.

[0137] Various aspects of the present technology can be used alone, in combination, or in various arrangements not specifically described in the previously described embodiments, and therefore its application is not limited to the details and arrangements of components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment can be combined in any manner with aspects described in other embodiments.

[0138] In addition, some embodiments may be implemented as one or more methods, examples of which have been provided. The actions performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from the order shown, which may include performing some actions simultaneously, even though shown as sequential actions in the illustrative embodiments.

[0139] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not by itself imply any priority, precedence, or order of one claim element relative to another claim element or temporal order of the acts of performing the method. These terms are merely used as labels to distinguish one claim element with a particular name from another element with the same name (but using an ordinal term).

[0140] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof, is intended to encompass the items listed thereafter and additional items.

[0141] Having described several embodiments in detail, it will be readily apparent to those skilled in the art that various modifications and improvements will be within the spirit and scope of the present technology. Therefore, the foregoing description is intended to be exemplary only and is not intended to be limiting.

Claims

1. A method comprising: defining a region of interest within an image captured by a sensor of a robot, the region of interest including an asset in an environment of the robot, in, The asset is associated with an asset identifier; configuring at least one parameter of a computer vision model based on the asset identifier; processing image data within the region of interest using the computer vision model to determine whether an alert should be generated; as well as When it is determined that the alarm should be generated, the alarm is output.

2. The method according to claim 1, in, Defining the area of ​​interest includes defining the area of ​​interest using asset information stored in a data structure associated with a task record.

3. The method according to claim 2, in, The data structure is associated with the action of capturing the image at a first waypoint indicated in the mission record, and The asset identifier is included in the data structure.

4. The method according to claim 2, in, The data structure includes the at least one parameter of the computer vision model.

5. The method according to claim 1, in, The image captured by the sensor of the robot is a thermal image.

6. The method according to claim 5, in, The at least one parameter of the computer vision model includes a temperature threshold.

7. The method according to claim 6, in, Processing image data within the region of interest using the computer vision model to determine whether an alert should be generated comprises: determining a temperature of the asset based on an analysis of the thermal image within the region of interest; comparing the determined temperature of the asset to the temperature threshold; as well as When the determined temperature meets or exceeds the temperature threshold, it is determined that an alarm is generated.

8. The method according to claim 1, in, The at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold.

9. The method according to claim 1, in, Outputting the alert includes displaying on a display a representation of the image annotated with an indication of the alert.

10. The method according to claim 1, in, Outputting the alert includes sending a message to a computing device via at least one network, the message including the alert.

11. The method according to claim 1, further comprising: The image and metadata indicative of the asset identifier are stored on at least one storage device.

12. The method according to claim 1, in, The region of interest is a first region of interest including a first asset, and the method further comprises: defining a second region of interest within the image, the second region of interest including a second asset in the environment of the robot, Wherein, using the computer vision model to process the image data in the region of interest to determine whether the alarm should be generated comprises: processing the image data within the first region of interest using the computer vision model to determine a first result; processing the image data within the second region of interest using the computer vision model to determine a second result; and A determination is made whether the alert should be generated based at least in part on the first result and the second result.

13. The method according to claim 1, further comprising: for each of a plurality of images captured over time, each of the images having the region of interest defined therein, processing image data within the region of interest of the image using the computer vision model to determine at least one quantity associated with the asset; and Based on at least one quantity associated with the asset determined for the plurality of images, a trend analysis for the at least one quantity is generated.

14. The method according to claim 13, in, The at least one quantity comprises one or more of temperature, pressure, vibration and radiation quantity.

15. The method according to claim 13, further comprising: An indication of the trend analysis for the at least one quantity is provided on a user interface.

16. The method according to claim 14, further comprising: The alert is generated based at least in part on the trend analysis.

17. A robot comprising: a perception system comprising an image sensor configured to capture an image; and at least one computer processor configured to: defining a region of interest within an image captured by the image sensor, the region of interest including an asset in an environment of the robot, in, The asset is associated with an asset identifier; configuring at least one parameter of a computer vision model based on the asset identifier; processing image data within the region of interest using the computer vision model to determine whether an alert should be generated; as well as When it is determined that the alarm should be generated, the alarm is output.

18. The robot according to claim 17, in, Defining the area of ​​interest includes defining the area of ​​interest using asset information stored in a data structure associated with a task record.

19. The robot according to claim 18, in, The data structure is associated with the action of capturing the image at a first waypoint indicated in the mission record, and The asset identifier is included in the data structure.

20. The robot according to claim 18, in, The data structure includes the at least one parameter of the computer vision model.

21. The robot according to claim 17, in, The image captured by the image sensor of the robot is a thermal image.

22. The robot according to claim 21, in, The at least one parameter of the computer vision model includes a temperature threshold.

23. The robot according to claim 22, in, Processing image data within the region of interest using the computer vision model to determine whether an alert should be generated comprises: determining a temperature of the asset based on an analysis of the thermal image within the region of interest; comparing the determined temperature of the asset to the temperature threshold; as well as When the determined temperature meets or exceeds the temperature threshold, it is determined that an alarm is generated.

24. The robot according to claim 17, in, The at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold.

25. The robot according to claim 17, in, Outputting the alert includes displaying on a display a representation of the image annotated with an indication of the alert.

26. The robot according to claim 17, in, Outputting the alert includes sending a message to a computing device via at least one network, the message including the alert.

27. The robot according to claim 17, in, The at least one computer processor is further configured to: The image and metadata indicative of the asset identifier are stored on at least one storage device.

28. The robot according to claim 17, in, The area of ​​interest is a first area of ​​interest including a first asset, wherein the at least one computer processor is further configured to: defining a second region of interest within the image, the second region of interest including a second asset in the environment of the robot, Wherein, using the computer vision model to process the image data in the region of interest to determine whether the alarm should be generated comprises: processing the image data within the first region of interest using the computer vision model to determine a first result; processing the image data within the second region of interest using the computer vision model to determine a second result; and A determination is made whether the alert should be generated based at least in part on the first result and the second result.

29. The robot according to claim 17, in, The at least one computer processor is further configured to: for each of a plurality of images captured over time, each of the images having the region of interest defined therein, processing image data within the region of interest of the image using the computer vision model to determine at least one quantity associated with the asset; as well as Based on at least one quantity associated with the asset determined for the plurality of images, a trend analysis for the at least one quantity is generated.

30. The robot according to claim 29, in, The at least one quantity comprises one or more of temperature, pressure, vibration and radiation quantity.

31. The robot according to claim 29, in, The at least one computer processor is further configured to: An indication of the trend analysis for the at least one quantity is provided on a user interface.

32. The robot according to claim 30, in, The at least one computer processor is further configured to: The alert is generated based at least in part on the trend analysis.

33. A non-transitory computer readable medium encoded with a plurality of instructions that, when executed by at least one computer processor, perform a method comprising: defining a region of interest within an image captured by a sensor of a robot, the region of interest containing an asset in an environment of the robot, in, The asset is associated with an asset identifier; configuring at least one parameter of a computer vision model based on the asset identifier; processing image data within the region of interest using the computer vision model to determine whether an alert should be generated; as well as When it is determined that the alarm should be generated, the alarm is output.

34. The non-transitory computer readable medium of claim 33, in, Defining the area of ​​interest includes defining the area of ​​interest using asset information stored in a data structure associated with a task record.

35. The non-transitory computer readable medium of claim 34, in, The data structure is associated with the action of capturing the image at a first waypoint indicated in the mission record, and The asset identifier is included in the data structure.

36. The non-transitory computer readable medium of claim 34, in, The data structure includes the at least one parameter of the computer vision model.

37. The non-transitory computer readable medium of claim 33, in, The image captured by the sensor of the robot is a thermal image.

38. The non-transitory computer readable medium of claim 37, in, The at least one parameter of the computer vision model includes a temperature threshold.

39. The non-transitory computer readable medium of claim 38, in, Processing image data within the region of interest using the computer vision model to determine whether an alert should be generated comprises: determining a temperature of the asset based on an analysis of the thermal image within the region of interest; comparing the determined temperature of the asset to the temperature threshold; as well as When the determined temperature meets or exceeds the temperature threshold, it is determined that an alarm is generated.

40. The non-transitory computer readable medium of claim 33, in, The at least one parameter of the computer vision model includes one or more of a pressure threshold, a vibration threshold, or a radiation threshold.

41. The non-transitory computer readable medium of claim 33, in, Outputting the alert includes displaying on a display a representation of the image annotated with an indication of the alert.

42. The non-transitory computer readable medium of claim 33, in, Outputting the alert includes sending a message to a computing device via at least one network, the message including the alert.

43. The non-transitory computer readable medium of claim 33, in, The method further comprises: The image and metadata indicative of the asset identifier are stored on at least one storage device.

44. The non-transitory computer readable medium of claim 33, in, The region of interest is a first region of interest including a first asset, and the method further comprises: defining a second region of interest within the image, the second region of interest including a second asset in the environment of the robot, Wherein, using the computer vision model to process the image data in the region of interest to determine whether the alarm should be generated comprises: processing the image data within the first region of interest using the computer vision model to determine a first result; processing the image data within the second region of interest using the computer vision model to determine a second result; and A determination is made whether the alert should be generated based at least in part on the first result and the second result.

45. The non-transitory computer readable medium of claim 33, in, The method further comprises: for each of a plurality of images captured over time, each of the plurality of images having the region of interest defined therein, processing image data within the region of interest of the image using the computer vision model to determine at least one quantity associated with the asset; as well as Based on at least one quantity associated with the asset determined for the plurality of images, a trend analysis for the at least one quantity is generated.

46. ​​The non-transitory computer readable medium of claim 45, in, The at least one quantity comprises one or more of temperature, pressure, vibration and radiation quantity.

47. The non-transitory computer readable medium of claim 45, in, The method further comprises: An indication of the trend analysis for the at least one quantity is provided on a user interface.

48. The non-transitory computer readable medium of claim 46, in, The method further comprises: The alert is generated based at least in part on the trend analysis.

49. A method comprising: Navigating a mobile robot to traverse a route through an environment; generating a mission record during navigation of the mobile robot along the route, the mission record comprising a plurality of waypoints and edges connecting pairs of the plurality of waypoints; receiving a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint among the plurality of waypoints; receiving, via the user interface, a second input from the user, the second input identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint; as well as A first asset identifier is associated with the first action in the task record, wherein the first asset identifier uniquely identifies the first asset in the environment.

50. The method of claim 49, further comprising: commanding the mobile robot to execute a first task corresponding to the task record; When the mobile robot reaches the first waypoint along the route, automatically capturing second sensor data; and The asset identifier is automatically stored on at least one storage device as metadata associated with the second sensor data.

51. The method of claim 50, further comprising: commanding the mobile robot to execute a second task corresponding to the task record; When the mobile robot reaches the first waypoint along the route, automatically capturing third sensor data; and The asset identifier is automatically stored on the at least one storage device as metadata associated with the third sensor data.

52. The method of claim 51 further comprising: An indication of the second sensor data and the third sensor data is displayed on a user interface.

53. The method according to claim 52, in, The second sensor data and the third sensor data include second image data and third image data, respectively, and wherein the method further includes: processing each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, The indication of displaying the second sensor data and the third sensor data includes: displaying the second sensor data with an indication of the first output; and The third sensor data is displayed with an indication of the second output.

54. The method of claim 49, further comprising: The user is prompted on the user interface to provide the second input, the second input identifying the first asset within the first sensor data.

55. The method of claim 49, further comprising: receiving, via the user interface, a third input from the user, the third input identifying a second asset within first sensor data captured by the mobile robot at the first waypoint; and associating a second asset identifier with the first action in the task record, in, The second asset identifier uniquely identifies the second asset in the environment.

56. The method of claim 49, further comprising: receiving a third input from a user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among the plurality of waypoints; receiving, via the user interface, a fourth input from the user, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint; and The first asset identifier is associated with the second action in the task record.

57. A robot comprising: a navigation system configured to navigate the mobile robot to traverse a route through an environment; A perception system comprising an image sensor configured to capture an image ； as well as at least one computer processor configured to: generating a mission record during navigation of the mobile robot along the route, the mission record comprising a plurality of waypoints and pairs of edges connecting the plurality of waypoints; receiving a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint among the plurality of waypoints; receiving, via the user interface, a second input from the user, the second input identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint; as well as A first asset identifier is associated with the first action in the task record, wherein the first asset identifier uniquely identifies the first asset in the environment.

58. The robot according to claim 57, in, The at least one computer processor is further configured to: commanding the mobile robot to execute a first task corresponding to the task record; automatically capturing second sensor data when the mobile robot reaches the first waypoint along the route; as well as The asset identifier is automatically stored on at least one storage device as metadata associated with the second sensor data.

59. The robot according to claim 58, in, The at least one computer processor is further configured to: commanding the mobile robot to execute a second task corresponding to the task record; When the mobile robot reaches the first waypoint along the route, automatically capturing third sensor data; as well as The asset identifier is automatically stored on the at least one storage device as metadata associated with the third sensor data.

60. The robot according to claim 59, in, The at least one computer processor is further configured to: An indication of the second sensor data and the third sensor data is displayed on a user interface.

61. The robot according to claim 60, in, The second sensor data and the third sensor data include second image data and third image data, respectively, and wherein the at least one computer processor is further configured to: processing each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, The indication of displaying the second sensor data and the third sensor data includes: displaying the second sensor data with an indication of the first output; and The third sensor data is displayed with an indication of the second output.

62. The robot according to claim 57, in, The at least one computer processor is further configured to: The user is prompted on the user interface to provide the second input, the second input identifying the first asset within the first sensor data.

63. The robot according to claim 57, in, The at least one computer processor is further configured to: receiving, via the user interface, a third input from the user, the third input identifying a second asset within first sensor data captured by the mobile robot at a first waypoint; as well as A second asset identifier is associated with the first action in the task record, wherein the second asset identifier uniquely identifies the second asset in the environment.

64. The robot according to claim 57, in, The at least one computer processor is further configured to: receiving a third input from a user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among the plurality of waypoints; receiving, via the user interface, a fourth input from the user, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint; as well as The first asset identifier is associated with the second action in the task record.

65. A non-transitory computer readable medium encoded with a plurality of instructions that, when executed by at least one computer processor, perform a method comprising: Navigating a mobile robot to traverse a route through an environment; generating a mission record during navigation of the mobile robot along the route, the mission record comprising a plurality of waypoints and pairs of edges connecting the plurality of waypoints; receiving a first input from a user via a user interface, the first input commanding the mobile robot to perform a first action by recording first sensor data at a first waypoint among the plurality of waypoints; receiving, via the user interface, a second input from the user, the second input identifying a first asset within the first sensor data captured by the mobile robot at the first waypoint; as well as A first asset identifier is associated with the first action in the task record, wherein the first asset identifier uniquely identifies the first asset in the environment.

66. The non-transitory computer readable medium of claim 65, the method further comprising: commanding the mobile robot to execute a first task corresponding to the task record; When the mobile robot reaches the first waypoint along the route, automatically capturing second sensor data; and The asset identifier is automatically stored on at least one storage device as metadata associated with the second sensor data.

67. The non-transitory computer readable medium of claim 66, the method further comprising: commanding the mobile robot to execute a second task corresponding to the task record; When the mobile robot reaches the first waypoint along the route, automatically capturing third sensor data; and The asset identifier is automatically stored on the at least one storage device as metadata associated with the third sensor data.

68. The non-transitory computer readable medium of claim 67, the method further comprising: An indication of the second sensor data and the third sensor data is displayed on a user interface.

69. The non-transitory computer readable medium of claim 68, in, The second sensor data and the third sensor data include second image data and third image data, respectively, and wherein the method further includes: processing each of the second sensor data and the third sensor data using a computer vision model to generate a first output and a second output, respectively, The indication of displaying the second sensor data and the third sensor data includes: displaying the second sensor data with an indication of the first output; and The third sensor data is displayed with an indication of the second output.

70. The non-transitory computer readable medium of claim 65, the method further comprising: The user is prompted on the user interface to provide the second input, the second input identifying the first asset within the first sensor data.

71. The non-transitory computer readable medium of claim 65, the method further comprising: receiving, via the user interface, a third input from the user, the third input identifying a second asset within the first sensor data captured by the mobile robot at the first waypoint; and associating a second asset identifier with the first action in the task record, in, The second asset identifier uniquely identifies the second asset in the environment.

72. The non-transitory computer readable medium of claim 65, the method further comprising: receiving a third input from a user via the user interface, the third input commanding the mobile robot to perform a second action by recording second sensor data at a second waypoint among the plurality of waypoints; receiving, via the user interface, a fourth input from the user, the fourth input identifying the first asset within the second sensor data captured by the mobile robot at the second waypoint; and The first asset identifier is associated with the second action in the task record.

73. A method of using a mobile robot to monitor physical assets in an environment over time, the method comprising: determining, within a region of interest defined within each of a plurality of images captured over time by the mobile robot, at least one quantity associated with a physical asset represented in the region of interest, in, performing the determining using a computer vision model, the computer vision model configured based on asset information associated with the physical asset; generating a trend analysis for the at least one quantity associated with the asset based on the at least one quantity of the plurality of images; as well as An indication of the trend analysis is output.

74. The method according to claim 73, in, The at least one quantity comprises one or more of temperature, pressure, vibration and radiation quantity.

75. The method according to claim 73, in, Outputting the indication of the trend analysis includes providing the indication of the trend analysis on a user interface.

76. The method of claim 73, further comprising: An alert is generated based at least in part on the trend analysis.

Images