Rendering of mission action into industrial environment map

Abstract

To render a mission action into an industrial environment map.SOLUTION: A mission action to be rendered can include a first mission action rendered in a first mode and a second mission action rendered in a second mode different from the first mode. For example, the first mission action can be rendered by using a first graphical feature on the basis of first metadata (e.g., the first metadata designating a robot attitude and a camera attitude) of the first mission action having a granularity of a first level. The second mission action can be rendered by using a second graphical feature on the basis of second metadata (e.g., the second metadata designating the robot attitude but not designating any camera attitude) of the second mission action having a granularity of a second level.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Rendering mission actions within industrial environment maps. [Background technology]

[0002] Multiple robots may be deployed within an industrial facility for various purposes, such as capturing images of instruments, measuring gas readings, capturing images for anomaly detection, etc. For example, at a given time, a first robot may be performing a first mission with a set of mission actions, each defining a corresponding robot pose and a corresponding camera pose for the robot's camera, at which images of points of interest (POIs) may be captured. Thus, performing the first mission may involve performing a set of mission actions to navigate the robot to various robot poses and capture corresponding images in those poses (e.g., with the camera in the corresponding camera poses). At a given time, a second robot may be performing a second mission with a unique set of mission actions, each defining a corresponding robot pose, without also defining a camera pose (e.g., the second robot lacks any camera but includes a gas sensor). Thus, executing the second mission may involve performing a unique series of mission actions to navigate the robot to various robot poses and use the gas sensor to measure gas readings at those poses, regardless of any camera pose.

[0003] Mission actions may be initially defined by a human operator for a controlled robot (e.g., through the use of a handheld controller and / or a computer interface) and specified by inputs provided by the human operator. The robot may be an actual robot controlled in an actual industrial facility or a simulated robot controlled in a simulation of the actual industrial facility. For example, in defining a mission action, a human may utilize a handheld controller to navigate the robot to a particular robot pose (e.g., robot location and orientation), adjust the robot's camera to a particular camera pose (e.g., camera location and orientation) while the robot is in the particular robot pose, and provide user interface inputs that define the particular robot pose and the particular camera pose as metadata for the particular mission action. After a robot mission action is initially defined, that robot (or a different robot) can subsequently utilize the defined robot mission action to perform the mission action (e.g., navigate to the robot pose and adjust the camera to the camera pose) autonomously (i.e., without requiring any human operator control). Summary of the Invention [Means for solving the problem]

[0004] As mentioned above, robots in an industrial environment (e.g., including an industrial facility) can perform corresponding defined missions, and in doing so, can each implement a sequence of mission data defined for the respective mission. For example, a first robot can perform a first defined mission by performing a first sequence of mission actions, and a second robot different from the first robot can perform a second defined mission by performing a second sequence of mission actions. The first robot may be the same as or different from the second robot. For example, the first robot may differ from the second robot in that the first and second robots are from different manufacturers, are of different types (e.g., quadruped, multi-wheeled, self-balancing single-wheel, crawler, aerial, etc.), and / or have different payloads (e.g., the first robot includes a camera and / or gas sensors, while the second robot lacks any camera and / or gas sensors). The first defined mission may be the same as or different from the second defined mission.

[0005] In some embodiments, different mission actions within the same sequence (e.g., a first sequence or a second sequence) may be defined at the same level of granularity. In some other embodiments, different mission actions within the same sequence (e.g., a first sequence or a second sequence) may be defined at different levels of granularity. For example, some mission actions may be defined more finely (e.g., robot pose and camera parameters) than other mission actions (e.g., robot pose only). As a non-limiting example, a first sequence of mission actions may include a first mission action, a second mission action, and a third mission action, which are defined at different levels of granularity. In this non-limiting example, a first mission action may be defined by a first robot pose of a first robot, but may lack any definition of a camera pose of a camera that is part of the first robot's payload; a second mission action may be defined by a second robot pose of the first robot and a first camera pose of the camera when the first robot is in the second robot pose; and a third mission action may be defined by a third robot pose of the first robot, a second camera pose of the camera, and information about the POI (e.g., component ID, component type, etc.) being captured by the camera in the second camera pose when the first robot is in the third robot pose. Thus, in this non-limiting example, the third mission action may be defined at the highest level of granularity (robot pose, camera pose, and POI information), the first mission action may be defined at a lower level of granularity (robot pose only), and the second mission action may be defined at a level of granularity that is between the granularity level of the first mission action and the granularity level of the second mission action (robot pose and camera pose).

[0006] In some embodiments, a mission action may optionally or additionally be defined with POI parameters that define characteristics of the POI being captured by sensor readings in the mission action. For example, a mission action (e.g., a second mission action) in which a robot (e.g., a first robot) captures an image of a particular instrument may be defined with a particular instrument type (e.g., a temperature type, a pressure type, etc.) and / or a unique identifier for the particular instrument (e.g., X1234AB). The POI characteristics defined for the mission action may be provided by an operator defining the mission action and / or may be automatically determined from analysis of images and / or other visual data captured in the mission action.

[0007] In some embodiments, different mission actions in different sequences (e.g., a first sequence and a second sequence) may be defined at the same level of granularity. In some other embodiments, different mission actions in different sequences (e.g., a first sequence and a second sequence) may be defined at different levels of granularity. For example, a mission action in a first sequence may be to capture an image of an instrument, and an additional mission action in a second sequence may be to measure a gas reading. In this example, the mission action may be defined in both (a) the robot pose (e.g., X-coordinate, Y-coordinate, Z-coordinate, and optionally, rotation and / or translation) and / or other robot parameters of the robot (e.g., the robot type of the robot) and (b) the camera parameters, such as the camera pose (e.g., rotation and / or translation), camera zoom level, and / or camera focus setting, of the camera used to capture the image of the instrument. The additional mission action may be defined in the robot pose (and sometimes only the robot pose) without any camera parameters.

[0008] Note that certain mission actions may be defined and / or navigated using certain types of robots, in which case these mission actions may not be defined at different levels of granularity.

[0009] In some embodiments, once defined, a mission action may be associated with metadata that defines the mission action and stored for subsequent access (e.g., within an industrial environment). For example, a defined mission action may be added by a human operator to a given mission (or a new mission) for a particular robot to ensure operation of an industrial facility within an industrial environment. Note that different mission actions may be defined and stored with metadata having different levels of granularity, or the same mission action may be defined and stored with metadata having different and varying levels of granularity in association with different robots.

[0010] Although a mission (e.g., the first defined mission or the second defined mission described above) may not change over time, there may be situations in which it is desirable to at least temporarily add a given mission action to a given mission (if the given mission did not previously include the given mission action). For example, a given mission may include mission actions A-K, but may not include mission action L (even though mission action L is defined in an alternative mission), yet there may be a desire to add mission action L to the given mission. For example, there may be a desire to add mission action L to a given mission if a potentially harmful condition exists near mission action L, if there is a potential problem with an instrument associated with mission action L, etc.

[0011] There may also be situations in which it is desirable to create a new mission that includes a new collection of predefined mission actions from one or more predefined missions. For example, if a potential leak or other problem exists within a given area of ​​an industrial facility, there may be a desire to create a new mission that includes mission actions (e.g., predefined) within or near that area. Such mission actions may include two missions from a first mission, two missions from a second mission, and one mission from a third mission.

[0012] Existing techniques may prohibit or prevent adding a given mission action to a given mission and / or may prohibit or prevent creating a new mission that includes a new collection of predefined mission actions from other missions. For example, existing techniques may not be able to overlay mission actions on a map of an industrial environment, making it difficult and / or time-consuming for a human operator to identify mission actions that may be relevant to an area of ​​the industrial environment. As another example, the overlay of mission actions on a map or other graphical representation of mission actions may not indicate the granularity at which the mission actions are defined, making it difficult and / or time-consuming for a human operator to identify potentially relevant mission actions. For example, it may not be possible to distinguish between mission actions defined solely by robot pose and those defined by both robot pose and camera parameters. Also, for example, it may not be possible to quickly ascertain mission actions related to a particular type of sensor (e.g., identifying mission actions related to a temperature sensor).

[0013] Embodiments disclosed herein can overlay mission actions on a map of an industrial environment, such as a bird's-eye view map of the industrial environment. Some of these embodiments may further render mission actions in dependence on the metadata of the mission actions. For example, the mission actions may be rendered according to the granularity at which each of the mission actions is defined (as reflected by the metadata of the mission actions). For example, a mission action defined solely in terms of robot pose (and / or other robot parameters) may be rendered in a unique manner relative to a mission action defined in terms of both robot pose (and / or other robot parameters) and camera parameters. As another example, a mission action related to pressure sensor readings may be rendered in a unique manner relative to a mission action related to temperature sensor readings, and both may be rendered in a unique manner relative to a mission action related to gas sensor readings.

[0014] As a particular example, mission actions defined in robot poses alone may be rendered on the industrial environment map in a first manner (e.g., without images or descriptors of any instruments or industrial POIs if no instruments or other industrial POIs are associated with these mission actions), and mission actions defined in robot poses and camera parameters may be rendered on the industrial environment map in a second manner (e.g., with images and / or descriptors of instruments and / or other POIs captured in the mission action). This may allow a human operator to more quickly identify mission actions to add to a given mission or to include within a new mission. For example, if a mission action is added to collect additional gas readings or additional panoramic images, the camera parameters are immaterial and the operator can select from any of the rendered mission actions. On the other hand, if a mission action is added to collect additional instrument readings or additional images of a particular POI (or a particular type of POI), the operator can focus their attention on the mission action defined in the camera parameters.

[0015] In some embodiments, additionally or alternatively, which mission actions are rendered at a given time may be selected based on user input (e.g., selection of a filter element, or defining an area to limit the showing of mission actions within that area, etc.) For example, only a subset of mission actions may be rendered at a given time (e.g., on an industrial environment map) based on user input specifying characteristics of the subset of mission actions.

[0016] As a particular example, an operator may select a filter element to show all mission actions. On the other hand, the operator may select an alternative filter element to show only mission actions defined by camera parameters (e.g., as reflected in their corresponding metadata) and / or associated only with certain types of POI (e.g., instruments). For example, in response to suboptimal operation of a cooling system, the operator may select a filter element to render only mission actions that are (i) defined by camera pose and (ii) associated with temperature instruments. This may allow the operator to quickly select one or more of the mission actions from such rendered mission actions to add to a given mission (e.g., to inspect the cooling system) or to include in a new mission.

[0017] In some embodiments, additionally or alternatively, which mission actions are rendered at a given time may be selected based on detecting an event. For example, only a subset of mission actions may be rendered (e.g., on an industrial environment map) at a given time based on detecting an event and determining that the event relates to characteristics of a subset of the mission actions. In a non-limiting example, in response to detecting an oil spill (or other event) on the floor of an open space in an industrial environment, a mission action may be rendered in the open space, but no other mission actions may be rendered in other spaces of the industrial environment. In this non-limiting example, mission actions defined at different levels of granularity may be rendered in different ways.

[0018] Continuing with the above non-limiting example, in some embodiments, not all mission actions within the open space are rendered. For example, a subset of mission actions within the open space, each defined by camera parameters, may be selectively rendered such that a human operator can select a particular mission action from the rendered subset of mission actions so that the robot can navigate to that particular mission action to capture one or more camera images of the oil spill for further analysis or before taking another action. In this case, it should be noted that mission actions defined at different levels of granularity may be rendered differently to facilitate selection by the human operator.

[0019] In some embodiments, a subset of mission actions may be automatically (e.g., without user input) rendered on the industrial environment map at a given time in response to the detection of an event. For example, the detection of a potential gas leak may cause all mission actions for the industrial environment to be automatically shown (because any of the mission actions may be used to measure gas readings). As another example, if the detection of a potential gas leak includes location information specifying an area (or target location), then in response to the detection of a potential gas leak with that location information, a subset of all mission actions within the specified area (or within a predetermined distance from the target location) may be automatically rendered. In these embodiments, optionally, the rendered mission actions may be automatically rendered in different ways, e.g., at different levels of granularity, based on metadata of the rendered mission actions that defines these mission actions.

[0020] Thus, embodiments disclosed herein render mission actions for an industrial facility within a map of the industrial facility according to metadata associated with the mission actions, which may enable a human operator to identify mission actions to add to a given mission and / or for inclusion within a new mission with reduced latency and / or a reduced amount of user input.

[0021] In various embodiments, a method implemented using one or more processors is provided, the method including identifying metadata for each of a plurality of mission actions for an industrial environment. The industrial environment may be or may include an industrial facility. The industrial facility may take the form of a chemical processing plant, an industrial office environment, an oil or natural gas refinery, a catalyst plant, a manufacturing plant, an offshore oil plant, or any other applicable plant. The metadata for each of the plurality of mission actions may be identified, for example, from a storage or database (e.g., accessible by an operator of the industrial environment) that stores metadata defined for all available mission actions for the industrial environment.

[0022] Mission actions for industrial environments may be initially defined, for example, by a human operator controlling the robot (e.g., through use of a handheld controller and / or computer interface) and providing input to specify the mission action. The input may provide, for example, the robot pose and / or other robot parameters (e.g., robot type, robot model, robot manufacturer, etc.) of the robot when capturing images in the mission action. The input may further provide, for example, camera parameters (e.g., camera pose, zoom level, etc.) of a camera used by the robot to capture images in the mission action. Additionally or alternatively, the input may provide, for example, POI parameters of a POI captured in the image, where the POI may be an industrial POI (e.g., an instrument) fixed or installed near the mission action.

[0023] In some embodiments, the robot may be an actual robot and the industrial facility may be an actual industrial facility. In some embodiments, the robot may be a simulated robot controlled in a simulation of the actual industrial facility. In some embodiments, the actual robot or the simulated robot may be controlled to navigate to and / or perform one or more actions in a mission action.

[0024] In some embodiments, metadata for a sequence of mission actions for a robot to perform a mission (e.g., detecting a gas leak relative to the environment surrounding each of the sequence of mission actions) may be stored in association with the mission and / or associated with the robot. For example, metadata for a first sequence of mission actions for a first robot to perform a first mission may be stored in association with the first mission in a database, and metadata for a second sequence of mission actions for a second robot to perform a second mission may be stored in association with the second mission in a database.

[0025] In some embodiments, metadata for a sequence of mission actions may be generated or derived from such input from a human operator. In some other embodiments, metadata for a particular mission action may be generated or derived from such input from a human operator. In some embodiments, metadata for a particular mission action may additionally or alternatively be identified or gathered based at least in part on communications with the robot capturing images of the mission actions, e.g., by retrieving from the robot the robot pose of the robot (and / or the camera pose of a camera used by the robot to capture the images) when the robot captures images of the mission actions.

[0026] In some embodiments, the metadata of the mission action may include a robot pose (for the robot) associated with the mission action. In some embodiments, additionally or alternatively, the metadata of the mission action may include one or more camera parameters for a camera of the robot used to capture images in the mission action. For example, if a human operator controls a robot for a mission action and controls a camera of the robot to capture images of instruments observable by the robot in the mission action, the metadata of the mission action may include the camera parameters of the camera. The one or more camera parameters for the camera of the robot (e.g., to be navigated in the mission action) may include, for example, a camera pose, a zoom level, and / or a focus.

[0027] In some embodiments, instead of or in addition to a camera (or two or more cameras), the robot may include or carry one or more other sensors (e.g., infrared sensors, etc.). In this case, the metadata of the mission action may alternatively or additionally include one or more sensor parameters of the one or more other sensors carried by the robot. In some embodiments, additionally or alternatively, the metadata of the mission action may include one or more POI parameters related to a POI (or two or more POIs) sensed (e.g., previously sensed or to be sensed) by the robot in the mission action. The one or more POI parameters may include, for example, a type (e.g., sensor) of the POI (e.g., an industrial pressure instrument connected to a pipe or tank in the mission action) and / or an identifier (e.g., G-10012) of the POI.

[0028] In various embodiments, the method may further include rendering the first subset of mission actions with a first graphical feature within a map interface for the industrial environment. The map interface may, for example, be a plan view (e.g., a bird's-eye view / overhead map) of the industrial environment. The first graphical feature may include, by way of non-limiting example, a first graphical symbol that displays the first mission action (from the first subset) within the map interface. The first graphical symbol may, for example, have a particular shape, color, pattern, and / or other characteristic that indicates that the first mission action belongs to the first subset of mission actions. In some embodiments, rendering the first subset of mission actions with the first graphical feature is responsive to determining that corresponding metadata for each of the mission actions in the first subset matches one or more first criteria.

[0029] In various embodiments, the method may additionally or alternatively include rendering the second subset of mission actions with a second graphical feature within a map interface for the industrial environment along with the first subset of mission actions. The second graphical feature may include, by way of non-limiting example, a second graphical symbol representing a second mission action from the second subset. The second graphical symbol may have, for example, a particular shape, color, pattern, and / or other characteristic that indicates that the second mission action belongs to the first subset of mission actions. In some embodiments, rendering the second subset of mission actions with the second graphical feature is responsive to determining that corresponding metadata for each of the mission actions in the second subset matches one or more second criteria. The first criteria is different from the second criteria. The first graphical feature is visually different from the second graphical feature.

[0030] In some embodiments, determining that the corresponding metadata for each of the mission actions of the first subset matches the one or more first criteria may be achieved / implemented by determining that the corresponding metadata for each of the mission actions of the first subset includes a robot pose for each mission action from the first subset (such that the robot can navigate to the robot pose to undertake / image the respective mission action) and one or more visual sensor parameters for capturing visual data for the respective mission action while the robot is in the corresponding robot pose. In some embodiments, the mission actions of the first subset may be associated with a hazardous material (or other object or industrial point of interest) to be imaged.

[0031] In some embodiments, the one or more visual sensor parameters include a visual sensor pose, a visual sensor zoom level, and / or a visual sensor focus parameter associated with a visual sensor carried by a robot (e.g., a mobile robot) in the industrial environment. In some embodiments, additionally or alternatively, the metadata of the first subset of mission actions includes one or more POI parameters related to a POI sensed (e.g., by the robot) during one of the first subset of mission actions. The one or more POI parameters may include, for example, a POI identifier (and / or type) of the POI.

[0032] In some embodiments, determining that the corresponding metadata for each of the mission actions of the second subset matches one or more second criteria is performed by determining that the corresponding metadata for each of the mission actions of the second subset includes a robot pose for the respective mission action from the second subset but lacks any camera parameters for capturing images of the mission action. In these embodiments, determining that the corresponding metadata for each of the mission actions of the second subset matches one or more second criteria may include, or be performed by, determining that the corresponding metadata for each of the mission actions of the second subset includes only a corresponding robot pose for each of the mission actions of the second subset. In some embodiments, the mission actions of the second subset may each be a location for which no images have been captured (e.g., because a human operator has not defined these mission actions with camera poses).

[0033] In some embodiments, determining that the corresponding metadata for each of the first subset of mission actions matches one or more first criteria includes, or is performed by, determining that the corresponding metadata for each of the first subset of mission actions indicates that the mission actions of the first subset each include a first type of sensor. In some embodiments, determining that the corresponding metadata for each of the second subset of mission actions matches one or more second criteria includes, or is performed by, determining that the corresponding metadata for each of the second subset of mission actions indicates that the mission actions of the second subset each include a second type of sensor. The second type may be different from the first type.

[0034] In various embodiments, the method may further include receiving a selection of the first subset or the second subset of given mission actions via a map interface for the industrial environment. The selection may be in the form of touch input, voice input, or any other suitable form from a user (e.g., a human operator). In some embodiments, the first subset of mission actions and / or the second subset of mission actions may each be rendered as a graphical user interface (GUI) selectable element for selection by the user (e.g., a mouse click of the given mission action). For example, the first subset of mission actions may each be rendered as a first GUI selectable element and the second subset of mission actions may each be rendered as a second GUI selectable element, where the second GUI selectable element is visually distinguishable from the first GUI selectable element.

[0035] In various embodiments, the method may further include, in response to receiving the selection, adding the given mission action to the predefined robot mission or including the given mission action within a new mission. As a non-limiting example, the predefined robot mission may include navigating the robot to a first mission action for capturing a first image in the first mission action and further navigating the robot to a second mission action for capturing a second image in the second mission action. In this non-limiting example, the given mission action may be located between the first mission action and the second mission action, and adding the given mission action to the predefined robot mission may include having the robot navigate to the given mission action for capturing an image in the given mission action after navigating to the first mission action and before navigating to the second mission action.

[0036] Alternatively, a given mission action may be prioritized relative to a first mission action and a second mission action, for example, when a hazard is detected near (e.g., within a predefined range) the given mission action at the same time that a routine inspection is required in the first mission action and the second mission action. In this case, adding the given mission action to the predefined robot mission may cause the robot to navigate to the given mission action for capturing an image of the given mission action prior to navigating to the first and second mission actions. Alternatively, the first mission action and the second mission action may be prioritized relative to a given mission action, for example, when the given mission action requires a routine inspection at the same time that the first mission action and the second mission action require an immediate inspection. In this case, adding the given mission action to the predefined robot mission may cause the robot to navigate to the given mission action for capturing an image of the given mission action after navigating to the first mission action and the second mission action.

[0037] In other words, a given mission action may be added to a predefined robot mission based on the location of the given mission action relative to the locations of other predefined mission actions of the predefined robot mission (e.g., the first and second mission actions in the non-limiting example above) and / or based on the priority level of the tasks in the given mission action (e.g., relative to the priority levels of the tasks in other predefined mission actions of the predefined robot mission).

[0038] In various embodiments, a method implemented using one or more processors is provided, the method including identifying corresponding metadata for each of a plurality of mission actions for an industrial environment. In some embodiments, the industrial environment may be or may include an industrial facility. The industrial facility may take the form of a chemical processing plant, an industrial office environment, an oil or natural gas refinery, a catalyst plant, a manufacturing plant, an offshore oil plant, or any other applicable plant. The metadata for each of the plurality of mission actions may be stored, for example, in a storage or database (e.g., accessible by an operator of the industrial environment) after being defined, such that the metadata, or portions thereof, can be identified or retrieved for later use (e.g., to select one or more mission actions to be rendered, to render different mission actions in different styles, etc.).

[0039] In some embodiments, identifying corresponding metadata for each of the plurality of mission actions may be performed in response to activation and rendering via a display of a computing device of a map interface of the industrial environment map. The computing device may, for example, be a local device within the industrial environment or a server device in communication with the industrial environment. The plurality of mission actions may (but need not necessarily) be rendered in response to the map interface being rendered. For example, a portion of the plurality of mission actions may be rendered in the map interface using the techniques / methods described herein.

[0040] In various embodiments, the method further includes determining one or more filtering criteria based on user interface input and / or based on events detected within the industrial environment, The one or more filtering criteria may be, for example, rules that filter the plurality of mission actions for rendering in the industrial environment map.

[0041] As one example, a user interface input may specify that the metadata of a mission action (to be rendered) must include a camera pose. In this example, a first filtering rule / criterion may be determined to filter out mission actions that are not defined with any camera poses from being rendered on the industrial environment map. As another example, a detected event may be a detected abnormal reading of a temperature instrument for a cooling system in an industrial environment. In this example, a second filtering rule / criterion may be determined to filter out mission actions that are not defined with any camera poses (necessary to capture an image of the cooling system) and that are not associated with a temperature instrument. For example, a mission action stored with metadata that does not include any camera poses for imaging a temperature instrument (sometimes, particularly, for imaging a temperature instrument for a cooling system) and does not include any instrument information for the temperature instrument may be filtered out using the second filtering criterion so that such a mission action is not rendered on the industrial environment map. This not only reduces the total number of mission actions rendered for user selection, but also reduces computing and other resources (such as battery and network) for the industrial environment.

[0042] In various embodiments, the method further includes selecting a subset of mission actions from the plurality of mission actions. In some embodiments, selecting the subset of mission actions is responsive to determining that corresponding metadata for each of the subset's mission actions matches one or more filtering criteria. As a non-limiting example, the one or more filtering criteria may include a first filtering rule requiring that the rendered mission action have metadata indicating a particular type of sensor (e.g., a pressure sensor) positioned relative to the rendered mission action. In this non-limiting example, the selected subset of mission actions may include, and may only include, a first mission action having metadata indicating that a first pressure sensor is observable in the first mission action, and a second mission action having metadata indicating that a second pressure sensor is observable in the second mission action. In other words, in this non-limiting example, no mission action may be selected (e.g., for rendering) from the plurality of mission actions whose metadata indicates that no pressure sensors are observable in the corresponding mission action.

[0043] As another non-limiting example, the corresponding metadata for each of the plurality of mission actions includes a corresponding robot pose for the respective robot in the corresponding mission action, and determining the filtering criteria includes determining one or more pose-based filtering criteria. The one or more pose-based filtering criteria may include a first pose-based filtering rule that requires selecting a subset of mission actions, each of which is within a predetermined distance from a target location. The target location may be, for example, a location selected by a user or a location where an event of interest (e.g., an oil spill) was detected.

[0044] In various embodiments, the method further includes rendering the subset of mission actions within a map interface for the industrial facility without rendering other mission actions that are not of the subset. In some embodiments, rendering the subset of mission actions may include rendering a graphical representation for each mission action of the subset. In some embodiments, for each mission action of the subset, the graphical representation may include a selectable element that, when selected, causes metadata associated with the corresponding mission action (or a portion thereof, e.g., a pressure sensor ID) to be rendered (e.g., as an overlay on the map interface). In some embodiments, alternatively or additionally, for each mission action of the subset, the graphical representation may be or include an additional selectable element that, when selected, causes the corresponding mission action to be added to a given mission or a new mission for the robot. In some embodiments, a user may be able to drag a symbol (e.g., a graphical representation) representing the corresponding mission action to add it to the robot mission (e.g., a given mission or a new mission).

[0045] In various embodiments, the method further includes receiving a selection of a subset of given mission actions via a map interface for the industrial facility. Continuing with the non-limiting example above, in which the subset of mission actions includes (only) a first mission action (having metadata indicating that a first pressure sensor is accessible / observable in the first mission action) and a second mission action (having metadata indicating that a second pressure sensor is accessible / observable in the second mission action), the metadata for the first mission action and the second mission action may include a first pressure reading received from the first pressure sensor and a second pressure reading received from the second pressure sensor. In this non-limiting example, the first pressure reading may be abnormal (e.g., higher than the normal range) and the second pressure reading may be within the normal range. In this case, the first mission action may be selected via the map interface as the given mission action requiring observation (e.g., image capturing), and the second mission action is not selected. The first mission action may be selected by a user viewing the map interface via touch input, voice input, or mouse click, or any other applicable type of input.

[0046] In various embodiments, the method further includes, in response to receiving the selection, adding the given mission action to a predefined robot mission or including the given mission action within a new robot mission. For example, the new robot mission may include navigating a particular type of robot to the given mission action. In this case, the method may further include causing the particular type of robot to perform the new robot mission, which includes navigating to the given mission action.

[0047] In various embodiments, a method implemented by one or more processors is provided, the method including: identifying corresponding metadata for each of a plurality of mission actions of an industrial environment, the metadata for each of the plurality of mission actions including at least a robot pose for the robot in the corresponding mission action; detecting an object or event to be inspected within the industrial environment; and, in response to detecting the object or event, rendering a first mission action and a second mission action within a map interface for the industrial environment, wherein rendering the first mission action and the second mission action is responsive to determining that the first mission action and the second mission action are within a predetermined distance to a location of the object or event, the first mission action and the second mission action being rendered with different graphical features based on the metadata of the first mission action and the second mission action having different levels of granularity; receiving a selection of the first mission action or the second mission action via the map interface for the industrial facility; and, in response to receiving the selection, adding the first mission action and the second mission action to the robot mission.

[0048] Additionally, some embodiments include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause any of the aforementioned methods to be performed. Some embodiments also include one or more non-transitory computer-readable storage media having stored thereon computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0049] It should be appreciated that all combinations of the foregoing and additional ideas detailed herein are contemplated as being part of the presently disclosed subject matter, e.g., all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the presently disclosed subject matter. [Brief explanation of the drawings]

[0050] [Figure 1A] FIG. 1 illustrates a schematic diagram of an exemplary environment in which selected aspects of the present disclosure may be implemented, according to various embodiments. [Figure 1B] FIG. 1 illustrates an exemplary storage of metadata associated with one or more mission actions, according to various embodiments. [Figure 2A] FIG. 10 illustrates an exemplary user interface showing a map with all available mission actions rendered, in accordance with various embodiments. [Figure 2B] FIG. 10 illustrates another exemplary user interface showing a map with all available mission actions rendered, in accordance with various embodiments. [Figure 2C] FIG. 10 illustrates another exemplary user interface showing a map rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 2D] 10A-10C illustrate further exemplary user interfaces showing maps rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 2E] 10A-10C illustrate additional exemplary user interfaces showing a map rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 2F]10A-10C illustrate further exemplary user interfaces showing maps rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 3] 1A-1D illustrate exemplary methods for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 4] FIG. 1 illustrates another exemplary method for implementing selected aspects of the present disclosure, according to various embodiments. [Figure 5] FIG. 1 illustrates a schematic diagram of an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0051] Embodiments described herein relate to rendering one or more mission actions on a map of an industrial environment. Some embodiments may further render mission actions depending on the granularity at which the mission action is defined (as reflected by the mission action's metadata). For example, a mission action defined in terms of only a robot pose may be rendered in a unique manner relative to a mission action defined in terms of both a robot pose and camera parameters. As another example, a mission action related to pressure sensor readings may be rendered in a unique manner relative to a mission action related to a temperature sensor reading, and both may be rendered in a unique manner relative to a mission action related to a gas sensor reading.

[0052] As yet another example, at a given time, mission actions may be selectively rendered based on user input (e.g., selection of a filter element) and / or based on detecting an event. For example, only a subset of mission actions may be rendered at a given time based on user input specifying characteristics of the subset of mission actions and / or based on detecting an event (e.g., a triggered alarm) and determining that the detected event relates to characteristics of the subset of mission actions. As a particular example, mission actions defined by robot pose alone may be rendered on the industrial environment map in a first manner, and mission actions defined by robot pose and camera parameters may be rendered on the map in a second manner (e.g., along with images and / or descriptors of instruments and / or other points of interest captured in the mission action). This may allow a human operator to more quickly identify mission actions to add to a given mission or to include within a new mission. For example, if a mission action should be added to collect additional gas readings or additional panoramic images, the camera parameters are not important and the operator can select from any of the rendered mission actions. On the other hand, if a mission action should be added to collect additional instrument readings or additional images of a particular POI (or a particular type of POI), the operator can focus their attention on the mission action with the camera parameters.

[0053] As another specific example, an operator may select a filter element to show all mission actions, or the detection of a potential gas leak may cause all mission actions to be automatically shown (because any of the mission actions may be used to measure gas readings). On the other hand, an operator may select an alternative filter element to show only mission actions that have camera parameters and / or are associated only with certain types of POIs (e.g., instruments), or the detection of certain events may cause such mission actions to be automatically shown. For example, in response to suboptimal operation of a cooling system, an operator may select a filter element that (i) defines a camera pose and (ii) causes only mission actions associated with a temperature instrument to be rendered. This may allow an operator to quickly select one or more of such mission actions to add to a given mission or to include within a new mission.

[0054] Accordingly, embodiments disclosed herein render mission actions for an industrial facility within a map of the industrial facility according to metadata associated with the mission actions, which may enable a human operator to identify mission actions to add to a given mission and / or for inclusion within a new mission with reduced latency and / or a reduced amount of user input.

[0055] FIG. 1A schematically illustrates an exemplary environment in which selected aspects of the present disclosure may be implemented, according to various embodiments. Referring now to FIG. 1A, an exemplary environment 100 in which various aspects of the present disclosure may be implemented is generally shown. The exemplary environment 100 may be or include an industrial facility 130, which may take many forms. For example, the exemplary environment 100 may be designed to perform any number of at least partially automated processes. The industrial facility 130 may take the form of a chemical processing plant, an industrial office environment, an oil or gas refinery, a catalyst plant, a manufacturing facility, an offshore oil plant, or any other applicable facility.

[0056] The exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled to a process automation network 106 within an industrial facility. Client device 103-A or 103-B may be implemented as a computer (e.g., laptop, desktop, notebook), tablet, robot, smart appliance (e.g., smartphone), messaging device, wearable device (e.g., watch), or any other applicable device. The process automation network 106 may be implemented using various wired and / or wireless communication technologies, including, but not limited to, Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), cellular networks such as 3GPP® Long Term Evolution (“LTE”) or other wireless protocols such as 3G, 4G, 5G, and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).

[0057] The exemplary environment 100 may further include one or more mobile robots. For example, the exemplary environment 100 may include a robot fleet having a first robot 111 and / or a second robot 112. The first robot 111 may be the same as or different from the second robot 112. The first robot 111 may be, for example, a four-legged robot (e.g., a robot dog), a wheel-driven robot, an unmanned aerial vehicle (e.g., a drone), a crawler robot, or any other applicable robot capable of moving within or around the industrial facility 130. The different robots may be of different types and may be manufactured by different manufacturers. In some embodiments, the robot fleet may include a subset of robots of the same type and / or manufactured by the same manufacturer.

[0058] 1A , the first robot 111 may be a robotic dog that navigates to one or more mission actions for the industrial facility 130 to inspect one or more points of interest (POIs) in or near the one or more mission actions. The second robot 112 may be a drone that navigates to one or more additional mission actions to inspect structures in the industrial facility 130 (e.g., within a chimney or vessel). The one or more mission actions navigated by the first robot 111, as well as the one or more additional mission actions navigated by the second robot 112, may each be defined (e.g., by a human operator) with metadata including, for example, the robot pose of the corresponding robot in the mission action and / or other information related to the mission action (e.g., camera parameters, POI parameters, etc.). Optionally, the one or more mission actions and the one or more additional mission actions may form all predefined mission actions for the industrial facility 130, for example, when the first robot 111 and the second robot 112 are the only robots deployed for the industrial facility 130 and no other mission actions have been defined by a human operator for the industrial facility 130.

[0059] The first robot 111 or the second robot 112 may be other types of robots, such as a robot for transporting supplies and products, a robot dog for patrolling and monitoring an industrial facility for anomalies, a spider robot for inspecting the exterior of a pipeline, a snake robot for inspecting the interior of a pipeline, etc. The description of the specific type and functionality of the first robot 111, the second robot 112, or any other applicable robot is not limiting herein. For example, in some embodiments, the first robot 111 (or the second robot 112) may be an actual robot, and the industrial facility 130 may be an actual industrial facility. In some other embodiments, the first robot 111 (or the second robot 112) may be a simulated robot controlled in a simulation of the industrial facility 130, where the simulated robot may be utilized by a human operator to define one or more mission actions (and / or one or more additional mission actions).

[0060] In some embodiments, the first robot 111 and the second robot 112 may each include one or more sensors for performing one or more missions. For example, the first robot 111 (or other robots, such as the second robot 112) may include (or, in some cases, be equipped with) a light detection and ranging (lidar) sensor for imaging objects by creating a 3D model of the imaged object. Additionally or alternatively, the first robot 111 (or other robots) may include other sensors, such as a visual sensor, ultrasonic testing immersion transducers for detecting surface irregularities and flaws (e.g., corrosion), one or more gas sensors for detecting the presence and concentration of harmful gases or vapors, and / or a temperature sensor for measuring temperature. The visual sensor may be a monographic camera, a stereographic camera, a thermal camera, or any other applicable visual sensor for capturing one or more images of one or more specific points of interest (POIs) in the industrial facility 130. The visual sensors may be removably coupled to the first robot 111 or may be integral with the first robot 111. In some embodiments, the visual sensors may change location and / or orientation relative to the first robot 111, for example, by rotation or other movement. As a non-limiting example, the first robot 111 may include front and rear high-resolution cameras removably coupled to the first robot 111.

[0061] In various embodiments, the exemplary environment 100 may include a map 140 of the industrial environment (sometimes referred to as an “industrial environment map”), where the industrial environment map 140 may be rendered as or in a map interface via a display of the client device 103-A (or of the client device 103-B, or of another device, such as the server device 105). The industrial environment map 140 may be, for example, a plan view (e.g., a bird's-eye view or overhead view) of the industrial facility 130.

[0062] Client device 103-A or 103-B may each include input and / or output devices for user interaction with industrial environment map 140. For example, user input via the input device of client device 103-B may cause all available mission actions (or a portion thereof) to be rendered in industrial environment map 140 or may cause all available mission actions (or a portion thereof) to be removed from appearing on or via industrial environment map 140.

[0063] Mission actions for the industrial facility 130 may be initially defined, for example, by a human operator controlling a robot (e.g., the first robot 111), for example, through the use of a handheld controller (and / or computer interface). A mission action may be a robotic mission action in which the robot navigates to observe a point of interest (POI) or to perform another action (e.g., a measurement). A mission action may correspond, for example, to a location where a POI (e.g., equipment, etc.) in the industrial facility 130 is located, a location where routine maintenance or inspection is expected, a location where information needs to be gathered (e.g., a location where pressure instrument readings need to be gathered / determined), or a location where an anomaly has been identified. After the mission action is initially defined, the robot can utilize the defined mission action (e.g., a robot pose defined in metadata associated with the mission action) to autonomously navigate to the mission action (e.g., without requiring any control from a human operator).

[0064] In some embodiments, a human operator may initially define a mission action by providing input to specify the mission action. For example, the human operator may provide user interface input that specifies an ID number for the mission action, one or more robots to use in the mission action (e.g., a particular robot for inspecting the surrounding environment in the mission action), robot poses for one or more robots in the mission action (e.g., for inspecting an object observable from the mission action), one or more camera poses (associated with one of the robot poses) for a particular robot's camera for capturing images of the object in the mission action, and / or any applicable information. In some embodiments, the user interface input may be stored as metadata associated with the mission action. In some embodiments, the user interface input may be processed to generate metadata associated with the mission action. For example, the metadata for the mission action may be stored as an entry for the mission action in a mission action database (see, for example, FIG. 1B ).

[0065] In some embodiments, the mission action metadata may include a robot pose (for the robot) associated with the mission action. In some embodiments, the mission action metadata may additionally or alternatively include one or more camera parameters related to the robot's camera. The one or more camera parameters related to the robot's camera may include, for example, a camera pose (e.g., at which the robot's camera captures a desired image of the mission action), a zoom level (e.g., of the camera for capturing a desired image of the mission action), and / or a focus (e.g., of the camera for capturing a desired image of the mission action). In some embodiments, the mission action metadata may additionally or alternatively include one or more POI parameters related to a POI (or two or more POIs) sensed in the mission action. The one or more POI parameters may include, for example, a type of POI (e.g., sensor) and / or an identifier of the POI.

[0066] In some embodiments, mission actions may be defined or updated to have various levels of granularity. In some embodiments, different mission actions may be defined with or associated with different sets of metadata, where the different sets of metadata may have different levels of granularity. For example, when a mission action is defined to capture images of an instrument, it may be defined with both (a) robot pose (e.g., X-axis, Y-axis, Z-axis, and optionally rotation and / or translation) and (b) camera parameters such as camera pose (e.g., rotation and / or translation), camera zoom level, and / or camera focus setting. When an additional mission action is defined / used to capture gas readings, it may be defined with (a) robot pose only and not with any camera parameters. A mission action (or additional mission actions) may optionally or additionally be defined with POI parameters that define the characteristics of the POI being captured by the sensor readings in the mission action. For example, a mission action defined to capture a particular instrument reading may include metadata defining the instrument type (e.g., temperature type, pressure type, etc.) and / or the instrument's unique identifier (e.g., X1234AB).

[0067] As another example, a first mission action defined to capture images of an instrument may be defined by both (a) robot pose (e.g., X-axis, Y-axis, Z-axis, and optionally rotation and / or translation) and (b) camera parameters, such as camera pose (e.g., rotation and / or translation), camera zoom level, and / or camera focus setting. In this example, a second mission action defined to capture gas readings may be defined by (a) robot pose only and not any camera parameters. Stated differently, some mission actions may be defined more finely (e.g., by robot pose and camera parameters) than other mission actions (e.g., robot pose only). In some embodiments, the metadata of the first mission action and / or second mission action may alternatively or additionally include, for example, point of interest (POI) parameters that define characteristics of a point of interest being captured by sensor readings in the first mission action or second mission action.

[0068] In various embodiments, exemplary environment 100 may further include a server computing device 105 (sometimes simply referred to as a “server device”). Server computing device 105 may include a mission action (MA) selection engine 1051, a rendering engine 1053, and / or storage 15. In some embodiments, optionally, server computing device 105 may include a mission action definition engine 1054. In some embodiments, industrial environment map 140 may be stored in and / or accessible via storage 15. In some embodiments, optionally, exemplary environment 100 may include two or more industrial environment maps (e.g., industrial environment maps determined in different years, etc.).

[0069] In various embodiments, the mission action definition engine 1054 may define one or more mission actions for the industrial facility. In some embodiments, the mission action definition engine 1054 may determine one or more mission actions for the industrial facility and / or determine metadata associated with the one or more mission actions based on user input. As a non-limiting example, the mission action definition engine 1054 may determine one or more mission actions for the industrial facility 130 based on user input that provides one or more defined robotic missions. For example, the user input may provide a first robotic mission that describes a first set of mission actions for a robot (e.g., the first robot 111 or the second robot 112, or a simulated robot of the first robot or the second robot) to navigate through, and may provide a second robotic mission that describes a second set of mission actions for a robot (e.g., the first robot 111 or the second robot 112, or a simulated robot of the first robot or the second robot) to navigate through. The robot may then navigate to perform the first robotic mission and / or the second robotic mission.

[0070] For example, a robot (e.g., a first robot 111) may be controlled for a first mission action belonging to a first set of mission actions to observe the surrounding environment of the first mission action. The robot may detect an instrument in the first mission action, and the robot's camera may be controlled to capture a specific image of the instrument with appropriate camera parameters (e.g., a specific camera pose, zoom level, etc.). In this case, the mission action definition engine 1054 may receive values ​​of those camera parameters from the robot and include these values ​​of the camera parameters in metadata associated with the first mission action. The mission action definition engine 1054 may alternatively or additionally receive the type and / or ID of the instrument (or other POI) based on user input provided by a human operator observing the specific image.

[0071] Continuing with the above non-limiting example, user input from a human operator may specify a particular robot (e.g., model, manufacturer, type, function, etc.) to navigate to the mission action, a robot pose of a particular robot to perform one or more actions (e.g., image capturing, modification, etc.) in the mission action, and / or a sensor pose of a sensor carried by the particular robot when the particular robot is in the robot pose, etc. In this case, the particular robot (e.g., model, manufacturer, type, function, etc.), the robot pose of the particular robot, and / or the sensor pose of the sensor may be determined and / or stored as metadata for the mission action.

[0072] In some embodiments, after the mission action definition engine 1054 defines and / or stores multiple mission actions for the industrial facility 130, the rendering engine 1053 may render the multiple mission actions (e.g., as an overlay) on or through the industrial environment map 140. The rendering engine 1053 may render different mission actions from the multiple mission actions in different ways. In some embodiments, the rendering engine 1053 may render the mission actions according to the granularity at which each mission action is defined (as reflected by the mission action's metadata). For example, a mission action defined only in terms of robot pose may be rendered in a unique manner relative to a mission action defined in terms of both robot pose and camera parameters. As another example, a mission action related to pressure sensor readings may be rendered in a unique manner relative to a mission action related to temperature sensor readings, and both may be rendered in a unique manner relative to a mission action related to gas sensor readings. As yet another example, which mission action is rendered at a given time may be selected based on user input (e.g., selection of a filter element) and / or based on detecting an event. For example, only a subset of the mission actions may be rendered at a given time based on user input specifying characteristics of the subset of mission actions and / or based on detecting an event and determining the event related to the characteristics of the subset of mission actions.

[0073] As a particular example, the rendering engine 1053 may render mission actions defined by robot pose alone on the industrial environment map 140 in a first manner, and render mission actions defined by robot pose and camera parameters on the industrial environment map 140 in a second manner (e.g., along with images and / or descriptions of instruments and / or other POIs captured in the mission action). This may allow a human operator to more quickly identify mission actions to add to a given mission or to include within a new mission. For example, if a mission action is to be added to collect additional gas readings or additional panoramic images, the camera parameters are not important and the operator may select from any of the rendered mission actions. On the other hand, if a mission action is to be added to collect additional instrument readings or additional images of a particular POI (or a particular type of POI), the operator may focus their attention on the mission action with the camera parameters.

[0074] As another particular example, the operator may select a filter element to cause all mission actions to be shown, or the detection of a potential gas leak may cause all mission actions to be automatically shown (as any of the mission actions may be used to measure gas readings). On the other hand, the operator may select an alternative filter element, and the mission action selection engine 1051 may show only mission actions that have camera parameters and / or that are related only to certain types of POIs (e.g., instruments) based on the selected alternative filter element. In some embodiments, the detection of an event may cause the mission action selection engine 1051 to automatically and selectively render one or more mission actions on the industrial environment map 140. 1A , in response to suboptimal operation (e.g., abnormal pressure) of industrial system 161 (symbolized by a bold rectangle, which may be, for example, a cooling system or a pipe system having its pressure measured at one or more locations), an operator may select a filter element to (i) define the robot pose and camera pose, and (ii) cause the rendering of only mission actions (e.g., mission action 163, mission action 165, and mission action 167, as shown in FIG. 1A ) associated with a particular type of instrument (e.g., a pressure instrument for abnormal pressure, a thermometer for abnormal temperature, etc.). This may allow the operator to quickly select one or more of the mission actions from the available or defined mission actions (e.g., mission action 163, mission action 165, and mission action 167) to add to a given mission or include within a new mission.For example, an operator can create a new mission for a robot (e.g., first robot 111) that carries a visual sensor, adding all of mission action 163, mission action 165, and mission action 167 so that the robot can use the visual sensor at the camera poses indicated by the metadata associated with each of mission actions 163, 165, and 167 to navigate to mission actions 163, 165, and 167, respectively, to take images of a pressure instrument at the robot poses indicated by the metadata associated with each of mission actions 163, 165, and 167.

[0075] In the above example, the operator may wish to inspect a particular portion of the system 161 (e.g., by providing a user-defined area 162 indicated by a dashed rectangle), and the rendering engine 1053 may render only the mission actions 163 and mission actions 165 defined in the robot poses and camera poses associated with the instrument and that are within the user-defined area 162. In this example, the user may, for example, select a mission action 163 and an additional mission action 165 from the rendered mission actions 163. Such selection of a mission action 163 may, for example, cause the first robot 111 to autonomously navigate to the robot pose associated with the mission action 163 to perform the mission (e.g., further inspection, etc.).

[0076] In some embodiments, the graphical representations of mission action 163, mission action 165, and / or mission action 167 may have a particular shape, color, pattern, or other characteristic. For example, mission action 163, mission action 165, and / or mission action 167 may each have a triangular shape, based on the fact that all of them are defined with the parameters of the robot pose, camera pose, and particular instrument (e.g., pressure instrument). The graphical representation of mission action 163 (or 165, or 167) may be selectable and, when selected or dragged, may cause mission action 163 (or 165, or 167) to be added to the mission.

[0077] 1A , the graphical representation of mission action 165 may optionally include additional selectable elements 1651 that, for example, when selected, render / pop up an image (or other graphical representation) of a particular instrument 1653 and / or other information (such as an instrument ID, such as ID_555, and / or a recently collected / recorded reading). In some embodiments, the graphical representation of mission action 165 may alternatively or additionally include further selectable elements associated with the graphical representation of mission action 165, where the further selectable elements (not shown) may, when selected, provide additional information, such as a robot pose of the robot, an image or representation of the robot, the model and / or manufacturer of the robot, a camera pose and type of the robot, an image of the camera, etc. The description of specific configurations (e.g., shape, color, pattern, POI, etc.) for each of the mission actions (e.g., mission actions 163, 165, 167, etc.) is not intended to be limiting.

[0078] Accordingly, embodiments disclosed herein render mission actions for an industrial facility within a map of the industrial facility. The mission actions may be rendered in a manner that depends, for example, on metadata associated with the mission actions. This may enable a human operator to identify mission actions to add to a given mission and / or to include within a new mission with reduced latency and / or a reduced amount of user input.

[0079] In some embodiments, the exemplary environment 100 may further include one or more additional POIs. For example, the exemplary environment 100 may include a robot control / simulation engine 1052 configured to control or simulate one or more mobile robots (e.g., first robot 111). For example, the robot control / simulation engine 1052 may cause the simulated robot to be rendered in a simulation of the industrial environment map 140, and a human operator may define a robot mission for the simulated robot, each of which includes a plurality of defined mission actions for the simulated robot (or a corresponding real robot) to perform one or more actions.

[0080] 1B illustrates a non-limiting example of storage of metadata associated with one or more mission actions, according to various embodiments. As shown in FIG. 1B, a mission action database 151 may be generated and stored (e.g., in storage 15), where mission action database 151 may include one or more entries for each mission action from one or more mission actions defined for an industrial facility (e.g., industrial facility 130). For example, the one or more entries may include a first entry 1511 for a first mission action, a second entry 1512 for a second mission action, and a third entry 1513 for a third mission action. The first entry 1511 may include metadata related to the first mission action, including, for example, one or more robot parameters of a robot for navigating the first mission action, where the one or more robot parameters may include a robot type (e.g., robot dog), a robot ID (e.g., robot dog_1111), a robot pose (X1, Y1, Z1, α1, β1, γ1), a robot mission (e.g., mission_1) that includes the first mission action, and / or other robot information. The first entry 1511 may further include one or more camera parameters of a camera carried by the robot, where the one or more camera parameters may include, for example, a camera pose (X1, Y1, Z1, α1, β1, γ1), a zoom level (2x), a focus setting (single autofocus mode), and / or other camera information. The first entry 1511 may further include one or more POI parameters related to a POI sensed or fixed in the first mission action. When the POI is a sensor in the first mission action, the one or more POI parameters may include, for example, a sensor type (e.g., a visual sensor, a temperature sensor, a pressure sensor, etc.) and a sensor ID (e.g., visual sensor_1111).The first entry 1511 may or may not include other information.

[0081] The second entry 1512 may include metadata related to the second mission action, which may include, for example, a robot type (e.g., drone), a robot ID (e.g., drone_1112), a robot mission (e.g., mission_2) including the second mission action, and one or more robot parameters such as a robot pose (X2, Y2, Z2, α2, β2, γ2). The second entry 1512 may further include one or more camera parameters including, for example, a camera pose (X2, Y2, Z2, α2, β2, γ2), a zoom level (e.g., 1x), and / or a focus setting (e.g., continuous autofocus mode). In some embodiments, the second entry 1512 may or may not include information other than values ​​for one or more robot parameters and values ​​for one or more camera parameters.

[0082] The third entry 1513 may include metadata related to the third mission action, including one or more robot parameters, such as the robot pose (X3, Y3, Z3, α3, β3, γ3), the robot type (e.g., crawler robot), the robot ID (e.g., crawler robot_1113), and the robot mission (e.g., mission_3) that includes the third mission action. The third entry 1513 may or may not include information other than values ​​for the one or more robot parameters.

[0083] As shown in FIG. 1B , the first mission action and the second mission action may be defined at different levels of granularity. The first mission action and the third mission action may be defined at different levels of granularity. The second mission action and the third mission action may be defined at different levels of granularity. In other words, different mission actions may be defined at different levels of granularity.

[0084] 1B is shown as separately storing a first entry 1511, a second entry 1512, and a third entry 1513 for a first mission action, but the description of mission action database 151 is not so limited. For example, mission action database 151 may store and divide entries based on robot mission. For example, mission action database 151 may include a first set of entries each corresponding to a mission action from a first robot simulation performed by a first robot (simulated or real robot), a second set of entries each corresponding to a mission action from a second robot mission performed by a second robot (simulated or real robot), ..., and an Nth set of entries each corresponding to a mission action from an Nth robot mission performed by an Nth robot (simulated or real robot).

[0085] In some embodiments, by way of non-limiting example, one or more mission actions (e.g., only the first mission action) may be selectively rendered (e.g., on the industrial environment map 140) depending on the granularity with which the one or more mission actions are defined (as reflected by the mission action's metadata). For example, a user may select a filter element to filter out any mission actions defined without a POI parameter for a particular instrument (e.g., a pressure instrument). In this case, the second and third mission actions shown in FIG. 1B would be filtered out and, for example, not rendered on the industrial environment map 140, and the first mission action may or may not be rendered depending on whether the metadata shown in the first entry 1511 includes a value for any POI parameter indicating that the pressure instrument is sensed or fixed in the first mission action. Note that the format of the mission action database 151 is not limited to the format shown in FIG. 1B, which is shown for illustrative purposes only.

[0086] 2A illustrates a non-limiting example of a user interface 201 of a computing device 109 showing a map on which all available mission actions are rendered, according to various embodiments. As illustrated in FIG. 2A , multiple mission actions (e.g., mission action_A, mission action_B, mission action_C, mission action_D, mission action_E, mission action_F, mission action_G) may be predefined for a first robotic mission and may be rendered in a first manner (e.g., represented by a graphical representation with a circle), and additional multiple mission actions (e.g., mission action_a, mission action_b, mission action_c, mission action_d, and mission action_e) may be predefined for a second robotic mission and may be rendered in a second manner that is different from the first manner (e.g., represented by a graphical representation with a triangle). In some embodiments, the multiple mission actions (mission action_A-G) and the additional multiple mission actions (mission action_a-mission action_e) may together form all defined / available mission actions for the industrial facility 130, although this is not required.

[0087] In some embodiments, mission actions (i.e., mission action_A through mission action_G) may be rendered in a first manner based on the metadata for such mission actions being defined at a first level of granularity, and mission actions (i.e., mission action_a through mission action_e) may be rendered in a second manner based on the metadata for such mission actions being defined at a second level of granularity. The first level of granularity may, for example, be higher or finer than the second level of granularity. For example, the metadata for mission action_A through mission action_G may be at a first level of granularity by including a robot pose for each of mission action_A through mission action_G and one or more camera parameters (e.g., which may define a camera pose, zoom level, and / or camera focus setting). For example, the metadata for mission action_a through mission action_e may be at a second level of granularity by including a robot pose for each of mission action_A through mission action_G and not including any camera parameters (e.g., camera pose).

[0088] As a non-limiting example, metadata for a mission action (i.e., mission action_A) may include a first robot pose (location and / or orientation) for a first robot associated with mission action_A and may include a first camera pose for a first camera carried by the first robot (e.g., to capture images for mission action_A). Such metadata for Mission action_A may be gathered and / or defined based on, for example, a previous instance in which the first robot was previously dispatched in the first robot pose, positioning the first camera in the first camera pose to capture images for mission action_A.

[0089] As a non-limiting example, metadata for a mission action (i.e., mission action_a) may include a second robot pose (location and / or orientation) for a second robot associated with mission action_a, and may not include any camera parameters. Such metadata for mission action_a may be gathered and / or defined, for example, based on previous instances in which the second robot was previously dispatched with the second robot pose. The second robot may be the same as or different from the first robot.

[0090] In some embodiments, by way of non-limiting example, the graphical representations for mission action_A through mission action_G are all rendered to have circular shapes, but some graphical representations (e.g., mission action_A, mission action_B, mission action_C) may have a different color or pattern than other graphical representations (e.g., mission action_D, mission action_E, mission action_F, mission action_G). The graphical representations for mission action_A through mission action_C on industrial environment map 140 may have circular shapes based on metadata for each of mission action_A through mission action_C, including both robot pose and camera pose, and such graphical representations may have a first color or a first pattern based on metadata for each of mission action_A through mission action_C, including one or more POI parameters (defining one or more characteristics of a POI sensed in the mission action). The graphical representation for mission action_D through mission action_G on the industrial environment map 140 may have a circular shape based on the metadata of each of mission action_D through mission action_G including both the robot pose and the camera pose, and such graphical representation may have a second color (e.g., different from the first color) or a second pattern (e.g., different from the first pattern) based on the metadata of each of mission action_D through mission action_G that does not include any POI parameters. The POI may be, for example, an instrument (or equipment, or other POI) fixed in the mission action.

[0091] In other words, the graphical representation for mission action_A (mission action_B, or mission action_C) may be circular and may have a first color or a first pattern based on metadata including one or more POI parameters related to a POI in mission action_A, in addition to including robot pose and one or more camera parameters. The POI may be identified or detected in mission action_A (e.g., by a first camera carried by the first robot), and the one or more POI parameters may include a type of POI (e.g., a temperature sensor, a pressure sensor, etc.) and / or an identifier of the POI. The graphical representation for mission action_D (mission action_E, mission action_F, or mission action_G) may be circular and may have a second color or a second pattern based on metadata that does not include any POI parameters because no industrial POI is identified in mission action_D (which may be an open space location).

[0092] In some embodiments, by way of non-limiting example, the graphical representations for mission action_a through mission action_e are all rendered to have triangles, but some graphical representations (e.g., of mission action_a, mission action_b, and mission action_c) may have a different color (or pattern) than other graphical representations (e.g., of mission action_d and mission action_e). The graphical representations for mission action_a through mission action_c on the industrial environment map 140 may have triangles based on the metadata for each of mission action_a through mission action_c, including the robot pose but not any camera pose, and such graphical representations may have a first color or a first pattern based on the metadata for each of mission action_a through mission action_c, including one or more POI parameters (defining one or more characteristics of a POI sensed in the mission action). The graphical representation for mission action_d through mission action_e on the industrial environment map 140 may have a triangle based on the metadata of each of mission action_d through mission action_e that includes the robot pose and does not include any camera pose, and such graphical representation may have a second color or a second pattern based on the metadata of each of mission action_D through mission action_G that does not include any POI parameters.

[0093] Stated differently, the graphical representation for mission action_a (or mission action_b, or mission action_c) may be triangular and may have a first color or a first pattern based on metadata including one or more POI parameters related to a POI in mission action_A, in addition to including robot pose and one or more camera parameters. The POI may be, for example, a sensor (or other POI, such as a pipe, equipment, etc.) identified or detected in mission action_a (e.g., by a first camera carried by the first robot), and the one or more POI parameters may include a POI type (e.g., a temperature sensor, a pressure sensor, a pipe, etc.) and / or a POI identifier (e.g., 123E5). As a non-limiting example, the graphical representation for mission action_d (or mission action_e) may be triangular and may have a second color or a second pattern based on metadata that does not include any POI parameters because no industrial POIs are identified in mission action_d.

[0094] It should be noted that in some embodiments, the graphical representation of a mission action defined with camera parameters on the industrial environment map 140 may be visually different / distinguishable from the graphical representation of a mission action defined without camera parameters. In some embodiments, the graphical representation of a mission action defined with one or more POI parameters on the industrial environment map 140 may be visually different from the graphical representation of a mission action defined without any POI parameters. Optionally, the graphical representation of a mission action defined with one or more POI parameters for a first POI on the industrial environment map 140 may be visually different from the graphical representation of a mission action defined with one or more POI parameters for a second POI that is different from the first POI.

[0095] FIG. 2B illustrates a non-limiting example of a user interface 203 showing a map on which all available mission actions are rendered, according to various embodiments. As shown in FIG. 2B , the user interface 203 may include one or more filter elements (e.g., each corresponding to a filter criterion as previously described) rendered in the user interface 203 along with a map 140 on which the mission actions (e.g., mission action_A through mission action_G, mission action_a through mission action_e) are rendered. The map of FIG. 2B may be the same as or different from the map 140 of FIG. 1A . In some embodiments, the one or more filter elements may include a first filter element (e.g., filter element_1), a second filter element (e.g., filter element_2), and / or a third filter element (e.g., filter element_3). In some embodiments, additionally or alternatively, the one or more filter elements may include a customized filtering definition field 213 for receiving user-defined filtering criteria (e.g., via typed input, voice input, etc.).

[0096] In some embodiments, filter element_1 may be, for example, a first selectable element that, when selected, applies a first filter rule / criteria to selectively render mission actions defined by robot parameters, each of which specifies a particular robot type. filter element_2 may be, for example, a second selectable element that, when selected, applies a second filter rule to selectively render mission actions defined by camera parameters. filter element_3 may be, for example, a third selectable element that, when selected, applies a third filter rule to selectively render mission actions defined by POI parameters (e.g., specifying a particular type of sensor or instrument).

[0097] In some embodiments, filter element_1 may be configured to trigger a drop-down menu (listing multiple robot types) for a user to selectively render mission actions based on the robot type defined for the mission action, filter element_2 may be configured to trigger a drop-down menu for selectively rendering mission actions based on camera parameters (listing values ​​for one or more camera parameters, with a “no camera parameters” option indicating a preference for mission actions defined without any camera parameters), and filter element_3 may be configured to trigger a drop-down menu for a user to selectively render mission actions based on POI parameters. Note that the number / format of filter elements or filter coefficients are not limited herein and may be any applicable number or format. For example, one or more mission actions may be selectively rendered based on a detected event (e.g., a cooling tower failure). In this case, one or more mission actions defined in both the camera pose and pressure sensor type (in metadata associated with the one or more mission actions) may be automatically selectively rendered in the industrial environment map 140 in response to the detected event (e.g., a cooling tower failure).

[0098] FIG. 2C illustrates an exemplary user interface 205 showing a map on which one or more selected mission actions are rendered for implementing selected aspects of the present disclosure, according to various embodiments. A user of the industrial environment map 140 may provide filter parameters for selectively rendering mission actions defined in both the robot pose and the camera pose. In this case, as shown in FIG. 2C , mission action_A through mission action_G may be selectively rendered in response to receiving filter parameters for selectively rendering mission actions defined in both the robot pose and the camera pose. Mission action_A through mission action_G may be selected from all available mission actions (e.g., mission action_A through mission action_G, mission action_a through mission action_e) for display on the industrial environment map 140 based on each of them being defined in a robot pose and a camera pose, and mission action_a through mission action_e are filtered from being displayed based on each of them not being defined in a camera pose.

[0099] 2D illustrates another exemplary user interface 207 showing a map rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. A user of the industrial environment map 140 may provide additional filter parameters to selectively render mission actions defined with robot pose, camera pose, and POI parameters. In this case, as shown in FIG. 2D , mission action_A through mission action_C may be selected from all available mission actions (e.g., mission action_A through mission action_G, mission action_a through mission action_G) for display on the industrial environment map 140 based on the fact that each of mission action_A through mission action_C is defined with POI parameters as well as robot pose and camera pose, whereas mission action_a through mission action_e and mission action_D through mission action_G are filtered out (e.g., not displayed) because each of mission action_a through mission action_e and mission action_D through mission action_G is not defined with POI parameters.

[0100] 2E illustrates a further exemplary user interface 209 showing a map rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. In this example, as shown in FIG. 2E, one or more mission actions (e.g., mission action_a, mission action_C, mission action_D) may be selectively and automatically displayed within the industrial environment map 140 in response to the detection of a target event (e.g., an anomaly) at a particular location T within the industrial facility 130.

[0101] One or more mission actions (e.g., mission action_a, mission action_C, mission action_D) may be displayed based on the distance between a particular location T and the respective mission action (of mission action_a, mission action_C, and mission action_D) being within a predefined distance threshold (e.g., 2 m). Alternatively or additionally, one or more mission actions (e.g., mission action_C and mission action_D) may be displayed based on metadata associated with each of the one or more mission actions that specifies the robot pose and camera pose. Alternatively or additionally, one or more mission actions (e.g., mission action_C and mission action_D) may be displayed based on user input, where the user input may, for example, modify a predefined distance threshold (e.g., from 1.0 m to 0.5 m).

[0102] 2F illustrates a further exemplary user interface 211 illustrating a map rendered with one or more selected mission actions for implementing selected aspects of the present disclosure, according to various embodiments. In this example, as shown in FIG. 2F, one or more mission actions (e.g., mission action_a, mission action_C, mission action_D, mission action_d) may be selectively displayed within the industrial environment map 140 in response to the detection of a target event (e.g., anomaly) at a particular location T within the industrial facility 130 and in response to a user defining an area (defined by a dashed line in FIG. 2F) that includes the particular location T. The area may be defined, for example, via a user touch on the display of the computing device 109.

[0103] 3 illustrates an exemplary method 300 for implementing selected aspects of the present disclosure, according to various embodiments. For convenience, the operations of the flowchart are described with reference to a system on which the operations are performed. This system may include various POIs on various computer systems, such as one or more POIs on server computing device 105 (and / or additional computing devices, such as client device 103-A or 103-B). Moreover, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0104] In various embodiments, in block 302, the system may identify, by a server such as server computing device 105, corresponding metadata for each of a plurality of mission actions for an industrial environment. The industrial environment may be or include an industrial facility. The industrial facility may take the form of a chemical processing plant, an industrial office environment, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil plant, or any other applicable facility.

[0105] In some embodiments, prior to identifying the metadata for each of the multiple mission actions, the system may determine or identify the multiple mission actions. The multiple mission actions may be determined, for example, based on user input and / or data transmitted from one or more robots deployed in the industrial environment. For example, the user input may define a first robotic mission for a first robot and a second robotic mission for a second robot (the same or different from the first robot). The first robotic mission includes a first mission action, a second mission action, and a third mission action defined by a human operator for the first robot to navigate to and / or perform the corresponding actions. The second robotic mission includes mission action_1, mission action_2, mission action_3, and mission action_4 defined by a human operator or another operator for the second robot to navigate to and / or perform the corresponding actions. In this example, the first through third mission actions, mission action_1 through mission action_4, and one or more additional mission actions (if available) form a plurality of mission actions predefined by a human operator. For example, the plurality of mission actions may include a fourth mission action determined based on data (e.g., lidar sensor data) transmitted from a robot (e.g., carrying a lidar sensor) indicating detection of an object or event (e.g., an oil spill or recurring leak) by the robot in the fourth mission action. The plurality of mission actions may also be determined in other manners and are not limited to the description herein.Optionally, the first mission action, the second mission action, and the third mission action may be robot mission actions that form a path along which the first robot observes one or more aspects of the industrial environment, and mission action_1 through mission action_4 may form a path along which the second robot observes one or more aspects of the industrial environment.

[0106] As a non-limiting example, mission actions for an industrial facility may be initially defined by a human operator controlling a robot (e.g., through the use of a handheld controller and / or a computer interface) and providing input to specify the mission action. Controlling the robot may be controlling a real robot in a real industrial facility or a simulated robot in a simulation of the industrial facility. For example, the human operator may utilize a handheld controller to navigate the robot to a particular robot pose (including location and orientation information for configuring the robot) and adjust a camera carried by the robot to a particular camera pose when the robot is in the particular robot pose (so that the camera can capture a clear image of a target object or location of interest to the human operator). In this example, the human operator may provide user interface input to define the robot pose of the robot and the camera pose of the camera carried by the robot as metadata for the mission action.

[0107] In the above non-limiting example, after a mission action is initially defined, the robot (or another robot of the same type or model) may utilize the defined mission action to autonomously navigate to the mission action (i.e., without requiring any human operator control from a human operator).

[0108] In some embodiments, different mission actions may be defined and / or navigated at different levels of granularity. For example, a mission action to capture an image of an instrument may define both (a) the robot pose (e.g., X-axis, Y-axis, Z-axis, and optionally, rotation and / or translation) and (b) camera parameters such as the camera pose (e.g., rotation and / or translation), camera zoom level, and / or camera focus setting. On the other hand, a mission action to capture a gas reading may define only (a) the robot pose and not any camera parameters. Mission actions may optionally or additionally be associated with and stored metadata including POI parameters that define characteristics of the POI being captured by the sensor readings in the mission action. For example, a mission action to capture a particular instrument reading may define the instrument type (e.g., temperature type, pressure type, etc.) and / or the instrument's unique identifier (e.g., X1234AB).

[0109] In other words, metadata associated with multiple mission actions may have different levels of granularity for different mission actions. In some embodiments, the metadata may be received or collected based on user input and / or data transmitted from the robot associated with the mission action (e.g., Lidar sensor data, configuration data, etc.). In some embodiments, the metadata defining each of the multiple mission actions may be stored in and accessed via a mission action database.

[0110] In various embodiments, at block 304, the system may render, by a server, such as server computing device 105, a first subset of the mission actions with the first graphical feature in a map interface for the industrial environment, where rendering the first subset of the mission actions with the first graphical feature is responsive to a determination that corresponding metadata for each of the mission actions in the first subset matches one or more first criteria. In some embodiments, the map interface is a floor plan view of the industrial environment.

[0111] In some embodiments, determining that the corresponding metadata for each of the first subset of mission actions matches one or more first criteria may include determining that the corresponding metadata for each of the first subset of mission actions indicates that the first subset of mission actions includes readings from a first type of sensor.

[0112] In some embodiments, determining that the corresponding metadata for each of the first subset of mission actions matches the one or more first criteria may include determining that the corresponding metadata for each of the first subset of mission actions includes a corresponding robot pose for the mission action and one or more corresponding visual sensor parameters for capturing visual data for the mission action while in the corresponding robot pose. In some embodiments, the one or more corresponding visual sensor parameters may include a visual sensor pose, a visual sensor zoom level, and / or a visual sensor focus parameter associated with a visual sensor carried by the mobile robot in the industrial environment.

[0113] In some embodiments, the metadata for the first subset of mission actions includes one or more POI parameters related to a POI (e.g., a measuring device, a hazardous object, etc.) in one of the mission actions of the first subset. The one or more POI parameters may include, for example, a POI identifier and / or a type of POI.

[0114] In various embodiments, at block 306, the system may render, by a server, such as server computing device 105, within a map interface for the industrial environment, the second subset of mission actions with the second graphical feature along with the first subset of mission actions, where rendering the second subset of mission actions with the second graphical feature is responsive to determining that corresponding metadata for each of the mission actions of the second subset matches one or more second criteria. In some embodiments, the first subset of mission actions and the second subset of mission actions may be rendered simultaneously. In some embodiments, the first subset of mission actions and the second subset of mission actions may not be rendered simultaneously. The first criteria is different from the second criteria. The first graphical feature may be visually different from the second graphical feature. In other words, the first subset of mission actions and the second subset of mission actions may be rendered in different ways (e.g., different shapes, colors, patterns, etc.) based, for example, on the fact that the first subset of mission actions are defined in metadata having a first level of granularity and the second subset of mission actions are defined in metadata having a second level of granularity (which may be finer or coarser than the first level of granularity).

[0115] In some embodiments, determining that the corresponding metadata for each of the second subset of mission actions matches one or more second criteria may include determining that the corresponding metadata for each of the second subset of mission actions indicates that the second subset of mission actions includes readings from a second type of sensor, which may be different from the first type described above.

[0116] In some embodiments, determining that the corresponding metadata for each of the second subset of mission actions matches one or more second criteria may include determining that the corresponding metadata for each of the second subset of mission actions includes a corresponding robot pose for the mission action but lacks any camera pose for capturing an image of the mission action.

[0117] In some embodiments, determining that the corresponding metadata for each of the mission actions of the second subset matches one or more second criteria may include determining that the corresponding metadata for each of the mission actions of the second subset includes only a corresponding robot pose for each of the mission actions of the second subset.

[0118] In various embodiments, in block 308, the system may receive a selection of the first subset or the second subset of given mission actions via a map interface for the industrial environment, for example, by a server, such as server computing device 105. In various embodiments, in block 310, the system may add the given mission actions to a predefined robot mission or include the given mission actions within a new robot mission in response to receiving the selection, for example, by a server, such as server computing device 105. The given mission actions may be automatically added to the predefined robot mission or the new robot mission.

[0119] For example, a predefined robot mission may include, e.g., mission actions A-K, but may not include mission action L (even though mission action L is defined in an alternate mission). In some embodiments, mission action L may be selected from one or more mission actions selectively rendered on a map interface based on user input. In this case, there may be a desire to add mission action L to a predefined robot mission, for example, if a potentially hazardous condition exists near mission action L (e.g., it is within a predetermined distance to mission action L) or if there is a potential problem with an instrument associated with mission action L.

[0120] 4 illustrates another exemplary method 400 for implementing selected aspects of the present disclosure, according to various embodiments. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various POIs on various computer systems, such as one or more POIs on server computing device 105 (and / or additional computing devices, such as client device 103-A or 103-B). Moreover, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0121] In various embodiments, at block 402, the system may identify, by a server such as server computing device 105, corresponding metadata for each of a plurality of mission actions for the industrial environment.

[0122] In some embodiments, the corresponding metadata for each of the plurality of mission actions includes a corresponding robot pose for the respective robot in the corresponding mission action, and determining the filtering criteria includes determining one or more pose-based filtering criteria (e.g., for selectively rendering mission actions that are within an area that includes a target location). The target location may be, for example, a location where an anomaly (e.g., an oil spill) has been detected, or a location where maintenance is required, etc. The area that includes the target location may be defined based on user input in a map interface, or the area may be automatically defined based on a predetermined distance to the target location.

[0123] In some embodiments, the corresponding metadata for each of the subset of mission actions includes one or more visual sensor parameters associated with a visual sensor for capturing images at a corresponding robot pose associated with the corresponding mission action, and determining the filtering criteria includes determining one or more visual-based filtering criteria for, e.g., selectively rendering mission actions defined at particular values ​​of the visual sensor parameters (e.g., at a 5x zoom level, etc.).

[0124] In some embodiments, the metadata for each of the subset of mission actions includes one or more POI parameters associated with a POI in the corresponding mission action, hi some embodiments, the one or more POI parameters include a POI identifier and / or a POI type.

[0125] In various embodiments, in block 404, the system may determine one or more filtering criteria based on user interface input and / or based on events detected within the industrial environment, for example, by a server such as server computing device 105. In some embodiments, the one or more filtering criteria include requiring that the metadata for each of the subset of mission actions include, or be of a type, a POI having a POI identifier.

[0126] In various embodiments, in block 406, the system may select, by a server, such as server computing device 105, a subset of the mission actions from the mission actions, where selecting the subset of mission actions is responsive to a determination that corresponding metadata for each of the mission actions in the subset matches one or more filtering criteria.

[0127] In various embodiments, in block 408, the system may, for example, by a server such as server computing device 105, render the subset of mission actions within the map interface for the industrial facility without rendering other mission actions that are not part of the subset of mission actions.

[0128] In some embodiments, the rendered subset may include a first mission action rendered with a first graphical feature and a second mission action rendered with a second graphical feature, where the first graphical feature is visually distinct from the second graphical feature. In some embodiments, the metadata associated with the first mission action is at a higher level of granularity than the metadata associated with the second mission action.

[0129] In various embodiments, at block 410, the system may receive, by a server such as the server computing device 105, a selection of the subset of given mission actions via a map interface for the industrial facility.

[0130] In various embodiments, at block 412, the system may add the given mission action to a predefined robot mission or include the given mission action within a new robot mission in response to receiving the selection, for example, by a server such as server computing device 105.

[0131] 5 is a block diagram of an exemplary computing device 510 that may optionally be utilized to implement one or more aspects of the techniques described herein. The computing device 510 generally includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input / output devices enable user actions with the computing device 510. The network interface subsystem 516 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0132] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 510 or onto a communication network.

[0133] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual presentation, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or to another machine or computing device.

[0134] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of the methods of Figure 3 or Figure 4, as well as for implementing the various components shown in Figure 1.

[0135] These software modules are generally executed by the processor 514, alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including a main random access memory (RAM) 530 for recording instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored within the storage subsystem 524 by the file storage subsystem 526 or within other machines accessible by the processor 514.

[0136] The bus subsystem 512 provides a mechanism for allowing the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0137] Computing device 510 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 510 shown in Figure 5 is intended only as a specific example to illustrate some embodiments. Many other configurations of computing device 510 are possible, having more or fewer components than the computing device shown in Figure 5.

[0138] While several embodiments have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the present teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Accordingly, it should be understood that the foregoing embodiments are presented by way of example only, and that, within the scope of the appended claims and equivalents thereto, embodiments other than those specifically described and claimed may be practiced. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0139] 15. Storage 100 Example Environments 103-A Local client device 103-B Local client device 105 Server Computing Devices 106 Processing Automation Network 109 Computing Devices 111 The First Robot 112 The Second Robot 130 Industrial Facilities 140 Industrial Environment Map 151 Mission Action Database 161 Industrial Systems 163 Mission Action 165 Mission Action 167 Mission Action 201 User Interface 203 User Interface 205 User Interface 207 User Interface 209 User Interface 211 User Interface 213 Filtering definition field 510 Computing Devices 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Storage Subsystem 525 Memory Subsystem 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 Read-Only Memory (ROM) 1051 Mission Action (MA) Selection Engine 1052 Robot Control / Simulation Engine 1053 Rendering Engine 1054 Mission Action Definition Engine 1511 First Entry 1512 Second Entry 1513 Third Entry 1651 selectable elements 1653 Specific Instruments

Claims

1. 1. A method implemented by one or more processors, comprising: identifying metadata for each of a plurality of mission actions for an industrial environment; rendering a first subset of the mission actions with a first graphical feature within a map interface for the industrial environment, wherein rendering the first subset of the mission actions with a first graphical feature is responsive to a determination that corresponding metadata for each of the mission actions in the first subset matches one or more first criteria; rendering a second subset of the mission actions with second graphical features in the map interface for the industrial environment along with the first subset of the mission actions, wherein rendering the second subset of the mission actions with second graphical features is in response to determining that corresponding metadata for each of the mission actions of the second subset matches one or more second criteria; the first criterion is different from the second criterion; the first graphical feature is visually distinct from the second graphical feature; Steps and receiving, via the map interface for the industrial environment, a selection of a given mission action of the first subset or the second subset; responsive to receiving the selection, adding the given mission action to a predefined robot mission or including the given mission action within a new robot mission; A method comprising:

2. Determining that the corresponding metadata for each of the mission actions of the first subset matches the one or more first criteria includes: determining that the corresponding metadata for each of the mission actions of the first subset includes a reading from a first type of sensor; Including, determining that the corresponding metadata for each of the mission actions of the second subset matches the one or more second criteria; determining that the corresponding metadata for each of the mission actions of the second subset includes a reading from a sensor of a second type, the second type being different from the first type; 2. The method of claim 1, comprising:

3. Determining that the corresponding metadata for each of the mission actions of the first subset matches the one or more first criteria includes: determining that the corresponding metadata for each of the mission actions of the first subset includes a corresponding robot pose for the mission action and corresponding one or more visual sensor parameters for capturing visual data of the mission action while in the corresponding robot pose; 2. The method of claim 1, comprising:

4. The method of claim 3 , wherein the mission actions of the first subset include hazardous materials.

5. The method of claim 3 , wherein the one or more visual sensor parameters include a visual sensor pose, a visual sensor zoom level, and / or a visual sensor focus parameter associated with a visual sensor carried by a mobile robot within the industrial environment.

6. The method of claim 3 , wherein the metadata for each of the mission actions of the first subset further includes one or more POI parameters related to a POI in a corresponding mission action of the mission actions of the first subset.

7. The method of claim 6 , wherein the one or more POI parameters include a POI identifier of the POI and / or a type of the POI.

8. determining that the corresponding metadata for each of the mission actions of the second subset matches the one or more second criteria; determining that the corresponding metadata for each of the mission actions of the second subset includes a corresponding robot pose for the mission action but lacks any camera parameters for capturing an image of the mission action; 2. The method of claim 1, comprising:

9. determining that the corresponding metadata for each of the mission actions of the second subset matches the one or more second criteria; determining that the corresponding metadata for each of the mission actions of the second subset includes only a corresponding robot pose for each of the mission actions of the second subset; 9. The method of claim 8, comprising:

10. The method of claim 1 , wherein the map interface is a floor plan of the industrial environment.

11. 2. The method of claim 1 , wherein the first graphical feature includes a first selectable graphical user interface (GUI) element representing a first mission action from the first subset of mission actions, and the second graphical feature includes a second selectable graphical user interface (GUI) element representing a second mission action from the second subset of mission actions, and the first selectable GUI element is different from the second selectable GUI element.

12. 1. A method implemented by one or more processors, comprising: identifying metadata for each of a plurality of mission actions for an industrial environment; determining one or more filtering criteria based on user interface input and / or based on events detected within the industrial environment; selecting a subset of the mission actions from the mission actions, wherein selecting the subset of mission actions is responsive to a determination that corresponding metadata for each of the mission actions in the subset matches the one or more filtering criteria; rendering the subset of mission actions within a map interface for the industrial environment without rendering any other mission actions from the plurality of mission actions that are not of the subset; receiving a selection of a given mission action of the subset via the map interface for the industrial environment; responsive to receiving the selection, adding the given mission action to a predefined robot mission or including the given mission action within a new robot mission; A method comprising:

13. the metadata for each of the plurality of mission actions includes a robot pose of the respective robot in the corresponding mission action; The method of claim 12 , wherein the one or more filtering criteria include pose-based filtering rules for selectively rendering mission actions that are within an area that includes a target location.

14. The method of claim 13 , wherein the area that includes the target location is defined based on user input in the map interface, or the area is automatically defined based on a predetermined distance to the target location.

15. the metadata for each of the mission actions in the subset includes one or more visual sensor parameters associated with a visual sensor for capturing images at the robot pose associated with a corresponding mission action; The method of claim 13 , wherein the one or more filtering criteria include visual-based filtering rules for selectively rendering specific types of mission actions.

16. The method of claim 12 , wherein the metadata for each of the mission actions of the subset includes one or more POI parameters associated with a POI in the corresponding mission action.

17. The method of claim 15 , wherein the one or more POI parameters include a POI identifier of the POI and / or a type of the POI.

18. Rendering the subset of mission actions comprises: rendering a first mission action from the subset of mission actions using a first graphical characteristic; rendering a second mission action from the subset of mission actions using a second graphical characteristic, the first graphical characteristic being visually distinct from the second graphical characteristic; 13. The method of claim 12, comprising:

19. 20. The method of claim 18, wherein metadata associated with the first mission action is at a higher level of granularity than metadata associated with the second mission action.

20. 1. A method implemented by one or more processors, comprising: identifying metadata for each of a plurality of mission actions for an industrial environment, the metadata for each of the plurality of mission actions including at least a robot pose for a robot in a corresponding mission action; detecting an object or event to be inspected within the industrial environment; responsive to detecting the object or the event, rendering a first mission action and a second mission action within a map interface for the industrial environment; wherein the step of rendering a first mission action and a second mission action is responsive to determining that the first mission action and the second mission action are within a predetermined distance to a location of the object or the event; the first mission action and the second mission action are rendered with different graphical characteristics based on metadata of the first mission action and the second mission action having different levels of granularity; Steps and receiving a selection of the first mission action or the second mission action via the map interface for the industrial environment; adding the first mission action or the second mission action to a robot mission in response to receiving the selection; A method comprising:

Images