REPRESENTATION OF MISSION ACTIONS ON AN INDUSTRIAL ENVIRONMENT MAP

Abstract

Method (300) implemented by one or more processors, the method comprising: Identifying (302) metadata for each of a plurality of mission actions for an industrial environment; Reproducing (304), in a map surface for the industrial environment, a first subset of mission actions with first graphic features, wherein the reproduction of the first subset of mission actions with the first graphic features is performed in response to a determination that appropriate metadata for each of the mission actions of the first subset matches one or more first criteria; Reproduce (306), in the map surface for the industrial environment and together with the first subset of mission actions, a second subset of mission actions with second graphic features, wherein the reproduction of the second subset of mission actions with the second graphic features is performed in response to a determination that appropriate metadata for each of the mission actions of the second subset matches one or more second criteria, the first criteria differ from the second criteria, and where the first graphic features are visually distinct from the second graphic features; Receiving (308) a selection, via the map surface for the industrial environment, of a given mission action of the first subgroup or the second subgroup; and in response to receiving the selection, adding the given mission action to a predefined robot mission or incorporating the given mission action into a new robot mission.

Description

[0001] In an industrial plant, multiple robots can be deployed for various purposes, such as capturing images from measuring instruments, performing gas measurements, capturing images for anomaly detection, and so on. For example, at a given time, a first robot can execute an initial mission, which comprises a series of mission actions. Each mission action defines a corresponding robot pose and a corresponding camera pose for the robot's camera, enabling the capture of images of points of interest (POIs). Accordingly, executing the initial mission might involve implementing the series of mission actions, resulting in the robot navigating to different robot poses and capturing corresponding images in those poses (e.g., with the camera in a specific camera pose).At the given time, a second robot can also execute a second mission, which comprises a unique set of mission actions, each defining a corresponding robot pose without defining a camera pose (e.g., the second robot may not have a camera but includes gas sensor(s)). Accordingly, executing the second mission may involve implementing the unique set of mission actions, resulting in the robot navigating to various robot poses and taking gas measurements in those poses using gas sensors and without regard to a camera pose.

[0002] A mission action can initially be defined by a human operator for a robot, which is controlled and specified by inputs provided by the human operator (e.g., using a hand controller and / or a computer interface). The robot can be a real robot controlled in a real industrial plant or a simulated robot controlled in a simulation of the real industrial plant. For example, when defining a mission action, a human might use a hand controller to navigate a robot to a specific robot pose (e.g., location and orientation of the robot), or point the robot's camera at a specific camera pose (e.g.,The robot can set the location and orientation of the camera while in the specified robot pose and provide a user interface input that defines the specific robot pose and camera pose as metadata for the specific mission action. Once the robot mission action is initially defined, the robot (or another robot) can then use the defined robot mission action to autonomously (i.e., without requiring control from a human operator) implement the mission action (e.g., navigate to the robot pose and adjust the camera to the camera pose).

[0003] DE 11 2022 001 224 T5 describes a system or procedure which includes defining missions based on factors associated with the missions or environmental data associated with the system, assigning missions to the robot fleet based on the capabilities of the robots, generating a plan of missions and robots, and managing the robot fleet using feedback.

[0004] EP 3 557 358 B1 describes an autonomous system for use with a vehicle in an environment. The autonomous system comprises a processor functionally connected to a storage device, a variety of sensors functionally connected to the processor, a vehicle control unit, a situation awareness module, a task planning module, and a task execution module. The situation awareness module is configured to determine the state of the environment, at least partially, based on sensor data from at least one of the multiple sensors. The task planning module is configured to identify, via the processor, multiple tasks to be performed by the vehicle and to create a task assignment list from these multiple tasks, based at least partially on predetermined optimization criteria.The task execution module is configured to instruct the vehicle control unit to execute the multitude of tasks according to the task assignment list. The task execution module can also be configured to monitor the vehicle or the vehicle control unit during the execution of the task assignment list in order to identify any errors.

[0005] US Patent 2022 / 0250658A1 discloses methods and systems for automatically selecting and operating autonomous vehicles to collect inspection data relating to one or more inspected facilities in response to a received inspection request specifying the facilities to be inspected. Specifically, based on the inspection request, the required inspection data is automatically determined by analyzing one or more structural models representing the inspected facility(ies), and mission parameters for an inspection mission to collect the inspection data are calculated. Subsequently, the operating parameters of a variety of autonomous vehicles are analyzed with respect to the calculated mission parameters to identify one or more autonomous vehicles capable of collecting the inspection data.In addition, operating instructions are calculated for one or more autonomous vehicles selected for the inspection mission in order to collect the required inspection data and are transmitted to the appropriate control of the selected autonomous vehicles.

[0006] US 2022 / 0147059 A1 describes embodiments that provide systems, procedures, and computer-readable storage media for performing robotic tasks in a distributed and coordinated manner. A fleet of robots can use a sequence of messages to appoint supervisors for a set of tasks, with the supervisor robots being responsible for ensuring that their supervised tasks are performed by other robots in the fleet. The supervisors can solicit requests from other robots to perform available tasks and select a robot to perform an available task. Once a robot is appointed, the supervisor and the worker can use message sequences to monitor the status of the task and the robots involved (e.g., the supervisor and the worker).Monitoring allows the supervisor to detect a disabled worker and enables other robots to detect a disabled supervisor. When disabled robots are detected, the robot(s) that detect the failure can initiate actions to take over the role(s) of the disabled robot(s), ensuring the smooth operation of the robot fleet and efficient task completion.

[0007] The disclosure in US 2020 / 0150687A1 relates generally to autonomous machines (AMs) and specifically to techniques for intelligently planning, managing, and executing various tasks using AMs. It discloses a control system (referred to as a fleet management system or FMS) for managing a set of resources at a site, which may include AMs. The FMS is configured to control and manage the AMs at the site so that the AMs can perform tasks autonomously. An AM can communicate directly with another AM at the site to complete a task without needing to be in constant communication with the FMS during task execution. The FMS is configured to use various optimization techniques to allocate resources (e.g., AMs) for task execution at the site.Resource allocation is carried out in such a way as to maximize the use of available AMs while ensuring that tasks are completed on time.

[0008] JP 2023 - 154 576 A describes, as a solution to the problem of contributing to the automation of facility management, a storage device that stores existing data, including data on equipment management and an actual result, information on the path structure of the existing data, mission information about a facility management task, and information on the path structure of the mission information. A computing device extracts, based on the existing data and the path structure of the existing data, information about the meaning of terms contained in the existing data and generates information about the degree of association between a multitude of terms.Furthermore, based on the mission information, the path structure of the mission information, and information about the degree of association between the terms, the computing device generates mission management information, including information about the meaning of each of a multitude of facility management tasks and information about the time series of the multitude of tasks. Subsequently, based on the mission management information, the computing device generates control display data for a facility management device.

[0009] A technical problem to be solved by the methods of the independent claims is to improve the representation of mission actions in an industrial environment map. The method according to claim 1 solves this problem, in particular, by representing, in a map surface for an industrial environment, different subsets of mission actions with visually distinguishable graphic features. The method according to claim 12 solves this problem, in particular, by selecting a subset of mission actions in response to a determination that corresponding metadata for each of the mission actions in the subset matches one or more filter criteria, and representing, in a map surface for an industrial plant, the subset of mission actions without representing any other mission action from the plurality of mission actions that does not belong to the subset.The method according to claim 20 solves this problem in particular by displaying, in a map surface for an industrial environment, a first mission action and a second mission action in response to the detection of an object or event, wherein the first mission action and the second mission action are displayed with different graphic features based on the fact that the metadata of the first mission action and the second mission action have different levels of granularity.

[0010] As mentioned above, robots in an industrial environment (e.g., an industrial plant) can perform predefined missions, implementing a sequence of mission data defined for each mission. For example, a first robot can perform a first predefined mission by implementing a first sequence of mission actions, and a second robot, different from the first, can perform a second predefined mission by implementing a second sequence of mission actions. The first robot can be the same as the second robot or different from it. For instance, the first robot might differ from the second robot in that they are from different manufacturers or are of different types (e.g.,...).The robots may be four-legged, multi-wheeled, self-balancing with a single wheel, tracked, aircraft, etc.) and / or carry different payloads (e.g., the first robot includes a camera and / or a gas sensor, and the second robot does not). The first defined mission may be the same as or different from the second defined mission.

[0011] In some implementations, different mission actions within the same sequence (e.g., the first or the second sequence) can be defined with the same level of granularity. In other implementations, different mission actions within the same sequence (e.g., the first or the second sequence) can be defined with different levels of granularity. For example, some mission actions might be defined with more granularity (e.g., robot pose and camera parameters) than other mission actions (e.g., robot pose only). As a non-restrictive example, the first sequence of mission actions could include a first mission action, a second mission action, and a third mission action, each defined with different levels of granularity.In this non-restrictive example: the first mission action can be defined with a first robot pose of the first robot, but not with a definition of a camera pose of a camera that is part of the first robot's payload; the second mission action can be defined with a second robot pose of the first robot and a first camera pose of the camera when the first robot is in the second robot pose; and the third mission action can be defined with a third robot pose of the first robot, a second camera pose of the camera, and with information from a POI (e.g., a component ID, a component type, etc.) captured by the camera in the second camera pose when the first robot is in the third robot pose.Accordingly, in this non-restrictive example, the third mission action can be defined with the greatest degree of granularity (robot pose, camera pose, and POI information), the first mission action can be defined with the lowest degree of granularity (robot pose only), and the second mission action can be defined with a degree of granularity (robot pose and camera pose) that lies between that of the first and second mission actions.

[0012] In some implementations, a mission action can optionally or additionally be defined with POI parameters that define characteristics of POIs captured by sensor measurement(s) during the mission action. For example, a mission action (e.g., the second mission action) in which a robot (e.g., the first robot) captures an image of a specific measuring device can be defined by the type of that measuring device (e.g., temperature type, pressure type, etc.) and / or a unique identifier of that measuring device (e.g., X1234AB). The POI characteristic(s) defined for the mission action can be provided by an operator defining the mission action and / or automatically determined by analyzing images and / or other vision data captured during the mission action.

[0013] In some implementations, different mission actions can be defined in different sequences (e.g., the first sequence and the second sequence) with the same level of granularity. In other implementations, different mission actions can be defined in different sequences (e.g., the first sequence and the second sequence) with different levels of granularity. For example, a mission action in the first sequence might be to capture an image of a measuring device, and an additional mission action in the second sequence might be to take gas measurements. In this example, the mission action can be defined using either of the following: (a) a robot pose (e.g., X, Y, Z coordinates and optionally rotation and / or translation) of a robot and / or other robot parameters (e.g., a robot type); and (b) camera parameter(s), such as a camera pose (e.g.,Rotation and / or translation), a camera zoom level and / or a camera focus setting of a camera used to capture the image of the measuring device. The additional mission action can be defined with a robot pose (and occasionally only the robot pose) and without camera parameters.

[0014] It should be noted that a specific mission action(s) can be defined and navigated to using a specific type of robot. In this case, these mission action(s) may not be defined with varying degrees of granularity.

[0015] In some implementations, once defined, a mission action can be stored, along with metadata that defines the mission action, for subsequent access (e.g., within the industrial environment). For example, a human operator can add the defined mission action to a given (or new) mission of a specific robot to ensure the operation of the industrial plant within the industrial environment. It should be noted that different mission actions can be defined and stored with metadata exhibiting varying levels of granularity, or that the same mission action can be defined and stored with metadata exhibiting varying levels of granularity, which may vary depending on the robot.

[0016] Although a mission (e.g., the aforementioned first defined mission or second defined mission) may remain unchanged over time, there may be situations where it is desirable to add a given mission action, at least temporarily, to a given mission (where the given mission did not previously include the given mission action). For example, the given mission may include mission actions AK but not mission action L (even though mission action L is defined in an alternative mission), but there may be a desire to add mission action L to the given mission. For instance, there may be a desire to add mission action L to the given mission if there is a potentially hazardous condition near mission action L, if there is a potential problem with a gauge associated with mission action L, and so on.

[0017] There may also be situations where it is desirable to create a new mission that includes a new set of predefined mission actions from one or more predefined missions. For example, if there is a potential leak or other problem in a given area of ​​an industrial facility, there may be a desire to create a new mission that includes mission actions (e.g., predefined ones) in or near that area. Such mission actions could include two from a first mission, two from a second mission, and one from a third mission.

[0018] Existing techniques may hinder or prevent adding a given mission action to a given mission and / or hinder or prevent creating a new mission that includes a new collection of predefined mission action(s) from other missions. For example, existing techniques may not overlay mission actions on a map of the industrial environment, making it difficult and / or time-consuming for a human operator to identify a mission action that might affect an area of ​​the industrial environment. As another example, any overlay of mission actions on a map, or any other graphical representation of mission actions, may not indicate the granularity at which the mission actions are defined, making it difficult and / or time-consuming for a human operator to identify a mission action that might be relevant.For example, it may not be possible to distinguish between a mission action defined only by a robot pose and a mission action defined by both a robot pose and camera parameter(s). Furthermore, it may not be possible to quickly identify mission actions that involve a specific type or types of sensors (e.g., identifying mission actions involving temperature sensors).

[0019] The implementations disclosed herein can overlay mission actions onto a map of an industrial environment, such as a bird's-eye view map / overview map of the industrial environment. Some of these implementations can further render the mission actions based on mission action metadata. For example, mission actions can be rendered based on the granularity with which each mission action is defined (as reflected by mission action metadata). Thus, mission actions defined only by a robot pose (and / or other robot parameters) can be rendered uniquely compared to mission actions defined by both robot poses (and / or other robot parameters) and camera parameters.As another example, mission actions involving pressure sensor measurements can be represented in a unique way as opposed to mission actions involving temperature sensor measurements, and both can be represented in a unique way as opposed to mission actions involving gas sensor measurements.

[0020] As a specific example, mission actions defined solely by a robot pose can be represented on an industrial environment map in one way (e.g., without an image or descriptor of a measuring device or other industrial POI if no measuring device or other industrial POI is associated with those mission actions), and mission actions defined by a robot pose and camera parameters can be represented on the industrial environment map in a second way (e.g., with an image or images and / or a descriptor or descriptors of a measuring device(s) and / or other POI(s) captured during the mission action). This can enable a human operator to more quickly identify a mission action to add to a given mission or to include in a new mission.If, for example, the mission action is to be added to collect an additional gas measurement or a panoramic image, then one or more camera parameters are not important, and the operator can select from all of the displayed mission actions. However, if the mission action is to be added to collect an additional gauge measurement or an additional image of a specific POI (or POI type), the operator can focus on mission actions defined by one or more camera parameters.

[0021] In some implementations, it is possible to select which mission actions are displayed at a given time, either additionally or alternatively, based on user input(s) (e.g., selecting a filter element or defining a range to restrict the display to mission actions within that range, etc.). For example, based on user input(s) specifying a characteristic(s) of the subset of mission actions, only a subset of mission actions can be displayed at a given time (e.g., over an industrial environment map).

[0022] As a specific example, an operator can select a filter element to display all mission actions. Alternatively, an operator can select a different filter element to display only mission actions defined by camera parameter(s) (e.g., as reflected by their corresponding metadata) and / or associated only with specific types of POIs (e.g., measuring instruments). For example, in response to suboptimal cooling system operation, an operator might select a filter element to display only mission actions that are (i) defined by a camera pose and (ii) associated with a temperature measuring instrument.This can allow the operator to quickly select one or more of the mission actions from these displayed mission actions to add to a given mission or to include in a new mission (e.g., to inspect the cooling system).

[0023] In some implementations, it is possible to select which mission actions are played at a given time based on the detection of a specific event or events. For example, based on the detection of a particular event and the determination that the event affects the characteristic(s) of a subset of mission actions, only a subset of mission actions might be played at that time (e.g., over an industrial environment map). As a non-restrictive example, in response to the detection of an oil leak (or other event) on the floor of an open space within an industrial environment, mission actions within that open space might be played, but no other mission actions in any other space or spaces within the industrial environment might be played.In this non-restrictive example, mission actions defined with different levels of granularity can be represented in different ways.

[0024] Continuing the above, non-restrictive example, some implementations do not reproduce all mission actions within the open space. For instance, a subset of mission actions within the open space, each defined by camera parameters, can be selectively reproduced, allowing a human operator to choose a specific mission action from the reproduced subset. A robot can then navigate to that specific mission action to capture one or more camera images of the oil leak for further analysis or before performing another action. It should be noted that in this case, mission actions defined with different levels of granularity can also be reproduced in different ways to facilitate selection by the human operator.

[0025] In some implementations, a subset of mission actions can be automatically replayed (e.g., without user input) at a given time and over an industrial environment map in response to the detection of a specific event. For example, the detection of a potential gas leak can automatically trigger the display of all mission actions for an industrial environment (since any of the mission actions can be used to take a gas measurement). As another example, if the detection of a potential gas leak includes location information that specifies an area (or a target location), a subset of all mission actions within the specified area (or within a predetermined distance from the target location) can be automatically replayed in response to the potential gas leak detection, along with the location information.In these implementations, the reproduced mission actions can optionally be automatically reproduced in different ways, e.g., based on metadata of the reproduced mission actions that define these mission actions with different levels of granularity.

[0026] Accordingly, the implementations disclosed herein represent (a) mission action(s) of an industrial plant on a map of the industrial plant and depending on metadata associated with the mission actions. This can enable a human operator to identify (a) mission action(s) to add to a given mission and / or to include in a new mission with reduced latency and / or a reduced number of user inputs.

[0027] Various implementations provide a method, implemented using one or more processors, that involves identifying metadata for each of a plurality of mission actions for an industrial environment. The industrial environment can be or include an industrial plant. The industrial plant can be a chemical processing plant, an industrial work environment, an oil or gas refinery, a catalyst factory, a manufacturing plant, an offshore oil platform, or any other suitable facility. The metadata for each of the plurality of mission actions can be identified, for example, from a memory or database (accessible, for example, to an operator or operators of the industrial environment) that stores the metadata defined for all available mission actions for the industrial environment.

[0028] A mission action for an industrial environment can, for example, be initially defined by a human operator controlling a robot (e.g., using a hand controller and / or a computer interface) who provides input(s) to specify the mission action. The input(s) can, for example, provide a robot pose when capturing an image during the mission action and / or other robot parameters (e.g., robot type, robot model, robot manufacturer, etc.). The input(s) can also, for example, provide camera parameters (e.g., a camera pose, a zoom level, etc.) of a camera used by the robot to capture the image during the mission action. Additionally or alternatively, the input(s) can, for example, provide POI parameters of a POI captured in the image, where the POI is an industrial POI (e.g., a trade fair or industrial park).a measuring device) that is fixed or installed near the mission operation.

[0029] In some implementations, the robot can be a real robot, and the industrial plant can be a real industrial plant. In other implementations, the robot can be a simulated robot controlled within a simulation of the real industrial plant. In some implementations, either the real robot or the simulated robot can be controlled to navigate to the mission action and / or perform one or more actions within the mission action.

[0030] In some implementations, metadata for a sequence of mission actions for a robot to perform a mission (e.g., detecting a gas leak for an environment surrounding each of the mission actions) can be stored associated with the mission and / or associated with the robot. For example, metadata for a first sequence of mission actions for a first robot to perform a first mission can be stored in a database associated with the first mission, and metadata for a second sequence of mission actions for a second robot to perform a second mission can be stored in the database associated with the second mission.

[0031] In some implementations, metadata of the sequence of mission actions can be generated or derived from the input(s) mentioned above by the human operator. In some other implementations, metadata of a specific mission action can be generated or derived from the input(s) mentioned above by the human operator. In some implementations, metadata of the specific mission action can additionally or alternatively be identified or collected, for example, at least partially based on communications with the robot capturing the image of the mission action, such as by retrieving a robot pose (and / or a camera pose of a camera used by the robot to capture images) from the robot when it captures an image of the mission action.

[0032] In some implementations, mission action metadata may include a robot pose associated with the mission action. Additionally, or alternatively, the mission action metadata may include one or more camera parameters for a robot camera used to capture images during the mission action. For example, if a human operator guides a robot to the mission action and controls a robot camera to capture an image of a measuring device observable by the robot during the mission action, the mission action metadata may include camera parameters. These camera parameters (e.g., for navigating to the mission action) may include a camera pose, zoom level, and / or focus.

[0033] In some implementations, the robot may include or carry one or more other sensors (e.g., an infrared sensor, etc.) instead of or in addition to a camera (or more than one camera). In this case, the mission action metadata may alternatively or additionally include one or more sensor parameters of the one or more other sensors carried by the robot. In some implementations, the mission action metadata may also alternatively include one or more POI parameters for a POI (or more than one POI) that the robot acquires during the mission action (e.g., one that was previously acquired or is to be acquired). The one or more POI parameters may, for example, include the type (e.g., sensor) of the POI (e.g., an industrial pressure gauge associated with a pipe or tank during the mission action) and / or an identifier (e.g., G-10012) of the POI.

[0034] In various implementations, the procedure may further include the representation, on a map surface of the industrial environment, of a first subset of mission actions with initial graphical features. The map surface may, for example, be a two-dimensional view (e.g., bird's-eye view / overview) of the industrial environment. As a non-restrictive example, the initial graphical features may include an initial graphical symbol representing an initial mission action (from the first subset) on the map surface. The initial graphical symbol may, for example, have a specific shape, color, pattern, and / or other features indicating that the initial mission action belongs to the first subset of mission actions.In some implementations, the first subset of mission actions is rendered with the first graphical features in response to a determination that the corresponding metadata for each of the mission actions in the first subset matches one or more first criteria.

[0035] In various implementations, the procedure may additionally or alternatively include a representation, on the map surface of the industrial environment, of a second subset of mission actions with second graphical features, along with the first subset of mission actions. As a non-restrictive example, the second graphical features may include a second graphical symbol representing a second mission action from the second subset. This second graphical symbol may, for example, have a specific shape, color, pattern, and / or other characteristics indicating that the second mission action belongs to the first subset of mission actions.In some implementations, the second subset of mission actions is rendered with the second set of graphical features in response to a determination that the corresponding metadata for each of the mission actions in the second subset matches one or more of the second set of criteria. The first set of criteria differs from the second set of criteria. The first set of graphical features differs visually from the second set of graphical features.

[0036] In some implementations, determining that the relevant metadata for each of the mission actions in the first subset matches one or more of the first criteria can be implemented by determining that the relevant metadata for each of the mission actions in the first subset includes: a robot pose for each mission action from the first subset (so that a robot can navigate to the robot pose to approach / map the respective mission action) and one or more vision sensor parameters to capture vision data of the respective mission action (while the robot is) in the relevant robot pose. In some implementations, the mission actions of the first subset can be associated with a hazardous object (or other object or industrial POI) to be mapped.

[0037] In some implementations, the one or more vision sensor parameters include a vision sensor pose, a vision sensor zoom level, and / or a vision sensor focus parameter, which are associated with a vision sensor carried by the robot (e.g., a mobile robot) within the industrial environment. In some implementations, the mission action metadata of the first subset additionally or alternatively includes one or more POI parameters for a POI, which are acquired (e.g., by a robot) during one of the mission actions of the first subset. The one or more POI parameters may, for example, include a POI identifier (and / or type) of the POI.

[0038] In some implementations, determining that the relevant metadata for each of the mission actions in the second subset matches one or more of the second criteria is implemented by determining that the relevant metadata for each of the mission actions in the second subset includes a robot pose for that mission action from the second subset, but does not include camera parameters for capturing an image of the mission action. In these implementations, determining that the relevant metadata for each of the mission actions in the second subset matches one or more of the second criteria may include, or be implemented by, determining that the relevant metadata for each of the mission actions in the second subset includes only one relevant robot pose for each of the mission actions in the second subset.In some implementations, the mission actions of the second subgroup may each be a location where no image is captured (e.g., because a human operator has not defined these mission actions with (a) camera pose(s)).

[0039] In some implementations, determining that the corresponding metadata for each of the mission actions in the first subgroup matches one or more of the first criteria involves determining that the corresponding metadata for each of the mission actions in the first subgroup indicates that each of the mission actions in the first subgroup includes a sensor of a first type, or is implemented by doing so. Similarly, in some implementations, determining that the corresponding metadata for each of the mission actions in the second subgroup matches one or more of the second criteria involves determining that the corresponding metadata for each of the mission actions in the second subgroup indicates that each of the mission actions in the second subgroup includes a sensor of a second type, or is implemented by doing so. The second type may differ from the first type.

[0040] In various implementations, the procedure can further include receiving, via the map interface for the industrial environment, a selection of a given mission action from the first subset or the second subset. The selection can be made by a user (e.g., a human operator) via touch input, audio input, or any other suitable method. In some implementations, the first subset and / or the second subset of mission actions can each be represented as a selectable element of a graphical user interface (GUI) for selection by the user (e.g., a mouse click on the given mission action).For example, the first subset of mission actions can each be represented as a first selectable GUI element, and the second subset of mission actions can each be represented as a second selectable GUI element, with the second selectable GUI element being visually distinguishable from the first selectable GUI element.

[0041] In various implementations, the procedure, in response to receiving the selection, may further include adding the given mission action to a predefined robot mission or incorporating the given mission action into a new robot mission. As a non-restrictive example, the predefined robot mission may include a robot navigating to a first mission action to capture a first image during the first mission action, and then navigating to a second mission action to capture a second image during the second mission action.In this non-restrictive example, the given mission action can be located between the first mission action and the second mission action, and adding the given mission action to the predefined robot mission can cause the robot to navigate to the given mission action after navigating to the first mission action and before navigating to the second mission action, in order to capture an image at the given mission action.

[0042] Alternatively, the given mission action can be prioritized over the first and second mission actions, for example, if a hazardous object is detected near (e.g., within a predefined area) the given mission action, while routine inspections are required for the first and second mission actions. In this case, adding the given mission action to the predefined robot mission can cause the robot to navigate to the given mission action to capture an image of it before navigating to the first and second mission actions. Alternatively, the first and second mission actions can be prioritized over the given mission action, for example, if the first and second mission actions require immediate inspection, while the given mission action requires a routine inspection.In this case, adding the given mission action to the predefined robot mission can cause the robot to navigate to the given mission action after navigating to the first and second mission actions, in order to capture an image of the given mission action.

[0043] In other words, the given mission action can be added to the predefined robot mission based on a location of the given mission action in relation to locations of other predefined mission actions (e.g., the first mission action and the second mission action in the preceding non-restrictive example), and / or based on a priority level of (a) task(s) in the given mission action (e.g., in relation to priority levels of tasks in other predefined mission actions of the predefined robot mission).

[0044] In various implementations, a procedure is provided, implemented using one or more processors, that involves identifying appropriate metadata for each of a plurality of mission actions for an industrial environment. In some implementations, the industrial environment may be or include an industrial plant. The industrial plant may be a chemical processing plant, an industrial work environment, an oil or gas refinery, a catalyst factory, a manufacturing plant, an offshore oil platform, or any other suitable facility. The metadata for each of the plurality of mission actions, once defined, may be stored, for example, in a memory or database (e.g., a database).stored (which can be accessed by operators of the industrial environment) so that the metadata or part of it can be identified or retrieved for subsequent use (e.g., to select one or more mission actions to be played back, to play back different mission actions in different ways, etc.).

[0045] In some implementations, identifying the relevant metadata for each of the majority of mission actions can be performed in response to the launching and rendering of an industrial environment map surface via a computer device display. The computer device could be, for example, a local device within the industrial environment or a server device communicating with the industrial environment. The majority of mission actions may (but need not) be rendered in response to the rendering of the map surface. For example, some of the majority of mission actions can be rendered on the map surface using the techniques / procedures described herein.

[0046] In various implementations, the process further includes determining one or more filter criteria based on user interface input and / or based on a detected event in the industrial environment. The one or more filter criteria could, for example, be rules that filter the majority of mission actions for display on an industrial environment map.

[0047] As an example, the user interface input can specify that the metadata of a mission action (to be reproduced) must include a camera pose. In this example, a first filter rule / criterion can be defined to filter out mission actions that are not defined with a camera pose, so that they are not reproduced on the industrial environment map. As another example, the detected event can be an abnormal temperature reading from a temperature sensor for a cooling system within the industrial environment. In this example, a second filter rule / criterion can be defined to filter out mission action(s) that are not defined with a camera pose (which is required to capture images of the cooling system) and that are not associated with a temperature sensor.For example, a mission action stored with metadata that does not include a camera pose for imaging the temperature sensor (occasionally, specifically for imaging the temperature sensor for the cooling system) and that does not include any sensor information for a temperature sensor can be filtered out using the second filter criteria, so that such a mission action is not displayed on the industrial environment map. This not only reduces the total number of mission actions displayed for user selection, thus simplifying the selection process, but also reduces the computing resources and other resources (battery, network, etc.) required for the industrial environment.

[0048] In various implementations, the procedure further involves selecting a subset of mission actions from the majority of mission actions. In some implementations, the selection of the subset of mission actions occurs in response to a determination that the corresponding metadata for each of the mission actions in the subset matches one or more filter criteria. As a non-restrictive example, the one or more filter criteria may include an initial filter rule requiring that a reproduced mission action has metadata specifying a particular sensor type (e.g., a pressure sensor) located relative to the reproduced mission action.In this non-restrictive example, the subset of selected mission actions can include, and only includes, a first mission action that has metadata indicating that a first pressure sensor is observable during the first mission action, and a second mission action that has metadata indicating that a second pressure sensor is observable during the second mission action. In other words, this non-restrictive example may not select any mission action from the plurality of mission actions (e.g., to replay) whose metadata indicates that no pressure sensor is observable during a corresponding mission action.

[0049] As another non-restrictive example, the relevant metadata for each of the plurality of mission actions includes a corresponding robot pose of that robot during that mission action, and determining the filter criteria involves determining one or more pose-based filter criteria. The one or more pose-based filter criteria may include an initial pose-based filter rule that requires selecting the subset of mission actions that are each located at a predetermined distance from a target location. The target location may be, for example, a user-selected location or a location where an event of interest (e.g., an oil leak) is detected.

[0050] In various implementations, the procedure further includes displaying, on a map surface of the industrial plant, the subset of mission actions, excluding other mission actions that do not belong to the subset. In some implementations, the display of the subset of mission actions may include displaying a graphical representation for each mission action in the subset. In some implementations, the graphical representation for each mission action in the subset may include a selectable element which, when selected, causes metadata (or a portion thereof, e.g., a pressure sensor ID) associated with a corresponding mission action to be displayed (e.g., as an overlay on the map surface).In some implementations, the graphical representation for each mission action in the subset may alternatively or additionally include an extra selectable element which, when selected, causes the corresponding mission action to be added to a given mission or a new mission for a robot. In some implementations, a user may be able to drag an icon (e.g., the graphical representation) representing the corresponding mission action to add it to a robot mission (e.g., the given mission or the new mission).

[0051] In various implementations, the procedure further includes receiving a selection of a given subset mission action via the map surface for the industrial plant. Continuing the preceding non-restrictive example, in which the subset of mission actions includes (and includes only) the first mission action (with metadata indicating that the first pressure sensor is accessible / observable during the first mission action) and the second mission action (with metadata indicating that the second pressure sensor is accessible / observable during the second mission action), the metadata of the first and second mission actions can include a first pressure measurement received from the first pressure sensor and a second pressure measurement received from the second pressure sensor. In this non-restrictive example, the first pressure measurement can be abnormal (e.g.,The first pressure measurement is higher than normal, and the second pressure measurement is within normal range. In this case, the first mission action can be selected via the map interface as the given mission action requiring observation (e.g., image capture), while the second mission action is not selected. The first mission action can be selected by a user viewing the map interface via touch input, audio input, mouse click, or any other suitable type of input.

[0052] In various implementations, the procedure, in response to receiving the selection, further includes adding the given mission action to a predefined robot mission or incorporating the given mission action into a new robot mission. The new robot mission might, for example, involve a robot of a specific type navigating to the given mission action. In this case, the procedure might further involve causing the robot of the specific type to perform the new robot mission, which includes navigating to the given mission action.

[0053] In various implementations, a procedure is provided that is implemented by one or more processors, and the procedure includes the following: identifying appropriate metadata for each of a plurality of mission actions in an industrial environment, wherein the metadata for each of the plurality of mission actions includes at least one robot pose for a robot during an appropriate mission action; detecting an object or event to be inspected within the industrial environment;In response to the detection of the object or event, the robot will render, on a map surface for the industrial environment, a first mission action and a second mission action, wherein the rendering of the first and second mission actions occurs in response to a determination that the first and second mission actions are within a predetermined distance of a location of the object or event, and wherein the first and second mission actions are rendered with different graphical features based on the fact that the metadata of the first and second mission actions have different levels of granularity; receive a selection, via the map surface for the industrial facility, of the first mission action or the second mission action; and, in response to receiving the selection, add the first or second mission action to a robot mission.

[0054] Additionally, some implementations include one or more processors of one or more computer devices, wherein the one or more processors are operable to execute instructions stored in associated main memory, and wherein the instructions are designed to cause the execution of one of the aforementioned procedures. Some implementations also include one or more non-volatile, computer-readable storage media that store computer instructions executable by one or more processors to perform one of the aforementioned procedures.

[0055] It should be noted that all combinations of the aforementioned concepts and additional concepts described in more detail herein are to be considered as part of the subject matter disclosed herein. For example, all combinations of claimed subject matter listed at the end of this disclosure are to be considered as part of the subject matter disclosed herein. Fig. Figure 1A schematically shows an exemplary environment in which selected aspects of the present revelation can be implemented according to different implementations. Fig. Figure 1B shows an example of storing metadata associated with one or more mission actions, according to different implementations. Fig. Figure 2A shows an example user interface displaying a map with all available mission actions, according to different implementations. Fig. Figure 2B shows another exemplary user interface, displaying a map with all available mission actions, according to different implementations. Fig. 2C shows another exemplary user interface, which displays a map represented by one or more selected mission actions for carrying out selected aspects of the present revelation, according to different implementations. Fig. 2D shows another exemplary user interface, displaying a map rendered with one or more selected mission actions for carrying out selected aspects of the present revelation, according to various implementations. Fig. 2E shows an additional exemplary user interface that displays a map represented by one or more selected mission actions for carrying out selected aspects of the present revelation, according to different implementations. Fig. 2F shows another exemplary user interface, displaying a map represented by one or more selected mission actions for carrying out selected aspects of the present revelation, according to different implementations. Fig. Figure 3 illustrates an exemplary procedure for carrying out selected aspects of the present disclosure, according to different implementations. Fig. Figure 4 illustrates a further exemplary procedure for carrying out selected aspects of the present disclosure according to different implementations. Fig. Figure 5 schematically illustrates an exemplary computer architecture on which selected aspects of the present revelation can be implemented.

[0056] The implementations described herein involve rendering one or more mission actions on a map of an industrial environment. Some of the implementations can further render the mission actions depending on the granularity with which the mission action is defined (as reflected by mission action metadata). For example, mission actions defined only by a robot pose can be rendered uniquely compared to mission actions defined by both robot poses and camera parameters. As another example, mission actions involving pressure sensor measurements can be rendered uniquely compared to mission actions involving temperature sensor measurements, and both can be rendered uniquely compared to mission actions involving gas sensor measurements.

[0057] As another example, mission actions can be selectively played back at a given time based on user input (e.g., selecting a filter element) and / or based on the detection of a specific event or events. For instance, only a subset of mission actions can be played back at a given time based on user input specifying a characteristic of that subset, and / or based on the detection of a specific event (e.g., a triggered alarm) and determining that the detected event relates to the characteristic(s) of the subset of mission actions.As a specific example, mission actions defined solely by a robot pose can be represented on an industrial environment map in one way, while mission actions defined by a robot pose and camera parameters can be represented in a second way (e.g., with an image(s) and / or a descriptor (descriptors of a measuring device(s) and / or another point of interest (POI) captured during the mission action). This can enable a human operator to more quickly identify a mission action to add to a given mission or to incorporate into a new one. For example, if the mission action is to be added to capture an additional gas measurement or a panoramic image, then the camera parameter(s) are not critical, and the operator can select from all of the represented mission actions.If, on the other hand, the mission action is to be added to collect an additional meter reading or an additional image of a specific POI (or a specific POI type), the operator can focus their attention on mission actions with camera parameters.

[0058] As another specific example, an operator can select a filter element to display all mission actions, or a potential gas leak detection can automatically trigger the display of all mission actions (since any mission action can be used to take a gas measurement). Alternatively, an operator can select a different filter element to display only mission actions with camera parameters and / or associated with specific types of POIs (e.g., measuring devices), or a specific event detection can automatically trigger the display of such mission actions.For example, in response to suboptimal cooling system operation, an operator can select filter elements to cause the playback of only mission actions that (i) define a camera pose; and (ii) are associated with (a) temperature sensor(s). This can allow the operator to quickly select one or more of these mission actions to add to a given mission or to include in a new mission.

[0059] Accordingly, the implementations disclosed herein represent (a) mission action(s) of an industrial plant on a map of the industrial plant and in relation to metadata associated with the mission actions. This can enable a human operator to identify (a) mission action(s) for addition to a given mission and / or inclusion in a new mission with reduced latency and / or a reduced number of user inputs.

[0060] Fig. Figure 1A schematically shows an exemplary environment in which selected aspects of the present revelation can be implemented according to various implementations. With reference now to Fig. Figure 1A is an exemplary environment 100, schematically represented, in which various aspects of the present disclosure can be implemented. The exemplary environment 100 can be or comprise an industrial plant 130, which can take numerous forms. For example, the exemplary environment 100 can be designed to implement any number of at least partially automated processes. The industrial plant 130 can be configured as a chemical processing plant, an industrial work environment, an oil or gas refinery, a catalyst factory, a manufacturing plant, an offshore oil platform, or any other suitable facility.

[0061] The exemplary environment 100 can include one or more client devices (e.g., local client devices 103-A and 103-B) that are operationally coupled to a process automation network 106 in the industrial plant. The client device 103-A or 103-B can be implemented as a computer (e.g., laptop, desktop, notebook), a tablet, a robot, a smart device (e.g., smartphone), a messaging device, a wearable device (e.g., watch), or any other suitable device. The process automation network 106 can be implemented using various wired and / or wireless communication technologies, including, but not limited to, the Institute of Electrical and Electronics Engineers (IEEE) 802.3 (Ethernet) standard, IEEE 802.11 (Wi-Fi), cellular networks, such as…3GPP Long Term Evolution (“LTE”) or other wireless protocols referred to as 3G, 4G, 5G and higher, and / or other types of communication networks with different types of topologies (e.g., mesh).

[0062] The exemplary environment 100 may further include one or more mobile robots. For example, the exemplary environment 100 may include a fleet of robots comprising a first robot 111 and / or a second robot 112. The first robot 111 may be the same as the second robot 112 or different from it. The first robot 111 may, for example, be a quadrupedal robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle (e.g., a drone), a tracked robot, or any other suitable robot that can move within or around the industrial facility 130. Different robots may be of different types and may be manufactured by different companies. In some implementations, the fleet of robots may include a subset of robots of the same type and / or manufactured by the same company.

[0063] As a non-restrictive example, the first robot 111 can be used, with reference to Fig. Robot 1A, a robot dog, navigates to one or more mission actions for industrial plant 130 to inspect one or more points of interest (POIs) at or near those mission actions. Robot 112, a second robot, can be a drone that navigates to one or more additional mission actions (e.g., inside a chimney or container, etc.) to inspect a structure of industrial plant 130. The mission actions to which Robot 111 navigates, as well as the additional mission actions to which Robot 112 navigates, can each be defined (e.g., by a human operator) with metadata that includes, for example, a robot pose of the respective robot at a mission action and / or other information (e.g., camera parameters, POI parameters, etc.) associated with the mission action.Optionally, the one or more mission actions and the one or more additional mission actions can constitute all mission actions predefined for the industrial plant 130, for example, if the first robot 111 and the second robot 112 are the only robots used for the industrial plant 130 and no other mission actions are defined by human operators for the industrial plant 130.

[0064] The first robot 111 or the second robot 112 could be other types of robots, such as a robot for transporting materials and products, a robot dog for patrolling and monitoring the industrial plant for anomalies, a spider robot for inspecting external pipes, a snake robot for inspecting internal pipes, etc. Descriptions of specific types and functions of the first robot 111, the second robot 112, or any other suitable robot are not limited herein. In some implementations, for example, the first robot 111 (or the second robot 112) could be a real robot, and the industrial plant 130 could be a real industrial facility.In some other implementations, the first robot 111 (or the second robot 112) can be a simulated robot controlled in a simulation of the industrial plant 130, where the simulated robot can be used by a human operator to define the one or more mission actions (and / or the one or more additional mission actions).

[0065] In some implementations, the first robot 111 and the second robot 112 may each include one or more sensors for performing one or more missions. For example, the first robot 111 (or another robot, such as the second robot 112) may include (or otherwise be equipped with) a light detection and distance (Lidar) sensor for mapping objects by generating a 3D model of the mapped objects. Additionally or alternatively, the first robot 111 (or another robot) may include other sensors, such as a vision sensor, ultrasonic immersion test transducers for detecting surface irregularities and defects (e.g., corrosion), one or more gas sensors for detecting the presence and concentration of hazardous gases or vapors, and / or a temperature sensor for measuring temperatures, etc.The vision sensor can be a monographic camera, a stereographic camera, a thermal camera, or any other suitable vision sensor for capturing one or more images of one or more specific points of interest (POIs) of the industrial plant 130. The vision sensor can be detachably coupled to or integrated into the first robot 111. In some implementations, the vision sensor can change its position and / or orientation relative to the first robot 111, for example, by rotating or otherwise moving. As a non-restrictive example, the first robot 111 can include front and rear high-resolution cameras that are detachably coupled to it.

[0066] In various implementations, the exemplary environment 100 may include a map 140 for an industrial environment (occasionally referred to as the "industrial environment map"), where the industrial environment map 140 may be displayed on a map surface via a client device 103-A (or client device 103-B, or other devices, such as the server device 105). The industrial environment map 140 may, for example, be a two-dimensional view (e.g., a bird's-eye view map / overview map) of the industrial plant 130.

[0067] Client device 103-A or 103-B can each include input devices and / or output devices for user interaction with the industrial environment map 140. For example, user input via an input device of client device 103-B can cause all available mission actions (or some of them) to be displayed on the industrial environment map 140, or cause all available mission actions (or some of them) to no longer appear on or above the industrial environment map 140.

[0068] A mission action for industrial plant 130 can, for example, be initially defined by a human operator controlling a robot (e.g., the first robot 111), for instance, using a hand controller (and / or a computer interface). The mission action can be a robot mission action to which a robot navigates in order to observe a point of interest (POI) or perform another action (e.g., a measurement). The mission action can, for example, correspond to a location where a POI (e.g., equipment, etc.) of industrial plant 130 is located, a location where routine maintenance or inspection is expected, a location where information needs to be collected (e.g., a location where measurements from a pressure gauge need to be collected / determined), or a location where an anomaly is identified. Once the mission action is initially defined, a robot can execute the defined mission action (e.g.,(the robot pose, which is defined in metadata associated with the mission action) to navigate autonomously to the mission action (e.g., without requiring control by a human operator).

[0069] In some implementations, the human operator can initially define the mission action by providing input(s) to specify the mission action. For example, the human operator can provide a user interface input that specifies a mission action ID number, one or more robots to be used in the mission action (e.g., a specific robot to inspect a surrounding environment during the mission action), robot pose(s) of the one or more robots during the mission action (e.g., to inspect an object observable from the mission action), one or more camera poses (associated with one of the robot pose(s)) for a camera of the specific robot to capture images of the object during the mission action, and / or any other suitable information.In some implementations, user interface input can be stored as metadata associated with the mission action. In some implementations, the user interface input can be processed to generate metadata associated with the mission action. For example, the mission action metadata can be stored as an entry for the mission action in a mission action database (see...). Fig. 1B as an example).

[0070] In some implementations, mission action metadata may include a robot pose associated with the mission action. Additionally, or alternatively, in some implementations, the mission action metadata may include one or more camera parameters for the robot's camera. These camera parameters may include, for example, a camera pose (in which the robot's camera captures desired images of the mission action), a zoom level (the camera used to capture the desired images during the mission action), and / or a focus (the camera used to capture the desired images). In some implementations, the mission action metadata may also include one or more POI parameters for a POI (or more than one POI) captured during the mission action.One or more POI parameters can include, for example, a type (e.g., sensor) of the POI and / or an identifier of the POI.

[0071] In some implementations, a mission action can be defined or updated to have varying levels of granularity. Different mission actions can be defined or associated with different sets of metadata, with each set of metadata having a different level of granularity. For example, a mission action defined to capture an image of a measuring instrument might be defined with both of the following: (a) a robot pose (e.g., X, Y, Z coordinates and optionally rotation and / or translation); and (b) camera parameter(s), such as a camera pose (e.g., rotation and / or translation), a camera zoom level, and / or a camera focus setting. An additional mission action, defined / used to capture a gas measurement, might be defined with only (a) a robot pose and not define any camera parameter(s).The mission action (or the additional mission action) can optionally or additionally be defined with POI parameter(s) that define a feature of POI(s) to be captured by sensor measurement(s) during the mission action. For example, the mission action defined to capture measurements from a specific measuring instrument can include metadata defining the type of measuring instrument (e.g., temperature type, pressure type, etc.) and / or a unique identifier for the measuring instrument (e.g., X1234AB).

[0072] As another example, a first mission action defined to capture an image of a measuring device might be defined with both of the following: (a) a robot pose (e.g., X, Y, Z coordinates and optionally rotation and / or translation); and (b) camera parameter(s), such as a camera pose (e.g., rotation and / or translation), a camera zoom level, and / or a camera focus setting. In this example, a second mission action defined to capture a gas measurement might be defined with only (a) a robot pose and not define any camera parameters. In other words, one or more mission actions might be defined with more granularity (e.g., with a robot pose and camera parameter(s)) than another mission action (e.g., with only a robot pose). In some implementations, metadata for the first and / or second mission action might alternatively or additionally include, for example, a POI parameter.which define(s) feature(s) of POI(s) that are captured by sensor measurement(s) during the first or second mission action.

[0073] In various implementations, the exemplary environment 100 may further include a server computer device 105 (which may simply be referred to as the "server device"). The server computer device 105 may include a mission action (MA) selection engine 1051, a replay engine 1053, and / or a memory 15. In some implementations, the server computer device 105 may optionally include a mission action definition engine 1054. In some implementations, the industrial environment map 140 may be stored in and / or accessible through the memory 15. In some implementations, the exemplary environment 100 may optionally include more than one industrial environment map (e.g., industrial environment maps determined in different years, etc.).

[0074] In various implementations, the Mission Action Definition Engine 1054 can define one or more mission actions for the industrial plant. In some implementations, the Mission Action Definition Engine 1054 can determine one or more mission actions for the industrial plant based on user input and / or determine metadata associated with the one or more mission actions. As a non-restrictive example, the Mission Action Definition Engine 1054 can determine one or more mission actions for the industrial plant 130 based on user input(s) that provide one or more defined robot missions. The user input(s) can, for example, provide an initial robot mission that describes an initial set of mission actions to which a robot (e.g., a robot, a robot, or a robot) can perform.The first robot 111 (or a simulated robot of the first robot 111) is to navigate to, and can provide a second robot mission that describes a second set of mission actions to which a robot (e.g., the first robot 111 or the second robot 112, or a simulated robot of the first or the second robot) is to navigate. The robot can then navigate to perform the first robot mission and / or the second robot mission.

[0075] The robot (e.g., the first robot 111) can, for example, be controlled to perform an initial mission action, which belongs to the first set of mission actions, to observe the surrounding environment of the first mission action. During this first mission action, the robot can detect a measuring device, and a camera on the robot can be controlled to capture a specific image of the measuring device with appropriate camera parameters (e.g., a specific camera pose, zoom level, etc.). In this case, the mission action definition engine 1054 can receive values ​​of these camera parameters from the robot and incorporate these values ​​into the metadata associated with the first mission action.The Mission Action Definition Engine 1054 can alternatively or additionally receive a type and / or ID of the measuring instrument (or other POI) based on user input provided by a human operator observing the specific image.

[0076] Continuing the above non-restrictive example, user input from a human operator can specify a particular robot (e.g., model, manufacturer, type, function, etc.) to navigate to the mission action, a robot pose of that particular robot to perform one or more actions (e.g., image capture, repair, etc.) during the mission action, and / or a sensor pose of a sensor carried by that particular robot when the robot is in the robot pose, etc. In this case, the particular robot (e.g., model, manufacturer, type, function, etc.), the robot pose of that particular robot, and / or the sensor pose of the sensor can be determined and / or stored as mission action metadata.

[0077] In some implementations, after the mission action definition engine 1054 defines and / or stores a plurality of mission actions for the industrial facility 130, the replay engine 1053 can replay the plurality of mission actions (e.g., as an overlay) on or over the industrial environment map 140. The replay engine 1053 can replay different mission actions from the plurality of mission actions in different ways. In some implementations, the replay engine 1053 can replay the mission actions depending on the granularity with which the mission action is defined (as reflected by mission action metadata). For example, mission actions defined with only a robot pose can be replayed in a unique way compared to mission actions defined with both robot poses and camera parameters.As another example, mission actions involving pressure sensor measurements can be uniquely differentiated from mission actions involving temperature sensor measurements, and both can be uniquely differentiated from mission actions involving gas sensor measurements. As yet another example, based on user input (e.g., selecting a filter element) and / or based on the detection of a specific event or events, the mission actions to be executed at a given time can be selected.For example, only a subset of mission actions can be played back at the given time, based on user input(s) that specify a feature(s) of the subset of mission actions, and / or based on detecting a specific event and determining that the specific event relates to the feature(s) of the subset of mission actions.

[0078] As a specific example, the playback engine 1053 can, in a first way, render mission actions defined only by a robot pose onto an industrial environment map 140, and in a second way render mission actions defined by a robot pose and camera parameter(s) onto the industrial environment map 140 (e.g., with image(s) and / or descriptor(s) of measuring device(s) and / or other POI(s) captured during the mission action). This can enable a human operator to more quickly identify a mission action to add to a given mission or to include in a new mission. For example, if the mission action is to be added to collect an additional gas measurement or an additional panoramic image, then one or more camera parameters are not important, and the operator can select from all of the rendered mission actions.If, on the other hand, the mission action is to be added to collect an additional meter reading or an additional image of a specific POI (or a specific POI type), the operator can focus their attention on mission actions with (a) camera parameter(s).

[0079] As another specific example, an operator can select a filter element to cause all mission actions to be displayed, or the detection of a potential gas leak can automatically cause all mission actions to be displayed (since any of the mission actions can be used to take a gas measurement). On the other hand, an operator can select an alternative filter element, and the mission action selection engine 1051 can, based on the selected alternative filter element, cause only mission actions with camera parameter(s) and / or associated with only certain types of POI(s) (e.g., measuring instruments) to be displayed. In some implementations, the detection of a specific event can automatically cause the mission action selection engine 1051 to selectively display one or more mission actions on the industrial environment map 140. With reference to Fig. 1A For example, in response to suboptimal operation (e.g., unusual pressure) of an industrial system 161 (symbolized by a closed rectangle, as shown, which could be, for example, a cooling system or a piping system whose pressure is measured at one or more locations, etc.), an operator can select (a) filter element(s) to reproduce only such mission actions (e.g., a mission action 163, a mission action 165, and a mission action 167, as in Fig. (shown in Figure 1A) to cause (i) a robot pose and a camera pose to be defined; and (ii) associated with a specific type of measuring device (e.g., pressure gauge in case of unusual pressure, thermometer in case of unusual temperature, etc.). This can allow the operator to quickly select one or more of the mission actions from all available or defined mission actions (e.g., mission action 163, mission action 165, and mission action 167) to add to a given mission or to include in a new mission. For example, the operator can add all of mission actions 163, 165, and 167 to create a new mission for a robot (e.g.,to create the first robot 111), which carries a vision sensor, so that the robot can navigate to mission action 163, 165 and 167 respectively, to capture images of pressure gauges in robot poses specified by metadata associated with each mission action 163, 165 and 167, using the vision sensors in camera poses specified by the metadata associated with mission action 163, 165 and 167.

[0080] In the preceding example, the operator may wish to inspect a specific part of the system 161 (e.g., by providing a user-defined area 162, represented by the dashed rectangle), and the playback engine 1053 can only play back mission action 163 and mission action 165, which are defined by a robot pose and a camera pose, respectively, associated with the measuring instrument(s), and located within the user-defined area 162. In this example, a user might select mission action 163 from the played-back mission action 163 and the additional mission action 165. Such a selection of mission action 163 might, for example, cause the first robot 111 to autonomously navigate to a robot pose associated with mission action 163 to perform a mission (e.g., further inspection, etc.).

[0081] In some implementations, a graphical representation of mission action 163, mission action 165, and / or mission action 167 may have a specific shape, color, pattern, or other characteristics. For example, mission action 163, mission action 165, and / or mission action 167 may each have a triangular shape, based on the fact that they are all defined by a robot pose, a camera pose, and parameters of a specific measuring device (such as a pressure gauge). The graphical representation of mission action 163 (or 165 or 167) may be selectable, and when selected or dragged, it may cause mission action 163 (or 165 or 167) to be added to a mission.

[0082] In some implementations, as in Fig. As shown in Figure 1A, a graphical representation of mission action 165 may optionally include, for example, an additional selectable element 1651 which, when selected, causes an image (or other graphical representation) and / or other information (measuring instrument ID, such as ID_555, and / or a last recorded measurement) of the specified measuring instrument 1653 to be displayed / expanded. In some implementations, the graphical representation of mission action 165 may alternatively or additionally include another selectable element associated with the graphical representation of mission action 165, wherein the additional selectable element (not shown), when selected, may provide additional information such as the robot's pose, an image or representation of the robot, a model and / or manufacturer of the robot, a camera pose and type of camera on the robot, an image of the camera, etc.The descriptions of the specific configuration (e.g., shape, color, pattern, POIs, etc.) for each of the mission actions (e.g., mission action 163, 165, 167, etc.) should not be restrictive.

[0083] Accordingly, the implementations disclosed herein represent (a) mission action(s) of an industrial plant on a map of the industrial plant. The mission action can be represented, for example, in ways that depend on metadata associated with the mission actions. This can enable a human operator to identify (a) mission action(s) for addition to a given mission and / or inclusion in a new mission with reduced latency and / or a reduced number of user inputs.

[0084] In some implementations, the exemplary environment 100 may also include one or more additional points of interest (POIs). For example, the exemplary environment 100 may include a robot control / simulation engine 1052 configured to control or simulate one or more mobile robots (e.g., the first robot 111). The robot control / simulation engine 1052 may, for instance, cause a simulated robot to be represented in a simulation of the industrial environment map 140, and a human operator may define a robot mission for the simulated robot, which includes a plurality of defined mission actions for the simulated robot (or a corresponding real-world robot) to perform one or more actions.

[0085] Fig. Figure 1B shows a non-restrictive example of storing metadata associated with one or more mission actions according to different implementations. As in Fig. As shown in Figure 1B, a mission action database 151 can be created and stored (e.g., within memory 15), wherein the mission action database 151 can contain one or more entries, each for a respective mission action of the one or more mission actions defined for an industrial plant (e.g., industrial plant 130). The one or more entries can, for example, include a first entry 1511 for a first mission action, a second entry 1512 for a second mission action, and a third entry 1513 for a third mission action. The first entry 1511 can contain metadata associated with the first mission action, such as one or more robot parameters of a robot for navigation to the first mission action, wherein the one or more robot parameters include a robot type (e.g., robot dog), a robot ID (e.g.,Robot_dog_1111), a robot pose (X1, Y1, Z1, α1, β1, γ1), a robot mission encompassing the first mission action (e.g., Mission_1), and / or other robot information. The first entry, 1511, may further include one or more camera parameters of a camera carried by the robot, where the one or more camera parameters are, for example, a camera pose (X1, Y2, Z1, α1, β). 1,The first entry, 1511, may include γ1), a zoom level (2x), a focus setting (single autofocus mode), and / or other camera information. Furthermore, it may include one or more POI parameters for a POI that is acquired or locked during the first mission action. If the POI is a sensor during the first mission action, the one or more POI parameters may include, for example, a sensor type (e.g., vision sensor, thermal sensor, pressure sensor, etc.) and a sensor ID (e.g., vision sensor_1111). The first entry, 1511, may or may not include other information.

[0086] The second entry, 1512, can include metadata associated with the second mission action, such as one or more robot parameters, like a robot type (e.g., drone), a robot ID (e.g., drone_1112), a robot mission encompassing the second mission action (e.g., mission_2), and a robot pose (X2, Y2, Z2, α2, β2, γ2). The second entry, 1512, can also include one or more camera parameters, such as a camera pose (X2, Y2, Z2, a2, β2, γ). 2) , a zoom level (e.g., 1x) and / or a focus setting (e.g., continuous autofocus mode). In some implementations, the second entry 1512 may or may not contain information other than (one) value(s) for the one or more robot parameters and (one) value(s) for the one or more camera parameters.

[0087] The third entry, 1513, can include metadata associated with the third mission action, such as one or more robot parameters, like a robot pose (X3, Y3, Z3, α3, β3, y3), a robot type (e.g., a tracked robot), a robot ID (e.g., TrackedRobot_1113), and a robot mission encompassing the third mission action (e.g., Mission_3). The third entry, 1513, may or may not include information other than a value for the one or more robot parameters.

[0088] As in Fig. As shown in Figure 1B, the first mission action and the second mission action can be defined with different levels of granularity. The first mission action and the third mission action can be defined with different levels of granularity. The second mission action and the third mission action can be defined with different levels of granularity. In other words, different mission actions can be defined with different levels of granularity.

[0089] While the mission action database 151 in Fig. Since Figure 1B is shown as storing the first entry 1511 for the first mission action, the second entry 1512, and the third entry 1513 separately, descriptions of the mission action database 151 are limited accordingly. For example, the mission action database 151 can store and divide entries based on robot missions. The mission action database 151 can, for instance, include a first set of entries, each corresponding to a mission action from a first robot mission performed by a first robot (simulated or real robot); a second set of entries, each corresponding to a mission action from a second robot mission performed by a second robot (simulated or real robot); and so on, and an Nth set of entries, each corresponding to a mission action from an Nth robot mission performed by an Nth robot (simulated or real robot).

[0090] In some implementations, as a non-restrictive example, one or more mission actions (e.g., the first mission action and only the first mission action) can be selectively reproduced depending on a granularity with which the one or more mission actions are defined (as reflected by mission action metadata) (e.g., on the industrial environment map 140). For example, a user can select a filter element that removes any mission action defined without a POI parameter for a particular gauge (e.g., pressure gauge). In this case, the second mission action and the third mission action, which are defined in Fig. The data shown in Figure 1B are filtered out and not displayed, for example, on the industrial map 140, whereas the first mission action may or may not be displayed depending on whether the metadata, as shown in the first entry 1511, includes values ​​of the POI parameters indicating that a pressure gauge is detected or fixed during the first mission action. It should be noted that the format of the mission action database 151 does not correspond to the data shown in Figure 151. Fig. 1B illustrated format(s) is limited, and that Fig. 1B is shown for illustrative purposes only.

[0091] Fig. Figure 2A shows a non-restrictive example of a user interface 201 of a computer device 109, which displays a map showing all available mission actions, according to various implementations. As in Fig. As shown in Figure 2A, a plurality of mission actions (e.g., Mission Action_A, Mission Action_B, Mission Action_C, Mission Action_D, Mission Action_E, Mission Action_F, Mission Action_G) can be predefined for a first robot mission and represented in a first way (e.g., represented by a graphic representation that has a round shape), and an additional plurality of mission actions (e.g., Mission Action_a, Mission Action_b, Mission Action_c, Mission Action_d, and Mission Action_e) can be predefined for a second robot mission and represented in a second way (e.g., represented by a graphic representation that has a triangular shape) that differs from the first way.In some implementations, the majority of mission actions (MissionAction_A-G) and the additional majority of mission actions (MissionAction_a-MissionAction_e) together can constitute all defined / available mission actions for Industrial Facility 130, however, this is not required.

[0092] In some implementations, mission actions (i.e., Mission Action_A-Mission Action_G) can be rendered in the first way based on metadata defined at a first level of granularity, and mission actions (i.e., Mission Action_a-Mission Action_e) can be rendered in the second way based on metadata defined at a second level of granularity. The first level of granularity can be higher or finer than the second. For example, the metadata of Mission Action_A-Mission Action_G at the first level of granularity might include a robot pose for each of Mission Action_A-Mission Action_G and one or more camera parameters (which might define, for example, a camera pose, zoom level, and / or camera focus setting).For example, the metadata of Missionsaction_a-Missionsaction_e can be in the second level of granularity by including a robot pose for each of Missionsaction_A-Missionsaction_G and not including a camera parameter (e.g., camera pose).

[0093] As a non-restrictive example, mission action metadata (i.e., Mission Action_A) can include an initial robot pose (location and / or orientation) for a first robot associated with Mission Action_A, and can include an initial camera pose for a first camera carried by the first robot (e.g., to capture images of Mission Action_A). This Mission Action_A metadata can, for example, be collected and / or defined based on a previous instance in which the first robot was previously sent into the first robot pose and has the first camera positioned in the first camera pose to capture images of Mission Action_A.

[0094] As a non-restrictive example, the metadata of a mission action (i.e., mission action_a) can include a second robot pose (location and / or orientation) for a second robot associated with mission action_a and may not include a camera parameter. This mission action_a metadata can, for example, be collected and / or defined based on a previous instance in which the second robot was previously sent into the second robot pose. The second robot can be the same as the first robot or different from it.

[0095] In some implementations, as a non-restrictive example, while the graphical representations for Missionsaction_A-Missionsaction_G are all rendered as having a round shape, some graphical representations (e.g., of Missionsaction_A, Missionsaction_B, Missionsaction_C) may have a different color or pattern than other graphical representations (e.g., of Missionsaction_D, Missionsaction_E, Missionsaction_F, Missionsaction_G).The graphical representations for Mission_A-Mission_Action_C on the industrial environment map 140 may have a round shape based on the fact that the metadata of each Mission_A-Mission_Action_C includes both a robot pose and a camera pose, and these graphical representations may have an initial color or pattern based on the fact that the metadata of each Mission_A-Mission_Action_C includes one or more POI parameters (which define one or more features of a POI that is captured during the mission action).The graphical representations for Mission_Action_D and Mission_Action_G on the industrial environment map 140 may have a round shape, based on the fact that the metadata for each Mission_Action_D and Mission_Action_G includes both a robot pose and a camera pose. Furthermore, these graphical representations may have a second color (e.g., different from the first color) or a second pattern (e.g., different from the first pattern), based on the fact that the metadata for each Mission_Action_D and Mission_Action_G does not include POI parameters. The POI could, for example, be a measuring device (or a piece of equipment or another POI) that is fixed during the mission action.

[0096] In other words, a graphical representation for Mission Action_A (Mission Action_B or Mission Action_C) can be circular and, based on the metadata including a robot pose and one or more camera parameters, can include one or more POI parameters for a POI at Mission Action_A, and may have an initial color or pattern. The POI can be a sensor identified or detected at Mission Action_A (e.g., by the first camera carried by the first robot), and the one or more POI parameters can include a type of POI (e.g., heat sensor, pressure sensor, etc.) and / or an identifier for the POI.A graphical representation for Missionsaction_D (Missionsaction_E, Missionsaction_F or Missionsaction_G) may be circular and may have a second color or pattern based on the fact that the metadata does not include a POI parameter, as no industrial POI is identified for Missionsaction_D (which may be a location in an open space).

[0097] In some implementations, as a non-restrictive example, while the graphical representations for Missionsaction_a-Missionsaction_e are all rendered as having a triangular shape, some graphical representations (e.g., of Missionsaction_a, Missionsaction_b, Missionsaction_c) may have a different color (or pattern) than other graphical representations (e.g., of Missionsaction_d, Missionsaction_e).The graphical representations for Missionsaction_a-Missionsaction_c on the industrial environment map 140 may have a triangular shape based on the fact that the metadata of each of Missionsaction_a-Missionsaction_c includes a robot pose but no camera pose, and these graphical representations may have color or the first pattern based on the fact that the metadata of each of Missionsaction_a-Missionsaction_c includes one or more POI parameters (which define one or more features of a POI that is captured during the mission action).The graphical representations for Missionsaktion_d-Missionsaktion_e on the industrial environment map 140 may have a triangular shape based on the fact that the metadata of each of Missionsaktion_d-Missionsaktion_e includes a robot pose and not a camera pose, and these graphical representations may have the second color or pattern based on the fact that the metadata of each of Missionsaktion_d-Missionsaktion_e does not include a POI parameter.

[0098] In other words, a graphical representation for Mission Action_a (Mission Action_b or Mission Action_c) can be triangular and, based on the metadata including a robot pose and one or more camera parameters, can include one or more POI parameters for a POI at Mission Action_A, which exhibit the first color or pattern. The POI can be, for example, a sensor (or another POI, such as a pipe, device, etc.) identified or detected at Mission Action_a (e.g., by the first camera carried by the first robot), and the one or more POI parameters can include a type of POI (e.g., thermal sensor, pressure sensor, pipe, etc.) and / or an identifier (e.g., 123E5) of the POI.As a non-restrictive example, a graphical representation for Missionsaction_d (or Missionsaction_e) may be triangular and may, based on the fact that the metadata does not include a POI parameter, since no industrial POI is identified at Missionsaction_d, exhibit the second color or pattern.

[0099] It should be noted that the graphical representations of mission actions defined with camera parameters on the Industrial Environment Map 140 may be visually different / distinguishable from graphical representations of mission actions defined without camera parameters in some implementations. Similarly, in some implementations, graphical representations of mission actions defined with one or more POI parameters on the Industrial Environment Map 140 may be visually different from graphical representations of mission actions defined without POI parameters.Optionally, graphical representations of mission actions defined with one or more POI parameters of a first POI on the industrial environment map 140 may be visually different from graphical representations of mission actions defined with one or more POI parameters of a second POI that differs from the first POI.

[0100] Fig. Figure 2B shows a non-restrictive example of a user interface 203 that displays a map with all available mission actions, according to various implementations. As in Fig. As shown in Figure 2B, the user interface 203 can include one or more filter elements (e.g., each corresponding to a filter criterion, as described previously) that are displayed on the user interface 203 together with the map 140, which shows mission actions (e.g., Mission Action_A-Mission Action_G, Mission Action_a-Mission Action_e). The map in Fig. 2B can be the same as card 140 in Fig. 1A or differ from it. In some implementations, the one or more filter elements may include a first filter element (e.g., FilterElement_1), a second filter element (e.g., FilterElement_2), and / or a third filter element (e.g., FilterElement_3). In some implementations, the one or more filter elements may additionally or alternatively include a user-defined filter definition field 213 for receiving (e.g., via typed input or audio input, etc.) user-defined filter criteria.

[0101] In some implementations, for example, filter element_1 might be a first selectable element which, when selected, causes a first filter rule / criterion to be applied to selectively reproduce mission action(s), each defined by a robot parameter specifying a particular robot type. Filter element_2 might be a second selectable element which, when selected, causes a second filter rule to be applied to selectively reproduce mission action(s), each defined by a camera parameter. Filter element_3 might be a third selectable element which, when selected, causes selective mission action(s) to be reproduced, each defined by a POI parameter (which, for example, specifies a particular type of sensor or measuring device).

[0102] In some implementations, filter element_1 can be configured to trigger a dropdown menu (listing multiple robot types) for a user to selectively play back mission action(s) based on a robot type defined for a mission action; filter element_2 can be configured to trigger a dropdown menu (listing values ​​for one or more camera parameters, where an option "no camera parameter" can indicate a preference for a mission action defined without a camera parameter) to selectively play back mission action(s) based on a camera parameter; and filter element_3 can be configured to trigger a dropdown menu for a user to selectively play back mission action(s) based on a POI parameter.It should be noted that the number / format of filter elements or filter factors is not limited herein and can be any suitable number or format. For example, one or more mission actions can be selectively played back based on a detected event (e.g., cooling tower failure). In this case, the one or more mission actions, defined by both a camera pose and a pressure sensor type (in metadata associated with the one or more mission actions), can be automatically and selectively played back in the industrial environment map 140 in response to the detection of the event (e.g., the cooling tower failure).

[0103] Fig. 2C shows an exemplary user interface 205 that displays a map rendered with one or more selected mission actions for performing selected aspects of the present disclosure, according to various implementations. A user of the industrial environment map 140 can provide a filter parameter to selectively render mission actions defined with both a robot pose and a camera pose. In this case, as shown in Fig. As shown in 2C, MissionsAction_A and MissionsAction_G can be selectively played back in response to receiving the filter parameter, which serves to selectively play back mission actions defined with both a robot pose and a camera pose. MissionsAction_A and MissionsAction_G can be selected for display on the industrial environment map 140 from all available mission actions (e.g., MissionsAction_A, MissionsAction_G, MissionsAction_a, MissionsAction_e) based on the fact that each is defined with a robot pose and a camera pose. MissionsAction_a and MissionsAction_e, however, are filtered out based on the fact that each is not defined with a camera pose, so they are not displayed.

[0104] Fig. Figure 2D shows another exemplary user interface 207, which displays a map rendered with one or more selected mission actions for performing selected aspects of the present disclosure, according to various implementations. A user of the industrial environment map 140 can provide an additional filter parameter to selectively render mission actions defined by: a robot pose, a camera pose, and a POI parameter. In this case, as shown in Fig. In 2D representation, MissionsAction_A-MissionAction_C can be selected from all available mission actions (e.g., MissionsAction_A-MissionAction_G, MissionsAction_a-MissionAction_e) for display on the industrial environment map 140, based on the fact that each of MissionsAction_A-MissionAction_C is defined not only with a robot pose and a camera pose, but also with a POI parameter. MissionsAction_A-MissionAction_G and MissionsAction_D-MissionAction_G are filtered out (e.g., not displayed) because each of MissionsAction_A-MissionAction_e and MissionsAction_D-MissionAction_G is not defined with a POI parameter.

[0105] Fig. Figure 2E shows another exemplary user interface 209, which displays a map represented by one or more selected mission actions for carrying out selected aspects of the present revelation, according to various implementations. In this example, as in Fig. 2E shows that one or more mission actions (e.g., mission action_a, mission action_C, mission action_D) are selectively and automatically displayed in the industrial environment map 140 in response to the detection of a target event (e.g., an anomaly) at a specific location T within industrial facility 130.

[0106] One or more mission actions (e.g., Mission Action_a, Mission Action_C, Mission Action_D) can be displayed based on the distance between a specific location T and a given mission action (Mission Action_a, Mission Action_C, and Mission Action_D) being within a predefined distance threshold (e.g., 2 m). Alternatively or additionally, one or more mission actions (e.g., Mission Action_C and Mission Action_D) can be displayed based on metadata associated with each mission action specifying a robot pose and a camera pose. Alternatively or additionally, one or more mission actions (e.g., Mission Action_C and Mission Action_D) can be displayed based on user input, which can, for example, modify the predefined distance threshold (e.g., from 1.0 m to 0.5 m).

[0107] Fig. Figure 2F shows another exemplary user interface 211, which displays a map represented by one or more selected mission actions for carrying out selected aspects of the present revelation, according to various implementations. In this example, as in Fig. As shown in 2F, one or more mission actions (e.g., mission action_a, mission action_C, mission action_D, mission action_d) can be selectively triggered in response to the detection of a target event (e.g., an anomaly) at a specific location T within industrial plant 130, and in response to a user defining an area (in Fig. 2F (defined by a dashed line) defines the specific location T, which is displayed on the industrial environment map 140. The area can be defined, for example, by a user touch on a display of a computer device 109.

[0108] Fig. Figure 3 illustrates an exemplary procedure 300 for performing selected aspects of the present disclosure according to various implementations. For simplicity, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various points of interest (POIs) of different computer systems, such as one or more POIs of the server computer device 105 (and / or additional computer devices, such as the client device 103-A or 103-B). Although the operations of procedure 300 are presented in a specific order, this is not to be construed as a restriction. One or more operations may be reordered, omitted, or added.

[0109] In various implementations, the system at block 302 can, for example, using a server such as server computer device 105, identify appropriate metadata for each of a plurality of mission actions for an industrial environment. The industrial environment can be or comprise an industrial plant. The industrial plant can be a chemical processing plant, an industrial work environment, an oil or gas refinery, a catalyst factory, a manufacturing plant, an offshore oil platform, or any other suitable facility.

[0110] In some implementations, the system can determine or identify the plurality of mission actions before identifying the metadata for each of the plurality. The plurality of mission actions can be determined, for example, based on user input and / or data transmitted by one or more robots deployed in the industrial environment. The user input might define, for instance, a first robot mission for a first robot and a second robot mission for a second robot (which may be the same as or different from the first robot). The first robot mission comprises a first mission action, a second mission action, and a third mission action, defined by a human operator for the first robot to navigate to and / or perform a corresponding action.The second robot mission comprises Mission Action_1, Mission Action_2, Mission Action_3, and Mission Action_4, which are defined by the human operator or another operator for the second robot to navigate to and / or perform a corresponding action. In this example, the first three mission actions, Mission Action_1 through Mission Action_4, and one or more additional mission actions (if available) constitute the plurality of mission actions predefined by a human operator. The plurality of mission actions might, for example, include a fourth mission action determined based on data (e.g., lidar sensor data) transmitted by a robot (e.g., one carrying a lidar sensor) indicating the detection of an object or event (e.g., oil spillage or repeated leakage) by the robot during the fourth mission action.The majority of mission actions can also be determined in other ways and are not limited to the descriptions herein. Optionally, the first, second, and third mission actions can be robot mission actions that form a path along which the first robot observes one or more aspects of the industrial environment, and mission actions 1 through 4 can form a path along which the second robot observes one or more aspects of the industrial environment.

[0111] As a non-restrictive example, a mission action for an industrial plant can initially be defined by a human operator controlling a robot (e.g., using a hand controller and / or a computer interface) and providing input(s) to specify the mission action. Controlling a robot can mean controlling a real robot in the actual industrial plant or controlling a simulated robot in a simulation of the industrial plant.For example, a human operator can use a hand controller to navigate a robot to a specific robot pose (including providing location and orientation information to configure the robot) and adjust a camera worn by the robot to a specific camera pose when the robot is in that pose (so that the camera can capture a clear image of a target object or location of interest to the human operator). In this example, the human operator can provide a user interface input to define the robot pose and the camera pose of the camera worn by the robot as metadata for the mission action.

[0112] In the above non-restrictive example, the robot (or another robot of the same type and model) can use the defined mission action to navigate autonomously (i.e., without requiring control from the human operator) to the mission action.

[0113] In some implementations, different mission actions with varying degrees of granularity can be defined and / or navigated to. For example, a mission action to capture an image of a measuring device can define both of the following: (a) a robot pose (e.g., X, Y, Z coordinates and optional rotation and / or translation); and (b) a camera parameter, such as a camera pose (e.g., rotation and / or translation), a camera zoom level, and / or a camera focus setting. On the other hand, a mission action to capture a gas measurement can define only (a) a robot pose and not define any camera parameters. A mission action can optionally or additionally be stored associated with metadata that includes a POI parameter, which defines a feature(s) of POIs that is / are captured by a sensor measurement during the mission action.A mission action to capture measurements from a specific instrument can, for example, define a type of instrument (e.g., temperature type, pressure type, etc.) and / or a unique identifier of the instrument (e.g., X1234AB).

[0114] In other words, metadata associated with multiple mission actions can have different levels of granularity for different mission actions. In some implementations, the metadata can be received or collected based on user input and / or data (e.g., lidar sensor data, configuration data, etc.) transmitted by robot(s) associated with the mission actions. In some implementations, the metadata defining each of the multiple mission actions can be stored in and accessed through a mission action database.

[0115] In various implementations, at block 304, the system can, for example, using a server such as server computer device 105, render a first subset of mission actions with initial graphical features in a map interface for the industrial environment. The rendering of this first subset of mission actions with their initial graphical features occurs in response to a determination that the corresponding metadata for each mission action in the first subset matches one or more initial criteria. In some implementations, the map interface is a two-dimensional view of the industrial environment.

[0116] In some implementations, determining that the relevant metadata for each of the mission actions of the first subset matches one or more first criteria may include determining that the relevant metadata for each of the mission actions of the first subset indicate that the mission actions of the first subset include measurements from a first type of sensor.

[0117] In some implementations, determining that the relevant metadata for each of the mission actions in the first subset matches one or more of the first criteria may involve determining that the relevant metadata for each of the mission actions in the first subset includes: a relevant robot pose for the mission action and one or more relevant vision sensor parameters for capturing vision data of the mission action in the relevant robot pose. In some implementations, the one or more relevant vision sensor parameters may include a vision sensor pose, a vision sensor zoom level, and / or a vision sensor focus parameter associated with a vision sensor worn by a mobile robot within the industrial environment.

[0118] In some implementations, the metadata of the mission actions of the first subgroup includes one or more POI parameters for a POI (measuring device, hazardous object, etc.) during one of the mission actions of the first subgroup. The one or more POI parameters may, for example, include a POI identifier and / or a POI type.

[0119] In various implementations, at block 306, for example, using a server such as server computer device 105, the system can display a second subset of mission actions with second graphical features in the map interface for the industrial environment, along with the first subset of mission actions. The display of the second subset of mission actions with the second graphical features occurs in response to a determination that the corresponding metadata for each of the mission actions in the second subset matches one or more second criteria. In some implementations, the first subset of mission actions and the second subset of mission actions can be displayed simultaneously. In other implementations, the first subset of mission actions and the second subset of mission actions may not be displayed simultaneously.The first criteria differ from the second criteria. The first graphical features can be visually distinct from the second graphical features. In other words, the first subset of mission actions and the second subset of mission actions can be represented in different ways (e.g., different shapes, colors, patterns, etc.), for example, based on the first subset of mission actions being defined with metadata that has a first level of granularity, and the second subset of mission actions being defined with metadata that has a second level of granularity (which is finer or coarser than the first level).

[0120] In some implementations, determining that the relevant metadata for each of the mission actions of the second subset matches one or more of the second criteria may include determining that the relevant metadata for each of the mission actions of the second subset indicate that the mission actions of the second subset include measurements from a second type of sensor. The second type may differ from the aforementioned first type.

[0121] In some implementations, determining that the relevant metadata for each of the mission actions of the second subset matches one or more of the second criteria may include determining that the relevant metadata for each of the mission actions of the second subset includes a relevant robot pose for the mission action, but may not include camera parameters for capturing an image of the mission action.

[0122] In some implementations, determining that the relevant metadata for each of the mission actions of the second subset matches one or more of the second criteria may include determining that the relevant metadata for each of the mission actions of the second subset includes only one relevant robot pose for each of the mission actions of the second subset.

[0123] In various implementations, the system at block 308 can receive a selection, via the map interface for the industrial environment, of a given mission action from the first subgroup or the second subgroup, for example, using a server such as server computer device 105. In various implementations, the system at block 310, for example, using a server such as server computer device 105, can, in response to receiving the selection, add the given mission action to a predefined robot mission or incorporate the given mission action into a new robot mission. The given mission action can be added to the predefined robot mission or the new robot mission automatically.

[0124] For example, the predefined robot mission might include mission actions AK, but not mission action L (even though mission action L is defined in an alternative mission). In some implementations, mission action L can be selected from one or more mission actions that are selectively displayed on the map surface based on user input. In this case, there might be a desire to add mission action L to the predefined robot mission, for example, if there is a potentially hazardous condition near mission action L (e.g., within a predetermined distance of mission action L), or if there is a potential problem with a gauge associated with mission action L, and so on.

[0125] Fig. Figure 4 illustrates another exemplary procedure 400 for performing selected aspects of the present disclosure according to various implementations. For simplicity, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various points of interest (POIs) of different computer systems, such as one or more POIs of the server computer device 105 (and / or additional computer devices, such as the client device 103-A or 103-B). Although the operations of procedure 400 are shown in a particular order, this is not to be understood as a restriction. One or more operations may be reordered, omitted, or added.

[0126] In various implementations, the system can identify appropriate metadata for each of a plurality of mission actions for an industrial environment at block 402, e.g., by means of a server such as the server computer device 105.

[0127] In some implementations, the corresponding metadata for each of the multiple mission actions includes a corresponding robot pose of that robot during that mission action, and determining the filter criteria involves defining one or more pose-based filter criteria (e.g., to selectively play back mission actions that are within an area containing a destination). The destination might be, for example, a location where an anomaly (e.g., an oil leak) is detected, or where maintenance is required, etc. The area containing the destination can be defined based on user input on the map surface, or the area can be automatically defined based on a predetermined distance relative to the destination.

[0128] In some implementations, the corresponding metadata for each of the subset's mission actions includes one or more corresponding vision sensor parameters associated with a vision sensor to capture an image in the corresponding robot pose associated with a corresponding mission action, and where determining the filter criteria includes determining one or more vision-based filter criteria to, for example, selectively reproduce mission actions defined with a specific value (e.g., 5x zoom level, etc.) of vision sensor parameters.

[0129] In some implementations, the metadata for each of the subset's mission actions includes one or more POI parameters associated with a POI at the corresponding mission action. In some implementations, the one or more POI parameters include a POI identifier and / or a POI type.

[0130] In various implementations, the system can determine one or more filter criteria at block 404, for example, using a server such as server computer device 105, based on user interface inputs and / or based on a detected event in the industrial environment. In some implementations, the one or more filter criteria include the requirement that the metadata for each of the mission actions of the subset includes the POI that has the POI identifier or is of the specified type.

[0131] In various implementations, at block 406, the system can, for example, by means of a server such as server computer device 105, select a subset of mission actions from the mission actions, the selection of the subset of mission actions being made in response to determining that the corresponding metadata for each of the mission actions in the subset matches one or more filter criteria.

[0132] In various implementations, the system at Block 408 can, for example, by means of a server such as the server computer device 105, display the subset of mission actions in a map interface for the industrial plant without displaying other mission actions that do not belong to the subset.

[0133] In some implementations, the rendered subset may include a first mission action, represented by first graphical features; and a second mission action, represented by second graphical features, where the first graphical features are visually distinct from the second. In some implementations, metadata associated with the first mission action has a higher level of granularity than the metadata associated with the second mission action.

[0134] In various implementations, the system at Block 410, e.g., by means of a server such as the server computer device 105, can receive a selection, via the map surface for the industrial plant, of a given mission action of the first subgroup.

[0135] In various implementations, at block 412, for example by means of a server such as the server computer device 105, the system can add the given mission action to a predefined robot mission or incorporate the given mission action into a new robot mission in response to receiving the selection.

[0136] Fig. Figure 5 is a block diagram of an exemplary computer device 510, which can optionally be used to perform one or more aspects of the techniques described herein. The computer device 510 typically includes at least one processor 514, which communicates with a number of peripheral devices via the bus subsystem 512. These peripheral devices may include a memory subsystem 524, which may include, for example, a memory subsystem 525 and a file storage subsystem 526, user interface output devices 520, user interface input devices 522, and a network interface subsystem 516. The input and output devices allow user interaction with the computer device 510. The network interface subsystem 516 provides an interface to external networks and is coupled with corresponding interface devices in other computer devices.

[0137] The user interface input devices 522 may include a keyboard, pointing devices such as a mouse, trackball, touchpad or graphics tablet, a scanner, a touchscreen integrated into the display, audio input devices such as speech recognition systems, microphones, and / or other types of input devices. In general, the use of the term "input device" is intended to encompass all possible types of devices and means of inputting information into the computer device 510 or into a communications network.

[0138] User interface output devices 520 may include a display subsystem, a printer, a fax machine, or non-visual displays, such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat panel display device, such as a liquid crystal display (LCD), a projection device, or any other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through audio output devices. In general, the use of the term "output device" is intended to include all possible types of devices and means of outputting information from computer devices 510 to the user or to another machine or computer device.

[0139] Memory subsystem 524 stores program and data constructs that provide the functionality of some or all of the modules described herein. For example, memory subsystem 524 can store the logic for performing selected aspects of the procedures from Fig. 3 or Fig. 4 as well as for the implementation of various components that are in Fig. The figures shown in 1 include:

[0140] These software modules are generally executed by processor 514 alone or in combination with other processors. The main memory 525, used in the memory subsystem 524, can comprise several memories, including random access memory (RAM) 530 for storing instructions and data during program execution and read-only memory (ROM) 532, which stores fixed instructions. A file storage subsystem 526 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges.The modules that implement the functionality of certain implementations can be stored by the file storage subsystem 626 in the storage subsystem 524 or in other machines that the processor(s) 514 can access.

[0141] Bus subsystem 512 provides a mechanism by which the various components and subsystems of the computer device 510 can communicate with each other as intended. Although bus subsystem 512 is schematically represented as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0142] The computer device 510 can be of various types, including a workstation, server, computer cluster, blade server, server farm, or any other data processing system or computer device. Due to the constantly evolving nature of computers and networks, the description in Fig. The computer device 510 shown in Figure 5 is only a specific example for the purpose of illustrating some implementations. Many other configurations of the computer device 510 are possible, with more or fewer components than those shown. Fig. 5 computer device shown.

Claims

[1] Method (300) implemented by one or more processors, the method comprising: Identifying (302) metadata for each of a plurality of mission actions for an industrial environment; Reproducing (304), in a map surface for the industrial environment, a first subset of mission actions with first graphic features, wherein the reproduction of the first subset of mission actions with the first graphic features is performed in response to a determination that appropriate metadata for each of the mission actions of the first subset matches one or more first criteria; Reproduce (306), in the map surface for the industrial environment and together with the first subset of mission actions, a second subset of mission actions with second graphic features, wherein the reproduction of the second subset of mission actions with the second graphic features is performed in response to a determination that appropriate metadata for each of the mission actions of the second subset matches one or more second criteria, the first criteria differ from the second criteria, and where the first graphic features are visually distinct from the second graphic features; Receiving (308) a selection, via the map surface for the industrial environment, of a given mission action of the first subgroup or the second subgroup; and in response to receiving the selection, adding the given mission action to a predefined robot mission or incorporating the given mission action into a new robot mission. [2] Method according to claim 1, wherein: Determining that the relevant metadata for each of the mission actions of the first subgroup matches one or more of the first criteria includes the following: Determine that the relevant metadata for each of the mission actions of the first subgroup include a measurement from a first type of sensor, and Determining that the relevant metadata for each of the mission actions of the second subgroup matches one or more of the second criteria includes the following: Determine that the relevant metadata for each of the mission actions of the second subgroup includes a measurement from a second type of sensor, the second type being different from the first type. [3] The method of claim 1, wherein determining that the relevant metadata for each of the mission actions of the first subgroup matches the one or more first criteria comprises: Determine that the appropriate metadata for each of the mission actions of the first subset includes: an appropriate robot pose for the mission action and one or more appropriate vision sensor parameters for capturing vision data of the mission action in the appropriate robot pose. [4] Method according to claim 3, wherein the mission actions of the first subgroup comprise a dangerous object. [5] Method according to claim 3, wherein one or more vision sensor parameters comprise a vision sensor pose, a vision sensor zoom level and / or a vision sensor focus parameter, which are associated with a vision sensor carried by a mobile robot within the industrial environment. [6] Method according to claim 3, wherein metadata for each of the mission actions of the first subgroup further comprises one or more POI parameters for a POI at a corresponding mission action of the mission actions of the first subgroup. [7] Method according to claim 6, wherein the one or more POI parameters comprise a POI identifier of the POI and / or a type of the POI. [8] The method of claim 1, wherein determining that the relevant metadata for each of the mission actions of the second subgroup matches one or more of the second criteria comprises: Determine that the relevant metadata for each of the mission actions of the second subset includes a corresponding robot pose for the mission action, but does not include camera parameters for capturing an image of the mission action. [9] The method of claim 8, wherein determining that the relevant metadata for each of the mission actions of the second subgroup matches one or more of the second criteria comprises: Determine that the corresponding metadata for each of the mission actions of the second subgroup includes only one corresponding robot pose for each of the mission actions of the second subgroup. [10] Method according to claim 1, wherein the map surface is a two-dimensional view of the industrial environment. [11] Method according to claim 1, wherein the first graphic features comprise a first selectable element of a graphical user interface (GUI) that represents a first mission action from the first subgroup of mission actions, and the second graphic features comprise a second selectable element of a graphical user interface (GUI) that represents a second mission action from the second subgroup of mission actions, wherein the first selectable GUI element differs from the second selectable GUI element. [12] Method (400) implemented by one or more processors, the method comprising: Identifying (402) metadata for each of a plurality of mission actions for an industrial environment; Determine (404), based on a user interface input and / or based on a detected event in the industrial environment, one or more filter criteria; Select (406) from the mission actions, a subset of the mission actions, wherein the selection of the subset of mission actions is in response to a determination that appropriate metadata for each of the mission actions of the subset matches one or more filter criteria; Reproduce (408), in a map surface for the industrial plant, the subset of mission actions, without reproducing any other mission action from the plurality of mission actions that is not part of the subset; Receiving (410) a selection, via the map surface for the industrial plant, of a given mission action of the first subgroup; and in response to receiving the selection, adding (412) the given mission action to a predefined robot mission or incorporating the given mission action into a new robot mission. [13] Method according to claim 12, wherein: The metadata for each of the majority of mission actions includes the following: a robot pose of the respective robot during a corresponding mission action, and The one or more filter criteria may include: a pose-based filter rule to selectively reproduce mission actions that are within an area containing a destination. [14] Method according to claim 13, wherein the area containing a destination is defined based on user input on the map surface or the area is automatically defined based on a predetermined distance in relation to the destination. [15] Method according to claim 13, wherein The metadata for each of the subset's mission actions includes the following: one or more vision sensor parameters associated with a vision sensor to capture an image in the robot pose associated with a corresponding mission action, and The one or more filter criteria may include: a vision-based filter rule to selectively reproduce mission actions of a specific type. [16] Method according to claim 12, wherein the metadata for each of the mission actions of the subgroup comprises one or more POI parameters associated with a POI at a corresponding mission action. [17] Method according to claim 15, wherein the one or more POI parameters comprise a POI identifier of the POI and / or a type of the POI. [18] Method according to claim 12, wherein reproducing the subset of mission actions comprises: Reproducing an initial mission action from the subset of mission actions with initial graphical features; and Reproducing a second mission action from the subset of mission actions with second graphic features, where the first graphic features are visually distinct from the second graphic features. [19] Method according to claim 18, wherein metadata associated with the first mission action has a higher level of granularity than the metadata associated with the second mission action. [20] Method implemented by one or more processors, the method comprising: Identifying metadata for each of a plurality of mission actions for an industrial environment, wherein the metadata for each of the plurality of mission actions includes at least one robot pose for a robot during a corresponding mission action; Detecting an object or event that needs to be inspected within the industrial environment; in response to the detection of the object or event, playback, in a map surface for the industrial environment, a first mission action and a second mission action, wherein the retrieval of the first and second mission actions occurs in response to the determination that the first and second mission actions are within a predetermined distance of a location of the object or event, where the first mission action and the second mission action are represented with different graphical features based on the fact that the metadata of the first mission action and the second mission action have different levels of granularity; and Receiving a selection, via the map interface for the industrial plant, the first mission action, or the second mission action; and in response to receiving the selection, adding the first or second mission action to a robot mission.

Images