Systems, methods, and devices for building robotic missions

Abstract

A method and robot management system for building robotic missions based on mission metrics are described, including: providing a mission file data record associated with a recorded robotic mission configuration; receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to an environment; determining first context information associated with the one or more robot-independent tasks; searching the mission file data record and identifying one or more mission files including one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files; and providing the one or more identified mission files based on the ranking information.

Description

[Technical Field]

[0001] The following relates to managing robotic services, for example in industrial facilities, and more specifically to building robotic missions. [Background technology]

[0002] Industrial facilities, such as chemical plants, oil refineries, and power plants, can be extremely large and involve numerous simultaneous processes spread throughout the facility. Tasks to be completed in such facilities can include monitoring and adjusting equipment, sensing environmental or other conditions within the facility, etc. Some facility operators have begun to deploy robots for the completion of such industrial tasks. Improved techniques for managing swarms of robots in connection with task completion are desirable. Summary of the Invention

[0003] The technology described herein relates to improved methods, systems, devices, and apparatus for supporting a robotic management system or robotic system platform. Generally, the described techniques provide for building robotic missions based on mission metrics, a robot-independent GUI, and / or a mission inventory.

[0004] The described method includes: providing a data record of a mission file associated with a recorded robotic mission configuration; receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to an environment; determining first context information associated with the one or more robot-independent tasks; searching the data record of the mission file and identifying one or more mission files including one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files; and providing the one or more identified mission files based on the ranking information.

[0005] The robot management system includes: an interface for providing commands to a plurality of robotic devices of one or more robot types; and a data repository including mission file data records associated with a configuration of recorded robotic missions. The robot management system manages the one or more robotic missions and the plurality of robotic devices by: receiving instructions for one or more robot-independent tasks to be performed with respect to an environment; determining first context information associated with the one or more robot-independent tasks; searching the mission file data records and identifying one or more mission files including one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files; and providing the one or more identified mission files based on the ranking information.

[0006] The robotic system platform includes: an interface for providing commands to a plurality of robotic devices, wherein at least one robotic device of the plurality of robotic devices is independent of the robotic system platform; a data repository including mission file data records associated with a configuration of recorded robotic missions; and one or more circuits. The one or more circuits electronically receive instructions for one or more tasks to be performed with respect to an environment. The one or more tasks are absent instructions for at least one of the following: a target robot type associated with completing the one or more tasks; a target robot function associated with completing the one or more tasks; or a target quantity of the robotic device associated with completing the one or more tasks. The one or more circuits further search the mission file data records and, based on the search, provide one or more identified mission files corresponding to one or more recorded robotic missions that at least partially completed the one or more tasks. The one or more circuits generate ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files and provide the one or more identified mission files based on the ranking information.

[0007] The described method includes: electronically receiving a first set of commands related to operating a first robotic device to perform a first set of tasks; recording the first set of commands in a first mission file; electronically receiving a second set of commands related to operating a second robotic device to perform the second set of tasks; recording the second set of commands in a second mission file; identifying a third set of tasks common to the first set of tasks and the second set of tasks; and providing the first mission file and the second mission file in response to receiving a search request indicating one or more tasks in the third set of tasks.

[0008] A robot management system is described that includes a processor and a memory coupled to the processor, the memory storing data that, when executed by the processor, causes the processor to: display a graphical user interface (GUI); electronically receive via the GUI a first set of commands related to operating a first robotic device of a first robot type to perform a first set of tasks; record the first set of commands in a first mission file; electronically receive a second set of commands related to operating a second robotic device of a second robot type to perform a second set of tasks; record the second set of commands in a second mission file; identify a third set of tasks common to the first set of tasks and the second set of tasks; and provide the first mission file and the second mission file in response to receiving a search request indicating one or more tasks of the third set of tasks.

[0009] The robotic system platform includes: a database including data records of mission files associated with a configuration of recorded robotic missions; a processor; and a memory coupled to the processor, the memory storing data that, when executed by the processor, causes the processor to: present a GUI; electronically receive a search request via the GUI indicating one or more candidate tasks to be performed with respect to an environment, the search request associated with performing the one or more candidate tasks for which no indication of a target robotic device, a target robot type, or both exists; identify at least a first mission file and a second mission file from the mission file data records based on a mapping of the one or more candidate tasks to the first mission file and the second mission file; and provide notification of the first mission file and the second mission file via the GUI.

[0010] The described method includes: identifying one or more target tasks to be performed; identifying a recorded mission file based on the one or more target tasks, wherein identifying the recorded mission file at least partially achieves a target goal associated with the one or more target tasks; and providing one or more commands to one or more robotic devices in connection with implementing the one or more recorded tasks, wherein implementing the one or more recorded tasks at least partially achieves the target goal.

[0011] The robot management system has: an interface for providing commands to a plurality of robotic devices of one or more robot types; and a data repository containing data records of mission files associated with a set of recorded robotic missions. The robot management system manages the one or more robotic missions and the plurality of robotic devices by: identifying one or more goal tasks to be performed; identifying a recorded mission file based on the one or more goal tasks, where identifying the recorded mission file at least partially achieves a goal goal associated with the one or more goal tasks; and providing one or more commands to the one or more robotic devices in connection with implementing the one or more recorded tasks, where implementing the one or more recorded tasks at least partially achieves the goal goal.

[0012] The robotic system platform includes: an interface for providing commands to a plurality of robotic devices of one or more robot types, the plurality of robotic devices being independent of the robotic system platform; a data repository including data records of mission files associated with a configuration of recorded robotic missions; and one or more circuits, the one or more circuits performing the following: identifying one or more goal tasks to be performed with respect to the environment, the identifying the one or more goal tasks being performed in response to electronically receiving one or more goal task instructions, goal goal instructions, or both; identifying one or more recorded mission files from the mission file data records based on the one or more goal tasks, the one or more recorded mission files including one or more recorded tasks that at least partially achieve the goal goal; and providing one or more commands to one or more robotic devices of the plurality of robotic devices in connection with implementing the one or more recorded tasks. [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates an example of a system according to an embodiment of the present disclosure.

[0014] [Figure 2] 1 illustrates an example of tasks and components of an operations management system for an industrial facility.

[0015] [Figure 3] An example of a generalized model of robot activity within an operations management system is presented.

[0016] [Figure 4] 1 illustrates example components of a robotic service platform within a operations management system, in accordance with one or more exemplary embodiments.

[0017] [Figure 5]1 illustrates an example of a detailed architecture of a multi-vendor robotic service platform, according to one or more exemplary embodiments.

[0018] [Figure 6] 1 illustrates another example of a detailed architecture of a multi-vendor robotic service platform and associated components, according to one or more illustrative embodiments.

[0019] [Figure 7] 1 illustrates an example of a subsystem of an operations management system including a multi-vendor robotic service platform, according to one or more exemplary embodiments.

[0020] [Figure 8] 1 illustrates an example environment for robot control and communication within a multi-vendor robotic service platform, in accordance with one or more exemplary embodiments.

[0021] [Figure 9] 1 illustrates an example process for planning, allocating, and executing robotic tasks within a multi-vendor robotic service platform, in accordance with one or more illustrative embodiments.

[0022] [Figure 10] 1 illustrates an example of a dashboard for entering and displaying data, in accordance with one or more exemplary embodiments.

[0023] [Figure 11] 1 illustrates an example of a robot-independent GUI for inputting and displaying data, according to one or more exemplary embodiments.

[0024] [Figure 12] 10 illustrates an example of feedback related to a completed mission, in accordance with one or more illustrative embodiments.

[0025] [Figure 13] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 14] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 15] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 16] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 17] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 18] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 19] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 20] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 21] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 22] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 23] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. [Figure 24] 1 illustrates an exemplary process flow according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] Industrial facilities, such as chemical plants, oil refineries, and power plants, can be extremely large and may involve numerous simultaneous processes spread throughout the facility. Tasks to be completed in such facilities may include monitoring and adjusting equipment, sensing environmental or other conditions within the facility, performing manufacturing tasks, etc. Some of these tasks may require attention on a 24-hour-a-day schedule, while others may require exposure to potentially hazardous conditions or working in confined or otherwise difficult-to-access spaces.

[0027] Some facility operators are adopting autonomous mobile robots to complete such industrial tasks, for example, to reduce costs, increase facility uptime, or protect human workers from potentially dangerous conditions. For example, some facilities may use autonomous robots for industrial applications according to 4D principles, having robots perform tasks that are tedious, dirty, dangerous, or unfriendly. That is, tasks may include repetitive, unsanitary, dangerous, or costly (e.g., no room for error). However, the range of tasks required in a complex facility may require numerous robots of various types and capabilities provided by different manufacturers. In some cases, robots may not operate according to common protocols or commands or communicate data in a similar manner. Therefore, integrating robot swarms into industrial facilities is becoming increasingly complex as the number of robots, robot types, and robot-assigned tasks increases. This complexity can be a significant barrier to the adoption of robots in industrial facilities.

[0028] Some robot management systems allow a user to send a series of commands to a robot to control the robot's movements, record the command set, and later recreate the recorded actions as a predefined action sequence (e.g., for use in training operators).

[0029] However, such robot management systems may be specific to the vendor control software corresponding to the robot. For example, such robot management systems may lack integration with a robot service platform that can provide a robot-independent way to create and record sequences of commands as an inventory of mission files. In some cases, a robot management system may not be able to control a heterogeneous robot fleet using an inventory of mission files. The term "heterogeneous robot fleet" may refer to a fleet of robotic devices of different device types, different capabilities and functions, different vendors, etc. In some cases, a heterogeneous robot fleet may include robots controllable via an open-source robot operating system (e.g., ROS robots 555 described later in this specification with reference to FIG. 5) and vendor-specific robots (e.g., non-ROS robots 565 described later in this specification with reference to FIG. 5).

[0030] Such robot management systems do not provide a general way for a user to create a set of commands (e.g., construct a mission) for execution by one or more robots in a robot fleet. Instead, the robot management systems rely on robot-specific controllers to communicate missions to each robot of different models or vendors. Such reliance on robot-specific controllers may prevent the robot management system from effectively coordinating missions among a robot fleet that includes different types of robots and / or robots from different vendors.

[0031] In some other cases, some robot management systems do not provide a data-based approach to building missions, but rather require an operator to identify a sequence of commands for one or more robots from a robot fleet to execute. Thus, for example, the effectiveness of the sequence of commands depends on the operator's personal experience and capabilities, and in some cases the operator's best guess.

[0032] According to example aspects of the present disclosure, a robotic servicing platform is described that supports a robot-independent method of creating and recording sequences of commands as an inventory of a mission file. The mission file may include data corresponding to the missions described herein. According to example aspects of the present disclosure, a robotic servicing platform is described that controls robotic devices and, in some aspects, may implement robot-independent GUI support for building and recording robotic missions. Aspects of a robot management system described herein support remotely managing a heterogeneous robotic fleet using an inventory of previously recorded missions.

[0033] The terms "mission file," "robot mission file," and "mission data file" may be used interchangeably herein. The terms "mission" and "robot mission" may be used interchangeably herein. The terms "metric," "metric data," and "metric information" may be used interchangeably herein. The terms "robot management system" and "robotic management system" may be used interchangeably herein. The terms "robot" and "robot device" may be used interchangeably herein. The terms "robot independent GUI," "robot independent control GUI," "GUI," and "control GUI" may be used interchangeably herein.

[0034] A robot-independent method may include controlling a heterogeneous robot fleet using an inventory of mission files. In some aspects, a robot diagnostic method may include evaluating one or more missions in the mission inventory (e.g., multiple missions, all missions, a subset of missions, etc.) to determine various different metrics, and possibly recording the metrics as metadata in the recorded mission file. Aspects of the present disclosure support applying metrics in relation to recommended missions based on a given set of context (e.g., contextual information related to one or more tasks). The terms “metric,” “metric data,” and “metric information” may be used interchangeably herein.

[0035] The management system may be a robotics system platform for organizing an inventory of missions (also referred to herein as recorded robotic missions), including overlapping, non-robot-dependent tasks among a fleet of robots. Each recorded mission is linked to one or more tasks completed in that mission. Thus, for example, the management system supports accessing (e.g., calling up) a mission or a segment of a mission to accomplish one or more tasks. Aspects of the present disclosure support accessing a mission or a segment of a mission to accomplish a single task or a combination of tasks.

[0036] For robot-independent tasks shared across robot-specific recorded mission files, the management system can use the robot-independent task as an "anchor" to identify recorded missions that can be used for future procedures. For example, user input can specify a desired task (e.g., a robot-independent task) through the management system, and the management system can provide the user with one or more recorded missions that can perform the desired task. In response to user selection of a recorded mission, the management system can provide commands to one or more robotic devices in connection with completing the desired task.

[0037] The management system supports implementations that allow a user to input a desired task without requiring the user to have extensive robotic knowledge. For example, the management system supports input of robot-independent tasks, and the management system can successfully implement a user-desired task without requiring the user to input robot-specific commands and / or instruction codes.

[0038] In an example implementation, a user may desire to have tasks A-D performed with respect to a facility and / or environment. The user may provide input indicating tasks A-D via the management system. The management system may retrieve a mission (or missions) from a mission inventory of recorded missions (e.g., previously completed missions) that can perform tasks A-D, either alone or collectively with another mission. The management system may present the mission (or missions) to the user via a GUI. In some aspects, the management system may perform the mission (or missions) automatically and / or semi-autonomously (e.g., in response to user confirmation).

[0039] In some aspects, the management system may provide a user with multiple candidate missions capable of performing tasks A-D (e.g., individually or collectively). In some cases, the candidate missions may be different. In one example, a first candidate mission may include performing at least tasks A-C, and a second candidate mission may include performing at least task D.

[0040] Aspects of the present disclosure support implementations in which a user is presented with options to select a candidate mission (or combination of candidate missions) in association with performing tasks A-D. In some alternative and / or additional aspects, the management system may provide the user with different selectable mission combinations capable of performing tasks A-D.

[0041] Aspects of the present disclosure may be implemented by a management system (e.g., system 100 described below with reference to FIG. 1, operation management system 200 and / or robot management system 232 described below with reference to FIG. 2, etc.).

[0042] In some embodiments, the metric data for a completed mission may include the quality of data acquired in association with the completed mission. For example, the management system may support evaluation of images of equipment and measurement devices (e.g., gauges, joints, etc.) for resolution and clarity. In some embodiments, the management system may support machine learning models capable of generating a confidence index in response to processing the images. In some other embodiments, the metric data may include a time duration associated with completing the mission (e.g., the time required to complete the mission), a power consumption associated with completing the mission (e.g., the power required to complete the mission), a robot usage cost associated with completing the mission (e.g., wear and tear on the robot), a cost associated with performing the mission, and / or an indication of whether the completed robotic mission involved at least one manual intervention by an operator.

[0043] As described herein, the management system may collect and record metadata associated with recorded missions. In some aspects, the metadata may include metric data as described herein. The term "recorded mission" may refer to a completed mission (e.g., a fully completed mission, a partially completed mission, a failed mission, a mission in progress, etc.) for which corresponding data has been recorded in a data repository (e.g., a mission file data record).

[0044] In some other aspects, the metadata may include contextual information related to the recording and / or replay of a mission (e.g., the context in which the mission was recorded and / or replayed). For example, the contextual information may include an indication of the status of the facility when the mission was performed. In one example, the contextual information may include an indication of whether the mission was performed during a plant operation (e.g., normal plant operation), a maintenance operation, a calibration operation, a test operation, etc.

[0045] The metadata may include environmental factors associated with the recorded mission, non-limiting examples of which include lighting conditions (e.g., the amount of available light or lack thereof), temperature conditions, noise conditions, and ambient weather conditions (e.g., sunny, cloudy, rainy, windy, etc., for missions conducted in an exterior environment).

[0046] The metadata can include facility state information and / or contextual information related to the facility during previously recorded missions. In some embodiments, the management system can generate embeddings from multiple data points corresponding to the facility. The management system can keep track of the embeddings for comparison with one or more subsequent embeddings representing subsequent states of the facility (e.g., the current state of the facility).

[0047] In some embodiments, for previously recorded missions, the metadata may include contextual information associated with the robot (or robots) that performed the mission. Non-limiting examples of contextual information associated with a robot include battery quality, odometer, weight, time since last maintenance, or other factors that may affect mission performance from the robot's perspective.

[0048] In some embodiments, the management system can use metadata in connection with tracking mission success and / or failure in one or more contexts. For example, the metadata can include an indication of whether a recorded mission met or did not meet one or more target criteria. In one example, for a recorded mission that did or did not meet one or more target criteria, the metadata can include contextual information for the recorded mission, contextual information for one or more tasks in the recorded mission, and / or contextual information associated with a facility associated with the recorded mission.

[0049] As described herein, the management system can determine and record metric data as metadata for a recorded mission file. The management system can use (e.g., with reference to) the metric data to recommend a mission (or missions) in a particular context. In one example, the management system can determine the current context based on the urgency of the task to be performed, time considerations or time constraints (e.g., before a facility shutdown at 9:00 PM), a target accuracy level, other tasks the robot is assigned to or will be assigned to perform, conditions related to the facility where the task is to be performed, etc.

[0050] In an example implementation, the management system can extract metric data corresponding to the recorded missions from the metadata of the recorded mission files. Based on the metric data corresponding to the recorded missions, the management system can match one or more of the recorded robotic missions (and / or tasks included therein) with the current context.

[0051] In another example implementation, a first recorded mission (e.g., Mission 1) may have a higher quality picture compared to a second recorded mission (e.g., Mission 2). Further, for example, the first mission may have a relatively higher cost associated with completion (e.g., uses more battery life) and / or a relatively longer time to complete compared to the second recorded mission. In one example, the management system may recommend and / or select (e.g., autonomously select) the first mission if accuracy of measurements (e.g., from captured images) is of higher priority. In another example, the management system may recommend and / or select (e.g., autonomously select) the second mission if completion time and / or battery life are of higher priority.

[0052] Additionally or alternatively, the management system may select (e.g., semi-autonomously select) the first or second mission in combination with operator confirmation. For example, the management system may provide a recommended mission (or missions) to an operator (e.g., a human operator), and the management system may command one or more robotic devices to perform the mission in response to operator confirmation. In some cases, the management system may store the decision in a machine learning model, and the management system may utilize the machine learning model to improve decision-making (e.g., recommended missions) for future tasks.

[0053] Aspects of the present disclosure are described in the context of environments such as industrial facilities, processing facilities, processing plants, factories, etc. However, it should be understood that embodiments of the present disclosure are not limited to deployment in these types of environments. In some cases, the techniques described herein support building missions based on mission metrics in any type of environment.

[0054] An example process for supporting building robotic missions based on mission metrics is described herein. Aspects of the present disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and process flows.

[0055] The management system may be a robotics system platform that includes a robot-independent GUI that supports building robotic missions, controlling robotic devices based on the robotic missions, and recording the robotic missions. The GUI may include a robot "driver." The robot driver may provide functionality similar to, for example, a printer driver. For example, the robot driver may translate robot-independent commands issued in the GUI into proprietary commands tailored to specific robot hardware. In one example, the robot driver may translate robot-independent commands issued in the GUI (e.g., capturing an image of a measurement instrument) into vendor-specific commands that are compatible with controlling a robotic device associated with the vendor. In some embodiments, the vendor-specific commands are in a format that is compatible with controlling a robotic device associated with the vendor, and the format may be incompatible with controlling a robotic device associated with a different vendor.

[0056] Aspects of the present disclosure support recording commands (e.g., robot-independent commands) issued by a management system in connection with controlling a robotic device. In some aspects, the management system can record the commands in a data repository of recorded mission files. The management system can process the mission files in connection with finding common tasks among different recorded robotic missions. The tasks may be robot-specific tasks.

[0057] In some example implementations, the management system may support search requests that indicate target tasks to be performed with respect to a facility and / or environment. For example, in response to a search request that indicates a target task (or target goal), the management system may provide one or more recorded robotic missions that are capable of achieving the target task (or target goal).

[0058] In one example, a user can operate a robot-independent GUI to cause a robotic device, such as a flyable drone, to inspect (e.g., capture an image of) three target gauges. The user (or another user) can also operate the robot-independent GUI to cause another robotic device, such as a quadruped robot, to inspect the same three target gauges (e.g., with or without performing other tasks). The management system can distinguish between commands that do not translate between two robots (e.g., to raise a flyable drone to a particular altitude) and commands that do translate between the robots (e.g., to take a picture of a gauge located at coordinates specified in the command).

[0059] The management system can identify any commands that translate between different robots as common tasks across recorded missions. For example, the management system can identify the task of inspecting three gauges as a common task across recorded missions. Thus, for example, if a user (or another user) wants to inspect three gauges again, the management system can provide a search function (e.g., via a GUI) to identify recorded robot missions that inspect the three particular gauges. For example, based on user input indicating a goal task, the management system can identify and provide to the user recorded robot missions (and corresponding mission files) that can complete at least the goal task (e.g., the recorded robot missions can perform additional tasks other than the goal task).

[0060] Aspects of the present disclosure support implementations in which finding common tasks across recorded missions is manual and / or automatic. For example, the management system can autonomously and / or semi-autonomously (e.g., with user feedback) find common tasks across different recorded missions and generate a mapping between the common tasks and the different recorded missions.

[0061] The management system can maintain a database reflecting various robotic types of robotic devices and their respective capabilities. In some examples, each robot type may correspond to a mode of mobility (e.g., fly, crawl, walk, etc.) or payload (e.g., headlights, arms, cameras, other sensors, etc.) associated with the robotic device. In one example, the database can include an indication of the mode of mobility and / or payload for each robotic device. The database can include indicators of common capabilities linked across different robotic devices and / or different robot types.

[0062] Aspects of the present disclosure support automated approaches that support matching waypoints (e.g., navigation points) to recorded robotic missions, captured gauges (e.g., using machine learning applied to captured images of gauges), etc. In some cases, for tasks that the management system "suspects" to be shared across different recorded robotic missions, the management system can present a notification of suggested commonality to the user. For example, the suggested commonality may include a notification of different robotic missions that have one or more overlapping tasks. In one example, the suggested commonality may include a notification of robotic missions that can perform the same task (or tasks).

[0063] Based on a response from the user (e.g., user input accepting or rejecting the commonality suggested by the management system), the management system can store the commonality in a database. In some embodiments, the management system can generate additional robotic missions based on overlapping tasks. In one example, for two different recorded robotic missions that have overlapping tasks, the additional robotic mission can include the overlapping tasks but not any tasks that did not overlap between the different recorded robotic missions.

[0064] The terms used below may be interpreted in the broadest reasonable manner, even though they are used in conjunction with the detailed description of specific examples of this disclosure. Indeed, although certain terms may be emphasized below, any term intended to be interpreted in any limited manner is clearly and specifically defined as such in the detailed description provided herein.

[0065] Robot fleet management system

[0066] 1 illustrates an example of a system 100 according to embodiments of the present disclosure. In some examples, the system 100 may support the construction and management of robotic missions related to an environment 111 and / or a facility 113. The system 100 may be a robotic management system (also referred to herein as a robotic system platform) or a motion management system, exemplary embodiments of which are described later in this specification.

[0067] In some examples, environment 111 may correspond to one or more areas contained within facility 113. In some alternative and / or additional examples, environment 111 may correspond to one or more areas surrounding (e.g., external to) facility 113. Additionally or alternatively, example aspects of the technology described herein may be applied to other environments in which equipment may be monitored, maintained (e.g., repaired), and / or operated, such as, but not limited to, urban environments, residential environments, commercial locations (e.g., office buildings or businesses), hospital environments (e.g., in hospital rooms), laboratory environments (e.g., labs, clean rooms, etc.).

[0068] In some embodiments, system 100 may be a distributed process control system capable of executing and controlling processes related to manufacturing, conversion, production, system monitoring, equipment monitoring, other operations, etc. In some examples, system 100 may be a distributed process control system including a controller (e.g., implemented by device 105 and / or server 110) connected to equipment 123 monitored by measurement equipment 125 described herein. In some other examples, system 100 may be a distributed control system including a controller (e.g., implemented by device 105 and / or server 110) capable of controlling device 105 to extract measurement information from measurement equipment 125. Device 105 may support extracting the measurement information by capturing, processing, and analyzing images of measurement equipment 125.

[0069] The system 100 may include a device 105 (or multiple devices 105), a server 110, a database 115, a communication network 120, an appliance 123, and a measurement device 125. The device 105 may be a wireless communication device (e.g., device 105-b). Non-limiting examples of the device 105 may include, for example, a personal computing device or a mobile computing device (e.g., a laptop computer, a mobile phone, a smartphone, a smart device, a wearable device, a tablet, etc.). In some examples, the device 105 may be operable by or carried by a human user. In some aspects, the device 105 may perform one or more operations autonomously or in combination with input by a user, the device 105, and / or the server 110.

[0070] In some embodiments, device 105 may be a transport device configured to move around environment 111. For example, device 105 may be a motorized robot or drone (e.g., device 105-c through device 105-g), a mobile vehicle, etc. In another example, device 105 may be electronically and / or mechanically coupled to a transport device. In one example, the movement of device 105 (e.g., device 105-c (a motorized robot or drone), device 105 coupled to a transport device, etc.) may be controlled by system 100 (e.g., via commands by device 105 or server 110). In some other embodiments, the movement of device 105 (or a transport device) may be autonomous or semi-autonomous (e.g., based on a schedule or programming).

[0071] Aspects of the present disclosure support instructing devices 105 (e.g., devices 105-c through 105-g) to perform one or more tasks associated with environment 111 or a target area of ​​environment 111. In some cases, system 100 may support implementations that instruct devices 105 (e.g., devices 105-c through 105-g) to autonomously perform one or more missions. In some cases, performing a mission (or combination of missions) may include performing one or more tasks. Additional and / or alternative aspects of the present disclosure support instructing a user, via device 105 (e.g., device 105-b), to perform one or more missions (and included tasks) associated with environment 111 or a target area of ​​environment 111.

[0072] It should be understood that the missions described herein may be performed and completed using a single robotic device (e.g., device 105-c) or multiple robotic devices (e.g., two or more of device 105-c through device 105-g). In some aspects, a mission may be a “composite” mission that includes a task list of multiple tasks. In some cases, a “composite” mission may include multiple missions, each of which may include a respective task list of one or more tasks. In one example, given a task list associated with a mission (e.g., a composite mission), system 100 may support mission selection logic that commands and transmits to device 105 (e.g., device 105-c) or multiple devices 105 (e.g., device 105-c through device 105-g) to perform the tasks included in the task list.

[0073] In some aspects, the mission selection logic may support instructing multiple devices 105 to perform tasks simultaneously. In some other aspects, the mission selection logic may support instructing multiple devices 105 to perform tasks based on an ordering. In one example, the ordering may be an order in which a first task is completed, followed by one or more subsequent tasks. In some other examples, the mission selection logic may support instructing multiple devices 105 to perform tasks based on an ordering in which tasks performed by different devices 105 partially overlap (e.g., in terms of time, percentage of task completion, etc.).

[0074] Embodiments of the mission selection logic described herein may support a single robotic device (e.g., device 105-c) performing a set of actions assigned to the robotic device. In some other embodiments, the mission selection logic may support weighting and consider concurrent robotic tasks to find the best combination of robotic tasks to complete a larger mission (e.g., a composite mission). For example, the mission selection logic may apply a respective weighting factor to the robotic tasks. In one example, the weighting factor for a given robotic task may be the robotic task's contribution to completing the larger mission. In some embodiments, the weighting factor may indicate the relative importance (e.g., priority) of the robotic task in relation to completing the larger mission.

[0075] It should be understood that example implementations described herein using a single robotic device to perform one or more missions may support implementations using multiple robotic devices to perform one or more missions (e.g., a single multi-robot complex mission, multiple multi-robot complex missions, etc.) Example aspects of missions and tasks are described later in this specification.

[0076] An exemplary task includes capturing images of one or more pieces of equipment 123 (e.g., equipment 123-a, equipment 123-b, etc.). Another exemplary task may include capturing images of measurement devices 125 (e.g., measurement devices 125-a, measurement devices 125-b, etc.). In some cases, a task may include measuring readings on one or more measurement devices 125 (e.g., by extracting data from the measurement devices 125 based on the captured images, etc.). Other non-limiting examples of tasks include general operation of equipment 123 associated with facility 113, maintaining equipment 123 (e.g., maintenance operations), repairing equipment 123 (e.g., repair operations), lifting and / or moving equipment 123, etc.

[0077] Server 110 may be, for example, a cloud-based server. In some embodiments, server 110 may be a local server connected to the same network (e.g., LAN, WAN) associated with device 105. Database 115 may be, for example, a cloud-based database. In some embodiments, database 115 may be a local database connected to the same network (e.g., LAN, WAN) associated with device 105 and / or server 110. Database 115 can assist in data analysis, machine learning, and AI processes.

[0078] Database 115 may include an inventory of devices 105 included in system 100. In some embodiments, database 115 may include data records of capabilities, functions, vendor information, and operational status associated with each of devices 105. For example, database 115 may include an indication that device 105-b is a mobile computing device and devices 105-c through 105-g are robotic devices or drones. Database 115 may include data records of each mobility type (e.g., fly, crawl, walk, etc.) and / or payload type (e.g., headlight, arm, camera, other sensor, etc.) associated with devices 105-c through 105-g.

[0079] In some examples, each device 105 may include an identification tag (e.g., electronic and / or physical). The identification tag may include identification information (e.g., a tag ID including a combination of letters, numbers, and / or symbols) and / or configuration information (e.g., device type, device parameters, device characteristics, device features, device capabilities, etc.) associated with the device 105. The database 115 may include data records associating each identification tag with a corresponding device 105.

[0080] System 100 may include data records in mission files 117. Mission files 117 may be stored in database 115, another database (e.g., different from database 115), or some other memory (e.g., memory 140, memory 165, etc.). In some examples, mission files 117 may correspond to missions described herein (e.g., general operation, maintenance operation, repair operation, etc. of equipment 123). In one example, each mission file 117 may include a corresponding mission and one or more tasks associated with the mission.

[0081] The communications network 120 may facilitate machine-to-machine communications between any of the devices 105, the server 110, the equipment 123, the measurement equipment 125, or one or more databases (e.g., the database 115). The communications network 120 may include any type of known communications medium or collection of communications media and may use any type of protocol to transfer messages between endpoints. The communications network 120 may include wired communications technologies, wireless communications technologies, or any combination thereof.

[0082] The Internet is an example of a communications network 120 comprising an Internet Protocol (IP) network of multiple computers, computing networks, and other communications devices located in multiple locations, and the components (e.g., computers, computing networks, communications devices) within communications network 120 may be connected via one or more telephone systems and other means. Other examples of communications network 120 may include, but are not limited to, a standard Plain Old Telephone System (POTS), an Integrated Services Digital Network (ISDN), a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless LAN (WLAN), a Session Initiation Protocol (SIP) network, a Voice over Internet Protocol (VoIP) network, a cellular network, and any other type of packet-switched or circuit-switched network known in the art. In some cases, communications network 120 may include any combination of networks or network types. In some aspects, communications network 120 may include any combination of communications media, such as coaxial cable, copper cable / wire, fiber optic cable, or antennas, for communicating data (e.g., sending / receiving data).

[0083] In one example where system 100 is associated with an industrial facility (e.g., a processing facility, a processing plant, a factory), equipment 123 can include any equipment associated with the industrial facility. For example, equipment 123 can include any type of equipment having a measurable parameter. In some embodiments, the measurable parameter can be associated with the performance of equipment 123 and / or the equipment's 123 resource usage (e.g., power, gas, water, etc.). Examples of measurable parameters include, but are not limited to, pressure, fluid level, mass flow rate (e.g., kg / H), temperature, power usage, gas usage, etc. In some other embodiments, the measurable parameter can be associated with the available or remaining content in equipment 123 (e.g., available power, available liquid, available gas, etc.).

[0084] Measurement device 125 (e.g., measurement device 125-a, measurement device 125-b, etc.) may include a field device capable of monitoring or measuring a parameter associated with equipment 123. In some embodiments, measurement device 125 may include a meter (e.g., an analog meter dial, an analog meter, an analog gauge, a digital meter, a digital gauge, a level meter, etc.) corresponding to a parameter value measurable by measurement device 125. The meter may be located, for example, on a meter face of measurement device 125. In some other embodiments, measurement device 125 may include multiple meter dials, each corresponding to a parameter value measurable by measurement device 125. In some cases, the multiple meter dials may correspond to the same parameter value but to different respective equipment 123.

[0085] In some embodiments, for meters such as analog meter dials, the meter dial may include a measurement indicator (e.g., a dial indicator) that points to a number (or marking) on ​​the meter dial based on a parameter value (e.g., device performance, resource usage, available content on device 123, etc.) measured by measurement device 125. In some other embodiments, for meters such as digital displays, one or more numeric indicators on the digital display may reflect a parameter value measured by measurement device 125.

[0086] In some cases, the measurable parameter may be related to available or remaining contents (e.g., available liquid, available gas, etc.) in the measurement device 125. For example, the measurement device 125 may be a standalone device capable of indicating available resources in the measurement device 125, such as available energy, available liquid, available gas, available time (e.g., as in the case of a timer), etc.

[0087] In one example where system 100 is associated with an industrial facility (e.g., a processing facility, a process plant, a factory, etc.), a hospital environment, a laboratory environment, etc., measurement devices 125 included in environment 111 may include flow meters, pressure gauges, level gauges, temperature gauges, etc. In some cases, measurement devices 125 may be mechanically coupled to equipment 123.

[0088] System 100 can be associated with a residential or commercial environment. For example, the metering devices 125 included in environment 111 can include any type of metering device 125 capable of providing measured or tracked information. For example, in a residential or commercial environment, metering devices 125 can include analog utility meters (e.g., utility meters not connected to communication network 120), such as gas meters, electricity meters, etc. In some cases, metering devices 125 can include parking meters with analog measurement indicators (e.g., analog meter dials, pointers) or digital measurement indicators (e.g., digital displays).

[0089] In some examples, each measurement device 125 may include an identification tag that may include identification information (e.g., a tag ID including a combination of letters, numbers, and / or symbols) and / or configuration information (e.g., device type, device parameters, device characteristics, device features) associated with the corresponding measurement device 125.

[0090] In some embodiments, a metering device 125 may not be connected to a communications network 120. For example, in some cases, a metering device 125 may not support network capabilities for communicating over a communications network 120. Embodiments of the present disclosure may include missions that include tasks such as automated (and / or semi-automated) meter reading of a metering device 125 based on a captured image of the metering device 125, repair and / or replacement of the metering device 125, examples of which are invented herein.

[0091] In some other aspects, measurement device 125 may support network capabilities for communication with other devices (e.g., device 105, server 110, etc.) using communication network 120 (e.g., via protocols supported by communication network 120). For example, measurement device 125 may support communicating measurement data to device 105, server 110, etc. via communication network 120. In some examples, measurement device 125 may include Internet of Things (IoT) devices, including analog meter dials, digital meters, and / or level meters described herein.

[0092] Aspects of the present disclosure may, for example, use captured images of a metering device 125 to support missions that include tasks such as automatic meter readings of the metering device 125 (e.g., of a meter included in the metering device 125) when communications over the communication network 120 are interrupted (e.g., due to network issues, blocked network communications, a malfunctioning transceiver in the metering device 125, etc.). For example, when communications over the communication network 120 are interrupted, the system 100 and / or an operator may develop and implement missions that include tasks such as automatic inspection of equipment 123, automatic meter readings of the metering device 125, etc.

[0093] In various embodiments, settings for any of the devices 105, server 110, database 115, communication network 120, equipment 123, and measurement devices 125 can be configured and modified by any user and / or administrator of system 100. Settings can include thresholds or parameters described herein, as well as settings related to how data is managed. Settings can be configured to be personalized for one or more devices 105, users of devices 105, and / or groups of other entities, and may be referred to herein as profile settings, user settings, or organization settings. In some embodiments, rules and settings can be used in addition to or instead of the parameters or thresholds described herein. In some examples, rules and / or settings can be personalized by users and / or administrators for any variable, threshold, user (user profile), device 105, entity, or group thereof.

[0094] Aspects of device 105 and server 110 are further described herein. Device 105 (e.g., device 105-a) may include image capture device 127, processor 130, network interface 135, memory 140, and user interface 145. In some examples, the components of device 105 (e.g., processor 130, network interface 135, memory 140, user interface 145) may communicate via a system bus (e.g., control bus, address bus, data bus) included in device 105. In some cases, device 105 may be referred to as a computing resource.

[0095] Image capture device 127 may be a standalone camera device or a camera device integrated with device 105. Image capture device 127 may support the capture of still images and / or video. For example, image capture device 127 may support the autonomous capture of images (e.g., still images, video (and video frames thereof), video streams (and video frames thereof), video scans, etc.). In some examples, image capture device 127 may be a camera (e.g., a CCTV camera) installed at a fixed location.

[0096] In some embodiments, image capture device 127 may include a single image sensor or an array of image sensors (not shown). The image sensor may include a photodiode that is sensitive to (e.g., detectable by) light in any frequency band. For example, the image sensor may include any combination of photodiodes, photocathodes, and / or photomultiplier tubes. The image sensor may be configured to detect light within any defined wavelength range (e.g., the visible spectrum, the ultraviolet spectrum, etc.).

[0097] Image capture device 127 may be mechanically mounted to or within the housing of device 105 to allow rotational freedom of image capture device 127 and / or the image sensor. In another example, image capture device 127 may be mounted to any surface or any object. In some embodiments, the camera device may be a spherical camera device (e.g., to provide a spherical field of view).

[0098] Image capture device 127 (and / or image sensor) may include a location sensor configured to record location information associated with image capture device 127 (and / or image sensor). In one example, image capture device 127 may be configured to record and output coordinates, positioning information, orientation information, velocity information, etc. For example, image capture device 127 may include an accelerometer, a GPS transponder, an RF transceiver, a gyroscope sensor, or any combination thereof.

[0099] System 100 may support image processing techniques (e.g., image preprocessing) implemented in any of device 105, server 110, and image capture device 127. Examples of image processing supported by system 100 may include image reading, image resizing, image conversion (e.g., gray to red, green, blue (RGB), hue, saturation, value (HSV) to RGB, blue, green, red (BGR) to RGB, etc.), image enhancement (e.g., filtering with morphological operators), histogram equalization, noise removal, linear contrast adjustment, median filtering, unsharp mask filtering, contrast-limited adaptive histogram equalization (CLAHE), affine transforms (e.g., geometric distortion correction), image transformations (e.g., Fourier transform, Hough transform, wavelet, etc.), color processing, etc.

[0100] In some cases, device 105 may use network interface 135 to send or receive packets to one or more other devices (e.g., another device 105, server 110, database 115, appliance 123, measurement device 125 (if measurement device 125 supports network communications)) over communications network 120. Network interface 135 may include, for example, any combination of network interface cards (NICs), network ports, associated drivers, etc. Communications between components of device 105 (e.g., processor 130, memory 140) and one or more other devices connected to communications network 120 (e.g., another device 105, database 115, appliance 123, measurement device 125 (if measurement device 125 supports network communications)) may flow through network interface 135, for example.

[0101] Processor 130 may correspond to one or many computer processing devices. For example, processor 130 may include a silicon chip, such as an FPGA, an ASIC, any other type of IC chip, a collection of IC chips, etc. In some embodiments, the processor may include a microprocessor, a CPU, a GPU, or multiple microprocessors configured to execute a set of instructions stored in a corresponding memory (e.g., memory 140 of device 105). For example, when executing a set of instructions stored in memory 140, processor 130 may enable or perform one or more functions of device 105.

[0102] Memory 140 may include one or more computer memory devices. Memory 140 may include, for example, random access memory (RAM) devices, read-only memory (ROM) devices, flash memory devices, magnetic disk storage media, optical storage media, solid-state storage devices, core memory, buffer memory devices, combinations thereof, etc. Memory 140 may correspond to a computer-readable storage medium in some examples. In some embodiments, memory 140 may be internal or external to device 105.

[0103] The processor 130 can utilize the data stored in the memory 140 as a neural network (also referred to herein as a machine learning network). The neural network may include a machine learning architecture. In some embodiments, the neural network may be or include an artificial neural network (ANN). In some other embodiments, the neural network may be or include any machine learning network, such as, for example, a deep learning network, a convolutional neural network (CNN), etc. Some elements stored in the memory 140 are described as instructions or instruction sets, and some functions of the device 105 may be implemented using machine learning techniques. In some embodiments, the neural network may include a region-based CNN (RCNN), Fast RCNN, Faster RCNN, and / or Mask RCNN.

[0104] Memory 140 may be configured to store instruction sets, neural networks, and other data structures (e.g., as illustrated herein) in addition to temporarily storing data for processor 130 to perform various types of routines or functions. For example, memory 140 may be configured to store program instructions (instruction sets) executable by processor 130 to provide the functionality of machine learning engine 141 described herein. Memory 140 may also be configured to store data or information usable or accessible by instructions stored in memory 140. One example of data that may be stored in memory 140 for use by its components is data model 142 (e.g., a neural network model, an object detection model, or other model described herein) and / or training data 143 (also referred to herein as training data and feedback).

[0105] The machine learning engine 141 may include a single engine or multiple engines. The device 105 (e.g., the machine learning engine 141) may utilize one or more data models 142 to recognize and process information obtained from other devices 105, the server 110, and the database 115. In some embodiments, the device 105 (e.g., the machine learning engine 141) may update one or more data models 142 based on learned information contained in the training data 143. In some embodiments, the machine learning engine 141 and the data models 142 may support forward learning based on the training data 143. The machine learning engine 141 may access and use one or more data models 142. The machine learning engine 141 may support image annotation, image augmentation, model selection, model training, performance analysis, and fine-tuning using any combination of the data models 142 and / or the training data 143.

[0106] The data model 142 may be constructed and updated by the machine learning engine 141 based on the training data 143. The data model 142 may be provided in any number of formats or forms. Non-limiting examples of the data model 142 include a decision tree, a support vector machine (SVM), a nearest neighbor, and / or a Bayesian classifier. Other exemplary aspects of the data model 142, such as the generation (e.g., construction, training) and application of the data model 142, are described with reference to the figure legends herein.

[0107] According to aspects of the present disclosure, the data model(s) 142 may include an object detection model. In some aspects, the data model(s) 142 may be a single object detection model trained to detect equipment 123 and / or measurement devices 125 (e.g., meters) included in captured images of the environment 111. Aspects of the present disclosure can support training the data model 142 only once, even when equipment 123 and measurement devices 125 having similar visual characteristics as compared to registered equipment 123 and measurement devices 125 are added to the environment 111 for management by the system 100 (or registered with the system 100).

[0108] The data model 142 may support the management and detection of equipment 123 and / or measurement devices 125 of various types, designs, and / or sizes. The data model 142 may support the detection of equipment 123 and measurement devices 125 included in captured images regardless of the capture angle and / or lighting conditions associated with the capture of the image. In some embodiments, the data model(s) 142 may support the detection of equipment 123 and measurement devices 125 based on features that have the highest discrimination between the equipment 123 and measurement devices 125. In some embodiments, the data model(s) 142 may support image analysis techniques such as two-dimensional (2D) and three-dimensional (3D) object recognition, image classification, image segmentation, motion detection (e.g., single particle tracking), video tracking, 3D pose estimation, etc.

[0109] In some examples, training data 143 may include aggregate images of environment 111, facility 113, equipment 123, and measurement device 125. For example, training data 143 may include captured images of environment 111 (e.g., images captured during a mission or task, etc.) and registered images of measurement device 125 (e.g., captured during a previous mission or task, etc.). In some embodiments, training data 143 may include aggregated measurement data, such as aggregated measurement information (e.g., measurements) associated with measurement device 125 over one or more time periods. In some other embodiments, training data 143 may include aggregated measurement information associated with one or more missions or tasks implemented in association with environment 111 and measurement device 125. In some other examples, the training data 143 may include parameters and / or configurations of devices 105 (e.g., robotic devices) used in connection with the performance of a mission or task, scheduling information associated with the mission or task, locations (e.g., of devices 105, equipment 123, etc.) associated with the mission or task, paths for performing the mission (e.g., paths taken by devices 105 in connection with the performance of the mission), equipment 123 and / or measurement devices 125 monitored in connection with the mission, etc.

[0110] The machine learning engine 141 may be configured to analyze real-time and / or aggregated information (e.g., captured images of the equipment 123 and / or measurement devices 125, measurements of the measurement devices 125, calculated performance of the equipment 123, etc.). In some cases, the machine learning engine 141 may support the calculation of predictive information (e.g., measurements of one or more measurement devices 125, performance of the equipment 123, etc.). For example, the machine learning engine 141 can predict measurement information of the measurement devices 125 based on historical data associated with the measurement devices 125 (e.g., previously recorded measurement information). In some embodiments, the machine learning engine 141 can predict performance trends associated with the equipment 123 (e.g., predicted efficiency, predicted energy usage, predicted lifespan, predicted instances of repair, predicted flow rate, predicted pressure levels, etc.). In some cases, the system 100 can adjust operating parameters associated with the equipment 123 included in the environment 111 and / or notify an operator of any faults associated with the equipment 123 based on the actual measurement information and / or predicted measurement information. In some cases, the system 100 may adjust mission parameters and / or tasks related to managing the facility 113 and / or managing the equipment 123 based on actual measurement information and / or predicted measurement information.

[0111] In some other embodiments, the machine learning engine 141 may be configured to analyze and adjust parameters associated with managing the facility 113 based on data collected during a mission or task. For example, the machine learning engine 141 may assign tasks and / or structure missions based on results associated with completed missions or tasks. In some cases, based on the results (e.g., a failed result due to poor image quality of a captured image, a failed result due to incomplete repair of equipment 123, etc.), the server 110 may control a device 105 (e.g., device 105-c, etc.), control transport equipment, and / or output a notification to an operator (e.g., via device 105-a, etc.) to repeat the mission or task.

[0112] For example, for a mission that includes capturing images of equipment 123 and / or measuring device 125, server 110 may modify capture settings (e.g., capture angle, time of day, etc.) associated with capturing images of equipment 123 and / or measuring device 125. In some embodiments, server 110 may control device 105, control transport equipment, and / or output notifications to an operator to recapture images of equipment 123 and / or measuring device 125. In another example, for a mission that includes repairing equipment 123 (e.g., equipment 123-a) and / or measuring device 125 (e.g., measuring device 125-a), server 110 may modify the type of robotic device used for the mission. For example, a failed mission may have been implemented using a non-flyable robotic device (e.g., device 105-c). Server 110 may repeat the mission (or one or more tasks of the mission) using a flyable robotic device (e.g., device 105-e). In some other examples, a failed mission may have been implemented using a robotic device (e.g., device 105-f) where a defect with the payload (e.g., lack of an articulated arm or ability to control a tool) resulted in the failed mission. The server 110 can repeat the mission (or one or more tasks of the mission) using a robotic device (e.g., device 105-g) with a payload (e.g., an articulated arm, a drill, etc.) that can complete the mission.

[0113] Aspects of the present disclosure support, for example, mission and / or task selection by server 110 without user indication of robot type and / or robot capabilities. For example, server 110 can select and recommend missions to a user based on contextual information related to a target task indicated by the user. In some aspects, server 110 can select and recommend missions from among recorded missions corresponding to mission file 117 based on contextual information of the recorded missions. Exemplary aspects of contextual information are described later herein.

[0114] The machine learning engine 141 can analyze any of the information described herein historically or in real time. The machine learning engine 141 can be configured to receive or access information from the device 105, the server 110, the database 115, the equipment 123, and / or the measurement device 125 (e.g., via image capture and image analysis). The machine learning engine 141 can construct any number of profiles, such as, for example, a profile associated with the system 100 (e.g., a profile associated with a facility), a profile associated with a mission, a profile associated with the equipment 123, or a profile associated with the measurement device 125 (e.g., including the configuration information described herein), using automated processing, artificial intelligence, and / or input from one or more users associated with the device 105. A profile may be, for example, a configuration profile, a performance profile, etc. The machine learning engine 141 can determine, manage, and / or combine information related to the configuration profile using automated processing, artificial intelligence, and / or input from one or more users of the device 105. In some embodiments, the machine learning engine 141 can construct any number of missions and configure one or more tasks for each mission.

[0115] The machine learning engine 141 can determine the configuration profile information based on user interaction with the information. The machine learning engine 141 can update the configuration profile (e.g., continuously, periodically, based on trigger conditions, etc.) based on relevant new information. The machine learning engine 141 can receive new information from any device 105, server 110, database 115, appliance 123, or measurement device 125 (e.g., via image capture, or via communication network 120 if the measurement device 125 supports network communication). The profile information can be organized and categorized in various ways. In some embodiments, the organization and categorization of the configuration profile information can be determined by automated processing, by artificial intelligence, and / or by one or more users of the device 105.

[0116] Examples of user interaction with information may include mission selection, task selection, and / or parameter selection (e.g., mission parameters, task parameters, etc.) based on the type of facility 113, the type of environment 111, the type of equipment 123, the type of measurement device 125, etc. Example aspects of the present disclosure support implementations in which the user interaction does not indicate the robot type and / or robot capabilities.

[0117] The machine learning engine 141 can generate, select, and execute appropriate processing decisions. Exemplary processing decisions may include analysis of measurement information (e.g., historical, real-time, etc.), predicted measurement information, device 105 configuration, equipment 123 configuration, measurement equipment 125 configuration, mission configuration, task configuration, etc. Processing decisions may be processed automatically by the machine learning engine 141 with or without human input.

[0118] The machine learning engine 141 can store historical information (e.g., reference data, measurement data, predicted measurement data, configurations, etc.) in memory 140 (e.g., a database included in memory 140). Data in the database of memory 140 can be updated, modified, edited, or deleted by the machine learning engine 141. In some embodiments, the machine learning engine 141 can support continuous, periodic, and / or batch fetching of data and data aggregation (e.g., from the equipment 123, the measurement devices 125 (via the image capture and / or communication network 120), the central controller, the devices 105, etc.).

[0119] The apparatus 105 may render a presentation (e.g., visually, audibly, using haptic feedback, etc.) of an application 144 (e.g., browser application 144-a, application 144-b). The application 144-b may be an application related to constructing, executing, controlling, and / or monitoring a mission (or its tasks). For example, the application 144-b may enable control of the device 105, the equipment 123, and / or the measurement device 125 described herein. In some embodiments, the application 144-b may be an application for constructing a robotic mission based on mission metrics (e.g., of previously recorded missions), a mission inventory, and target mission metrics (e.g., as input by an operator).

[0120] Application 144-b may support representation of missions, mission objectives, and / or mission tasks in a human-readable format. In some embodiments, the human-readable format may be different from a format (e.g., a programming language, program code, etc.) that conforms to a command protocol for controlling a robotic device (e.g., device 105-c, etc.). For example, the human-readable format may be different from a vendor-specific command protocol for controlling a robotic device. Application 144-b may support input of robot-independent data, commands, and / or representations. In some embodiments, application 144-b may support user input of tasks in “plain language” (e.g., “Determine the efficiency of all equipment in a facility,” “Identify equipment that is likely to fail in the next month,” “Replace X component in all equipment that is likely to fail in the next month,” etc.).

[0121] In some embodiments, the term "robot-agnostic" may refer to a definition of a mission goal, task (e.g., robot task), or mission in an internal format common to system 100. In some cases, the term "robot-agnostic" may refer to a definition of a mission goal, task, or mission without an indication of robot type and / or robot function.

[0122] In one example, device 105 can render a presentation via user interface 145 (also referred to herein as a robot-independent interface). User interface 145 may include, for example, a display (e.g., a touchscreen display), an audio output device (e.g., a speaker, a headphone connector), or any combination thereof. In some embodiments, application 144 may be stored on memory 140. In some cases, application 144 may include a cloud-based application or a server-based application (e.g., supported and / or hosted by database 115 or server 110). Settings for user interface 145 may be partially or fully customizable and may be managed by one or more users, by an automated process, and / or by artificial intelligence.

[0123] In one example, any of the applications 144 (e.g., browser application 144-a, application 144-b) may be configured to receive data in an electronic format and present the data content via a user interface 145. For example, an application 144 may receive data from another device 105, a server 110, a database 115, an appliance 123, and / or a measurement device 125 (if it supports network communications) via a communications network 120, and the device 105 may display the content via a user interface 145.

[0124] Database 115 may include a relational database, a centralized database, a distributed database, an operational database, a hierarchical database, a network database, an object-oriented database, a graph database, a NoSQL (non-relational) database, etc. In some embodiments, database 115 may, for example, store and provide access to any of the stored data described herein.

[0125] Server 110 may include a processor 150, a network interface 155, database interface instructions 160, and memory 165. In some examples, the components of server 110 (e.g., processor 150, network interface 155, database interface 160, memory 165) may communicate via a system bus (e.g., control bus, address bus, data bus) included in server 110. Processor 150, network interface 155, and memory 165 of server 110 may include example aspects of processor 130, network interface 135, and memory 140 of apparatus 105 described herein.

[0126] For example, processor 150 may be configured to execute a set of instructions stored in memory 165, upon which processor 150 may enable or perform one or more functions of server 110. In some examples, server 110 may use network interface 155 to send or receive packets to one or more other devices (e.g., device 105, database 115, another server 110) over communications network 120. Communications between components of server 110 (e.g., processor 150, memory 165) and one or more other devices (e.g., device 105, database 115, appliance 123, measurement device 125, etc.) connected to communications network 120 may flow through network interface 155, for example.

[0127] In some examples, database interface instructions 160 (also referred to herein as database interface 160), when executed by processor 150, may enable server 110 to send data to and receive data from database 115. For example, database interface instructions 160, when executed by processor 150, may enable server 110 to generate database queries, provide one or more interfaces for a system administrator to define database queries, send database queries to one or more databases (e.g., database 115), receive responses to the database queries, access data associated with the database queries, and format responses received from the databases for processing by other components of server 110.

[0128] Memory 165 may be configured to store instruction sets, neural networks, and other data structures (e.g., as illustrated herein) in addition to temporarily storing data for processor 150 to perform various types of routines or functions. For example, memory 165 may be configured to store program instructions (instruction sets) executable by processor 150 to provide the functionality of machine learning engine 166. One example of data that may be stored in memory 165 for use by its components is data models 167 (e.g., any data models described herein, object detection models, neural network models, etc.) and / or training data 168.

[0129] Data model 167 and training data 168 may include examples of aspects of data model 142 and training data 143 described with reference to device 105. Machine learning engine 166 may include examples of aspects of machine learning engine 141 described with reference to device 105. For example, server 110 (e.g., machine learning engine 166) may utilize one or more data models 167 to recognize and process information obtained from device 105, another server 110, database 115, appliance 123, and / or image capture device 127. In some embodiments, server 110 (e.g., machine learning engine 166) can update one or more data models 167 based on learned information included in training data 168.

[0130] In some embodiments, components of the machine learning engine 166 may be provided in a separate machine learning engine that communicates with the server 110.

[0131] Aspects of the subject matter described herein may be implemented to realize one or more advantages. The described techniques may support exemplary improvements related to controlling heterogeneous robotic fleets (e.g., robotic fleets of different robot types and capabilities, different robot vendors, etc.). Aspects of the present technology provide for issuing commands to a robotic fleet through robot-independent task instructions. In some cases, aspects of system 100 provide a general method for a user to create a set of commands (e.g., construct a mission) for execution by one or more robotic devices from the robotic fleet. Thus, for example, system 100 supports implementations that reduce or eliminate any operator burden related to cooperative missions between robotic fleets including robots of different types and / or from different vendors, since the operator may provide commands to the robots via robot diagnostic tasks input via system 100. Additionally, for example, system 100 supports data recording of completed robotic missions, and system 100 can configure and / or suggest one or more robotic missions based on robot-independent tasks input by the operator. Thus, for example, the effectiveness of robot-independent tasks input by an operator is not hindered by the operator's personal experience and capabilities (e.g., with respect to the operator's knowledge of robot-specific commands, etc.).

[0132] An exemplary implementation that supports building robotic missions based on a mission inventory is described herein. The system 100 can provide a mission file 117 with data records (also referred to herein as a mission inventory) related to recorded robotic missions. The data records can include metric data 118 (e.g., performance data, time data, cost, etc.) related to the recorded robotic mission, context information 119 related to the recorded robotic mission, and an indication of whether the recorded robotic mission met previous target criteria (e.g., whether the mission was successful). In some embodiments, the mission file 117 can include metadata, which can include any of the metric data 118, the context information 119, and an indication of whether the recorded robotic mission met previous target criteria. Examples of the metric data 118 and the context information 119 are described herein.

[0133] The system 100 can receive user input 146 including instructions for a target task (or tasks) to be performed with respect to the facility 113 and / or the environment 111. Additionally or alternatively, the system 100 can autonomously and / or semi-autonomously identify the target task (e.g., based on an analysis of the results of previously recorded missions), for example, using machine learning techniques described herein. The target task can be a robot-independent task as described herein. In some embodiments, the target task can be a process automation inspection task.

[0134] The system 100 can determine contextual information 147 related to the target task. In some examples, the contextual information 147 can include criteria associated with the user (e.g., urgency of the task, etc.) and / or criteria associated with the facility 113 and / or environment 111 (e.g., time considerations or constraints, etc.).

[0135] The system 100 can search the data records in the mission files 117. The system 100 can identify one or more mission files (e.g., from the mission files 117) that include tasks that match the target task of the user input 146. In some aspects, the system 100 can identify multiple mission files that include completed tasks that match the target task. In some cases, the system 100 can generate ranking information for the identified mission files and provide the mission files based on a ranked order. In some examples, the system 100 can rank the mission files using metric data (e.g., metric data 118) related to mission completion and / or completion of associated tasks.

[0136] System 100 can provide the identified mission file to a user via user interface 145. In some aspects, system 100 can implement the identified mission file in response to user input (e.g., user confirmation, user modification, etc.). Additionally or alternatively, system 100 can automatically implement the identified mission file (or portion thereof) if a confidence value (e.g., as determined by system 100) associated with the identified mission file (or portion thereof) meets a confidence value threshold.

[0137] Described herein are example implementations that support building robotic missions based on user interface 145 (e.g., a robot-independent GUI). In connection with a first mission, system 100 may electronically receive a first set of commands associated with operating a first robotic device (e.g., device 105-c) to perform a first set of tasks. System 100 may record the first set of commands in mission file 117-a. In connection with a second mission (e.g., a subsequent mission), system 100 may electronically receive a second set of commands associated with operating a second robotic device (e.g., device 105-d) to perform a second set of tasks. System 100 may record the second set of commands in mission file 117-b.

[0138] System 100 can identify a third set of tasks common to the first set of tasks and the second set of tasks. In some embodiments, based on the third set of tasks, system 100 can identify at least one robotic capability common to the first robotic device (e.g., device 105-c) and the second robotic device (e.g., device 105-d). System 100 can, for example, generate mission file 117-c including the third set of tasks based on a mapping between the third set of tasks and mission file 117-a (e.g., first set of tasks) and mission file 117-b (e.g., second set of tasks).

[0139] Thus, for example, system 100 may receive a search request (e.g., via user input 146) indicating target task(s). Additionally or alternatively, system 100 may autonomously and / or semi-autonomously identify the target task(s), for example, using machine learning techniques described herein (e.g., based on analyzing the results of previously recorded missions). If system 100 identifies the target task as being included in the third task set, system 100 may provide mission file 117-a and / or mission file 117-b to the user for user selection and / or confirmation. The user may select mission file 117-a and / or mission file 117-b to perform the target task. Additionally or alternatively, system 100 may provide mission file 117-c, and the user may select mission file 117-c in connection with performing the target task.

[0140] Described herein are example implementations that support building robotic missions based on a mission inventory (e.g., data records in mission file 117). System 100 can identify target task(s) to be performed with respect to facility 113 and / or environment 111. For example, system 100 can receive user input 146 including instructions for the target task. Additionally or alternatively, system 100 may autonomously and / or semi-autonomously identify the target task (e.g., based on an analysis of results of previously recorded missions, based on an analysis of the state of each of equipment 123, based on an analysis of the state of facility 113, etc.), for example, using machine learning techniques described herein.

[0141] System 100 may identify mission file 117-a based on the target task. In some embodiments, system 100 may identify one or more recorded tasks in mission file 117-a that at least partially achieve a target goal associated with the target task. In one example, system 100 may determine whether one or more recorded tasks in mission file 117-a achieve a target goal. For example, system 100 may determine whether one or more recorded tasks in mission file 117-a completely overlap with the target task. If system 100 identifies that one or more recorded tasks in mission file 117-a completely achieve a target goal, system 100 may provide commands to a robotic device (e.g., device 105-c) or multiple robotic devices (e.g., device 105-c, device 105-d, etc.) in connection with implementing the one or more recorded tasks. Additionally or alternatively, the system 100 may provide instructions for the mission file 117-a to the user (e.g., via the user interface 145), and the system 100 may provide commands to the robotic device in response to user confirmation of the mission file 117-a.

[0142] In another example, if system 100 identifies that one or more recorded tasks of mission file 117-a do not fully achieve the target goal, system 100 may identify another mission file (e.g., mission file 117-b) based on the target task. System 100 may identify one or more recorded tasks of mission file 117-b that at least partially achieve the target goal associated with the target task(s). In one example, one or more recorded tasks of mission file 117-b may overlap with any target task(s) that are not overlapped by one or more recorded tasks of mission file 117-a. If system 100 identifies that the recorded tasks of mission file 117-a and mission file 117-b fully achieve the target goal, system 100 may provide commands to a robotic device (e.g., device 105-c) or multiple robotic devices (e.g., device 105-c, device 105-d, etc.) in connection with implementing the recorded tasks. Additionally or alternatively, system 100 may provide notification of mission file 117-a and mission file 117-b to a user (e.g., via user interface 145), and system 100 may provide commands to a robotic device in response to user confirmation of mission file 117-a and mission file 117-b.

[0143] In exemplary embodiments described herein for building a robotic mission, the task list can include any amount of tasks, and the task list can include any task type. For example, system 100 can set a task as a goal task to be performed based on a set of criteria. In some examples, system 100 can determine as a target task any task that meets target criteria related to being a tedious (e.g., repetitive), dirty (e.g., unhygienic), dangerous (e.g., posing a safety risk to a human operator), and / or more important (e.g., expensive) task. In some embodiments, system 100 can determine a target task based on a threshold for determining whether the task is a tedious (e.g., repetitive), dirty (e.g., unhygienic), dangerous (e.g., posing a safety risk to a human operator), and / or important (e.g., expensive) task.

[0144] An exemplary threshold for determining (e.g., classifying) a task as tedious (e.g., repetitive) may be a threshold number of times the task is performed over a period of time. An example of a target criterion for determining a task as tedious (e.g., repetitive) may be an indication of whether the candidate task can be successfully performed by a robotic device, whether the candidate task can be performed more efficiently by a robotic device (e.g., performed more times over the same period of time), etc. An example of a target criterion for determining a task as dirty (e.g., unhygienic) may include whether the task is related to waste disposal, waste cleanup, etc. Another example of a target criterion may include a classification related to waste (e.g., hazardous, toxic, etc.).

[0145] Exemplary thresholds for determining a task as dangerous (e.g., presenting a safety risk to a human operator) may include the degree of safety risk to the human operator, the number of times a human operator has been injured performing the task relative to the time duration and / or mission volume, etc. An example of target criteria for determining a task as dangerous may include environmental conditions associated with the task (e.g., accessibility of equipment associated with the task, temperature conditions associated with the equipment and the task, etc.). An exemplary threshold for determining a task as more important (e.g., expensive) may include cost (e.g., financial cost). Note that the threshold and target criteria described herein are by way of example only, and aspects of the present disclosure are not limited thereto.

[0146] 2 illustrates tasks and components of an operations management system 200 for an industrial facility 113. The operation of the industrial facility 113 may include tasks and components related to the operation of the facility, monitoring safety or other conditions within the facility, and maintaining the facility. Overall operations may be managed by an enterprise resource management system 202, which may control tasks such as production planning 204, production accounting 206, maintenance accounting 220, and planning for both turnaround and long-term maintenance 222, for example.

[0147] Tasks for operation and maintenance of the industrial facility 113 may be performed in an operations office or central control room (CCR) 234 or in a technical area 236, such as within a production area of ​​the industrial facility 113 itself.

[0148] Operations-related tasks performed within the office / CCR 234 may include, for example, production scheduling 208 , data reconciliation and production results reporting 210 , production operations management and reporting 212 , data historian 214 , panel operations 216 , and process control 218 .

[0149] Maintenance-related tasks performed within office / CCR 234 can include, for example, maintenance and reliability management and reporting 224, annual maintenance planning and scheduling 226, maintenance operations management and reporting 228, and third-party contractor management 230. Driving-related tasks performed in technical area 236 can include, for example, human-performed driving tasks 238 and robot-performed driving tasks 240. Maintenance-related tasks performed in technical area 236 can include, for example, human-performed maintenance tasks 242 and robot-performed maintenance tasks 244.

[0150] The robot-performed operation tasks 240 and robot-performed maintenance tasks 244, as well as other robot-performed tasks, may be managed by a robot management system 232. Exemplary embodiments of one or more such robot management systems are described in more detail below.

[0151] Robots operating within an operations management system may require additional information and metadata about each robot and robot activity within the system, including robot management, tracking, and analysis of the robots and their activities. For example, a robot may require information about other robots and their activities when multiple robots are collaborating on an activity or when the activities of multiple robots may compete. Figure 3 shows a generalized model 300 of robotic activity within an operations management system based on the ISA-95 standard for "Enterprise Control Systems Integration" (https: / / www.isa.org / standards-and-publications / isa-standards / isa-standards-committees / isa95).

[0152] Management of the robots 110 functioning within the industrial facility 113, including robot-performed operation tasks 240 and robot-performed maintenance tasks 244, may involve activities performed by the robots 110, data related to the robots 110 and the tasks performed, operations performed by the robot management system 232, and communications between the robot management system 232 and the robots 110. Such management may be structured according to a model of robot activity 300.

[0153] The data regarding the robots 110 may include, for example, a robot definition 305 for each robot 110 , a robot capability 310 for each robot 110 , a robot schedule 315 for each robot 110 , and robot performance metrics 320 for each robot 110 .

[0154] Operations performed by the robot management system 232 may include, for example, robot resource management 330, robot definition management 350, detailed robot scheduling 325, robot dispatching 345, robot execution management 360, robot tracking 335, robot data collection 355, and robot performance analysis 340. In a system utilizing robots provided by multiple robot vendors, these operations may reflect differences between the robots provided by different vendors. For example, each robot vendor may make data about each robot available in a different format specific to that robot or vendor. To address this, the robot management system 232 can standardize the data by obtaining and interpreting a subset of the data that may be relevant to a human operator. For example, two robots from different vendors may each have an error regarding a dead end in a mission, but the robots may format and present that error differently depending on their platform. The robot management system 232 can correctly interpret the different information presented by robots from multiple vendors through adapters, such as the robot-specific adapter 644 described below with respect to FIG. 6 .

[0155] During robot resource management 330, the robot management system 232 can update or reference the robot capability information 310 to maintain information about available robot resources and make those resources available for robot-assigned tasks. For example, during the course of performing a mission or task, a robot's capabilities may temporarily change, such as due to a depletion of the robot's battery, damage to or malfunction of one or more tools or sensors attached to the robot, removal or replacement of one or more tools or sensors attached to the robot, or other changes to the robot's capabilities. Additionally, a robot may be unable to move from its current location due to obstacles, environmental conditions (e.g., heavy rain on a robot not protected from wet conditions), damage to the robot's tracks, legs, or other propulsion means, loss of connectivity to the robot management system 232 or other plant systems, etc. Additionally, a robot operating in extreme heat, such as in a desert environment, may overheat robot components. Similarly, a robot operating in extremely cold temperatures may affect the operation of joints or other moving parts. Such extreme temperatures may shorten the life of the robot's battery.

[0156] The robot management system 232 can monitor the status of the robots and maintain robot capability information 310 to reflect such changes and make decisions about how to schedule or reschedule the robot fleet for best performance. In some embodiments, an operator or technician can be responsible for updating the robot capability information 310 in the robot management system 232. Each robot can also (or alternatively) report capability information, including payload status, to the robot management system 232 before a mission is performed. If the robot management system 232 determines that a robot's payload capability is insufficient for the mission, the robot management system 232 can select a different robot for the mission and update the specific robot capability.

[0157] The robot management system 232 can check future scheduled missions against updated robot capabilities. If no other capable robots are available when a mission is to be performed, the robot payload may be modified, replaced, and / or updated to allow the mission to be performed. Such modifications, replacements, or updates may affect the availability of the robot for the time the mission was originally intended or scheduled to be performed. A mission may indicate a period of time or other recurring circumstances under which the mission may be performed. In such cases, the mission may be time-scheduled to be performed later under conditions suitable for the mission. To avoid rescheduling, the robot management system 232 can verify the robot state prior to the start of the mission (the required time depends on the facility and personnel). This verification may require a pre-check, which may also include checking the cameras, microphones, sensors, payload, and / or other capabilities of the robot. This may be performed for all robots, including those with variable and invariant sensors.

[0158] During robot definition management 350, the robot management system 232 can maintain robot definition information and send robot-specific production rules 365 to the robot 110. For example, in addition to the temporary changes in robot capabilities described above, a particular robot may undergo permanent changes in capability and availability, such as the removal or installation of sensors or tools, or the relocation of the robot to another area of ​​the facility so that the robot is available to perform missions or tasks in a different, limited portion of the facility. Modifications to the robot may change other capabilities of the robot, such as, for example, a change to the robot's overall weight may cause changes to battery life, movement speed, or maximum distance for the robot to travel. Such changes may be reflected in updated robot definition information. During detailed robot scheduling 325, the robot management system 232 can determine and update information regarding the overall robot schedule. Robot scheduling is described in more detail below.

[0159] During robot dispatch 345, the robot management system 232 can release a selected robot 110 to perform a robot-assigned task. Robot dispatching is described in more detail below. During robot execution management 360, the robot management system 232 can send robot commands 370 to the robot 110 and receive responses 375 from the robot 110. Robot activity commands and data collection are described in more detail below. During robot tracking 335, the robot management system 232 can maintain information about the location, status, and performance of the robot during an assigned activity or during idle periods. Robot tracking is described in more detail below. During robot data collection 355, the robot management system 232 can receive robot-specific data 380 from the robot 110. Robot data collection is described in more detail below.

[0160] During robot performance analysis 340, the robot management system 232 can perform an analysis of the robot's performance upon completion of a robot-assigned task. This analysis can include updates to information about the robot definition 305 and robot capabilities 310 to improve the robot's performance of the assigned task. For example, the time it takes the robot to complete a mission or task, or the time it takes to perform a portion of a mission or task, such as moving from one location to another, navigating stairs, or a ramp, can be measured. Additionally, the accuracy of measurements and data (e.g., photographs, or other data) collected by the robot can be monitored to determine whether the robot generally returns accurate measurements and data. Such information can be taken into account when planning future missions or tasks for the robot, or can be used to determine that the robot needs maintenance.

[0161] Additionally, tasks assigned to robots may be modified after scheduling, during execution, or as a result of execution. Conditions surrounding the mission may change, necessitating such modifications. For example, as a result of opening a door, a change in temperature or air composition may be detected, such that different sensors are needed to provide accurate and actionable feedback about the environment surrounding the mission. As another example, the measured temperature may be lower than expected, causing opening a manual valve to require a higher torque than could be applied by the original robot assigned to the task.

[0162] As described above, the operations management system can interact with a robot management system, such as a robot service platform (RSP), to manage the robot 110 and robot-assigned tasks. Figure 4 illustrates components of a robot service platform (RSP) 400 within or in communication with the operations management system, according to one or more embodiments.

[0163] The activities of the robots 110 within the operations management system 200 may relate to various industrial operations 410, including, for example, pipeline inspection, pressure vessel inspection, surveillance, first responder, emission detection, tank inspection, and subsea platform surveillance. The identified operations 410 are merely exemplary, and the principles of the present disclosure may relate to any facility suitable for specialized services performed by a robotic fleet in a dynamic environment. Such services rely on the human's ability to recognize and adapt to dynamic environments and may be repetitive and / or dangerous for a human operator who may have previously placed a human operator in such a role.

[0164] Such dynamic environments may require a robotic service platform to recognize and coordinate the activities of individual robots or robot swarms to meet changing conditions, or to operate in environments that may not be designed for robotic operation while managing swarms of robots of varying technical and operational capabilities. For example, applicable facilities may include chemical and pharmaceutical manufacturing, mining, food and beverage production, water and wastewater treatment, etc. To support these operations, RSP 400 may receive commands from an external control system 420, such as operations management system 200 or an industrial automation (IA) system, and may include modules, for example, for interfacing with industrial automation systems 430, for data aggregation and analysis 440, for security and safety 460, for coordinating operations, collecting data, and controlling robots 470, and for robotic fleet management 480.

[0165] Access to these modules may be provided to a human operator by an integrated human-machine interface (HMI) 450. The industrial automation system interface 430 may provide services for industrial process automation, possibly in conjunction with external industrial automation (IA) systems. The data aggregation and analysis module 440 may receive data from tasks assigned to robots and support data aggregation and analysis. Data aggregation and analysis is described in more detail below. The security and safety module 460 may ensure the safety and security of facility operations and individual robots 110. The coordinate collection control module 470 may provide overall management of robot-assigned activities, collaboration or robot-assigned activities, and collection of data returned from robot-assigned activities. The robot fleet management module 480 may provide management of the robot fleet 110, including, for example, maintaining information regarding the current status, location, availability, and performance of individual robots 110, coordinating related or potentially conflicting activities of multiple robots 110, and coordinating necessary maintenance and / or removal of individual robots 110. Further details regarding robot management are provided herein.

[0166] Managing robotic activities in large or complex industrial facilities may require the coordination of robots provided by multiple vendors, each with different, potentially incompatible, command and control or data management and communication protocols. Thus, integrating a fleet of robots into an industrial facility can become complex as the number of robots and tasks assigned to them increases. Figure 5 shows a detailed architecture 500 of a multi-vendor robot services platform 510 for addressing these issues, according to one or more embodiments.

[0167] The multi-vendor robot service platform 510 may interface with external control systems, such as an operations management (OM) system 200 or an industrial automation (IA) system 505, through an IA system integration application programming interface (API) 515. The multi-vendor robot service platform 510 can provide humans to manage, control, or monitor the robot fleet with access to end-user applications 530, such as, for example, operations management for the overall operation of a facility, maintenance management for activities for maintaining a facility, site inspection for direct inspection of portions of a physical facility, and workflow management for coordinating multiple human and robot activities within a facility through an engineering and operations human-machine interface (HMI) 525.

[0168] The instructions for the activity from the OM 200 may be work instructions (WIs) directed for assignment to a human. Such instructions may be in a human-readable format. The multi-vendor robot service platform 510 can translate the WIs into robot commands. If there are multiple types of robots from multiple vendors, the translation may be performed differently for each type of robot, as discussed elsewhere in this disclosure. If the activity is performed by a human, the WIs may be assigned to an operator in the context of shift planning. If the activity is assigned to a robot, the robot service platform 510 can autonomously translate and assign the WIs to one or more robots, as discussed elsewhere in this disclosure.

[0169] In other embodiments, when an alarm generates a robotic mission, such as from an external application, action can be taken to ensure a capable robot is available. For example, if the alarm is a critical alarm, the robot service platform 510 can determine to abort the current mission and process the critical alarm mission if a robot or human is not available. Alternatively, an operator may manually take over the robot and address the alarm. An external system can send a request for task performance, and the robot service platform 510 can respond with robot availability or mission initiation information. Human and robotic tasks can each include capability requirements for the human or robot assigned to the task, such as the ability to work at a specified height above ground or the ability to use a specific tool. Additionally, tasks can have requirements related to the operational status of the facility, such as time limits, a requirement that the plant not be operating while the task is being performed, or changes in government or other regulations governing the operation of the facility. The robot service platform 510 may separate requests received from the OM system 200 or IA system 505 into requests that are robot-compatible tasks that can be sent to an associated robot for execution and requests that are robot-compatible tasks that can be returned to the OM system 200 or IA system 505 for allocation to a human.

[0170] The robot fleet management module 545 can interact with various robot platforms 560 that support interaction with the robots 110 and sensors 550 (e.g., environmental sensors). The robot platforms 560 can include an open-source robot operating system 555, such as the Robot Operating System (ROS), as well as a platform for interacting with vendor-specific (non-open source) robots 565. The robot fleet management module 545 can also interact directly with ROS robots 555 and non-ROS robots 565, rather than through the robot platform 560. Interactions between the robot fleet management module 545 and the robots 110 can include, for example, robot commands for mission and task completion based on information from the control and coordination function 540. Interactions between the robot fleet management module 545 and the robots 110 can further include, for example, collection of data and other information from the robots 110 and sensors 550, which is provided to the data collection and aggregation module 535 for further processing. Such processing may include storing the collected and processed data in a data store 520 and / or providing the collected and processed data to an end-user application 530. The data stored in the data store 520 may be further provided to the (IA) system 505 by an IA system integration API 515.

[0171] The coordination function 540 translates end-user application functions into robotic system functions. For example, if an end user requests a gauge inspection, the coordination function 540 can translate that request into a robot mission to take a picture of the gauge. For example, the coordination function 540 can determine a robot mission that includes detailed steps such as navigating from the robot's current location to the gauge's location, taking an image of the gauge, and returning either an image of the gauge or an analysis of the gauge reading based on the image. Other requested tasks are translated into other appropriate actions. To support robots from multiple vendors, gauge missions can be pre-configured by the robot fleet management module 545 to support multiple robots. For example, detailed actions corresponding to robots from multiple vendors may be pre-determined and stored. This can allow robot missions based on recurring task requests to be quickly dispatched or repeatedly scheduled.

[0172] The coordination function 540 can further coordinate robot tasks with operations performed by other human and robot operators, such as processes performed by operators in a control room, automated processes within an IA platform, and field operations. Each of these components works in tight synchronization to ensure the proper operation of the facility. In traditional facilities, much of this synchronization can be accomplished verbally, but in facilities employing robot fleets, the coordination function 540 can integrate robotic and automated systems into this synchronization. In some situations, full control over a robot is not possible at all times, and the robot may be allowed to operate semi-autonomously, using any specific capabilities of the robot to complete a task. When multiple robots are assigned missions, the coordination function 540 can assign several missions with appropriate time delays to avoid collisions between robots operating in the same area.

[0173] In facilities with robots from multiple vendors, robot fleet management, such as may be provided by robot fleet management module 545, robot activity control, such as may be provided by control and coordination functionality 540, and data analysis, such as may be provided by data collection and aggregation module 535, may utilize additional vendor-specific and robot type-specific capabilities in a multi-vendor robot service platform (RSP).

[0174] FIG. 6 further illustrates a detailed architecture 600 of a multi-vendor robotic service platform (RSP) according to one or more embodiments.

[0175] The robot service platform 672 can include, for example, an external system interface 624, a navigation control and data service 626, a fleet manager 628, and a robot interface 630.

[0176] The external system interface 624 may include, for example, a process control server 632, which may receive information for processes and tasks to be completed from the process control client 604 under the direction of a human operations manager or operator 602. For example, the received missions and tasks may include activities to be performed by a human or robot related to the overall operation of the facility, such as facility inspection, data collection, facility control (e.g., changing control settings), etc. The external system interface 624 may further include, for example, a first web server 634, which may provide data reports to the web client 606 for display to the human operator or analyst 608. The external system interface 624 may further include, for example, a second web server 636, which may provide a data feed to the historian data store 612.

[0177] The historian data store 612 can provide data to a report generator 610, which can provide additional reports to a human operator or analyst 608. The historian data store 612 can further provide data to an archive module 618 for storage in a database 620 within the robotic service platform 672. The database 620 can provide data to a machine learning (ML) and artificial intelligence (AI) module 614 for further analysis. Analysis results from the ML and AI module 614 can be stored in the database 620. The database 620 can further provide data to the historian data store 612 for access by the report generator 610. The report generator 610, the historian data store 612, and the ML and AI module 614 may be cloud-based or may operate separately from the robotic service platform 672.

[0178] Navigation control and data services 626 may include, for example, robot-specific adapters 644 that can provide platform-independent command, control, and data services for robots 110A, 110B, and 110C. Robot-specific adapters 644 may include, for example, adapters for communication of data collected by robot 110, such as photographs, videos, environmental data, sensor readings, etc., adapters for communication of data regarding robot 110, such as, for example, robot motion and position, robot health, and robot capabilities, and adapters for communication of command and control information with robot 110, such as, for example, multi-stage mission commands, individual operation commands (e.g., navigate to a destination, take photographs, take video recordings, take audio recordings, take environmental measurements, take material temperature measurements, take air temperature measurements, take humidity measurements, determine instrument readings, measure the presence or concentration of a gas or chemical, emit light of a particular wavelength, intensity, and / or light pattern, emit sound of a particular pitch, intensity, and / or light pattern, emit a radio frequency homing beacon), etc. The robot-specific adapter 644 may also receive information from the robot 110, including, for example, information regarding task or mission status and completion, non-data task completion information (e.g., mission completion or abandonment), task completion information (e.g., task ...

[0179] Command / control and data information may generally be maintained within RSP672 in one or more common internal formats. This allows the internal operation of RSP672, and interfaces and information shared externally to RSP672 (such as with other components of an operations management system (OM)), to operate independently of the robots or robot types active within the facility. This may therefore reduce the complexity of utilizing a diverse fleet of robots within a facility. To this end, each adapter may take information from RSP672 in one or more of the common internal formats, convert that information into specific information appropriate for the particular robot with which it is being communicated (e.g., in a format compatible with the particular robot), and then transmit the converted information to the particular robot.

[0180] For example, a robot task or mission definition may be converted from a common internal format to a format that conforms to the expected command protocol for the assigned robot. Conversely, data transmitted from the robot may be received by the adapter in a robot-specific format and converted by the adapter to a common internal format for use by RSP 672 and other components of the operations management system (OM), such as OM 700 described in FIG. 7. The common internal format for a robot task or mission definition may include a list of measurements or data to be captured within the facility. The adapter 644 can convert this list into robot-specific instructions, which may include, for example, directions to physically move to a data capture location and which equipment to use. Vendor-specific robot data may also be parsed by the adapter 644 to discover generalized or useful information for display to the operator. For example, data such as battery life may be parsed and displayed in different ways (percentage / remaining voltage / time remaining), which may be standardized.

[0181] Some such adapters 644 may be narrowly tailored to communicate with a specific robot according to robot type or vendor, while others may be compatible with multiple robot types. An adapter 644 can be thought of as a fleet-specific adapter, compatible with multiple robots of the same type, operating simultaneously or cooperatively. The adapter 644 may be manually coded based on common internal data formats and information about the data and command protocols for a specific vendor and robot. Alternatively, such adapters may be automatically generated based on algorithms or artificial intelligence methods. The adapter may be configured to provide a comparable level of basic control across multiple robot types. This may include, for example, navigation capabilities, data capture, and robot metric reporting. The adapter may further include additional functionality for each robot type, depending on the robot vendor's specific software limitations. For example, if a robot is not capable of recording its metric data, the adapter may record the robot's metric data to maintain compatibility with the standard adapter level for other components of the platform.

[0182] Common characteristics from different robot types can be abstracted into a common internal data format, such as map data or simulation data on a 3D model. The OM system 700 or IA system 505 can assign a work instruction (WI) checklist, a human-readable format such as a spreadsheet, consisting of instructions for recording information corresponding to various identifiers. The RSP 672 can convert the WI checklist into detailed information, such as absolute or relative position and orientation information, such as GPS coordinates or site-specific visual, electronic, or magnetic identifiers, for execution by the robot. Unstructured data recorded by the robot, e.g., media formats such as images, audio, or video, can be further processed by the RSP 672 using data processing applications to become structured data, such as text or numeric data, suitable for storage and processing in the OM system 700 or IA system 505.

[0183] The navigation control and data service 626 may further include, for example, a fleet management module 638 for receiving information for missions and tasks to be completed from the process control server 632. The fleet management module 638 may determine which of the robots 110A, 110B, and 110C should perform each task or mission and may provide detailed information regarding the robot tasks and missions to a robot-specific adapter 644. The fleet management module 638 may also provide information regarding the progress or completion of the missions and tasks to the process control server 632. The navigation control and data service 626 may further include, for example, a data management module 640 that may receive data from the robots 110A, 110B, and 110C via the robot-specific adapter 644. The data management module 640 may further include a machine learning (ML) and artificial intelligence (AI) module 642, which may further analyze the data received from the robots 110A, 110B, and 110C.

[0184] External systems interface 624 and navigation control and data services 626 can provide database 620 with information about tasks originated and performed by robots 110A, 110B, and 110C, task results returned by robots 110A, 110B, and 110C, and other information related to the management of robots 110A, 110B, and 110C, and missions performed by robots 110A, 110B, and 110C. For example, records of robot tasks and task results can be stored in log 622 within database 620.

[0185] Fleet manager 628 may manage the fleet of robots 110A, 110B, and 110C with respect to, among other things, scheduling and dispatching missions and tasks to robots 110A, 110B, and 110C, monitoring the health and maintenance status of robots 110A, 110B, and 110C and their components, and scheduling maintenance of robots 110A, 110B, and 110C. Fleet manager 628 may include vendor-specific fleet management modules 646 that may provide management of robots 110A, 110B, and 110C specific to one or more vendors of robots 110A, 110B, and 110C. A separate vendor-specific fleet management module 646 may be provided for each vendor-specific interface, such as vendor-specific interfaces 648 and 650.

[0186] The fleet manager 628 can assign missions and tasks to the robots 110 based on mission requirements and robot capabilities. For example, if the fleet manager 628 determines that a mission can only be supported by robot type A, the fleet manager 628 can assign the mission only to robots of type A among the robots 110A, 110B, and 110C. For example, a mission may require traversing an obstacle, such as a staircase, that can only be traversed by a particular type of robot. For another mission that can be assigned to robots of type A or B, the fleet manager 628 can assign the mission to robots of type A or B among the robots 110A, 110B, and 110C based on the availability of each robot 110A, 110B, and 110C. The availability of each robot 110A, 110B, and 110C can be determined by the fleet manager 628 using metrics such as communication status and battery life.

[0187] In some embodiments, the operation management system 200 can specify missions or tasks to be performed by a particular robot or robots in a particular group of robots. If a group of robots is specified, the fleet manager 628 can assign missions to robots selected from the specified group of robots. Additionally, robot types or some attributes of robots from a particular vendor may have known reliability, and the fleet manager 628 can apply direct oversight of the robots based on this known reliability.

[0188] The robot interface 630 may include, for example, a first vendor-specific robot interface 648, a second vendor-specific robot interface 650, and a fleet server 660. The first vendor-specific robot interface 648 and the second vendor-specific robot interface 650 may each provide an interface to the robot specific to that robot's vendor. For example, the first vendor-specific robot interface 648 may interact with the robot 110A through a robot operating system (ROS) interface, while the second vendor-specific robot interface 650 may interact with the robot 110B through a proprietary remote procedure call (RPC) interface. The number and types of vendor-specific robot interfaces provided by the robot interface 630 are not limited and may depend on the number and types of robots in the fleet managed by the robot service platform 672.

[0189] Vendor-specific interfaces have traditionally been required to access these robots. That is, robots from each vendor typically have proprietary software and are gated using each vendor's application programming interface (API). For example, if a vendor uses gRPC calls to access its proprietary API to control robots from that vendor, there may be no other alternative means to communicate with or control the robots. The fleet server 660 can interact with the robots of a coordinated fleet, such as robot 110C, via a fleet client 664. The robot interface 630 can further interact with a real-time communication module 662 to provide additional communication streams to a human operations manager or operator 602, for example, via the robots 110A, 110B, and 110C, the process control server 632 or the process control client 604, or other components of the RSP 672.

[0190] For real-time communication, the robot interface 630 can support multiple types of protocols for streaming data. For example, WebRTC is a streaming protocol for high-quality video / sound streaming that can be utilized for such communication. Such streaming can include, for example, receiving data to be displayed to an operator as well as performing real-time processing of the data using artificial intelligence (AI) or machine learning (ML) to support functions such as anomaly detection. Real-time communication can be important for fleet management to perform different tasks. Some robots may have the ability to operate during lost communication for a short period of time, while others may not. The fleet server 660 may need to frequently communicate with each robot based on its capabilities and track the robot during missions.

[0191] The vendor-specific fleet manager module 646 and fleet server 660 may be cloud-based or may operate separately from the fleet manager 628 and robot interface 630.

[0192] The robots 110A, 110B, and 110C may be provided by different vendors, may be of multiple types, and may have different instrumentation, tools, and sensors available. Each robot, possibly in conjunction with the robot management and fleet management modules of the RSP672, may manage information about the health and status of the robot 110, including battery or fuel levels, capabilities, location, malfunctions, and maintenance requirements.

[0193] Examples of robots 110A, 110B, and 110C may include robots that are fully autonomous, pre-programmed for specific tasks, movements, routes of administration, or activities, or under direct human control. Robots 110A, 110B, and 110C may be stationary or mobile, and mobile robots may be wheeled, tracked, bipedal, quadrupedal, multipedal, or include other means of propulsion. Robots 110A, 110B, and 110C may be equipped with tools, sensors, or other hardware to complete missions and tasks, such as articulated arms, grippers, claws, wrenches, screwdrivers, hammers, prybars, cameras, microphones, chemical detectors, noise sensors, and vibration sensors. Robots 110A, 110B, and 110C may include digital and physical storage devices such as photographs, videos, sounds, environmental readings, and environmental samples such as soil and chemicals. The robots 110A, 110B, and 110C may include a variety of communication capabilities, including analog (such as radio and video transmission) and digital (such as Wi-Fi, Bluetooth, other short-range communications).

[0194] A robotic service platform such as RSP 672 can interact with and depend on other subsystems of an operations management system. Figure 7 illustrates the subsystems of an operations management system 700, as well as a multi-vendor RSP, which may operate as part of, separate from, or in cooperation with the operations management system 700, according to one or more embodiments.

[0195] The operations management system 700 may include multiple subsystems for supporting the operation and management of an industrial facility, such as an operations assistance system 702, a patrol management system 704, a task management system 706, a process control system 710, and a robotics management system or robotics assistance platform (RSP) 672.

[0196] The process control system 710 may provide operational control and monitoring for the facility through control subsystems that may be distributed throughout the facility or located external to the facility. The process control system 710 may include, for example, a collaborative information server 722 that may support information sharing between the process control system 710 and other subsystems of the operations management system 700. The process control system 710 may also include a field control system 728 that may control processes within the facility and collect data from those process data via field devices, and a safety control system 730 that may monitor the safety state of processes within the facility and ensure that the processes return to a safe state if a deviation occurs. The process control system 710 may coordinate with the operations support system 702, the patrol management system 704, the task management system 706, and the robot management system or robotic support platform (RSP) 672.

[0197] The operations support system 702 may provide services that support the operation of the entire facility. The operations support system 702 may include, for example, a collaborative information server 722 that may facilitate information sharing between the operations support system 702 and other subsystems of the operations management system 700, a procedure information management module 714 that may store and manage information related to procedures and methods for operating the industrial facility, and a procedure execution management module 716 that may manage the execution of the procedures and methods.

[0198] The tour management system 704 can provide services related to the periodic patrol and monitoring of the facility for operations, safety, and security. The tour management system 704 can include, for example, a collaborative information server 722 that can support the sharing of information between the tour management system 704 and other subsystems of the operations management system 700. The tour management system 704 can also include a checklist management module 718, which can manage checklists to ensure all operational, safety, and security protocols are adequately covered; a checklist execution management module 724 that can manage the execution of tasks to fulfill checklist requirements, as may be determined by the checklist management module 718; and a schedule management module 720 that can schedule the completion of checklist tasks. Checklists and associated checklist tasks can be assigned to either human or robotic assets. The assignment and scheduling of robotic assignment tasks is described in more detail below.

[0199] The task management system 706 can provide for the creation, assignment, and monitoring of tasks within the facility. Tasks can be assigned by humans or robots. The task management system 706 can include, for example, a collaborative information server 722 that can support the sharing of information between the operation task management system 706 and other subsystems of the operation management system 700. The task management system 706 can also include a trigger management module 725 that can create new tasks triggered by incoming information, and a task execution management module 726 that can control the assignment and execution of tasks. Management of tasks assigned to robots is described in more detail below.

[0200] The RSP 672 can provide management and operation of a fleet of robots of various types provided by multiple vendors. The RSP 672 can include, for example, a collaborative information server 722 that can support the sharing of information between the RSP 672 and other subsystems of the operations management system 700. The RSP 672 can also include a robot fleet management module 628, a robot data management module 640, a robot common interface module such as a robot-specific adapter 644, a database 620, and a robot data analysis module such as the data management module 640 and a machine learning (ML) and artificial intelligence (AI) module 642, all of which are discussed above with respect to FIG. 6 . The RSP 672 can further include, for example, robot interface modules specific to different types of robots, e.g., a robot A interface 742 for interacting with a type robot 110A, a robot B interface 744 for interacting with a type B robot 110B, and a robot C interface 746 for interacting with a type C robot 110C.

[0201] The robot interfaces may connect directly to the robot 110, as shown for the robot B interface 744 and the robot C interface 746, or may connect to the robot 110 through an external robot controller, such as the robot control 748 that connects the robot A interface 742 with the robot 110A. Such an external robot controller may be a cloud-based server, as shown in FIG. 7, or may be connected by various computer networks. An exemplary connection between the robot support platform and the robot 110 is discussed below with respect to FIG. 8.

[0202] FIG. 8 illustrates an environment 800 for robot control and communication within a multi-vendor robotic service platform, according to one or more embodiments.

[0203] The operations management system 805 may include many systems and modules for the overall management of an industrial facility, such as, for example, the systems and modules shown in Figure 2 and described above. As mentioned above, the operations management system 805 may interact with the robotic support platform (RSP) 840 to provide information for missions and tasks related to the operation of the plant that may be performed by robots, such as robots 110A and 110B.

[0204] The connections between the motion management system 805, the RSP 840, and the robot 110 may be provided differently depending on the needs and capabilities of the facility and the robot 110. For example, in one embodiment, the RSP 840A may be provided as a cloud-based service that utilizes a global network 830, such as the Internet, to connect to the robot 110A. In another embodiment, the cloud-based RSP 840B may use a virtual private network 835 to securely connect to an internet service provider network 825 and from there connect to the robot 110A. In yet another embodiment, the RSP 840C may be provided as an application running on a local computer, such as a local personal computer 845. The RSP 840C may utilize a local web server 845 and a local network 820 to connect to the robot 110A. In yet another embodiment, the RSP 840C may utilize a web server 850 to connect to a vendor-specific robot server 815 and from there connect to the robot 110B. Although the vendor-specific robotics server 815 is shown as a cloud-based service, the vendor-specific robotics server 815 may be provided, for example, on a local personal computer 845, on a different local computer, or on a remote computer accessible over a global network such as the Internet. In another embodiment, the cloud-based RSP 840D can connect to the vendor-specific robotics server 815 and from there to the robot 110B.

[0205] FIG. 9 illustrates a method 900 for assigning, scheduling, and executing robotic tasks within a multi-vendor robot service platform (RSP) (e.g., RSP 672 of FIG. 6 ) according to one or more embodiments. In operation 910, an operations management system (OM) (e.g., OM 700 of FIG. 7 ) may determine that one or more tasks for the operation or maintenance of an industrial facility are desired to be performed (e.g., based on user input, based on a determination by the operations management system, based on a determination by the RSP, etc.). At least one of the tasks may be designated for completion by a robot in process by the RSP in operation 910. In operation 920, the OM may send a robot-designated task to the RSP. In operation 930, the RSP may convert the received robot-designated task into one or more robot missions.

[0206] Converting the received robot-specified task into a robotic mission may include determining mission factors, data related to the environment in which the task is located, and / or mission capabilities for accomplishing the task, as described herein. Defining a robotic mission is described in more detail below. In operation 940, the RSP may schedule one or more robotic missions, including, for example, selecting an appropriate robot (e.g., a robot capable of performing one or more tasks or functions associated with completing one or more robotic missions) and assigning the mission to the selected robot. Selecting and assigning a robot to a particular task is described in more detail below. In operation 950, the RSP may manage (e.g., initiate, monitor, and otherwise manage) one or more robotic missions. Managing a robotic mission may include, for example, dispatching robots, monitoring the progress of robots during a mission, and responding to events during the mission, such as by assigning additional or replacement robots to a mission.

[0207] Management of robotic missions is described in more detail below. In operation 960, the RSP may receive data generated by one or more robotic missions. Processing of data resulting from robotic missions and / or other feedback received from the facility before, during, or after a mission is described in more detail below. In operation 970, the RSP may update tasks received from the OM based on the received data. In operation 980, the OM may update various information regarding facility operations and completed tasks. This information may include plant operation and maintenance status and may be displayed through various user interfaces, such as, for example, a shift report or a facility status dashboard. The OM may also use the task completion status and data generated by the tasks to plan future tasks and / or missions in a new iteration beginning at operation 910.

[0208] FIG. 10 illustrates an example dashboard 1000 that supports aspects of the present disclosure.

[0209] In some examples, a user can define, assign, and / or manage tasks via direct input to a wired or wireless device via the dashboard 1000. A user can select or modify missions, robots, and / or mission schedules via a GUI such as the dashboard 1000. The dashboard 1000 or a similar device may allow a user to remotely enter commands, for example, via the web client 606 of FIG. 6 or directly, for example, via the process control client 604. The dashboard 1000 may display various headings, such as robots 1010, sensors 1020, missions 1030, and / or alarms / events 1040. These headings may indicate subcategories that can be selected via touch, mouse, etc. to receive, send, and / or view information related to missions and / or other events within the facility. The categories and / or subcategories may include images, numbers, and / or words denoting each category or subcategory. For example, a user can select the subcategory Configure Robot Type 1012, and that selection may bring the user to another screen listing all available robot types. A user can select the type of robot to be used for a particular mission, or a robot can be automatically selected based on a mission profile, as described herein. Alternatively, or additionally, configuring robot type 1012 allows a user to select human personnel to perform the mission.

[0210] Robot configurations 1014 may also be shown under categories 1010. Selecting a robot configuration 1014 may allow a user to select various functions or capabilities of the robot, such as including a particular sensor package. Robot configurations 1014 also allow a user to select packages, e.g., particular tools and / or sensors, for human use during a mission. It will be understood that functions or capabilities may be automatically selected based on, for example, a mission profile as described herein.

[0211] A third category, robot status 1016, may be shown under category 1010. Robot status 1016 may indicate the status of the robots, for example, via a table (not shown). For example, the first robot and the second robot may be shown in list format on dashboard 1000 when robot status 1016 is selected. Robot status 1016 may indicate various robot capabilities or other information about each robot, such as the robot name, robot type, robot status, communication status, battery level and battery status, odometer reading, mission status, mission progress, the latest (or last) communication update with the robot, and / or whether the robot has video capability. It will be understood that this list is only one example and that any robot capability or other information about a robot may be supported. It will also be understood that robot status is not limited to robots and may be used to indicate the capabilities or status of any operator, e.g., a robot or human personnel. In some embodiments, robot status 1016 may indicate a current mission and / or previously recorded missions associated with the robot, a current work task and / or previous work task associated with the robot, etc.

[0212] Continuing with reference to FIG. 10, four subcategories may be displayed under the Sensors 1020 category. For example, the subcategories may display Area Settings 1022, Sensor Settings 1024, Sensor Status 1026, and Graphic View 1028. Selecting each of these subcategories may change the display to show different information. For example, selecting Area Settings 1022 may allow the sensor to be configured based on where the sensor is located, such as zeroing the sensor for use in a particular area. This may be done manually or automatically based on feedback.

[0213] Similarly, selecting sensor settings 1024 allows a sensor to be configured. For example, the sensor may be an imaging sensor, and the capabilities of the image sensor, such as brightness, saturation, etc., may be changed automatically or manually by the user when sensor settings 1026 is selected. Selecting sensor status 1026 may provide the status of the sensor, similar to seeing the state of the robot when robot state 1016 is selected. For example, a list of one or more sensors may be shown, and the status or capabilities of those sensors may be displayed when sensor status 1026 is selected. Selecting graphic view 1028 may display the sensors as a graphic, for example, a temperature sensor as a thermometer, a pressure sensor as a pressure gauge, etc. Selecting graphic view 1028 may display other information of one or more sensors in a graphical format.

[0214] Under the category Mission 1030, the user can select one of Set Mission Template 1032, Set Mission 1033, Set Mission List 1034, Run Mission 1036, Mission History 1038, or Mission History (Review) 1039. By selecting Set Mission Template 1032, the user can change the factors, data, and / or mission capabilities associated with the mission or change how the factors are displayed. For example, it may be important to consider environmental temperature when defining a mission. In this case, the user can select Set Mission Template 1032 and modify the mission template to include temperature as a factor when defining a mission.

[0215] In response to user input selecting mission setting 1033, the robotic system platform can display user interface 1100 (shown later in FIG. 11 ), through which the user can indicate one or more tasks desired by the user. In one example, the task can include a task the user desires to perform with respect to a facility or environment (e.g., industrial facility 113 or industrial facility setting 100 of FIG. 1 ). The task can also be a robot-independent task as described herein. In some embodiments, the task can be a process automation inspection task related to the facility or environment. Based on the task, the robotic system platform can provide the user (e.g., via user interface 1100) with one or more candidate missions associated with completing the task. Exemplary embodiments and missing configurations of user interface 1100 are described later with reference to FIG. 11 .

[0216] By selecting mission list settings 1034, a user can manually modify scheduled missions. For example, when mission settings 1034 is selected, a list of missions and the robots assigned to each mission may be displayed. The user can view the robots and missions and change the mission schedule. For example, a mission may include obtaining sensor readings, and the user may have knowledge that the sensor is located at a particular height. The user may view missions scheduled for robots that do not have the ability to reach the sensor height to obtain the sensor data. As described herein, this data can be saved for future use, and the mission profile can be automatically updated when future data is obtained from this sensor.

[0217] Selecting Run Mission 1036 can start a mission or schedule a mission to start at a specific date and time in the future, for example. Additionally, if a mission is scheduled in the future, the mission can be manually or automatically selected to start earlier via Run Mission 1036 based on feedback. Selecting Mission History 1038 allows the user to view which missions have been started, what data has been acquired, and any warnings or other feedback from the mission. Selecting Mission History (Review) 1039 allows data to be reviewed manually or automatically. For example, if an analog sensor image has been acquired, the sensor readings may need to be reviewed.

[0218] Dashboard 1000 may be used to allow a user to define, assign, and / or manage missions in any manner described herein. It will be understood that some or all aspects of the mission definition, assignment, and / or management may be performed manually or automatically. Dashboard 1000 may also allow a user to verify or correct any data received by the system. Alternatively, dashboard 1000 may be used only by a supervisor to view the status of all missions.

[0219] 11 illustrates an example of a user interface 1100 through which a user can indicate one or more target tasks associated with a facility and / or environment. Through the user interface 1100, the robotic system platform can provide the user with one or more candidate missions related to completing the task.

[0220] Aspects of the user interface 1000 may support viewing, configuring, and building missions. In one example, via the user interface 1000, a user may provide input including instructions for a target goal 1105 and / or one or more target tasks 1110 (e.g., target task 1110-a through target task 1110-d).

[0221] In some embodiments, the robotic system platform can identify and / or configure one or more of the target tasks 1110 based on the goal 1105. For example, a user can provide input indicating the goal 1105, and the robotic system platform can configure one or more of the tasks 1110 in relation to achieving the goal 1105. Additionally or alternatively, a user may provide input indicating one or more target tasks 1110 with or without indicating the goal 1105.

[0222] Either the goal 1105 or the target task 1110 can be robot-independent. That is, for example, the goal 1105 and / or the target task 1110 can be expressed in a human-readable format rather than in a format (e.g., a programming language, program code, etc.) that conforms to a vendor-specific command protocol for controlling a robotic device. In some embodiments, the term “robot-independent” may refer to a definition of the goal 1105, the target task 1110 (target robot task), or the mission in an internal format common to the robotic system platforms described herein or to the motion management systems described herein. In some cases, the term “robot-independent” may refer to a definition of the goal 1105, the target task 1110, or the mission without an indication of the robot type and / or robot function.

[0223] In some embodiments, the target goal 1105 and / or the target task 1110 may include an indication of any target criteria (e.g., before a certain time, during a target time window, urgency, etc.). Based on the target goal 1105 and / or the target task 1110, the robotic system platform may provide candidate missions 1115, embodiments of which are described herein. In one example, the candidate missions 1115 may include performing a task that matches the target task 1110. In some cases, the robotic system platform may order and display the missions 1115 based on their respective rankings (e.g., “Mission 1” may have the highest ranking and be displayed at the top of the missions 1115, while “Mission 4” may have the lowest ranking and be displayed at the bottom of the missions 1115). In some cases, the robotic system platform may display the respective ranking information associated with the candidate missions 1115 using, for example, any combination of indicators (e.g., using highlighting, shading, color, text, icons, etc.).

[0224] In some embodiments, in response to a user input selecting a mission 1115, the robotic system platform may indicate (e.g., using highlighting, shading, color, text, icons, etc.) which of the tasks 1110 will be performed based on the selection. Additionally or alternatively, the robotic system platform may automatically indicate the correspondence between the tasks 1110 and the missions 1115. Referring to user interface 1100 in a non-limiting example, mission 1115-a ("Mission 1") corresponds to task 1110-a ("Task A") and task 1110-c ("Task C"), and mission 1115-c ("Mission 3") corresponds to task 1110-b ("Task B") and task 1110-d ("Task D"). It should be understood that in some example implementations, any amount of tasks 1110 and / or missions 1115 may be displayed on user interface 1100.

[0225] In some embodiments, for the target task 1110 and / or mission 1115, the robotic system platform may indicate a robot identifier 1120, a robot type 1125, and robot functions / capabilities 1130, aspects of which are described herein. In one example, the robot type 1125 may include an indication of a mobility type (e.g., fly, crawl, walk, etc.) and / or a payload type (e.g., headlights, arms, cameras, other sensors, etc.). In some cases, the robot functions / capabilities 1130 may include an indication of a payload type. In some other embodiments, any of the robot identifier 1120, robot type 1125, and robot functions / capabilities 1130 may be omitted (e.g., hidden from view).

[0226] Examples of tasks 1110 may be "take an image of a measuring instrument gauge," "measurement of instrument A," and "measurement of all instruments in the facility." In another example, the target goal 1105 may be "determine the efficiency of all instruments in the facility by 5 PM," and the robotic system platform may automatically suggest tasks A-D (e.g., "take an image of instrument gauge A," "take an image of instrument gauge B," "take an image of instrument gauge C," "take an image of instrument gauge D").

[0227] An example of mission 115 may include, for example, performing a field patrol mission of a facility (e.g., facility 113 in FIG. 1 ). In one example, the field patrol mission may include capturing images and / or measuring readings of a set of target devices within the facility (e.g., device 123-a and device 123-b, or all devices 123, etc.). In some cases, mission 115 may include identifying all devices 123 that require repair or replacement (e.g., based on the performance of devices 123) and performing the repairs or replacements.

[0228] Thus, for example, via the user interface 1100, a user can input a target task 1110 (or desired goal 1105) and initiate a search of data records in the mission file (e.g., via a search button 1112). The robotic system platform can identify and suggest one or more missions 1115 that can perform the target task 1110 based on the target task 1110 and other factors described herein (e.g., contextual information, target criteria, reliability information, etc.).

[0229] The user interface 1100 may support user input to confirm (e.g., confirm 1116), reject (e.g., reject 1117), and / or modify (e.g., modify 1117) the mission 1115 proposed by the robotic system platform.

[0230] 12 illustrates an exemplary embodiment of a mission history 1200 according to an embodiment of the present disclosure. Aspects of the present disclosure support displaying the mission history 1200 in response to a user input selecting mission history 1038 in FIG.

[0231] The mission history 1200 may include feedback related to completed missions, which may include the results of performing tasks associated with the missions.

[0232] In one example, the mission may include a task to capture sensor information for sensor "PG003" and sensor "PG002." The feedback may include robot information 1210, first information 1220 related to the reading of sensor "PG003," and second information 1230 related to the reading of sensor "PG002." Alternatively or additionally, this information may be retrieved from memory (e.g., historian 612 of FIG. 6). The robot data may be used to determine whether the robot was suitable to accomplish this mission (e.g., whether the robot performed the mission or tasks therein with better or worse results, reliability, efficiency in time or energy, etc., compared to other robots). In some embodiments, the robot data may be used as feedback to update planned or future missions (e.g., which robots are assigned to a particular mission).

[0233] In some cases, the robot management system may support the implementation of AI and / or machine learning techniques to automatically determine the respective values ​​of sensor "PG003" and sensor "PG002" based on image analysis. Thus, for example, the robot or robot management system may obtain readings of sensor "PG003" and sensor "PG002" based on image analysis.

[0234] In one example, after determining the results of a task (e.g., sensor readings), the robot can transmit image data along with information about the task. This information can be transmitted wirelessly from the location where the task was performed or can be transmitted via a wired connection when the robot returns to home base. In one example, the robot can transmit information to a robotic system platform. The information can include both the results (e.g., sensor readings) and any raw data or information collected during the mission (e.g., sensor image data). It will be understood that any information transmitted by the robot can be considered feedback, stored in a database, or used to define future missions.

[0235] In some embodiments, the robot and / or robot management system can determine metric data associated with the completed task (e.g., performance data, time data, power consumption, usage costs associated with the robot, etc.) Examples of such metric data are described herein.

[0236] 13 illustrates a process flow 1300 that supports building a robotic mission based on mission metrics according to aspects of the present disclosure. Process flow 1300 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of process flow 1300 may be implemented by system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0237] In the following description of process flow 1300, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1300, or other operations may be added to process flow 1300.

[0238] At 1305, the process flow 1300 can include providing an inventory of recorded robot mission files. In one example, each mission corresponding to a respective robot mission file is assigned one or more performance metrics.

[0239] At 1310, process flow 1300 may include receiving user input. In some embodiments, the user input identifies (e.g., includes instructions for) one or more tasks to be performed. The tasks may be process automation inspection tasks that do not rely on robotics.

[0240] At 1315, process flow 1300 may include determining current context information associated with the user and / or facility (e.g., an urgency associated with one or more tasks to be performed at the user and / or facility, etc.). In some cases, process flow 1300 may include determining current constraints associated with the user and / or facility (e.g., a time constraint associated with the user and / or facility).

[0241] At 1320, process flow 1300 may include searching an inventory of recorded robotic mission files for a mission file that matches one or more of the tasks (e.g., one or more of the non-robot dependent process automation inspection tasks). For example, process flow 1300 may include identifying a recorded robotic mission file having at least one task that matches the one or more tasks identified at 1310.

[0242] At 1325, process flow 1300 may include ranking and / or sorting the matching mission files based on any combination of performance metrics associated with the matching mission files and one or more tasks, current context information associated with the user and / or facility, and current constraints associated with the user and / or facility.

[0243] At 1330-a, process flow 1300 may include recommending one or more recorded robot mission files based on ranking and / or sorting. Additionally or alternatively, at 1330-b, process flow 1300 may include presenting the set of recorded robot mission files based on a ranked order.

[0244] In some aspects, process flow 1300 may include executing (e.g., re-executing) the recommended mission(s) at 1335. For example, 1335 may include automatically executing the recommended mission(s) in response to a confidence score associated with the recommended mission and its ranking exceeding a confidence threshold. In another example, 1335 may include semi-autonomously executing the recommended mission(s) (e.g., in response to user input confirming the recommended mission(s)).

[0245] Figure 14 shows a process flow 1400 that supports building a robotic mission based on mission metrics. Process flow 1400 may be implemented, for example, in the industrial facility 113 shown in Figure 1. Aspects of process flow 1400 may be implemented by system 100 (e.g., of Figure 1), an operations management system (e.g., operations management system 700 of Figure 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0246] In the following description of process flow 1400, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1400, or other operations may be added to process flow 1400.

[0247] Process flow 1400 may be implemented by a robot management system that includes an interface for providing commands to multiple robotic devices of one or more robot types and a data repository that includes data records of mission files associated with a set of recorded robotic missions. The robot management system manages the one or more robotic missions and the multiple robotic devices in accordance with aspects of process flow 1400.

[0248] At 1405, the process flow 1400 can include providing a data record of a mission file associated with the set of recorded robotic missions.

[0249] At 1410, process flow 1400 may include receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to the environment. In some embodiments, the one or more robot-independent tasks include one or more process automation inspection tasks.

[0250] At 1415, process flow 1400 may include determining first context information associated with the one or more robot-independent tasks. In some embodiments, determining the first context information is based on a set of target criteria associated with a user, an environment, or both.

[0251] At 1420, process flow 1400 may include searching data records of mission files and identifying one or more mission files that include one or more tasks that match the one or more robot-independent tasks based on the search and the first context information.

[0252] In some embodiments, the one or more identified mission files include metadata. In some embodiments, the metadata includes at least one of recorded metric data, second context information associated with the one or more identified mission files, and an indication of whether a recorded robot mission corresponding to the one or more identified mission files met one or more target criteria.

[0253] In some embodiments, the one or more identified mission files correspond to one or more recorded robot missions of a set of recorded robot missions.

[0254] In some embodiments, the recorded metric data includes at least one of performance data associated with the completion of one or more recorded robotic missions, time data associated with the completion of one or more recorded robotic missions, and power consumption associated with the completion of one or more recorded robotic missions.

[0255] In some embodiments, the recorded metric data is associated with the robot and includes a usage cost of completing one or more recorded robot missions.

[0256] In some embodiments, the recorded metric data includes an indication of whether one or more recorded robotic missions included at least one operator intervention.

[0257] At 1425, process flow 1400 may include generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files.

[0258] In some embodiments, generating ranking information associated with the one or more identified mission files is based on at least one of first context information associated with the one or more robot-independent tasks and second context information associated with the one or more identified mission files.

[0259] In some embodiments, the second context information includes at least one of an operational status associated with a facility included in the environment and an environmental factor associated with the environment.

[0260] In some embodiments, generating ranking information associated with the one or more identified mission files is based on a set of target criteria associated with at least one of a user and an environment.

[0261] In some embodiments, generating ranking information associated with the one or more identified mission files is based on the amount of one or more tasks that match one or more robot-independent tasks.

[0262] At 1430, process flow 1400 may include providing one or more identified mission files based on the ranking information. In some embodiments, providing the one or more identified mission files includes displaying the one or more identified mission files based on the ranking information.

[0263] At 1435, the process flow 1400 may include determining ranking information and a confidence value associated with the one or more identified mission files.

[0264] At 1440, process flow 1400 may include automatically performing at least a portion of one or more recorded robotic missions of the set of recorded robotic missions based on the confidence value meeting a threshold.

[0265] FIG. 15 shows a process flow 1500 that supports building robotic missions based on mission metrics. Process flow 1500 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of process flow 1500 may be implemented by system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0266] In the following description of process flow 1500, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1500, or other operations may be added to process flow 1500.

[0267] Aspects of process flow 1500 may be implemented by a robot management system that includes an interface for providing commands to multiple robotic devices of one or more robot types and a data repository that includes data records of mission files associated with recorded robotic mission configurations. The robot management system manages the one or more robotic missions and the multiple robotic devices in accordance with aspects of process flow 1500.

[0268] At 1501, process flow 1500 may include providing image data acquired in association with one or more recorded robotic missions of a set of recorded robotic missions to a machine learning model.

[0269] At 1503, the process flow 1500 can include receiving output in response to the machine learning model processing the image data.

[0270] In some embodiments, the output includes one or more measurements associated with one or more recorded robotic missions and one or more confidence values ​​associated with the one or more measurements.

[0271] At 1505, the process flow 1500 can include providing a data record of a mission file associated with a set of recorded robotic missions.

[0272] At 1510, the process flow 1500 may include receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to the environment.

[0273] At 1515, the process flow 1500 may include determining first context information associated with the one or more robot-independent tasks.

[0274] At 1520, process flow 1500 may include searching data records of mission files and identifying one or more mission files that include one or more tasks that match the one or more robot-independent tasks based on the search and the first context information.

[0275] At 1525, process flow 1500 may include generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files. In some embodiments, the recorded metric data includes one or more measurements, one or more confidence values, or both (e.g., as output by the machine learning model at 1503).

[0276] At 1530, the process flow 1500 may include providing one or more identified mission files based on the ranking information.

[0277] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0278] 16 illustrates a process flow 1600 that supports building robotic missions based on mission metrics. Process flow 1600 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of process flow 1600 may be implemented by system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0279] In the following description of process flow 1600, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1600, or other operations may be added to process flow 1600.

[0280] Aspects of process flow 1600 may be implemented by a robotic system platform that includes an interface for providing commands to a plurality of robotic devices, where the plurality of robotic devices are independent of the robotic system platform, a data repository that includes mission file data records associated with recorded robotic mission configurations, and one or more circuits.

[0281] At 1605, process flow 1600 can include electronically receiving instructions for one or more tasks to be performed with respect to the environment. In some embodiments, the one or more tasks are missing instructions for at least one of a target robot type associated with completing the one or more tasks, a target robot function associated with completing the one or more tasks, and a target quantity of robotic device associated with completing the one or more tasks.

[0282] At 1610, process flow 1600 may include searching mission file data records and, based on the search, providing one or more identified mission files corresponding to one or more recorded robot missions that at least partially completed one or more tasks.

[0283] At 1615, the process flow 1600 may include generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files.

[0284] At 1620, the process flow 1600 may include providing one or more identified mission files based on the ranking information.

[0285] 17 shows a flowchart 1700 that supports building robotic missions based on a robot-independent GUI. The process flow 1700 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of the process flow 1700 may be implemented by the system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0286] In the following description of process flow 1700, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1700, or other operations may be added to process flow 1700.

[0287] At 1705, the process flow 1700 can include presenting a robot independent control GUI. For example, 1705 can include presenting the robot independent control GUI in response to executing a set of instructions.

[0288] At 1710, the process flow 1700 may include receiving a first command sequence at a robot-independent control GUI to operate a first type of robot to perform a first set of tasks (e.g., several tasks).

[0289] At 1715, the process flow 1700 may include recording the first sequence of commands to a first robot mission file.

[0290] At 1720, process flow 1700 may include associating one or more tasks with one or more robot capabilities (e.g., mobility type, payload type, etc.). For example, 1720 may include associating tasks included in a first robot mission file with robot capabilities.

[0291] At 1725, the process flow 1700 may include associating the robot capabilities with the robot type.

[0292] At 1730, the process flow 1700 may include identifying one or more robot capabilities that are common to two or more robot types.

[0293] At 1735, process flow 1700 may include receiving, at the robot-independent control GUI, a second command sequence for operating a second type of robot to perform a set of tasks (e.g., several tasks). In some embodiments, the first set of tasks performed by the first robot type (at 1710) and the second set of tasks performed by the second robot type at least partially overlap. For example, between the first set of tasks and the second set of tasks, at least one task may be included in both the first set of tasks and the second set of tasks (e.g., is common).

[0294] At 1740, the process flow 1700 may include recording the second sequence of commands to a second robot mission file.

[0295] At 1745, process flow 1700 may include mapping one or more robot-independent process automation inspection tasks across the first robot mission file and the second robot mission file. For example, 1745 may include mapping the one or more robot-independent process automation inspection tasks. In some embodiments, the one or more robot-independent process automation inspection tasks may include at least one task that is included in (e.g., common to) both the first set of tasks and the second set of tasks.

[0296] At 1750, the process flow 1700 may include receiving a search request for one or more robot-independent process automation inspection tasks.

[0297] At 1755, the process flow 1700 may include returning (eg, presenting) the first robot mission file and the second robot mission file in response to the search request and the mapping.

[0298] 18 shows a flowchart 1800 that supports building robotic missions based on a robot-independent GUI. Process flow 1800 may be implemented, for example, in the industrial setting 113 shown in FIG. 1. Aspects of process flow 1800 may be implemented by system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0299] In the following description of process flow 1800, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 1800, or other operations may be added to process flow 1800.

[0300] Process flow 1800 can be implemented by a robot management system including a processor and a memory coupled to the processor; the memory stores data that, when executed by the processor, enables the processor to perform aspects of process flow 1800 described herein.

[0301] At 1805, the process flow 1800 may include electronically receiving a first set of commands related to operating a first robotic device to perform a first set of tasks.

[0302] At 1810, the process flow 1800 may include recording a first set of commands to a first mission file.

[0303] At 1815, process flow 1800 may include electronically receiving a second set of commands related to operating a second robotic device to perform a second set of tasks.

[0304] In some embodiments, the first set of commands comprises a first sequence of commands and the second set of commands comprises a second sequence of commands.

[0305] At 1820, the process flow 1800 may include recording a second set of commands to a second mission file.

[0306] At 1825, the process flow 1800 may include identifying a third set of tasks that are common to the first set of tasks and the second set of tasks.

[0307] At 1826, the process flow 1800 may include mapping a third set of tasks to the first mission file and the second mission file.

[0308] At 1827, process flow 1800 may include confirming a mapping of the third set of tasks to the first mission file and the second mission file based on at least one of a user input confirming the mapping and a comparison result of a confidence value associated with the mapping to a threshold value.

[0309] At 1828, process flow 1800 may include generating a third mission file including the third set of tasks in response to mapping the third set of tasks to the first mission file and the second mission file. For example, process flow 1800 may include generating the third mission file in response to verifying the mapping.

[0310] At 1830, process flow 1800 may include providing the first mission file and the second mission file in response to receiving a search request indicating one or more tasks of the third task set. In some aspects, at 1830, process flow 1800 may include providing a third mission file.

[0311] In some embodiments, providing the first mission file and the second mission file is based on mapping.

[0312] In some cases, process flow 1800 may include displaying a GUI and at least one of receiving a first set of commands, receiving a second set of commands, receiving a search request, and providing a first mission file and a second mission file being performed via the GUI.

[0313] In some embodiments, the first set of tasks, the second set of tasks, and the third set of tasks comprise robot-independent tasks that lack an indication of the target robot type.

[0314] In some embodiments, the first set of tasks, the second set of tasks, and the third set of tasks comprise process automation inspection tasks.

[0315] In some embodiments, the first set of tasks, the second set of tasks, and the third set of tasks are devoid of robot type instructions.

[0316] In some embodiments, a first set of candidate tasks executable by a first robotic device at least partially overlaps with a second set of candidate tasks executable by a second robotic device.

[0317] 19 shows a flowchart 1900 that supports building robotic missions based on a robot-independent GUI. The process flow 1900 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of the process flow 1900 may be implemented by the system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0318] In the following description of process flow 1900, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Some operations may be omitted from process flow 1900, or other operations may be added to process flow 1900.

[0319] Process flow 1900 can be implemented by a robot management system that includes a processor and a memory coupled to the processor; the memory stores data that, when executed by the processor, enables the processor to perform aspects of process flow 1900 described herein.

[0320] At 1905, process flow 1900 may include electronically receiving a first set of commands related to operating a first robotic device to perform a first set of tasks.

[0321] At 1910, the process flow 1900 may include recording a first set of commands to a first mission file.

[0322] At 1919, process flow 1900 may include electronically receiving a second set of commands related to operating a second robotic device to perform a second set of tasks.

[0323] At 1920, the process flow 1900 may include recording a second set of commands to a second mission file.

[0324] At 1925, the process flow 1900 may include identifying a third set of tasks that are common to the first set of tasks and the second set of tasks.

[0325] At 1930, the process flow 1900 may include, in response to receiving a search request indicating one or more tasks of the third task set, providing the first mission file and the second mission file.

[0326] At 1935, process flow 1900 may include identifying at least one robot capability common to the two or more robot types based on the third set of tasks. In some embodiments, the at least one robot capability includes a mobility type and a payload type.

[0327] In one example, process flow 1900 may include associating one or more tasks of the third set of tasks with one or more robot capabilities; associating the one or more robot capabilities with two or more robot types; and identifying at least one robot capability common to the two or more robot types based on associating the one or more robot capabilities with the two or more robot types.

[0328] 20 shows a flowchart 2000 that supports building robotic missions based on a robot-independent GUI. Process flow 2000 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of process flow 2000 may be implemented by system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0329] In the following description of process flow 2000, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Some operations may be omitted from process flow 2000, or other operations may be added to process flow 2000.

[0330] Process flow 2000 can be implemented by a robotic system platform including a database containing data records of mission files associated with a set of recorded robotic missions, a processor, and a memory coupled to the processor; the memory stores data that, when executed by the processor, enables the processor to perform aspects of process flow 2000 described herein.

[0331] At 2005, the process flow 2000 may include presenting a graphical user interface (GUI).

[0332] At 2010, process flow 2000 may include electronically receiving, via a GUI, a search request indicating one or more candidate tasks to be performed with respect to the environment. In some embodiments, the search request does not include an indication of a target robotic device, a target robot type, or both, in association with performing the one or more candidate tasks.

[0333] In 2015, the process flow 2000 may include identifying at least a first mission file and a second mission file from the data records of the mission files based on a mapping of the one or more candidate tasks to the first mission file and the second mission file. In some embodiments, the data records of the mission files are associated with a set of recorded robot missions.

[0334] At 2020, the process flow 2000 may include providing, via the GUI, instructions for the first mission file and the second mission file.

[0335] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0336] The exemplary systems and methods of the present disclosure are described in connection with an example robot management system and build robotic missions using a robot-independent GUI. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits certain known structures and devices. This omission should not be construed as a limitation on the scope of the claimed disclosure. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein.

[0337] 21 shows a flowchart 2100 that supports building a robotic mission based on a mission inventory. The process flow 2100 may be implemented, for example, in the industrial facility 113 shown in FIG. 1. Aspects of the process flow 2100 may be implemented by the system 100 (e.g., of FIG. 1), an operations management system (e.g., operations management system 700 of FIG. 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0338] In the following description of process flow 2100, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 2100, or other operations may be added to process flow 2100.

[0339] At 2105, the process flow 2100 can include obtaining two or more recorded robot mission files.

[0340] At 2110, the process flow 2100 may include mapping one or more robot-independent process automation inspection tasks across two or more recorded robot mission files.

[0341] At 2115, the process flow 2100 may include receiving user input. In some embodiments, the user input identifies one or more non-robot dependent process automation inspection tasks to be performed.

[0342] At 2120, process flow 2100 may include identifying, based on the mapping, one or more of the recorded robot mission files that, alone or in combination with at least one other recorded mission file, cause one or more robot-independent process automation inspection tasks requested by the user to be performed.

[0343] In some embodiments, the process flow 2100 may include segmenting one or more recorded mission files to map specific portions of one or more missions to the accomplishment of specific tasks.

[0344] Figure 22 shows a flowchart 2200 that supports building a robotic mission based on a mission inventory. Process flow 2200 may be implemented, for example, in the industrial facility 113 shown in Figure 1. Aspects of process flow 2200 may be implemented by system 100 (e.g., of Figure 1), an operations management system (e.g., operations management system 700 of Figure 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0345] In the following description of process flow 2200, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 2200, or other operations may be added to process flow 2200.

[0346] Aspects of process flow 2200 can be implemented by a robot management system that includes an interface for providing commands to a plurality of robotic devices of one or more robot types and a data repository that includes mission file data records associated with recorded robotic mission configurations, where the robot management system manages the one or more robotic missions and the plurality of robotic devices in accordance with aspects of process flow 2200.

[0347] At 2205, process flow 2200 may include receiving user input indicating one or more target tasks to be performed. Additionally or alternatively, at 2205, process flow 2200 may include receiving user input including an indication of a target goal.

[0348] In some embodiments, the one or more target tasks may include one or more robot-independent tasks. In some embodiments, the one or more target tasks may include one or more process automation inspection tasks. In some other embodiments, the one or more target tasks may not have a robot type indication. In some examples, the target goal may not have a robot type indication.

[0349] At 2210, the process flow 2200 may include identifying one or more target tasks to be performed.

[0350] In some examples, identifying the one or more target tasks to be performed may be based on receiving user input indicating the one or more target tasks to be performed. In some examples, identifying the one or more target tasks may be based on user input including an indication of a target goal.

[0351] At 2215, process flow 2200 may include identifying a recorded mission file based on the one or more target tasks. In some embodiments, identifying the recorded mission file includes identifying (at 2220) one or more recorded tasks of the recorded mission file that at least partially achieve a target goal associated with the one or more target tasks.

[0352] In some embodiments, a recorded mission file may be associated with a recorded robot mission.

[0353] In some embodiments, the process flow 2200 may include providing, from the data repository, a set of recorded mission files associated with implementing one or more recorded tasks at 2225. The set of recorded mission files may include a recorded mission file.

[0354] At 2230, the process flow 2200 may include receiving a user input indicating a recorded mission file.

[0355] At 2235, process flow 2200 may include providing one or more commands to one or more robotic devices in connection with performing one or more recorded tasks. In some embodiments, implementing the one or more recorded tasks at least partially achieves a target goal.

[0356] In some embodiments, providing one or more commands to one or more robotic devices is in response to user input received in 2230.

[0357] In some embodiments, process flow 2200 may include automatically providing one or more commands based on a confidence value associated with a comparison between one or more recorded tasks and one or more target tasks.

[0358] In some cases, 2235 of process flow 2200 may include providing one or more commands to one or more robotic devices in connection with implementing (e.g., further implementing) one or more second recorded tasks associated with the second recorded mission file. In some embodiments, implementing the one or more recorded tasks and the one or more second recorded tasks completely achieves a target goal.

[0359] In some embodiments, process flow 2200 may include identifying at least one second recorded mission file based on the one or more target tasks, hi some embodiments, the at least one second recorded mission file includes one or more second recorded tasks that at least partially achieve the target goal.

[0360] In some cases, 2235 of process flow 2200 may include providing one or more commands to one or more robotic devices in connection with implementing the one or more recorded tasks and the one or more second recorded tasks. In some embodiments, implementing the one or more recorded tasks and the one or more second recorded tasks completely achieves a target goal.

[0361] In some embodiments, process flow 2200 may include segmenting one or more recorded mission files of a set of recorded mission files, mapping one or more segments of the one or more recorded mission files to one or more recorded objectives, one or more recorded tasks, or both, and storing the mapping in a data record of the recorded mission file.

[0362] Figure 23 shows a flowchart 2300 that supports building a robotic mission based on a mission inventory. The process flow 2300 may be implemented, for example, in the industrial facility 113 shown in Figure 1. Aspects of the process flow 2300 may be implemented by the system 100 (e.g., of Figure 1), an operations management system (e.g., operations management system 700 of Figure 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of the RSP (e.g., robot fleet manager 628).

[0363] In the following description of process flow 2300, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 2300, or other operations may be added to process flow 2300.

[0364] Aspects of process flow 2300 can be implemented by a robot management system that includes an interface for providing commands to a plurality of robotic devices of one or more robot types and a data repository that includes mission file data records associated with recorded robotic mission configurations, where the robot management system manages the one or more robotic missions and the plurality of robotic devices in accordance with aspects of process flow 2300.

[0365] At 2310, the process flow 2300 may include identifying one or more target tasks to be performed.

[0366] At 2315, the process flow 2300 may include identifying a recorded mission file based on one or more target tasks.

[0367] In some embodiments, at 2316, the process flow 2300 may include segmenting the recorded mission file in response to identifying the one or more target tasks.

[0368] In some cases, at 2317, process flow 2300 may include identifying a mapping between one or more segments of the recorded mission file and at least one of the objective goal and the one or more target tasks associated with the one or more target tasks.

[0369] In some embodiments, identifying (at 2315) the recorded mission file includes identifying (at 2318) one or more recorded tasks of the recorded mission file that at least partially achieve a target goal associated with the one or more target tasks.

[0370] In some embodiments, identifying 2318 one or more recorded tasks that at least partially achieve the target goal is based on the results of the mapping. In some embodiments, each of the segments includes at least one recorded task associated with a recorded mission file.

[0371] At 2320, the process flow 2300 may include providing one or more commands to one or more robotic devices in connection with performing one or more recorded tasks. In some cases, performing the one or more recorded tasks at least partially achieves a target goal.

[0372] In some cases, at 2325, the process flow 2300 may include segmenting the second recorded mission file in response to identifying that one or more recorded tasks do not fully achieve the target goal.

[0373] At 2330, process flow 2300 may include identifying a second mapping between one or more second segments of the second recorded mission file and at least one of the target goal and the one or more target tasks.

[0374] At 2335, process flow 2300 may include identifying one or more second recorded tasks of the second recorded mission file that at least partially achieve the target goal based on the results of the second mapping. In some embodiments, the one or more recorded tasks and the one or more second recorded tasks fully achieve the target goal and fully overlap with the one or more target tasks.

[0375] Figure 24 shows a flowchart 2400 that supports building a robotic mission based on a mission inventory. Process flow 2400 may be implemented, for example, in the industrial facility 113 shown in Figure 1. Aspects of process flow 2400 may be implemented by system 100 (e.g., of Figure 1), an operations management system (e.g., operations management system 700 of Figure 7), a user (e.g., via a control system process control client 604, web client 606, or via a direct input device), an RSP (e.g., RSP 672), another system in communication with the RSP, or a robot fleet manager of an RSP (e.g., robot fleet manager 628).

[0376] In the following description of process flow 2400, operations may be performed in a different order than shown, or operations may be performed in a different order or at different times. Certain operations may be excluded from process flow 2400, or other operations may be added to process flow 2400.

[0377] Aspects of process flow 2400 may be implemented by a robotic system platform that includes an interface for providing commands to multiple robotic devices of one or more robot types, where the multiple robotic devices are independent of the robotic system platform, a data repository that includes mission file data records associated with recorded robotic mission configurations, and one or more circuits.

[0378] At 2405, process flow 2400 includes identifying one or more target tasks to be performed with respect to the environment, where identifying the one or more target tasks is in response to electronically receiving one or more target task instructions, target goal instructions, or both.

[0379] At 2410, process flow 2400 may include identifying one or more recorded mission files from the mission file data records based on the one or more goal tasks, the one or more recorded mission files including one or more recorded tasks that at least partially accomplish the goal goal.

[0380] At 2415, the process flow 2400 may include providing one or more commands to one or more robotic devices of the plurality of robotic devices in connection with performing the one or more recorded tasks.

[0381] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0382] The exemplary systems and methods of the present disclosure have been described in connection with an example of building a robotic mission based on a mission inventory. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits certain known structures and devices. This omission should not be construed as a limitation on the scope of the claimed disclosure. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein.

[0383] Exemplary systems and methods of the present disclosure are described in connection with an example robotic management system. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits certain known structures and devices. This omission should not be construed as a limitation on the scope of the claimed disclosure. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein.

[0384] Furthermore, while the exemplary embodiments illustrated herein show various components of the system as collocated, some components of the system may be located remotely, in remote portions of a distributed network such as a communications network and / or the Internet, or in dedicated secure, unsecure, and / or encrypted systems. Accordingly, it should be understood that components of the system may be combined into one or more devices, such as a server, a communications device, or may be collocated on particular nodes / elements of a distributed network, such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. From the foregoing discussion and for reasons of computational efficiency, it will be appreciated that components of the system may be located anywhere within the distributed network of components without affecting the operation of the system.

[0385] Furthermore, it should be understood that the various communication links, including communication channels, connecting elements can be wired or wireless links, or any combination thereof, or any other known or later-developed element capable of providing and / or communicating data and / or signals to and from the connected elements. As used herein, the term module can refer to any known or later-developed hardware, software, firmware, or combination thereof capable of performing the function associated with that element. The terms “determining,” “calculating,” and “computing,” as well as variations thereof, are used interchangeably herein and include any type of methodology, process, mathematical operation, or technique. These wired or wireless links may also be secure links and capable of communicating encrypted information. Transmission media used as links can be any suitable carrier for electrical signals, including, for example, coaxial cable, copper wire, and optical fiber, and can take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0386] While process flows are discussed and illustrated with reference to a particular sequence of events, it should be understood that modifications, additions, and omissions to this sequence can be made without substantially affecting the operation of the embodiments. Additionally, the exact sequence of events need not occur as described in the disclosed embodiments; rather, steps may be performed by one or other devices within a system. Additionally, the exemplary techniques illustrated herein are not limited to the specifically illustrated embodiments and may be utilized with other exemplary embodiments, and each described feature may be claimed individually and separately. As will be appreciated by those skilled in the art, aspects of the present disclosure may be embodied as a system, method, and / or computer program product. Accordingly, aspects of the present disclosure may be implemented entirely in hardware, entirely in software (including, but not limited to, firmware, program code, resident software, microcode), or a combination of hardware and software. All such embodiments may be generally referred to herein as circuits, modules, or systems. Additionally, aspects of the present disclosure may be formed as a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.

[0387] The computer-readable medium described herein may be a computer-readable storage medium, including, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. As used herein, a computer-readable storage medium may be any non-transitory tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, device, computer, computing system, computer system, or any programmable machine or device that inputs, processes, and outputs instructions, commands, or data. A non-exhaustive list of specific examples of computer-readable storage media includes an electrical connection having one or more wires, a portable computer diskette, a floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), a USB flash drive, a non-volatile RAM (NVRAM or NOVRAM), an erasable programmable read-only memory (EPROM or flash memory), a flash memory card, an electrically erasable programmable read-only memory (EEPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), a DVD-ROM, an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer-readable storage medium can be any computer-readable medium that is not a computer-readable signal medium, such as a propagated data signal in which computer-readable program code is embodied.

[0388] The program code may be embodied as computer-readable instructions stored on or in a computer-readable storage medium, for example, as source code, object code, interpreted code, executable code, or a combination thereof. Any standard or proprietary programming or interpreted language can be used to generate the computer-executable instructions. Examples of such languages ​​include C, C++, C#, Pascal, JAVA, JAVA Script, BASIC, Smalltalk, Visual Basic, and Visual C++.

[0389] Transmission of the program code embodied on the computer-readable medium may be performed using any suitable medium, including, but not limited to, wireless, wired, fiber optic cable, radio frequency (RF), or any suitable combination thereof.

[0390] The program code may execute entirely on the user / operator / administrator's computer, partially as a standalone software package, partially on the user / operator / administrator's computer, partially on a remote computer, or entirely on a remote computer or server. Any such remote computer may be connected to the user / operator / administrator's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider).

[0391] Many variations and modifications of the present disclosure may be employed, and it is possible to provide some features of the present disclosure without providing others.

[0392] In yet another embodiment, the systems, methods, and protocols described herein can be implemented in connection with special-purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit elements, ASICs or other integrated circuits, digital signal processors, hardwired electronic or logic circuits, e.g., discrete element circuits, programmable logic devices or gate arrays, e.g., PLDs, PLAs, FPGAs, PALs, special-purpose computers, any equivalent means, etc. Generally, any apparatus or means capable of implementing the methods described herein can be used to implement various communication methods, protocols, and techniques according to aspects of the present invention. Exemplary hardware that can be used for the present disclosure includes computers, handheld devices, telephones (e.g., cellular, Internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Furthermore, alternative software implementations, including, but not limited to, distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, can also be configured to implement the methods described herein.

[0393] Examples of processors described herein include, but are not limited to, at least one of the following: Qualcomm® Snapdragon® 610 and 615 with 4G LTE Integration and 64-bit computing, Apple® A7, A8X, A9X, or A10 processors with 64-bit architecture, Apple® M7, M8, M9, or M10 motion coprocessors, Samsung® Exynos® series, Intel® Xeon® processor family, Intel® Atom® processor family, Intel® Core i5-4670K and i7-4770K 2nm Haswell, Intel® i5-3570K 2nm Ivy Bridge, AMD® FX TM Processors, AMD® FX-4300, FX-6300, and FX-8350 2nm Vishera, AMD® Kaveri processors, Texas Instruments® Jacinto C6000 TM Texas Instruments® OMAP Automotive Infotainment Processor TM Automotive Mobile Processor, ARM® Cortex TM -M processors, ARM® Cortex-A and ARM926EJ-S® processors, Broadcom® AirForce BCM4704 / BCM4703 wireless networking processors, AR7100 wireless network processing units, and other industry-equivalent processors may perform computational functions using any known or future-developed standards, instruction sets, libraries, and / or architectures.

[0394] In yet another embodiment, the disclosed methods can be readily implemented in conjunction with software using object-oriented or object-oriented software development environments that provide portable source code that can be used on a variety of computer, workstation, or mobile device platforms. Alternatively, the disclosed systems can be implemented partially or fully in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the system's speed and / or efficiency requirements, specific functionality, and the particular software or hardware system or microprocessor or microcomputer system utilized. However, from the functional descriptions provided herein, the methods presented herein can be readily implemented in hardware and / or software using any known or later-developed system or structure, device, and / or software by those skilled in the art with a general basic knowledge of computer and image processing technology.

[0395] In yet another embodiment, the disclosed methods may be implemented in part in software stored on a storage medium and executable on a cooperatively programmed general-purpose computer, such as a controller and memory, a special-purpose computer, a mobile device, a smartphone, a microprocessor, etc. In these examples, the disclosed systems and methods may be implemented as a program embedded in a personal computer, such as an applet, a JAVA or CGI script, as a resource resident on a server or computer workstation, as a routine embedded in a dedicated measurement system, as a plug-in, as a system component, or the like. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system, such as the hardware and software system of an image processor.

[0396] Although this disclosure describes components and functions implemented in embodiments with reference to particular standards and protocols, this disclosure is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically superseded by faster or more efficient equivalents having essentially the same functionality. Such replacement standards and protocols having the same functionality are considered equivalents included in this disclosure.

[0397] The present disclosure includes, in various embodiments, configurations, and aspects, components, methods, processes, systems, and / or devices substantially as illustrated and described herein, including various embodiments, subcombinations, and subsets thereof. After understanding the present disclosure, one of ordinary skill in the art will understand how to make and use the system inventive methods disclosed herein. The present disclosure, in various embodiments, configurations, and aspects, includes providing devices and processes not depicted and / or described herein and / or in various embodiments, configurations, or aspects herein, including the absence of items used in previous devices or processes to, for example, improve performance, achieve ease, and / or reduce cost of implementation.

[0398] The foregoing discussion of the present disclosure has been presented for purposes of illustration and description. The foregoing is not intended to limit the present disclosure to the form(s) disclosed herein. For example, in the foregoing Detailed Description, various features of the present disclosure are grouped together in one or more embodiments, configurations, or aspects for the purpose of streamlining the disclosure. Features of the embodiments, configurations, or aspects of the present disclosure may be combined in alternative embodiments, configurations, or aspects other than those described above. This method of disclosure is not to be interpreted as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment, configuration, or aspect. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the present disclosure.

[0399] Furthermore, while the description of the present disclosure includes a description of one or more embodiments, configurations, or aspects, and certain variations and modifications, other variations, combinations, and modifications are within the scope of the present disclosure, for example, as would be within the skill and knowledge of one of ordinary skill in the art after understanding the present disclosure. It is intended to entitle, to the fullest extent permitted, alternative embodiments, configurations, or aspects, including alternative, interchangeable, and / or equivalent structures, functions, ranges, or steps to those claimed, whether or not such alternative, interchangeable, and / or equivalent structures, functions, ranges, or steps are disclosed herein, and no patentable subject matter is intended to be offered to the public.

[0400] The words "at least one," "one or more," "or," and "and / or" are open-ended expressions that function as both conjunctions and disjunctions. For example, the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B, and / or C," and "A, B, or C" each mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0401] The term "a" or "an" refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" may be used interchangeably.

[0402] The term "automatic" and variations thereof, as used herein, refers to any process or operation that is performed without human input, typically continuous or semi-continuous, as the process or operation is performed. However, a process or operation can be automated even if material or non-material human input is used in the execution of the process or operation, if that input is received before the process or operation is performed. Human input is considered critical if such input affects how the process or operation is performed. Human input that consents to the performance of a process or operation is not considered "critical."

[0403] Aspects of the present disclosure may take the form of entirely hardware embodiments, entirely software (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which may be generally referred to herein as "circuits," "modules," or "systems," utilizing any combination of one or more computer-readable medium. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0404] As used herein, the terms "determine," "calculate," "compute," and variations thereof are used interchangeably and include any type of methodology, process, mathematical operation, or technique.

[0405] While this technology has been described in conjunction with many embodiments, it is evident that many alternatives, modifications, and variations will be or will be apparent to those skilled in the applicable arts. Accordingly, it is intended to embrace all such alternatives, modifications, equivalents, and variations that are within the spirit and scope of this disclosure.

[0406] All references mentioned in documents filed herewith are incorporated herein by reference in their entirety.

[0407] Aspects of the present disclosure can be configured as follows: (1) A method comprising: Providing data records for mission files related to the configuration of recorded robotic missions; receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to the environment; determining first context information associated with one or more robot-independent tasks; searching the mission file data records and identifying one or more mission files comprising one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files; providing one or more identified mission files based on the ranking information; A method comprising: (2) The method of (1), wherein generating the ranking information associated with the one or more identified mission files includes: first context information relating to one or more robot-independent tasks; second context information associated with the one or more identified mission files; Includes: (3) In one or more of the methods of (1)-(2), the second context information includes at least one of the following: Operating conditions related to the facilities included in the environment; Environmental factors related to the environment. (4) One or more of the methods (1) to (3), the one or more identified mission files include metadata; The metadata includes at least one of the following: Recorded metric data; second context information associated with the one or more identified mission files; Notification of whether a recorded robotic mission corresponding to one or more identified mission files met one or more target criteria. (5) In one or more of the methods of (1) to (4), generating the ranking information associated with the one or more identified mission files is based on: A set of targeting criteria associated with at least one of a user and an environment. (6) In one or more of the methods of (1) to (5), generating the ranking information associated with the one or more identified mission files is based on: Amount of one or more tasks that match one or more robot-independent tasks. (7) One or more of the methods of (1) to (6), wherein determining the first context information is based on setting target criteria associated with the user, the environment, or both. (8) One or more of the methods of (1)-(7), wherein the one or more robotic diagnostic tasks include one or more process automation inspection tasks. (9) One or more of the methods of (1) to (8), wherein providing the one or more identified mission files comprises displaying the one or more identified mission files based on the ranking information. (10) One or more of the methods of (1) to (9), wherein the one or more identified mission files correspond to one or more recorded robot missions of the set of recorded robot missions. (11) One or more of the methods of (1) to (10), further comprising: determining a confidence value associated with the ranking information and the one or more identified mission files; and automatically conducting at least a portion of one or more recorded robot missions of the set of recorded robot missions based on the confidence value meeting a threshold. (12) The recorded metric data includes at least one of the following: performance data relating to the completion of one or more recorded robotic missions; temporal data relating to the completion of one or more recorded robotic missions; The power consumption associated with completing one or more recorded robotic missions. (13) One or more of the methods of (1)-(12), wherein the recorded metric data includes a usage cost associated with the robot and completes one or more recorded robot missions. (14) One or more of the methods of (1) to (13), wherein the recorded metric data includes an indication of whether the one or more recorded robotic missions include at least one operator intervention. (15) One or more of the methods (1) to (14) further include the following: providing image data acquired in association with one or more recorded robotic missions of the set of recorded robotic missions to a machine learning model; receiving an output in response to the machine learning model processing the image data, the output comprising: one or more measurements associated with one or more recorded robotic missions; one or more confidence values ​​associated with one or more measurements; Including, The recorded metric data includes one or more measurements, one or more confidence values, or both. (16) A robot management system, an interface for providing commands to a plurality of robotic devices of one or more robot types; a data repository containing data records of mission files associated with recorded robotic mission configurations; Equipped with The robot management system manages one or more robotic missions and a plurality of robotic devices by the following steps: receiving instructions for one or more robot-independent tasks to be performed with respect to the environment; determining first context information associated with one or more robot-independent tasks; searching the mission file data records and identifying one or more mission files comprising one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files; and providing one or more identified mission files based on the ranking information. (17) The system of (16), wherein generating the ranking information associated with the one or more identified mission files is based on at least one of the following: first context information relating to one or more robot-independent tasks; Second context information associated with the one or more identified mission files. (18) The system or systems of (16) to (17), wherein the second context information includes at least one of the following: Operating conditions related to the facilities included in the environment; Environmental factors related to the environment. (19) One or more systems among (16) to (18), the one or more identified mission files include metadata; The metadata includes at least one of the following: Recorded metric data; second context information associated with the one or more identified mission files; Notification of whether a recorded robotic mission corresponding to one or more identified mission files met one or more target criteria. (20) A robot system platform, an interface for providing commands to a plurality of robotic devices, the plurality of robotic devices being independent of the robotic system platform; a data repository containing data records of mission files associated with recorded robotic mission configurations; one or more circuits; Equipped with The one or more circuits perform the following: electronically receive instructions for one or more tasks to be performed with respect to the environment, the one or more tasks not having a notification of at least one of the following: The target robot type involved in completing one or more tasks; A goal robot function related to the completion of one or more tasks; a target quantity for the robotic device related to the completion of one or more tasks; searching the mission file data records and, based on the search, providing one or more identified mission files corresponding to one or more recorded robotic missions that at least partially completed one or more tasks; generating ranking information associated with the one or more identified mission files based on the recorded metric data associated with the one or more identified mission files; Based on the ranking information, one or more identified mission files are provided. (21) A method comprising: electronically receiving a first set of commands related to operating a first robotic device to perform a first set of tasks; Recording of first command set in first mission file; electronically receiving a second set of commands related to operating a second robotic device to perform a second set of tasks; Recording of second command set in second mission file; Identifying a third set of tasks that are common to the first set of tasks and the second set of tasks; In response to receiving a search request indicating one or more tasks of the third set of tasks, providing a first mission file and a second mission file. (22) The method according to (21), further comprising: Mapping the third set of tasks to the first mission file and the second mission file and providing the first mission file and the second mission file is based on the mapping. (23) One or more of the methods of (21)-(22), further comprising: and ascertaining a mapping of a third set of tasks to the first mission file and the second mission file based on at least one of the following: User input to confirm mapping; Comparison of confidence values ​​relative to the mapping to thresholds. (24) One or more of the methods of (21) to (23), further comprising: Generating the third mission file includes providing a third set of tasks in response to mapping the third set of tasks to the first mission file and the second mission file, and providing the third mission file. (25) One or more of the methods of (21) to (24), further comprising: Identifying at least one robot capability common to the two or more robot types based on the third set of tasks. (26) One or more of the methods of (21) to (25) further comprising: associating one or more tasks of the third set of tasks with one or more robot capabilities; Associating one or more robot capabilities with two or more robot types; identifying the at least one robot capability common to the two or more robot types based on associating the one or more robot capabilities with the two or more robot types. (27) One or more of the methods of (21)-(26), wherein the at least one robotic capability includes: Mobility type; Payload type. (28) One or more of the methods of (21) to (27), further comprising: identifying a set of waypoints common to the first mission file and the second mission file; Providing the first mission file and the second mission file is based on identifying a set of waypoints. (29) One or more of the methods of (21) to (28), further comprising: identifying one or more measurement devices common to the first set of tasks and the second set of tasks based on first image data acquired in association with the first set of tasks and second image data acquired in association with the second set of tasks; Providing the first mission file and the second mission file is based on identifying the one or more measurement devices. (30) One or more of the methods of (21) to (29), further comprising: Displaying a Graphical User Interface (GUI); At least one of receiving the first command set, receiving the second command set, receiving the search request, and providing the first mission file and the second mission file is via a GUI. (31) One or more of the methods of (21) to (30), wherein the first set of tasks, the second set of tasks, and the third set of tasks include robot-independent tasks that do not have an indication of a target robot type. (32) The method of one or more of (21) to (31), wherein the first set of tasks, the second set of tasks, and the third set of tasks comprise process automation inspection tasks. (33) One or more of the methods of (21) to (32), wherein the first set of tasks, the second set of tasks, and the third set of tasks are free of robot type notification. (34) One or more of the methods of (21) to (33), wherein a first set of candidate tasks executable by the first robotic device at least partially overlaps with a second set of candidate tasks executable by the second robotic device. (35) One or more methods among (21) to (34), the first set of commands includes a first sequence of commands; The second set of commands includes a second sequence of commands. (36) A robot management system comprising: processor; a memory coupled to the processor that stores data that, when executed by the processor, enables the processor to: Displaying a graphical user interface (GUI); electronically receiving, via the GUI, a first set of commands related to operating a first robotic device of a first robotic type to perform a first set of tasks; Record the first command set in the first mission file; electronically receiving a second set of commands related to operating a second robotic device of a second robotic type to perform a second set of tasks; Record a second set of commands in the second mission file; Identify a third set of tasks that are common to the first set of tasks and the second set of tasks; In response to receiving a search request indicating one or more tasks of the third set of tasks, a first mission file and a second mission file are provided. (37) The system of (36), wherein the data, when executed by the processor, further causes the processor to perform the following: mapping the third set of tasks to the first mission file and the second mission file; Providing the first mission file and the second mission file is based on the mapping. (38) One or more of the systems of (36) to (37), wherein the data, when executed by the processor, further causes the processor to: Identify a mapping of the third set of tasks to the first and second mission files based on at least one of the following: User input to confirm mapping; Comparison of confidence values ​​relative to the mapping to thresholds. (39) The data, when executed by the processor, further causes the processor to perform one or more of the systems of (36) to (38): In response to mapping the third task set to the first mission file and the second mission file, a third mission file including the third task set is generated. (40) A robot system platform, a database containing mission file data records related to recorded robotic mission configurations; processor; a memory coupled to the processor that stores data that, when executed by the processor, enables the processor to: Present a graphical user interface (GUI); receiving electronically via the GUI a search request indicating one or more candidate tasks to be performed with respect to the environment, wherein the search request does not include an indication of a target robotic device, a target robot type, or both, in association with performing the one or more candidate tasks; identifying at least a first mission file and a second mission file from the mission file data records based on a mapping of the one or more candidate tasks to the first mission file and the second mission file; Provide instructions for the first and second mission files via the GUI. (41) A method comprising: Identification of one or more target tasks to be performed; identifying a recorded mission file based on the one or more target tasks, wherein identifying the recorded mission file at least partially achieves a target goal associated with the one or more target tasks; providing one or more commands to one or more robotic devices in connection with implementing one or more recorded tasks; and Implementing one or more recorded tasks at least partially achieves the target goal. (42) The method according to (41), further comprising: identifying at least one second recorded mission file based on the one or more goal tasks, the at least one second recorded mission file comprising one or more second recorded tasks that at least partially achieve the goal; providing one or more commands to one or more robotic devices in connection with implementing the one or more recorded tasks and the one or more second recorded tasks; Implementing one or more recorded tasks and one or more second recorded tasks completely achieves the target goal. (43) One or more of the methods of (41)-(42), further comprising: segmenting one or more recorded mission files of the set of recorded mission files; mapping one or more segments of one or more recorded mission files to one or more recorded objectives, one or more recorded tasks, or both; The mapping is stored in a data record of the recorded mission file. (44) One or more of the methods of (41) to (43), further comprising: segmenting the recorded mission file in response to identifying one or more target tasks; Identify a mapping between one or more segments of the recorded mission file and at least one of the following: Target goal; the one or more target tasks; Identifying the one or more recorded tasks that at least partially achieve the target goal is based on a result of the mapping. (45) One or more methods of (41) to (44), Each of the segments includes at least one recorded task associated with a recorded mission file. (46) One or more of the methods of (41) to (45), further comprising: segmenting the second recorded mission file in response to identifying one or more recorded tasks not fully achieving the target goal; identifying a second mapping between one or more second segments of the second recorded mission file and at least one of the following: Target goal; one or more target tasks; identifying one or more second recorded tasks in the second recorded mission file that at least partially achieve the target goal based on results of the second mapping; The one or more recorded tasks and the one or more second recorded tasks: Completely achieve your goal, Completely overlap one or more target tasks. (47) One or more of the methods of (41) to (46), further comprising: Receiving user input indicating one or more target tasks to be performed. (48) One or more of the methods of (41) to (47), further comprising: receiving user input including an indication of a target goal; Identifying one or more target tasks is based on an indication of a target goal. (49) One or more of the methods of (41) to (48), further comprising: The one or more commands are automatically provided based on a confidence value associated with a comparison between the one or more recorded tasks and the one or more target tasks. (50) One or more of the methods of (41) to (49), further comprising: providing, from the data repository, a recorded mission file configuration associated with implementing one or more recorded tasks, the recorded mission file configuration comprising the recorded mission file; receiving a user input indicating a recorded mission file; Providing one or more commands to one or more robotic devices is responsive to user input. (51) One or more of the methods of (41) to (50), further comprising: providing one or more commands to the one or more robotic devices in association with implementing one or more second recorded tasks associated with the second recorded mission file; Implementing one or more recorded tasks and one or more second recorded tasks completely achieves the target goal. (52) One or more of the methods (41) to (51), wherein the recorded mission file is associated with the recorded robot mission. (53) One or more of the methods of (41) to (52), wherein the one or more target tasks comprise one or more robot-independent tasks. (54) One or more of the methods of (41) to (53), wherein the one or more target tasks include one or more process automation inspection tasks. (55) One or more of the methods of (41) to (54), wherein the one or more target tasks are free of robot-type instructions. (56) One or more of the methods (41) to (55), wherein the target goal is free of robot-type instructions. (57) A robot management system comprising: an interface for providing commands to a plurality of robotic devices of one or more robot types; a data repository containing data records of mission files associated with recorded robotic mission configurations; Equipped with The robot management system manages one or more robotic missions and a plurality of robotic devices by the following steps: Identification of one or more target tasks to be performed; identifying a recorded mission file based on the one or more target tasks, wherein identifying the recorded mission file at least partially achieves a target goal associated with the one or more target tasks; providing one or more commands to one or more robotic devices in connection with implementing one or more recorded tasks; Implementing one or more recorded tasks at least partially achieves the target goal. (58) The system of claim 57, wherein the data, when executed by the processor, further causes the processor to: identifying at least one second recorded mission file based on the one or more target tasks, the at least one second recorded mission file including one or more second recorded tasks that at least partially achieve the target goal; providing one or more commands to one or more robotic devices in connection with performing the one or more recorded tasks and the one or more second recorded tasks; Performing one or more recorded tasks and one or more second recorded tasks fully accomplishes the goal. (59) The data, when executed by the processor, causes the processor to perform one or more of the systems of (57) to (58): segmenting one or more recorded mission files of the set of recorded mission files; mapping one or more segments of one or more recorded mission files to one or more recorded objectives, one or more recorded tasks, or both; The mapping is stored in a data record of the recorded mission file. (60) A robotic system platform, an interface for providing commands to a plurality of robotic devices of one or more robot types, the plurality of robotic devices being independent of the robotic system platform; a data repository containing data records of mission files associated with recorded robotic mission configurations; one or more circuits; Equipped with The one or more circuits perform the following: identifying one or more target tasks to be performed with respect to the environment, wherein identifying the one or more target tasks is responsive to electronically receiving instructions for the one or more target tasks, instructions for the target goal, or both; identifying one or more recorded mission files from the mission file data records based on the one or more goal tasks, the one or more recorded mission files comprising one or more recorded tasks that at least partially accomplish the goal; Providing one or more commands to one or more robotic devices of the plurality of robotic devices in connection with implementing the one or more recorded tasks.

Claims

1. providing a data record of a mission file associated with a set of recorded robotic missions; receiving input from a user including instructions for one or more robot-independent tasks to be performed with respect to the environment; determining first context information associated with the one or more robot-independent tasks; searching the data records of the mission files and identifying one or more mission files that include one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files; providing the one or more identified mission files based on the ranking information; A method having the following.

2. generating the ranking information associated with the one or more identified mission files, the first context information related to the one or more robot-independent tasks; second context information associated with the one or more identified mission files; The method of claim 1 , wherein the method is based on at least one of:

3. The second context information is operational status of facilities contained within said environment; environmental factors related to said environment; The method of claim 2 , comprising at least one of:

4. the one or more identified mission files include metadata; The metadata includes: the recorded metric data; second context information associated with the one or more identified mission files; notifying whether a recorded robotic mission corresponding to the one or more identified mission files met one or more target criteria; The method of claim 1 , comprising at least one of:

5. generating the ranking information associated with the one or more identified mission files, A set of targeting criteria associated with at least one of the user and the environment. The method of claim 1 , based on

6. generating the ranking information associated with the one or more identified mission files, Amount of one or more tasks that match the one or more robot-independent tasks The method of claim 1 , based on

7. The method of claim 1 , wherein determining the first context information is based on a set of targeting criteria associated with the user, the environment, or both.

8. The method of claim 1 , wherein the one or more robot-independent tasks include one or more process automation inspection tasks.

9. The method of claim 1 , wherein providing the one or more identified mission files comprises displaying the one or more identified mission files based on the ranking information.

10. The method of claim 1 , wherein the one or more identified mission files correspond to one or more recorded robotic missions of the set of recorded robotic missions.

11. The method further comprises: determining a confidence value for the ranking information and the one or more identified mission files; automatically performing at least a portion of one or more of the set of recorded robotic missions based on the confidence value satisfying a threshold; 11. The method of claim 10, comprising:

12. The recorded metric data may include: performance data regarding the completion of the one or more recorded robotic missions; temporal data regarding the completion of the one or more recorded robotic missions; power consumption associated with completing the one or more recorded robotic missions; The method of claim 10, comprising at least one of:

13. The method of claim 10 , wherein the recorded metric data includes a usage cost associated with a robot and completion of the one or more recorded robot missions.

14. The method of claim 10 , wherein the recorded metric data includes an indication of whether the one or more recorded robotic missions included at least one operator intervention.

15. The method further comprises: providing image data acquired in association with one or more of the set of recorded robotic missions to a machine learning model; receiving an output in response to the machine learning model processing the image data, the output comprising: one or more measurements associated with the one or more recorded robotic missions; one or more confidence values ​​associated with the one or more measurements; the steps of: and the recorded metric data includes the one or more measurements, the one or more confidence values, or both; The method of claim 1.

16. A robot management system, an interface that provides commands to a plurality of robotic devices of one or more robot types; a data repository containing data records of mission files associated with a set of recorded robotic missions; Equipped with The robot management system includes: receiving instructions for one or more robot-independent tasks to be performed with respect to the environment; determining first context information associated with the one or more robot-independent tasks; searching the mission file data records and identifying one or more mission files including one or more tasks that match the one or more robot-independent tasks based on the search and the first context information; generating ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files; providing the one or more identified mission files based on the ranking information; managing one or more robotic missions and the plurality of robotic devices by Robot management system.

17. generating the ranking information associated with the one or more identified mission files, first context information related to the one or more robot-independent tasks; second context information associated with the one or more identified mission files; is based on at least one of 17. The system of claim 16.

18. The second context information is operational status of facilities contained within said environment; environmental factors related to said environment; 20. The system of claim 17, comprising at least one of:

19. the one or more identified mission files include metadata; The metadata includes: the recorded metric data; second context information associated with the one or more identified mission files; notifying whether a recorded robotic mission corresponding to the one or more identified mission files met one or more target criteria; at least one of:

17. The system of claim 16.

20. 1. A robotic system platform, comprising: an interface for providing commands to a plurality of robotic devices, the plurality of robotic devices being independent of the robotic system platform; a data repository containing data records of mission files associated with a set of recorded robotic missions; one or more circuits; Equipped with The one or more circuits electronically receiving instructions for one or more tasks to be performed with respect to the environment, said one or more tasks comprising: a target robot type for completion of the one or more tasks; a goal robot function related to the completion of the one or more tasks; a target amount for the robotic device regarding completion of the one or more tasks; not having at least one notification of searching the data records of the mission files and, based on the searching, providing one or more identified mission files corresponding to the one or more recorded robotic missions that at least partially completed the one or more tasks; generating ranking information associated with the one or more identified mission files based on recorded metric data associated with the one or more identified mission files; providing the one or more identified mission files based on the ranking information; A robot system platform that implements the above.

Images