Component Anomaly Detection Using Pose-Dependent Machine Learning Models

Abstract

To use pose-dependent machine learning (ML) models in determining whether an anomaly is present within an environment.SOLUTION: A first ML model is used to generate first output based on processing a first image that is captured by a camera of a mobile robot when the camera is at a first pose. The first ML model is used based on the first image being captured at the first pose and the first ML model being trained for the first pose. A second ML model is used to generate second output based on processing a second image that is captured by the camera when the camera is at a distinct second pose. The second ML model is used based on the second image being captured at the second pose and the second ML model being trained for the second pose. The first and second outputs can be used in determining whether an anomaly is present.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to component anomaly detection using pose-dependent machine learning models. [Background technology]

[0002] Complex industrial facilities, such as petrochemical refineries, chemical plants, etc., may contain numerous components utilized in the processing of liquids, gases, and / or other substances involved in the industrial process of the industrial facility. It is important to ensure that the components involved in the industrial process are operating as intended and / or that the substances involved in the industrial process are in the intended state.

[0003] Industrial facilities utilize a variety of sensors to monitor such components and / or materials for anomalies, including temperature sensors utilized to monitor temperature anomalies of components and / or materials, optical sensors utilized to monitor material composition anomalies (e.g., based on exciting the materials with a light source), and the like.

[0004] Industrial facilities may utilize a variety of sensors to monitor various anomalies, but such sensors are typically each located at a corresponding fixed location within the industrial facility. Therefore, a very large number of sensors may be required to efficiently monitor the industrial facility. Furthermore, providing power and / or communications to each of the sensors may require extensive wiring throughout the industrial facility. Furthermore, each of the large number of sensors is prone to failure or malfunction and, when that occurs, must be repaired or replaced, which may be difficult in large industrial facilities that include a large number of fixed sensors. Summary of the Invention [Means for solving the problem]

[0005] Implementations of the present disclosure are directed to utilizing a mobile robot to capture images of components and / or materials in an environment, such as an industrial facility or other environment, at a given pose via a camera of the mobile robot. The implementations are further directed to processing the captured images to generate an ML output using a machine learning (ML) model corresponding to the given pose. The machine learning model may be a neural network model, such as a convolutional neural network (CNN) model, including one or more convolutional layers. The images are processed using the ML model in response to the images captured at the given pose and in response to the ML model corresponding to the given pose at which the images were captured. The implementations are further directed to determining whether the components and / or materials captured by the images have an anomaly based on the generated ML output, and, if so, performing one or more remedial actions. Performing a remedial action may include, for example, rendering an alarm, stopping a process, and / or performing other corrective action.

[0006] As referenced above, an image is processed using an ML model in response to the image captured at a given pose and also in response to the ML model corresponding to the given pose at which the image was captured. For example, metadata can indicate that the image was captured at a given pose, and the ML model can select from among multiple candidate ML models to process the image based on the metadata indicating that the image was captured at the given pose and the ML model that also corresponds to the given pose. The metadata can be included as data incorporated as part of the image or as data separate from but associated with the image. The metadata can directly or indirectly reflect a given position and / or a given orientation at which the image was captured. The given position and / or given orientation reflected by the metadata can be referenced to a world frame, such as a relative world frame. In other words, the given pose can be referenced to some absolute or relative world frame rather than solely relative to the mobile robot. For example, if the camera capturing the images is non-adjustably coupled to the robot (i.e., it is always at a fixed pose relative to the robot), then the given pose (due to the non-adjustable coupling) will change as the robot moves, even though the camera remains at a fixed pose relative to the robot.

[0007] An ML model can correspond to a given pose based on being trained using ground truth images captured at or near the given pose (e.g., within a threshold value for the given pose). For example, the ML model can be trained based at least in part on negative training instances, each of which includes training instance inputs of corresponding images captured at or near the given pose when an anomaly was not present and a labeled training instance output indicating that the anomaly was not present. Also, for example, the ML model can additionally or alternatively be calibrated based at least in part on positive calibration instances, each of which includes calibration instance inputs of corresponding images captured at or near the given pose when an anomaly was present and a labeled calibration instance output indicating that the anomaly was present. The ML model can be indexed or otherwise associated with data indicating that it corresponds to the given pose and / or a pose within a threshold value for the given pose. For example, the ML model can be indexed with data indicative of the given pose based on being trained for the given pose and can be used to process an image based on being indexed with such data and also based on the image being associated with metadata indicative of the given pose.

[0008] In particular, as referenced above, multiple candidate ML models can be provided, each of which can correspond to a different corresponding pose. For example, a first ML model can correspond to a first pose based on being trained based on images captured at or near the first pose, and a second ML model can correspond to a second pose based on being trained based on images captured at or near the second pose, etc. For example, the first ML model can be trained based only on images captured at or near the first pose, and the second ML model can be trained based only on images captured at or near the second pose, etc. Also, for example, the first ML model can be a given ML model fine-tuned based on images captured at or near the first pose, and the second ML model can be a given ML model fine-tuned based on images captured at or near the second pose, etc. For example, the first ML model can be fine-tuned based on training instances, each of which includes corresponding images captured at or near the first pose and corresponding supervised labeled outputs indicating whether an anomaly is present in the corresponding images. For example, the corresponding supervised labeled output can be based on input from a human operator after reviewing the corresponding image, where the input indicates whether an anomaly is present in the image.

[0009] Thus, which of multiple candidate ML models is utilized when processing a given image captured by a mobile robot can depend on the given pose at which the image was captured. For example, if the given image was captured at or near a first pose, a first ML model corresponding to the first pose can be utilized. On the other hand, if the given image was instead captured at or near a second pose, a second ML model corresponding to the second pose can be utilized. Thus, the ML model utilized when processing the image to generate an output used to determine whether an anomaly exists in the component and / or material captured by the image can be selected depending on the pose of the image and can be selected based on the ML model having been trained on ground truth images also captured at or near the pose. Thus, the ML model utilized is applied to the pose of the image, resulting in a more accurate output when the image is processed using the ML model. A more accurate output can reduce the occurrence of false positive and / or false negative anomaly detections. Reducing the occurrence of false positives can prevent network and / or computing device resource utilization in erroneously performing remedial action, such as rendering a false positive alert. Reducing the occurrence of false negatives can prevent the occurrence of unsafe conditions in industrial facilities and / or damage to components and / or materials in industrial facilities.

[0010] In various implementations of the present disclosure, multiple ML models and images, each corresponding to a different pose, are utilized in determining whether an anomaly exists in a given component and / or material. As an example, a first ML model can be trained using first ground truth images capturing a given liquid tank from at or near a first pose. The first ML model can be used to process the first images capturing the given liquid tank from the first pose to generate a first output indicating whether an anomaly associated with the given liquid tank (e.g., an anomaly related to the given liquid tank itself and / or the liquid contained therein) exists. Continuing the example, a second ML model can be trained using second ground truth images capturing the same given liquid tank from at or near a second, separate pose. The second ML model can be used to process a second image capturing the same given liquid tank from a second, separate pose to generate a second output indicating whether an anomaly associated with the given liquid tank exists. Additionally, both the first output from the first ML model and the second output from the second ML model can be considered when determining whether an anomaly related to the liquid tank exists. For example, an anomaly can be detected if (a) the first model output meets a lower anomaly threshold and the second model output also meets a lower anomaly threshold, (b) either the first model output or the second model output meets an upper anomaly threshold, or (c) an average (or other combination) of the first and second model outputs meets a threshold (e.g., a lower, upper, or other threshold).

[0011] Considering both the first output and the second output when determining whether an anomaly related to the liquid tank exists can reduce the occurrence of false positives. For example, the first image may contain significant glare (e.g., due to current sunlight conditions), resulting in the first output (alone) potentially indicating an anomaly despite the absence of a true anomaly related to the liquid tank. However, the second image may contain less glare (or no glare), resulting in the second output (alone) not indicating an anomaly. Considering both the first output and the second output in such a situation can result in a correct no-anomaly determination, while considering only the first output alone can result in an incorrect anomaly determination. As another example, if each of the first and second outputs and / or the combination of the first and second outputs meets a lower anomaly threshold, an anomaly can be detected, and considering both the first and second outputs can result in the lower anomaly threshold being set more aggressively. Such a more aggressive lower anomaly threshold can reduce the occurrence of false negatives. In other words, if only a single output is considered, the lower anomaly threshold needs to be less aggressive (i.e., a higher lower bound) to prevent an excess of false positives.

[0012] While many of the foregoing examples describe the utilization of a first ML model and a second ML model in determining whether an anomaly exists in a given component and / or material, it should be noted that in various implementations, more than two ML models may be utilized for a given component and / or material.

[0013] The mobile robot utilized in capturing images from various poses may be a four-legged robot, a wheeled robot, an unmanned aerial vehicle, a powered guided robot, or any other robot that moves itself through an environment. Images are captured via the mobile robot's vision component. Each image is captured when the vision component is in a corresponding pose (i.e., a given position and orientation). The pose of the vision component when an image is captured at a given time is a function of the robot's pose at the given time. If the vision component is in a fixed pose relative to the robot, the pose of the vision component is purely a function of the robot's pose. If the pose of the vision component is independently adjustable relative to the robot, the pose of the vision component is a function of the robot's pose and the pose of the vision component relative to the robot. Images utilized herein may include, for example, an RGB image including red, green, and blue channels and captured by a monographic RGB camera; an RGB-D image including a depth channel in addition to red, green, and blue channels and captured by a stereographic camera; or a thermal image including one or more thermal channels and captured by a thermal camera.

[0014] By utilizing a mobile robot that includes a vision component and moves around an environment to capture images of components and / or objects of interest for anomaly monitoring / detection, the number of sensors monitoring the environment can be reduced and / or the extensive wiring required for such sensors can be avoided. Furthermore, the accuracy of anomaly detection based on processing each image can be improved by processing each of the one or more captured images of the component and / or material captured at a given pose using a pose-dependent ML model trained to process images corresponding to the given pose for anomaly detection. This allows anomaly detection to be more robust and / or more accurate.

[0015] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates a schematic diagram of an environment in which selected aspects of the present disclosure may be implemented, according to various implementations. [Figure 2] FIG. 1 illustrates a schematic diagram of an example of how the techniques described herein may be implemented, according to various implementations. [Figure 3] FIG. 1 illustrates an exemplary method for carrying out selected aspects of the present disclosure. [Figure 4] FIG. 1 illustrates a schematic diagram of an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0017] Implementations described herein relate to monitoring and selectively detecting anomalies in components (e.g., liquid tanks, piping, wiring) in an environment (e.g., an industrial automation facility) using pose-dependent machine learning (ML) models. In various implementations, images of the components are captured by a vision component (e.g., a camera) carried by or integrated with a mobile robot movable within the environment. The images are captured by the vision component at various poses (i.e., various positions and orientations), and ML models corresponding to the poses are selected to process the images. For example, a first image captured by the vision component at a first pose can be processed using a first ML model trained for anomaly detection based on images captured at or near the first pose, and a second image captured by the vision component at a second pose can be processed using a second ML model trained for anomaly detection based on images captured at or near the second pose. Based on processing the images with the selected ML models (e.g., processing the first image using the first ML model and processing the second image using the second ML model), one or more anomalies (if any) associated with the component can be identified / detected, and one or more remedial actions can be performed to address the detected anomaly or anomalies.

[0018] 1 , an exemplary environment 100 in which various aspects of the present disclosure may be implemented is generally illustrated. In some implementations, the exemplary environment 100 may be or include an industrial facility. The industrial facility may take a variety of forms and may be designed to implement any number of at least partially automated processes. The industrial facility may take the form of a chemical processing plant, an oil or natural gas refinery, a catalyst factory, a manufacturing facility, an offshore oil platform, etc.

[0019] The exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled to a process automation network 106 within an industrial facility. The client devices 103-A and 103-B may be implemented as computers (e.g., laptops, desktops, notebooks), tablets, robots, smart appliances (e.g., smartphones), messaging devices, wearable devices (e.g., watches), or any other applicable devices. The process automation network 106 may be implemented using various wired and / or wireless communication technologies, including, but not limited to, cellular networks such as the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), 3GPP® Long Term Evolution (“LTE”) or other wireless protocols designated as 3G, 4G, 5G, and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).

[0020] The exemplary environment 100 may further include a mobile robot 101 having or carrying a vision component 1011. The mobile robot 101 may be a four-legged robot (e.g., a robotic dog), a wheeled robot, an unmanned aerial vehicle, a robot that moves along elevated and / or non-elevated tracks in an environment, or any other applicable robot capable of moving within an industrial facility. The vision component 1011 may be a monographic camera, a stereographic camera, a thermal camera, or any other applicable vision component to capture one or more images of one or more specific components of the industrial facility (e.g., a liquid tank or tube 102 that stores or transports a liquid substance). The vision component 1011 may be removably coupled to the mobile robot 101 or may be integrated as a non-removable adhesive part of the mobile robot 101. In some implementations, the vision component 1011 may be independently positionable and / or oriented relative to the robot frame of the mobile robot 101, for example, by rotation or other movement via an actuator that independently controls the vision component 1011. In addition to the vision component 1011, the mobile robot 101 may include one or more additional vision components for navigating an industrial facility, sensing static or dynamic objects, and / or capturing images.

[0021] The exemplary environment 100 may further include a server device 105. The server device 105 may include a machine learning (ML) engine 1051 and an anomaly detection engine 1052. The server device 105 may further include or otherwise have access to one or more pose-dependent machine learning (ML) models 1053. The server device 105 may communicate with one or more local client devices (e.g., 122-A and 122-B) and / or with one or more remote client devices (not shown). The local client devices 122-A or 122-B may connect to the server device 105 via one or more local area networks (e.g., process automation network 106), and the remote client devices may connect to the server device 105 via one or more wide area networks (e.g., the Internet). The local and remote client devices may be operable by personnel, such as a system integrator, to configure and / or interact with various aspects of the exemplary environment 100.

[0022] In some implementations, the server device 105, in addition to the ML engine 1051 and the anomaly detection engine 1052, may include a database (not shown) that stores information used by the ML engine 1051 and / or the anomaly detection engine 1052 to implement selected aspects of the present disclosure. Various aspects of the server device 105, such as the ML engine 1051 and / or the anomaly detection engine 1052, may be implemented using any combination of hardware and software. In some implementations, the ML engine 1051, the anomaly detection engine 1052, or one or more pose-dependent ML models 1053 may be implemented across multiple computer systems as part of what is often referred to as a “cloud infrastructure” or simply a “cloud.” However, this is not required; in FIG. 1 , for example, the ML engine 1051 is implemented within an industrial facility, e.g., within a single building, or across a single canvas of a building or other industrial facility. In such implementations, the ML engine 1051 may be implemented on one or more local computing systems, such as one or more local server computers.

[0023] In some implementations, the mobile robot 101 can move through an industrial facility and arrive at a designated spot or waypoint. The vision component 1011 of the mobile robot 101 can be configured to a given pose to capture an image of the liquid tube 102 at the given pose (position and / or orientation). The vision component 1011 can be configured to a given pose as a result of the mobile robot 101 being in the corresponding pose and / or as a result of the vision component 101 being independently adjusted (if the vision component 1011 is independently adjustable relative to the robot frame of the mobile robot 101). The image captured by the vision component 1011 can capture and include pixels or other data corresponding to a region of interest about the liquid tube 102 (e.g., a joint J as shown in FIG. 1 or a region where a wet spot or crack indicating a potential leak was previously identified). Based on a given pose at which an image is captured by the vision component 1011 and / or based on the type of image captured, the ML engine 1051 may select a pose-dependent ML model from the ML models 1053 that corresponds to the given pose for processing the captured image.

[0024] The captured image can be processed by the ML engine 1051 as an input to a selected pose-dependent ML model corresponding to a given pose to generate an ML output of the selected pose-dependent ML model. The ML output indicates whether the region of interest in the liquid tube 102 contains one or more anomalies. In some implementations, the pose-dependent ML model corresponding to a given pose can be trained using training instances, each of which includes (1) an image captured by a visual component (or additional visual components) at a given pose (i.e., at a given location and a given orientation) and (2) a ground truth label indicating whether an anomaly is present in the image (e.g., whether no anomaly is present or whether one or more specific types of anomaly are present). For example, the pose-dependent ML model corresponding to a given pose can be trained using multiple training instances, each of which includes an image captured at a given pose and a ground truth label indicating whether an anomaly is present in the image captured at the given pose. In this case, the ML output can indicate whether the region of interest in the liquid tube 102 contains any anomalies. Training the pose-dependent ML model based on such training instances may include training the pose-dependent ML model only on such training instances, or fine-tuning the pose-dependent ML model by training on such training instances after initial training on non-pose-dependent training instances.

[0025] In some implementations, each ground truth label can be a single corresponding value, such as either a “presence of anomaly” label (e.g., “1”) or a “no anomaly” label (e.g., “0”). In those implementations, the output generated by processing the image using the ML model similarly generates a single corresponding value (e.g., a value from “0” to “1”), where the single value can indicate a corresponding likelihood that an anomaly is present. In some other implementations, each ground truth label can include multiple values, such as a “presence of anomaly” label or a “no anomaly” label for each of multiple regions. For example, the images of the training instances can be 512 pixels by 512 pixels (or other size), and each ground truth label can include nine (or other quantity) values, where each value can indicate a corresponding likelihood that an anomaly is present in a portion of the image. For example, the image can be divided into a 3×3 grid, and each value can indicate a corresponding likelihood that an anomaly is present in one of the nine cells of the 3×3 grid.

[0026] In some implementations, a pose-dependent ML model corresponding to a given pose can be trained using multiple training instances, each including an image captured at the given pose and a ground truth label indicating a particular type of anomaly present in the image captured at the given pose. In some of these implementations, such an ML output can indicate whether a region of interest in the liquid tube 102 includes a particular type of anomaly.

[0027] In various implementations, based on the ML output, the anomaly detection engine 1052 can determine whether an anomaly exists within a region (e.g., a region of interest) of the liquid tube 102 captured in the image processed by the selected pose-dependent ML model. For example, based on the ML output indicating the presence of an anomaly within the captured image, the anomaly detection engine 1052 can determine that the liquid tube 102 includes an anomaly and, in response, cause a remedial action to be performed. For example, the anomaly detection engine 1052 can generate an alert message, execute, or otherwise communicate with other engines / components (not shown) to perform one or more additional remedial actions (e.g., display an alert message 107 via the client devices 103-A and / or 103-B and / or pause an industrial process involving the liquid tube 102).

[0028] While only a single mobile robot 101 is shown in FIG. 1 , it is understood that multiple mobile robots can be deployed in an industrial environment and utilized in the implementations disclosed herein. For example, each of the multiple mobile robots can include a corresponding vision component used to capture images, and images from the multiple mobile robots are transmitted (along with pose data) to the server device 105 for processing by the ML engine 1051 and the anomaly detection engine 1052. Also, while the ML engine 1051, the ML model 1053, and the anomaly detection engine 1052 are shown in FIG. 1 as being implemented separately from the mobile robot 101, in some implementations, all or aspects can be implemented by the mobile robot 101. For example, the mobile robot 101 can include the ML engine 1051, the anomaly detection engine 1052, and at least a subset of the ML model 1053. For example, the mobile robot 101 may include a subset of the ML model 1053 at a given time based on the subset corresponding to the pose at which the mobile robot 101 captures an image in a mission to be performed by the mobile robot 101 at the given time.

[0029] 2 schematically illustrates an example of how the techniques described herein can be implemented according to various implementations. As shown in FIG. 2 , a vision component 1011 of a mobile robot 101 can capture a first image 201 corresponding to a first region (e.g., a first side) of a liquid tube 102 in an industrial facility (sometimes referred to as an “industrial automation facility”), where the first image 201 is captured by the vision component 1011 when the vision component 1011 is in a first pose in the environment. The mobile robot can be controlled (e.g., autonomously, optionally based on a previously recorded mission) to move from the first side to a second side of a particular component to enable the vision component 1011 to capture a second image 203, where the second image 203 is captured by the vision component 1011 when the vision component 1011 is in a second pose in the environment that is different from the first pose. The second image 203 can correspond to a second region (e.g., the right side) of the liquid tube 102. In some implementations, the second region can optionally supplement the first region such that a portion of the liquid tube 102 is captured in its entirety for anomaly detection. In some implementations, the first and second regions can include the same region of the liquid tube 102, but are captured from different perspectives (i.e., as a result of being captured from different poses of the vision component 1011).

[0030] In response to the first image 201 being captured at a first pose, the first ML model 211 may be selected from a plurality of trained ML models. The plurality of trained ML models may be trained based on images captured at a corresponding range of poses (e.g., the first pose, the second pose, etc.) to detect anomalies (e.g., cracks, leaks, thermal anomalies, corrosion, impurities, etc.) in the industrial facility. The first ML model 211 may be selected based on being trained to process images captured at the first pose and / or based on the type of anomaly to be detected from the first image 201. In response to the second image 203 being captured at a second pose, the second ML model 213 may be selected from the plurality of trained ML models based on being trained to process images captured at the second pose and / or based on the type of anomaly to be detected from the second image 203.

[0031] In some implementations, the first image 201 can be determined to be captured in a first pose based on metadata associated with the first image and received from the mobile robot indicating that the first image was captured in a first pose. In those implementations, selecting a first ML model to process the first image is responsive to determining that the first image was captured in a first pose based on metadata associated with the first image. In some implementations, the second image can be determined to be captured in a second pose based on metadata associated with a second image and received from the mobile robot indicating that the second image was captured in a second pose. In those implementations, selecting a second ML model to process the second image is responsive to determining that the second image was captured in a second pose based on metadata associated with the second image.

[0032] The first image 201 can be processed using the first ML model 211 to generate a first ML output 221 of the first ML model 211. The first ML output 221 can indicate whether an anomaly is present in the first image 201. In some implementations, the first ML output 221 can indicate whether a particular type of anomaly is present in the first image 201. The second image 203 can be processed using the second ML model 213 as an input to generate a second ML output 223 of the second ML model 213. The second ML output 223 can indicate whether an anomaly is present in the second image 203. In some implementations, the second ML output 223 can indicate whether a particular type of anomaly is present in the second image 203.

[0033] Based on the first ML output 221 and / or the second ML output 223, it can be determined whether an anomaly (possibly a particular type of anomaly) has been detected for the particular component. For example, the first and second regions can be different regions of one or more particular components, and based on the first ML output 221 indicating an anomaly (e.g., a leak) and the second ML output 223 indicating no anomaly, the anomaly detection engine (e.g., 1052 in FIG. 1 ) can determine that an anomaly has been detected for the particular component (in the first region). In this example, the anomaly detection engine and / or other components can perform one or more remedial actions, including, but not limited to, generating and delivering (audibly and / or visually) an alert message to one or more client devices of relevant personnel within the industrial facility to schedule maintenance or more detailed investigation, pausing or stopping any processes involving the particular component, and / or controlling a mobile robot to capture additional images of higher resolution or different type (than the first image 201 and the second image 203).

[0034] In some implementations, the anomaly detection engine can determine whether an anomaly has been detected for a particular component (e.g., determine whether to trigger the rendering of an alert indicating that an anomaly is present) by determining whether the first ML output 221 (e.g., a value indicating the presence or absence of an anomaly) meets a threshold and by determining whether the second ML output 223 (the “second output”) meets a threshold.

[0035] In some implementations, the anomaly detection engine can determine to cause the rendering of an alert only if both the first output (e.g., 0.8, where numbers closer to "1" are more indicative of the presence of an anomaly and values ​​closer to "0" are less indicative of an anomaly) meets a threshold (e.g., 0.7) and the second output (e.g., 0.9) meets the threshold (0.7). In some implementations, the anomaly detection engine can determine to cause the rendering of an alert if either the first output (e.g., 0.8) or the second output (e.g., 0.6) meets the threshold (0.7).

[0036] In some implementations, the anomaly detection engine can determine a combined output based on the first output and the second output and can determine whether to trigger the rendering of an alert based on the combined output. In some implementations, the anomaly detection engine can determine a level of alert to be rendered based on both the first output and the second output. For example, if the combined output determined from the first and second outputs is determined to meet a first threshold (e.g., 0.7) but not a second threshold (e.g., 0.8), a first type of alert (e.g., a non-emergency alert) can be determined and visually rendered via one or more displays of one or more computing devices in the industrial automation facility. If the combined output is determined to meet a second threshold (e.g., 0.8) but not a third threshold (e.g., 0.9), a second type of alert (e.g., an emergency alert) can be determined and visually rendered via one or more displays. If the combined output is determined to meet a third threshold (e.g., 0.9), a third type of alert (e.g., an emergency alert) may be determined and visually rendered via one or more displays, and a text alert may be delivered to a telephone number of a person responsible for the particular component. If the combined output is determined not to meet the first threshold (e.g., 0.7), no alert is generated or rendered.

[0037] As another example, if the first and second outputs are each determined to meet a first threshold (e.g., 0.7) but not a second threshold (e.g., 0.7), a first type of alert (e.g., a non-emergency alert) may be determined and visually rendered via a display of a computing device within the industrial automation facility. If the first and second outputs are each determined to meet a second threshold (e.g., 0.8) but not a third threshold (e.g., 0.9), a second type of alert (e.g., an emergency alert) may be determined and rendered via a display. If the first and second outputs are each determined to meet a third threshold (e.g., 0.9), a third type of alert (e.g., an emergency alert) may be determined and rendered via a display and / or via an additional computing device. If either the first or second outputs do not meet the first threshold (e.g., 0.7), no alert is generated or rendered.

[0038] In some implementations, the threshold value of the first output, the second output, or the combined output that triggers an alarm may be different from the threshold value (if any) that triggers the suspension of a process involving a particular component in which an anomaly is detected. For example, the threshold value of the first, second, and / or combined output for triggering the suspension of a process involving a particular component may be a higher numerical value (e.g., 0.95) than the aforementioned first, second, and / or third threshold values. In other words, in some implementations, in response to the combined output meeting the threshold value for triggering a suspension (e.g., 0.95) only if the combined output (or alternatively, the first and / or second output) meets / exceeds the numerical value of approximately 0.95, all processes involving a particular component are suspended.

[0039] The example of FIG. 2 is described with respect to capturing images of a particular component from a first pose and a second pose, using a first ML model to process the first pose images, and using a second ML model to process the second pose images, and using output generated from the processing in determining whether an anomaly exists. However, in various implementations, more ML models, or only a single ML model, can be utilized in determining whether an anomaly associated with a given component exists. For example, there may be three (or four or more) ML models, each trained for a given component, but each for a different corresponding pose of the visual component. In such an example, images of the given component may be captured from a first pose, a second pose, and a third pose, a first ML model is used to process the first pose images, a second ML model is used to process the second pose images, and a third ML model is used to process the third pose images, and output generated from the processing can be used in determining whether an anomaly exists.

[0040] 3 is a flowchart illustrating an example method 300 for implementing selected aspects of the present disclosure, according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of server device 105 (and / or additional computing devices, such as mobile robot 101 or client device 103-A), including ML engine 1051 and / or anomaly detection engine 1052. Furthermore, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0041] In block 302A, the system identifies a first image captured by the camera of the mobile robot when the camera is in a first pose. In block 302B, the system identifies a second image captured by the camera of the mobile robot when the camera is in a second pose different from the first pose. The first image and the second image both capture one or more particular components of the industrial automation facility, but capture the one or more particular components from different perspectives. Optionally, the second image can be captured by an additional mobile robot carrying an additional camera that is the same as, similar to, or different from the camera.

[0042] In some implementations, the mobile robot is a wheeled robot, an unmanned aerial vehicle, or a four-legged robot that can move within an industrial automation facility. The mobile robot can be controlled, for example, to move along a predefined path and / or for the camera (or any other vision component referenced above) to pause at one or more predefined spots to capture an image at a given pose (e.g., a first or second pose). In some implementations, the camera is fixed relative to the mobile robot such that controlling the mobile robot can control or adjust the pose of the camera. In some implementations, the camera is movable (e.g., rotationally and / or translationally) relative to the mobile robot while operably coupled to the mobile robot. In this case, after pausing the mobile robot at a given pose, the camera can be further controlled to capture an image at the given pose (first or second pose). In some implementations, two or more mobile robots can be deployed within an industrial automation facility to capture one or more images (e.g., in parallel). In some implementations, the camera of the mobile robot is a monographic camera, a stereographic camera, or a thermal camera.

[0043] In some implementations, the mobile robot can be controlled to move from a first predefined spot to a second predefined spot so that the camera can capture a first image in a first pose and further capture a second image in a second pose. In some implementations, the first image and the second image can capture the same region of one or more specific components at different resolutions. In some implementations, the first image and the second image can capture different regions of one or more specific components. The different regions may, although not necessarily in some situations, include overlapping regions. For example, the first image can capture a first portion of one or more specific components, and the second image can capture a second portion of the one or more specific components that complements the first portion, such that the one or more specific components are captured in their entirety. In some implementations, the first image and the second image can be of different types, although this is not necessarily the case.

[0044] At block 304A, a first machine learning (ML) model may be selected for processing the first image. In response to the first image being captured in a first pose, the system may select the first ML model based on the first ML model being trained based on training instances corresponding to the first pose. At block 306A, the system may process the first image using the selected first ML model to generate a first output indicating whether an anomaly is present in the first image.

[0045] In some implementations, the first ML model is trained using one or more training instances, each of which includes (1) an image captured by a camera and determined to correspond to a first pose in the absence of an anomaly for one or more particular components, and (2) a ground truth label indicating the absence of an anomaly for one or more particular components.

[0046] At block 304B, the system may select, by a server, such as server device 105, a second ML model to process the second image, where the selection of the second ML model is responsive to the second image being captured at a second pose and the second ML model being trained based on training instances corresponding to the second pose. At block 306B, the system may process the second image using the selected second ML model to generate a second output indicating whether an anomaly is present in the second image.

[0047] In some implementations, the second ML model is trained using one or more training instances, each of which includes (1) an image captured by a camera and determined to correspond to a second pose in the absence of an anomaly for one or more particular components, and (2) a ground truth label indicating the absence of an anomaly for one or more particular components.

[0048] At block 308, the system may determine whether to cause the rendering of an alert indicating that an anomaly exists for one or more particular components based on both the first output and the second output. In some implementations, the system may determine whether to cause the rendering of an alert indicating that an anomaly exists by determining whether the first output meets a threshold, determining whether the second output meets a threshold, and determining to cause the rendering of an alert if either the first output or the second output meets the threshold.

[0049] In some implementations, the system can determine whether to cause the rendering of an alarm indicating that an anomaly exists by determining whether the first output satisfies a threshold, determining whether the second output satisfies the threshold, and determining to cause the rendering of an alarm only if both the first output satisfies the threshold and the second output meets the threshold. In some implementations, the system can determine whether to cause the rendering of an alarm indicating that an anomaly exists by determining a combined output based on the first output and the second output, and determining whether to cause the rendering of an alarm based on the combined output.

[0050] In some implementations, the system can render an alert in response to a determination to trigger the rendering of an alert. The alert can be rendered audibly and / or visually. For example, the alert can be rendered audibly via one or more speakers located within the industrial automation facility. Additionally or alternatively, the alert can be rendered in the form of a message to one or more computing devices within or outside the industrial automation facility. For example, a desktop in a monitoring room may receive an alert message indicating that an anomaly has been detected for one or more particular components, the type of anomaly, and / or contact information for a person responsible for the one or more particular components.

[0051] Instead of or in addition to determining to trigger the rendering of an alarm, the system may perform one or more other remedial actions. The one or more other remedial actions may include pausing or stopping one or more industrial processes (e.g., automation processes) involving one or more particular components in which the anomaly was detected. For example, the system may determine to trigger a pause of a process involving one or more particular components within an industrial automation facility based on both the first output and the second output.

[0052] The methods described herein or in other aspects of the present disclosure may be performed via one or more processors of one or more computing devices separate from and not attached to the mobile robot, in which case the first and second images are transmitted by the mobile robot and identified by the one or more computing devices after being transmitted by the mobile robot.

[0053] 4 is a block diagram of an exemplary computing device 410 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 410 typically includes at least one processor 414 that communicates with several peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with the computing device 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0054] The user interface input devices 422 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 410 or a communications network.

[0055] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel display such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing visual images. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to encompass all possible types of devices and methods for outputting information from the computing device 410 to a user or to another machine or computing device.

[0056] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic for implementing the various components shown in Figures 1-2 as well as for performing selected aspects of the method of Figure 3.

[0057] These software modules typically execute on the processor 414 alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive, along with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 426 within the storage subsystem 424, or may be stored within another machine accessible by the processor 414.

[0058] The bus subsystem 412 provides a mechanism for allowing the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0059] Computing device 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 410 shown in Figure 4 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 410 are possible, having more or fewer components than the computing device shown in Figure 4.

[0060] While several implementations have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain one or more of the results and / or advantages described herein, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Accordingly, the foregoing implementations are presented by way of example only, and it should be understood that, within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Additionally, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0061] In some implementations, a method implemented using a processor is provided, the method including identifying a first image captured by a camera of a mobile robot when the camera is in a first pose and identifying a second image captured by the camera of the mobile robot when the camera is in a second pose different from the first pose. The first image and the second image both capture one or more specific components of the mobile robot's environment but capture the one or more specific components from different perspectives. The mobile robot can be, for example, a wheeled robot, an unmanned aerial vehicle, or a four-legged robot. The camera of the mobile robot can be, for example, a monographic camera, a stereographic camera, or a thermal camera. The one or more processors can be, for example, of one or more computing devices separate from and not attached to the mobile robot, and the first and second images can be transmitted by the mobile robot and identified by the computing devices after being transmitted by the mobile robot.

[0062] In various implementations, the method may further include selecting a first machine learning (ML) model for processing the first image, where selecting the first ML model is responsive to the first image being captured at a first pose and the first ML model being trained based on training instances corresponding to the first pose. In those implementations, the method may further include processing the first image using the selected first ML model to generate a first output indicating whether an anomaly (e.g., associated with one or more particular components of the industrial automation facility) is present.

[0063] In various implementations, the method can further include selecting a second ML model for processing the second image, where selecting the second ML model is responsive to the second image being captured at a second pose and the second ML model being trained based on training instances corresponding to the second pose. In those implementations, the method can further include processing the second image with the selected second ML model to generate a second output indicating whether an anomaly is present.

[0064] In various implementations, the method can further include determining, based on both the first output and the second output, whether to cause the rendering of an alert indicating that an anomaly exists. In some implementations, determining, based on both the first output and the second output, whether to cause the rendering of an alert indicating that an anomaly exists can include determining whether the first output satisfies a threshold value, determining whether the second output satisfies a threshold value, and deciding to cause the rendering of an alert if either the first output or the second output satisfies the threshold value. In these or other implementations, the method can further include rendering the alert in response to the determination to cause the rendering of the alert.

[0065] In some implementations, determining whether to cause the rendering of an alert indicating that an anomaly exists based on both the first output and the second output can include determining whether the first output satisfies a threshold value; determining whether the second output satisfies a threshold value; and determining to cause the rendering of an alert only if both the first output satisfies the threshold value and the second output satisfies the threshold value.

[0066] In some implementations, determining whether to cause the rendering of an alert indicating that an anomaly exists based on both the first output and the second output can include determining a combined output based on the first output and the second output, and determining whether to cause the rendering of an alert based on the combined output.

[0067] In some implementations, the first ML model is trained using one or more training instances, each of which includes (1) an image captured by a camera and determined to correspond to a first pose in the absence of an anomaly associated with one or more particular components, and (2) a ground truth label indicating the absence of an anomaly associated with one or more particular components.

[0068] In some implementations, the second ML model is trained using one or more training instances, each of which includes (1) an image captured by the camera or an additional camera corresponding to a second pose in the absence of an anomaly associated with one or more particular components, and (2) a ground truth label indicating the absence of an anomaly associated with one or more particular components.

[0069] In some implementations, instead of or in addition to determining to cause the rendering of an alert, the method may include determining, based on both the first output and the second output, to cause a suspension of a process involving one or more particular components within the industrial automation facility.

[0070] Additionally, some implementations include one or more processors of the mobile robot and / or one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the methods disclosed herein. Some implementations also include one or more transitory or non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the methods disclosed herein.

[0071] In some implementations, a system is provided that includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: identify a first image captured by a camera of a mobile robot when the camera is in a first pose; identify a second image captured by the camera of the mobile robot when the camera is in a second pose that is different from the first pose; select a first machine learning (ML) model for processing the first image; process the first image using the selected first ML model to generate a first output that indicates whether an anomaly is present; select a second ML model for processing the second image; process the second image using the selected second ML model to generate a second output that indicates whether an anomaly is present; and determine, based on both the first output and the second output, whether to trigger rendering of an alert indicating the anomaly is present.

[0072] In various implementations, execution of the instructions by the one or more processors further causes the one or more processors to determine, based on the first output and / or the second output indicating the presence of an anomaly associated with the one or more particular components, to cause the rendering of an alarm, causing the rendering of the alarm visually and / or audibly. In various implementations, execution of the instructions by the one or more processors further causes the one or more processors to determine, to pause one or more processes in the industrial automation facility that utilize the one or more particular components, pausing the one or more processes that utilize the one or more particular components.

[0073] In some implementations, a mobile robot is provided that includes a camera, a memory storing instructions and storing a trained machine learning (ML) model, and one or more processors. The processor is operable to execute instructions to receive from the camera a first image captured when the camera is in a first pose, receive a second image captured by the camera of the mobile robot when the camera is in a second pose different from the first pose, select a first machine learning (ML) model to process the first image, process the first image using the selected first ML model to generate a first output indicative of whether an anomaly is present, select a second ML model to process the second image, process the second image using the selected second ML model to generate a second output indicative of whether an anomaly is present, and determine, based on both the first output and the second output, whether to trigger rendering of an alert indicating the anomaly is present. [Explanation of symbols]

[0074] 100 Environment 101 Mobile Robot 102 Tube, Liquid Tube 103-A Local Client Device, Client Device 103-B Local Client Device, Client Device 105 Server Devices 106 Process Automation Network 107 Warning Message 122-A Local Client Device 122-B Local client device 201 First Image 203 Second Image 211 First ML Model 213 Second ML Model 221 First ML Output 223 Second ML Output 410 Computing Devices 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Memory Subsystem 425 Memory Subsystem 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM) 1011 Visual Components 1051 Machine Learning (ML) Engine, ML Engine 1052 Anomaly Detection Engine 1053 Pose-dependent machine learning (ML) model, Pose-dependent ML model, ML model

Claims

1. 1. A method implemented by one or more processors, comprising: identifying a first image captured by a camera of a mobile robot when the camera is in a first pose; identifying a second image captured by the camera of the mobile robot when the camera is in a second pose different from the first pose; the first image and the second image both capture one or more particular components, but capture the one or more particular components from different perspectives; selecting a first machine learning (ML) model for processing the first image, wherein selecting the first ML model is responsive to the first image being captured in the first pose and the first ML model being trained based on images corresponding to the first pose; processing the first image using the selected first ML model to generate a first output indicating whether an anomaly is present; selecting a second ML model for processing the second image, wherein selecting the second ML model is responsive to the second image being captured in the second pose and the second ML model being trained based on an image corresponding to the second pose; processing the second image using the selected second ML model to generate a second output indicating whether an anomaly exists for the one or more particular components; determining whether to cause the rendering of an alarm indicating that the anomaly exists based on both the first output and the second output; when the rendering of the alarm indicates that the anomaly exists, controlling the mobile robot to capture an additional image having a higher resolution than the first image and the second image; Including, A method wherein the camera pose is independently adjustable relative to the mobile robot, and the first pose and the second pose are functions of the pose of the mobile robot and the pose of the camera relative to the mobile robot.

2. determining whether to cause the rendering of the alarm indicating that the anomaly exists based on both the first output and the second output; determining whether the first output satisfies a threshold; determining whether the second output satisfies the threshold; determining to cause the rendering of the alert if either the first output or the second output meets the threshold; 2. The method of claim 1, comprising:

3. determining whether to cause the rendering of the alarm indicating that the anomaly exists based on both the first output and the second output; determining whether the first output satisfies a threshold; determining whether the second output satisfies the threshold; determining to cause rendering of the alert only if both the first output meets the threshold and the second output meets the threshold; 2. The method of claim 1, comprising:

4. determining whether to cause the rendering of the alarm indicating that the anomaly exists based on both the first output and the second output; determining a combined output based on the first output and the second output; determining whether to trigger rendering of the alert based on the combined output; 2. The method of claim 1, comprising:

5. The method of claim 1 , wherein the mobile robot is a wheeled robot, an unmanned aerial vehicle, or a four-legged robot.

6. The method of claim 1 , wherein the camera of the mobile robot is a monographic camera, a stereographic camera, or a thermal camera.

7. 2. The method of claim 1, wherein the first ML model is trained using one or more training instances, each of which includes (1) an image captured by the camera or an additional camera corresponding to the first pose in the absence of an anomaly associated with the one or more particular components, and (2) a ground truth label indicating the absence of an anomaly associated with the one or more particular components.

8. 10. The method of claim 1, wherein the second ML model is trained using one or more training instances, each of which includes (1) an image captured by the camera or an additional camera corresponding to the second pose in the absence of an anomaly associated with the one or more particular components, and (2) a ground truth label indicating that no anomaly is present.

9. 2. The method of claim 1, wherein the first image captures a first portion of the one or more particular components, and the second image captures a second portion of the one or more particular components that is complementary to the first portion.

10. The method of claim 1 , wherein the first image and the second image capture the same region of the one or more particular components.

11. The method of claim 1 , further comprising the step of rendering the alert in response to a determination to cause rendering of the alert.

12. based on both the first output and the second output; The method of claim 1 , further comprising determining to cause a suspension of processes involving the one or more particular components in an environment.

13. 2. The method of claim 1, wherein the one or more processors are of one or more computing devices separate from and not attached to the mobile robot, and the first and second images are transmitted by the mobile robot and identified by the computing devices after being transmitted by the mobile robot.

14. one or more computing devices including a camera and in at least selective network communication with a mobile robot deployed in an environment, the one or more computing devices comprising: receiving a first image from the mobile robot captured by the camera of the mobile robot when the camera is in a first pose; receiving a second image from the mobile robot captured by the camera of the mobile robot when the camera is in a second pose different from the first pose; receiving the first image and the second image, both capturing one or more particular components but capturing the one or more particular components from different perspectives; selecting a first machine learning (ML) model for processing the first image, wherein selecting the first ML model is responsive to the first image being captured in the first pose and the first ML model being trained based on images corresponding to the first pose; processing the first image using the selected first ML model to generate a first output indicating whether an anomaly is present; and selecting a second ML model for processing the second image, wherein selecting the second ML model is responsive to the second image being captured in the second pose and the second ML model being trained based on an image corresponding to the second pose; processing the second image using the selected second ML model to generate a second output indicating whether an anomaly is present; and determining whether to cause the rendering of an alarm indicating that an anomaly exists for the one or more particular components based on both the first output and the second output; and controlling the mobile robot to capture an additional image having a higher resolution than the first image and the second image when the rendering of the alarm indicates that the anomaly exists; and A system wherein the pose of the camera is independently adjustable relative to the mobile robot, and the first pose and the second pose are functions of the pose of the mobile robot and the pose of the camera relative to the mobile robot.

15. The system of claim 14 , wherein the environment is an industrial automation facility and the one or more computing devices are located within the industrial automation facility.

16. In determining whether to cause the rendering of the alert, the one or more computing devices: determining whether the first output meets a threshold; determining whether the second output satisfies the threshold; and determining to cause the rendering of the alarm if either the first output or the second output meets the threshold; and The system of claim 14, wherein

17. In determining whether to cause the rendering of the alert, the one or more computing devices: determining whether the first output meets a threshold; determining whether the second output satisfies the threshold; and determining to cause rendering of the alert only if both the first output meets the threshold and the second output meets the threshold; The system of claim 14, wherein

18. In determining whether to cause the rendering of the alert, the one or more computing devices: determining a combined output based on the first output and the second output; determining whether to trigger rendering of the alert based on the combined output; and The system of claim 14, wherein

19. the one or more computing devices: determining, based on metadata associated with the first image and received from the mobile robot, that the first image was captured in the first pose, wherein selecting the first ML model for processing the first image is responsive to determining that the first image was captured in the first pose. The system of claim 14 further comprising:

20. the one or more computing devices: determining that the first image was captured in the second pose based on metadata associated with the second image and received from the mobile robot, wherein selecting the second ML model for processing the second image is responsive to determining that the second image was captured in the second pose. The system of claim 14 further comprising:

Images