Machine learning-based image processing and component anomaly detection by considering upper and lower limits

Abstract

To provide a method for anomaly detection for a component through machine-learning based image processing and considering an upper bound value and a lower bound value.SOLUTION: A method processes an image by using a machine learning (ML) model, to generate a reduced dimensionality ML output. The image captures components of an environment (for example, industrial automation facility) and is captured via a vision component of a mobile robot within the environment (302). The ML output indicates, for each of N regions of the image, a corresponding anomaly detection probability that indicates whether an anomaly is present in a respective region of the N regions of the image (304). The method determines, based on the ML output, the quantity of anomaly detection probabilities that each satisfy a threshold, and further determines whether the quantity is larger than a lower bound value and smaller than an upper bound value (308) and if so, executes correction actions such as displaying an alert indicating that an anomaly is present for the components.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to machine learning based image processing and component anomaly detection by considering upper and lower limits. [Background technology]

[0002] Complex industrial facilities, such as petrochemical refineries, chemical plants, and the like, can include numerous components utilized in the processing of liquids, gases, and / or other substances involved in the industrial process of the industrial facility. It is important to ensure that the components involved in the industrial process are operating as intended and / or that the substances involved in the industrial process are in their intended state.

[0003] Various sensors have been utilized within industrial facilities to monitor for anomalies in such components and / or materials, including temperature sensors utilized to monitor temperature anomalies in components and / or materials, optical sensors utilized to monitor anomalies in the composition of materials (e.g., based on exciting the materials with a light source), and the like.

[0004] While various sensors may be utilized within an industrial facility to monitor various anomalies, each such sensor is typically deployed at a corresponding fixed location within the industrial facility. Accordingly, a large number of sensors may be required to effectively monitor the industrial facility. Furthermore, providing power and / or information to each of the sensors may require extensive wiring throughout the industrial facility. Furthermore, each of the large number of sensors is prone to failure or malfunction and must be repaired and / or replaced when such failure occurs, which may be difficult in a large industrial facility that includes a large number of fixed sensors. Summary of the Invention [Means for solving the problem]

[0005] Implementations of the present disclosure are directed to using a mobile robot to capture images of components and / or substances in an environment, such as within an industrial facility or other environment, at a given pose via the mobile robot's camera. The implementations are further directed to processing the captured images using a machine learning (ML) model to generate an ML output. The machine learning model may be a neural network model, such as a convolutional neural network (CNN) model, including one or more convolutional layers. The implementations are further directed to determining, based on the generated ML output, whether an anomaly exists with the components and / or substances captured by the image, and, if so, performing one or more corrective actions. Performing a corrective action may include, for example, displaying an alarm, stopping a process, and / or performing other corrective actions.

[0006] In many implementations, the utilized ML model may be a model trained to be used to process an image to generate a reduced-dimensional ML output that is of reduced dimensionality compared to the processed image. Each value in the reduced-dimensional ML output may directly or indirectly reflect the probability that a corresponding region of the image contains an anomaly. For example, each value may be a corresponding value between 0 and 1, and each value may correspond to a corresponding portion of the image encompassing multiple pixels of the image (e.g., a corresponding 64x64 pixel portion of the image). In these implementations, determining whether an anomaly exists may be based on determining the number of values that satisfy a threshold (e.g., a threshold such as 0.7) and determining that an anomaly exists only if the number is (a) greater than a lower bound and (b) less than an upper bound. For example, assume the ML output includes 16 values. An anomaly may be detected only if the number of values that satisfy the threshold is greater than 1 or 2 (or other lower bound) but less than 8 or 9 (or other upper bound).

[0007] The occurrence of false positive and / or false negative anomaly detections can be mitigated by determining that an anomaly exists only when the quantity is (a) greater than a lower limit and (b) less than an upper limit. For example, the occurrence of false positives in detecting anomalies can be prevented by considering an upper limit in addition to a lower limit. For example, the occurrence of false positives can be prevented by not determining that an anomaly exists when the quantity of values satisfying a threshold is greater than an upper limit. For example, when the quantity of values satisfying a threshold is greater than an upper limit, implementations disclosed herein recognize that this may not be due to an anomaly but may be due to other external factors in the environment, such as excessive light (e.g., from the sun and / or lighting devices), the presence of a person or other moving object in the image, and / or a problem with the camera that captured the image. Mitigating the occurrence of false positives can prevent the utilization of network and / or computing device resources in erroneously taking corrective action, such as, for example, displaying a false positive alert. Mitigating the occurrence of false negatives can prevent the occurrence of unsafe conditions in an industrial facility and / or damage to components and / or materials in the industrial facility.

[0008] In some implementations, the same lower limit value and / or the same upper limit value may be used in determining whether an anomaly exists for each of a plurality of distinct components and / or substances in an environment. For example, the same lower limit value (e.g., 2) and the same upper limit value (e.g., 11) may be utilized in analyzing both (a) a first ML output generated based on processing a first image of a tank (in a first area of the environment) with an ML model and (b) a second ML output generated based on processing a second image of a pipe (in a second area of the environment).

[0009] In some additional or alternative implementations, the lower and / or upper bounds used for the first component may differ from the lower and / or upper bounds used for the second component. For example, a first image determined to capture a first component (e.g., based on metadata of the first image) may be processed using an ML model to generate a first ML output, a first quantity of values of the first ML output that satisfy a threshold may be determined, and an anomaly for the first component may be determined only if the first quantity is between 2 and 9. Continuing the example, a second image determined to capture a second component (e.g., based on metadata of the second image) may be processed using an ML model to generate a second ML output, a second quantity of values of the second ML output that satisfy a threshold may be determined, and an anomaly for the second component may be determined only if the first quantity is between 3 and 8, which is a more restricted range than the range utilized for the first image determined to capture the first component. A more restricted range is less likely to result in an anomaly being determined. In some of these implementations, the less restrictive and / or more restrictive ranges may optionally be explicitly manually specified based on user interface input. The manual specification may reflect the importance of each component (e.g., a first component may be more important than a second component) and / or may be based on additional or alternative considerations, such as how dynamic the area in which the components are located is expected to be. In other implementations, the less restrictive and / or more restrictive ranges may optionally be set depending on implicit user interface input. For example, the range for the second component may initially be 2 to 9, but may be automatically adjusted to a more restrictive 3 to 8 based on precedent user feedback indicating that the anomaly determined based on the 2 to 9 range was not actually an anomaly. For example, the precedent user feedback may have been provided by an operator after reviewing an image of the first component and / or after the operator personally inspected the first component (e.g., feedback indicating that the previous anomaly detection was incorrect).

[0010] In some implementations, the same lower limit value and / or the same upper limit value may always be used in determining whether an anomaly exists for a given component and / or a given substance in an environment.

[0011] In some alternative implementations, the lower and / or upper limits used for a given component and / or substance can change very dynamically over time. For example, a more restrictive range can be used during certain temporal conditions (e.g., day of the week, time of day, date of month, year, etc.), and a less restrictive range can be used during certain other temporal conditions. As another example, a first range can be used for a first duration and then adjusted to a more or less restrictive range based on explicit or implicit user input as described above.

[0012] The captured image may be a high-resolution image having image dimensions (sometimes referred to as "pixel dimensions") of mxnxc pixels, where m is the width dimension, n is the height dimension, and c is the channel dimension. For example, c may be 3 when the image is a red, green, blue (RGB) image, and c may be 1 when the image is a grayscale image. The mxn value may be, for example, 1024x1024, 1024x768, 2048x2048, or other values. The reduced-dimensional ML output containing multiple probabilities may be arranged, for example, in a q1xq2 matrix, where the q1xq2 dimensions are smaller than the mxn dimensions. For example, the reduced-dimensional ML output may be a 2×2 matrix defining four distinct probabilities (p1, p2, p3, p4), where p1 is the probability that the anomaly is present in a first region of the image, p2 is the probability that the anomaly is present in a second region of the image, p3 is the probability that the anomaly is present in a third region of the image, and p4 is the probability that the anomaly is present in a fourth region of the image. The first, second, third, and fourth regions may jointly correspond to all regions of the image. The first, second, third, and fourth regions may correspond to entirely unique regions of the image, or one or more of the regions may partially overlap with other of the regions.

[0013] The mobile robot utilized to capture images processed in anomaly monitoring according to the techniques disclosed herein may be a quadruped robot, a wheeled robot, an unmanned aerial vehicle, a track-guided robot, or any other robot that moves under its own power within an environment. Images are captured via the mobile robot's vision component. Each image is captured when the vision component is at a corresponding pose (i.e., a given position and orientation). The pose of the vision component when the image is captured at a given time is a function of the robot's pose at the given time. When the vision component is at a fixed pose relative to the robot, the pose of the vision component is entirely a function of the robot's pose. When the vision component's pose is independently adjustable relative to the robot, the pose of the vision component is a function of the robot's pose and the pose of the vision component relative to the robot. An image as utilized herein may include, for example, an RGB image that includes a red channel, a green channel, and a blue channel and is captured by a monographic RGB camera; an RGB-D image that includes a depth channel in addition to red, green, and blue channels and is captured by a stereo camera; an ultraviolet (UV) image that includes a UV channel and is captured by a UV camera; or a thermal image that includes one or more thermal channels and is captured by a thermal camera.

[0014] By utilizing mobile robots that include vision components and move around industrial environments to capture images of components and / or materials of interest for anomaly monitoring / detection, the number of sensors monitoring the industrial environment may be reduced and / or the extensive wiring required for such sensors may be avoided. Moreover, the accuracy of anomaly detection may be improved by processing the captured images using an ML model to generate an ML output having multiple values and determining that an anomaly exists only when the number of those values that satisfy a threshold is (a) greater than a lower limit and (b) less than an upper limit. This allows anomaly detection to be more robust and / or more accurate.

[0015] It should be appreciated that all combinations of the foregoing and additional concepts described in greater detail herein are contemplated as being part of the presently disclosed subject matter, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the presently disclosed subject matter. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates a schematic diagram of an environment in which selected aspects of the present disclosure may be implemented, according to various implementations. [Figure 2] FIG. 1 illustrates a schematic diagram of an example of how the techniques described herein may be implemented, according to various implementations. [Figure 3] FIG. 1 illustrates an exemplary method for carrying out selected aspects of the present disclosure. [Figure 4] FIG. 1 illustrates a schematic diagram of an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0017] Implementations described herein relate to monitoring and detecting anomalies associated with components (e.g., liquid tanks, piping, wiring) in an environment (e.g., an industrial automation facility) based on ML outputs (e.g., reduced-dimensional outputs) generated by processing images of the components using a trained ML model (e.g., a trained neural network model). In various implementations, images of the components are captured by a vision component (e.g., a camera) carried by or integrated into a mobile robot capable of moving within the environment. The reduced-dimensional ML output may indicate, for each of N regions of the image, a corresponding probability that an anomaly is present in the respective region. Determining whether an anomaly exists may be based on analyzing the reduced-dimensional ML output. For example, each value in the reduced-dimensional ML output may reflect a probability that the corresponding region of the image contains an anomaly, and an anomaly may be determined to exist only when the number of values satisfying a threshold is greater than a lower limit and less than an upper limit. Implementations further relate to performing one or more corrective actions in response to detecting that an anomaly exists in the environment. As described above, each value in the reduced-dimensional output can reflect a probability that the corresponding region of the image contains an anomaly. In some implementations, the values can directly reflect the probability. For example, each value can be a corresponding value between 0 and 1 and can directly reflect the probability. For example, a value of 0.18 can reflect an 18% probability. In some implementations, the values can indirectly reflect the probability. For example, each value can be a corresponding value between 0 and 8 and can indirectly reflect the probability. For example, a value of 2 can reflect a 25 percent probability.

[0018] 1 , an exemplary environment 100 in which various aspects of the present disclosure may be implemented is generally illustrated. In some implementations, the exemplary environment 100 may be or include an industrial automation facility. Industrial facilities can take many forms and can be designed to implement any number of at least partially automated processes. Industrial automation facilities can take the form of chemical processing plants, oil or natural gas refineries, catalyst factories, manufacturing facilities, offshore oil platforms, etc.

[0019] The exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled with a process automation network 106 within an industrial automation facility. Client device 103-A or 103-B may be implemented as a computer (e.g., laptop, desktop, notebook), tablet, robot, smart appliance (e.g., smartphone), messaging device, wearable device (e.g., watch), or any other applicable device. The process automation network 106 may be implemented using various wired and / or wireless communication technologies, including, but not limited to, the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), 3GPP® Long Term Evolution (“LTE”) or other wireless protocols described as 3G, 4G, 5G and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).

[0020] The exemplary environment 100 may further include a mobile robot 101 having or carrying a vision component 1011. The mobile robot 101 may be a quadruped robot (e.g., a robotic dog), a wheeled robot, an unmanned aerial vehicle, a robot that moves along elevated and / or non-elevated tracks within an environment, or any other applicable robot capable of moving within an industrial facility. The vision component 1011 may be a monographic camera, a stereo camera, a thermal camera, or any other applicable vision sensor for capturing one or more images of one or more specific components of an industrial automation facility (e.g., a liquid tank T or tube 102 that stores or transports a liquid substance). The vision component 1011 may be removably coupled to the mobile robot 101 or may be integrated as a non-removable, intimate part of the mobile robot 101. In some implementations, the vision component 1011 may change location and / or orientation independently relative to the robot frame of the mobile robot 101, for example, by rotation or other movement via an actuator that solely controls the vision component 1011. In addition to the vision component 1011, the mobile robot 101 may include one or more additional vision components for navigating through an industrial facility, sensing static or dynamic objects, and / or capturing images.

[0021] The exemplary environment 100 may further include a server computing device 105. The server computing device 105 may include a machine learning (ML) engine 1051 and an anomaly detection engine 1052. The server computing device 105 may further include or otherwise have access to one or more trained machine learning (ML) models 1053, each trained to generate outputs indicative of multiple probabilities, with each probability indicative of the likelihood that an anomaly exists for a respective one of the N regions of the processed image. The server computing device 105 may be connected to multiple client devices. The server computing device 105 may communicate with one or more local client devices (e.g., 122-A and 122-B) and / or one or more remote client devices (not shown). The local client device 122-A or 122-B may be connected to the server computing device 105 via one or more local area networks (e.g., process automation network 106), and the remote client device may be connected to the server computing device 105 via one or more wide area networks (e.g., the Internet). The local and remote client devices may be operable by personnel, such as a system integrator, to configure and / or interact with various aspects of the exemplary environment 100.

[0022] In some implementations, the server computing device 105, in addition to the ML engine 1051 and the anomaly detection engine 1052, may include a database (not shown) that stores information used by the ML engine 1051 and / or the anomaly detection engine 1052 to perform selected aspects of the present disclosure. In some implementations, the server computing device 105 may include, in addition to the ML engine 1051 and the anomaly detection engine 1052, an image preprocessing engine 1055 that processes different images to have the same image dimensions. Various aspects of the server computing device 105, such as the ML engine 1051, the anomaly detection engine 1052, and / or the image preprocessing engine 1055, may be implemented using any combination of hardware and software. In some implementations, the ML engine 1051, the anomaly detection engine 1052, the image preprocessing engine 1055, or the trained ML model 1053 may be implemented across multiple computer systems as part of what is often referred to as a “cloud infrastructure” or simply “cloud.” However, this is not required, and in Figure 1, for example, the ML engine 1051 is implemented within an industrial facility, for example, within a single building or across a single campus of multiple buildings or other industrial infrastructure. In such an implementation, the ML engine 1051 may be implemented on one or more local computing systems, such as one or more local server computers.

[0023] In some implementations, the mobile robot 101 can navigate through an industrial facility and arrive at one or more designated spots or waypoints. The vision component 1011 of the mobile robot 101 can be (but need not necessarily be) configured at a given pose to capture an image of the liquid tube 102 at the given pose. The vision component 1011 can be configured at a given pose as a result of the mobile robot 101 being at the corresponding pose and / or as a result of the vision component 1011 being independently adjusted (when the vision component 1011 is independently adjustable relative to the robot frame of the mobile robot 101). The image captured by the vision component 1011 can include captured pixels or other data and can correspond to an area or component of interest, such as the liquid tank T and the liquid substance contained therein (shown in FIG. 1 ).

[0024] The captured image may be processed by the ML engine 1051 as an input to the trained ML model to generate an ML output. The ML output may be a reduced-dimensional output indicating, for each of the N regions of the image, a corresponding probability that an anomaly exists within the respective region of the N regions. The anomaly detection engine 1052 may process the ML output in determining whether an anomaly exists for the component, and, if an anomaly is determined to exist, may cause one or more corrective actions to be executed to address the detected anomaly. For example, the anomaly detection engine 1052 may cause an audible and / or visual alert to be displayed at the computing device, such as a visual alert 107 visually displayed at the client device 103-A.

[0025] While only a single mobile robot 101 is shown in FIG. 1 , it is understood that multiple mobile robots may be deployed in an industrial environment and utilized in the implementations disclosed herein. For example, each of the multiple mobile robots may include a corresponding vision component used to capture images, and images from the multiple mobile robots may be transmitted to the server device 105 for processing by the ML engine 1051 and the anomaly detection engine 1052. Similarly, while the ML engine 1051, the ML models 1053, the anomaly detection engine 1052, and the image pre-processing engine 1055 are shown in FIG. 1 as being implemented separately from the mobile robot 101, in some implementations, all aspects may be implemented by the mobile robot 101. For example, the mobile robot 101 may include the ML engine 1051, the anomaly detection engine 1052, and at least a subset of the ML models 1053. For example, the mobile robot 101 may include a subset of the ML model 1053 at a given time based on that subset's correspondence to a component, and the mobile robot 101 captures an image of that component with a task to be performed by the mobile robot 101 at the given time.

[0026] FIG. 2 schematically illustrates an example of how the techniques described herein may be implemented according to various implementations. As shown in FIG. 2, a vision component 1011 of a mobile robot 101 may capture an image 201. The image 201 captures a particular component (e.g., a liquid tank T and a liquid carried by the liquid tank) within an industrial automation facility. A vision sensor captures the image 201 at a given pose. The image 201 may have image dimensions of m×n pixels (and c channels) and may be processed using a trained ML model 211, which may be one (e.g., only one, or one of multiple) of the ML models 1053 of FIG. 1. The trained ML model 211 may be trained for use in processing the image to generate an ML output that indicates, for each of multiple regions of the image, a corresponding probability that an anomaly exists within the region.

[0027] In some implementations, the trained ML model 211 is selected from multiple candidate ML models 1053 to process the image 201. In some of those implementations, the trained ML model 211 is selected to process the image 201 based on the image 201 capturing a particular component (e.g., a liquid tank T), and the ML model 211 is trained based on training instances including training instance inputs of images also capturing the particular component (e.g., a liquid tank T) and training instance outputs of ground truth probabilities. For example, the trained ML model 211 may be fine-tuned based on such training instances. In some of those implementations, metadata embedded or included in the image 201 may directly or indirectly indicate a particular component, and such metadata may be used in selecting the trained ML model 211. For example, the metadata may indicate a location where the image was captured, and such location may be correlated to a particular component and / or the trained ML model 211.

[0028] In some additional or alternative implementations, the trained ML model 211 is a model trained based on training instances that each include a training instance input of an image that captures or nearly captures a particular component (e.g., the liquid tank T) using corresponding visual components at a given pose of the image 201, and a training instance output of a ground truth probability. For example, the trained ML model 211 can be fine-tuned based on such training instances. In some of those implementations, the trained ML model 211 is selected based on determining that the image 201 captures a particular component at a given pose, and that the trained ML model 211 is a model fine-tuned based on training instances for the particular component and the given pose. For example, metadata incorporated in or included in the image 201 can directly or indirectly indicate the given pose, and such metadata can be used in selecting the trained ML model 211. For example, the metadata can indicate a given pose of the visual components when the image was captured, and the given pose can be correlated with the trained ML model 211.

[0029] In some additional or alternative implementations, the trained ML model 211 can be selected from a plurality of trained ML models based on the image dimensions of the image 201.

[0030] The image 201 can be processed using the ML model 211 to generate an ML output 221 of the ML model 211. The ML output 221 can be a reduced-dimension output that indicates, for each of N regions of the image 201, a corresponding probability (also referred to herein as an “anomaly detection probability”) that an anomaly exists within the respective region. The ML output 221 can indicate or include a plurality of anomaly detection probabilities arrayed within a matrix (e.g., a vector or a multi-dimensional matrix), and the position of the i-th anomaly detection probability within the matrix corresponds to the location of the i-th (0 < i < N) region within the image 201.

[0031] 2, the reduced dimensional output may include first, second, ..., and sixteenth probabilities arranged in a matrix M1, also shown below: In matrix M1, the location of a probability (e.g., p13) may correspond to the location of a respective region (e.g., the thirteenth region shown in FIG. 2) in image 201, for which a probability (e.g., p13) is predicted.

[0032]

number

[0033] Based on the ML output 221 (i.e., the output of reduced dimensionality), it may be determined whether an anomaly is detected for a particular component. For example, based on the multiple anomaly detection probabilities indicated by the ML output 221, it may be determined whether an anomaly is detected for a particular component. In some implementations, a quantity of anomaly detection probabilities that satisfy a threshold (e.g., 0.7 or another threshold) may be determined from the N anomaly detection probabilities. With reference to FIG. 2 , p1, p2, p4, p5, p8, and p9 may be identified as satisfying a threshold (e.g., 0.7), and as a result, the quantity of anomaly detection probabilities that satisfy the threshold may be determined to be “6.” In these implementations, if the quantity (e.g., 6) of anomaly detection probabilities that satisfy the threshold (e.g., 0.7) is greater than a lower limit value (e.g., “1”) and less than an upper limit value (e.g., “11”), an image 201 of the particular component may be determined to capture an anomaly associated with the particular component. As described herein, the lower and / or upper limits may optionally depend on the particular component captured (e.g., as indicated by metadata of the image 201), temporal conditions, and / or other factors. Stated differently, in some implementations, the lower and / or upper limits may be dynamic for a given component and / or may be component-based.

[0034] In response to determining that the image 201 captures an anomaly, one or more corrective actions may be performed. The one or more corrective actions may include causing a visual and / or audible alarm to be displayed. Alternatively or additionally, the one or more corrective actions may include pausing one or more processes, such as one or more processes involving a particular component. Alternatively or additionally, the one or more corrective actions may include notifying one or more operators to inspect a particular component.

[0035] 3 is a flowchart illustrating an example method 300 of performing selected aspects of the present disclosure, according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computing devices, such as one or more components of the server computing device 105 (and / or additional computing devices, such as the mobile robot 101 or client device 103-A) that include the ML engine 1051 and / or the anomaly detection engine 1052. Moreover, although the operations of the method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0036] In block 302, the system identifies an image captured by a vision component (e.g., a camera) of the mobile robot, the image capturing one or more specific components of an industrial automation facility. The industrial automation facility may be a chemical processing plant, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable facility implementing one or more at least partially automated processes. The one or more specific components may include, by way of non-limiting example, a liquid tank and / or the liquid the liquid tank carries. The mobile robot may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot capable of moving within the industrial automation facility.

[0037] The vision component may be a monographic RGB camera, a stereo camera, a thermal camera, a UV camera, or any other applicable vision component. The image captured by the vision component may accordingly be an RGB image, an RGB-D image, a UV image, a thermal image, or any other applicable image. In some implementations, the vision sensor may be integrated with the mobile robot or removably coupled to the mobile robot.

[0038] In block 304, the system processes the image using an ML model (e.g., a CNN or other neural network model) to generate a reduced-dimensional output that indicates or predicts an anomaly detection probability (sometimes simply referred to as a "probability") for each of the N regions of the image, indicating whether an anomaly is present within the respective region of the N regions. The value of "N" may be, for example, 9 or greater, but this is not intended to be limiting. For example, the value N may be equal to 4, 6, 25, 36, or any other applicable positive integer.

[0039] The ML model used by the system in block 304 in processing the images may be trained based on one or more training instances, each training instance including a training instance input for a corresponding image and a training instance output for N corresponding ground truth labels, each ground truth label indicating whether an anomaly is present in a respective one of the N regions of the respective image. For example, each of the ground truth labels may be either an “anomaly present” label (e.g., “1”) or an “anomaly absent” label (e.g., 0). The “anomaly present” label, if present, may be applied to a label corresponding to a region of the corresponding image that contains an anomaly, and the “anomaly absent” label, if present, may be applied to a label corresponding to a region of the corresponding image that does not contain an anomaly. For example, if the anomaly is corrosion and corrosion is present in only two of the 16 regions of the image, the ground truth labels may include two “anomaly present” labels corresponding to the two anomalous regions of the image and 14 “anomaly absent” labels corresponding to the 14 non-anomalous regions of the image.

[0040] In some implementations, the system optionally selects an ML model to process the image from a plurality of trained ML models at block 3041. In some of these implementations, the system selects the ML model to process the image based on images capturing one or more particular components, where the ML model is trained based on images corresponding to the one or more particular components. In some further or alternative implementations of these implementations, the system selects the ML model from the plurality of trained ML models based on the image being captured at a given pose (e.g., a first pose corresponding to a first position and / or a first orientation of a visual component), where the ML model is trained based on images corresponding to the given pose. In some further or alternative implementations of these implementations, the system selects the ML model from the plurality of trained ML models based on the image based on image type (e.g., thermal, RGB, RGB-D) and / or image dimensions (e.g., 1024 x 768 pixels).

[0041] As a non-limiting example, the plurality of trained ML models may include a first ML model trained using an image captured by a visual sensor at a first pose, where the first ML model is trained to process the first image captured by the visual sensor at the first pose to output a first quantity (“N_1”) of anomaly detection probability relative to a total number of N_1 regions in the first image captured at the first pose. The plurality of trained ML models may include a second ML model trained using an image captured by the visual sensor at a second pose, where the second ML model is trained to process the second image captured by the visual sensor at the second pose to output a second quantity (“N_2”) of anomaly detection probability relative to a total number of N_2 regions in the second image captured at the second pose. The second quantity (“N_2”) may optionally be different from the first quantity (“N_1”), where the first pose is different from the second pose. The multiple trained ML models can include additional ML models, such as a third model trained based on images captured at a third pose.

[0042] In block 306, the system can determine a quantity of anomaly detection probabilities that each satisfy a threshold based on the reduced-dimensional output. In some implementations, the system identifies anomaly detection probabilities that satisfy a threshold from the reduced-dimensional output in block 3061. The system can determine a quantity based on a count of the identified anomaly detection probabilities that satisfy the threshold in block 3063. For example, assuming the threshold is 0.7, the reduced-dimensional output indicates a total of 16 anomaly detection probabilities: 0.65, 0.72, 0.65, 0.45, 0.55, 0.78, 0.85, 0.8, 0.85, 0.55, 0.45, 0.82, 0.78, 0.48, 0.58, and 0.63. In such an example, the anomaly detection probabilities that satisfy the threshold (0.7) may be identified to include: 0.72, 0.78, 0.85, 0.8, 0.85, 0.82, 0.78. In this example, the quantity of anomaly detection probabilities identified as satisfying the threshold may be determined to be "7."

[0043] In block 308, the system determines whether an anomaly exists based on comparing the quantity against both the lower limit and the upper limit. For example, the system may determine that an anomaly exists only when the quantity is higher than the lower limit and lower than the upper limit. In some implementations, the lower limit and / or upper limit are static. In some implementations, the lower limit and upper limit may be determined based on the value of "N" (i.e., the quantity of regions reflected in the reduced-dimensional output). For example, the lower limit may be less than or equal to 20% of N, such as about 10% of the value of "N." Also, for example, the upper limit may be greater than or equal to 50 percent of N, such as about 90 percent of the value of "N."

[0044] Alternatively or additionally, in some implementations, the lower limit value and / or the upper limit value may be dynamic. For example, one or both may be determined based on the type or other characteristics of the component captured by the image, current temporal conditions, detected or estimated light levels within the industrial automation facility, and / or other factors. For example, a lower limit value utilized during daylight hours and / or periods of higher light levels may differ from a lower limit value utilized during nighttime hours and / or periods of lower light levels.

[0045] As a non-limiting example, "N" can have 16 values. In this example, the lower limit may be predefined to be "1" or "2," and the upper limit may be predefined to be "7" or "8." When the quantity of anomaly detection probabilities identified as satisfying the threshold is determined to be "5," which is greater than the lower limit and less than the upper limit, an image with a predicted anomaly detection probability may be determined to capture anomalies for one or more specific components.

[0046] Additionally, in some implementations, the system may display an alert indicating that an anomaly exists for one or more particular components in response to determining that the quantity is greater than the lower limit and less than the upper limit. Alternatively or additionally, the system may pause one or more processes in response to determining that the quantity is greater than the lower limit and less than the upper limit.

[0047] In some implementations, the system may determine that the quantity is less than or equal to the lower limit, in which the system may determine that no anomaly is detected for one or more particular components in response to determining that the quantity is less than or equal to the lower limit.

[0048] In some implementations, the system can determine that the quantity is equal to or greater than an upper limit value. In these implementations, the system can generate a notification recommending manual inspection of the image to determine whether the image is suitable for detecting an anomaly for one or more specific components. For example, if an image is captured while the visual sensor is exposed to excessive light or if the image captures a moving object instead of one or more specific components, the image can be identified as not containing an anomaly for one or more specific components. Optionally, such an image, along with supervised labeled output that does not indicate an anomaly for any of the N regions, can be used to further train the ML model. Optionally, the system can further control the mobile robot to capture additional images via the visual sensor. Optionally, the additional images can be captured at a different pose than the images processed for anomaly detection, and a different ML model can be selected to process the additional images to see if an anomaly is detected.

[0049] Instead of or in addition to determining to display an alarm, the system may perform one or more other corrective actions. The one or more other corrective actions may include pausing or stopping one or more industrial processes (e.g., automated processes), such as industrial processes involving one or more particular components in which an anomaly is detected. For example, the system may determine to pause a process involving one or more particular components in an industrial automation facility based on both the first output and the second output.

[0050] In some implementations, at least some aspects of the methods described herein are performed by one or more processors of one or more computing devices separate from and not attached to the mobile robot, and in some of those implementations, the images are transmitted by the mobile robot to the computing device and identified by the computing device after being transmitted by the mobile robot.

[0051] 4 is a block diagram of an exemplary computing device 410 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 410 generally includes at least one processor 414 that communicates with several peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 that includes a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computing device 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0052] The user interface output devices 422 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad or a graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 410 or onto a communications network.

[0053] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 410 to a user or to another machine or computing device.

[0054] The storage subsystem 424 stores program and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 424 may include logic for implementing the various components shown in Figures 1 and 2, as well as for performing selected aspects of the method of Figure 3.

[0055] These software modules are generally executed by the processor 414 alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of some implementations may be stored by the file storage subsystem 426 within the storage subsystem 424 or within another machine accessible by the processor 414.

[0056] The bus subsystem 412 provides a mechanism for allowing the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus system 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0057] Computing device 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 410 shown in Figure 4 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 410 can have more or fewer components than the computing device shown in Figure 4.

[0058] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, it is intended that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is therefore to be understood that the foregoing implementations are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be practiced differently from what is specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, item, material, tool, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, tools, and / or methods is included within the scope of the present disclosure, provided that such features, systems, articles, materials, tools, and / or methods are not mutually inconsistent.

[0059] In various implementations, the method may be implemented using one or more processors and may include identifying an image captured by a camera of the mobile robot, the image capturing one or more specific components of an environment. The environment may optionally be an industrial automation facility implementing any number of at least partially automated processes, such as a chemical processing plant, an oil or natural gas refinery, a catalyst factory, a manufacturing facility, an offshore oil platform, or any other applicable industrial environment. The mobile robot may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot capable of moving within an industrial automation facility. The camera may be a Monograph RGB camera, a stereo camera, a thermal camera, or any other applicable visual sensor. Accordingly, the image may be an RGB image, an RGB-D image, a thermal image, or any other applicable image having image dimensions of m×n×c (m, n, and c are positive integers greater than or equal to 1).

[0060] In various implementations, the method may further include processing the image using a trained machine learning (ML) model to generate a reduced-dimensionality output indicating, for each of N regions of the image, a corresponding probability that an anomaly exists in the region, where N is a positive integer less than (m×n). As a non-limiting example, the image may be a 1024×768 RGB image, and the reduced-dimensionality output may indicate, for each of 3×3 (=9) regions of the image, a corresponding probability that an anomaly exists in each of the nine regions. In this non-limiting example, the reduced-dimensionality output may indicate, for example, a first probability p1 of one or more anomalies being detected for a first region of the image, a second probability p2 of one or more anomalies being detected for a second region of the image, ..., and a sixteenth probability p16 of one or more anomalies being detected for a sixteenth region of the image, where the first, second, ..., sixteenth regions together form the image.

[0061] As a non-limiting example, the reduced dimensionality output may include first, second, ..., and sixteenth probabilities arranged in a matrix. The location of the probability (e.g., p2) in the matrix may correspond to the location of the respective region (e.g., second region) in the image for which the probability (e.g., p2) is predicted.

[0062] In various implementations, the method may further include determining, based on the reduced dimension output, corresponding probability quantities, each of which satisfies a threshold. In various implementations, the method may further include determining whether the quantities are greater than a lower bound and less than an upper bound. Continuing with the non-limiting example above, the threshold may be approximately 0.7, and the reduced dimension output may indicate p1=0.6, p2=0.8, p3=0.5, p4=0.75, p5=0.77, p6=0.82, p7=0.85, p8=0.5, p9=0.9, p10=0.65, p11=0.77, p12=0.58, p13=0.83, p14=0.3, p15=0.48, and p16=0.28. In this example, the quantity of probabilities each satisfying a threshold value (e.g., 0.7) may be determined as 8 (p2=0.8, p4=0.75, p5=0.77, p6=0.82, p7=0.85, p9=0.9, p11=0.77, and p13=0.83). If the lower limit value is predefined to be "2" and the upper limit value is predefined to be "10," the quantity of probabilities satisfying the threshold value (e.g., 8) may be determined to be greater than the lower limit value (e.g., 2) but also less than the upper limit value (e.g., 10).

[0063] In various implementations, the method may further include displaying an alert indicating that an anomaly exists for one or more specific components in response to determining that the quantity is greater than the lower limit and less than the upper limit. Continuing with the non-limiting example above, in response to determining that 8 out of 16 probabilities, 2 ("lower limit") < 8 < 10 ("upper limit"), satisfy the threshold, an alert may be displayed warning that one or more specific components have an anomaly. The alert may be a text message (e.g., email, text, etc.) indicating one or more specific components experiencing an anomaly, a confidence level of the existence / detection of the anomaly for one or more specific components, and / or the time the anomaly was detected, etc. The alert may also be an audio message announcing the possible existence of an anomaly for one or more specific components. If the quantity is greater than the upper limit or less than the lower limit, no alert may be generated. For example, if the quantity is greater than the upper limit value, it is likely that the image does not accurately or completely reflect one or more particular components due to the presence of a moving object (e.g., a person) blocking one or more particular components when the image is being captured by the camera, exposure to excessive light when the image is captured, etc. If the quantity is lower than the lower limit value, it may be determined that no anomaly exists for one or more particular components.

[0064] In some implementations, instead of or in addition to determining to display an alert, the method may include pausing one or more processes involving one or more particular components in the environment in response to determining that the quantity is greater than the lower limit value and less than the upper limit value.

[0065] Additionally, some implementations include one or more processors of the mobile robot and / or one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause any of the methods disclosed herein. Some implementations additionally or alternatively include one or more transitory or non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods disclosed herein.

[0066] In some implementations, a system is provided that includes one or more processors and a memory that stores instructions that, in response to execution by the one or more processors, cause the one or more processors to: identify images captured by a camera of a mobile robot when the camera is in a first pose, the images capturing one or more particular components within an environment of the mobile robot; process the images using a machine learning (ML) model to generate reduced-dimensional outputs indicating, for each of N regions of the images, a corresponding probability that an anomaly is present in the region; determine corresponding probability quantities, each satisfying a threshold, based on the reduced-dimensional outputs; and determine whether the quantities are greater than a lower bound and less than an upper bound.

[0067] In various implementations, the system can further include instructions for, in response to determining that the quantity is greater than the lower limit and less than the upper limit, displaying an alert indicating that an anomaly exists for one or more particular components. In various implementations, the system can further include instructions for, in response to determining that the quantity is greater than the lower limit and less than the upper limit, pausing one or more processes that utilize the one or more particular components.

[0068] In some implementations, a mobile robot is provided that includes a camera, a memory storing instructions and storing a trained machine learning (ML) model, and one or more processors. The processor is operable to execute instructions to: receive from the camera images capturing one or more specific components of an environment in which the mobile robot is deployed; process the images using the trained ML model to locally generate reduced-dimensional outputs indicating, for each of N regions of the image, a corresponding probability indicating whether an anomaly is present within the respective region; determine corresponding probability quantities each satisfying a threshold based on the reduced-dimensional outputs; determine whether the quantities are greater than a lower bound and less than an upper bound; and, in response to determining that the quantities are greater than the lower bound and less than the upper bound, communicate with one or more additional computing devices to cause an alert to be displayed indicating that an anomaly is present for the one or more specific components. [Explanation of symbols]

[0069] 100 Example Environments 101 Mobile Robot 102 tubes 103-A Client Device, Local Client Device 103-B Client Device, Local Client Device 105 Server Computing Devices 106 Process Automation Network 107 Visual Alarm 122-A Local Client Device 122-B Local client device 201 images 211 pre-trained ML models 221ML output 410 Computing Devices 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Memory Subsystem 425 Memory Subsystem, Memory 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM) 1011 Visual Components 1051 Machine Learning (ML) Engine 1052 Anomaly Detection Engine 1053 trained machine learning (ML) models, ML models 1055 Image Preprocessing Engine

Claims

1. 1. A method implemented by one or more processors, comprising: identifying an image captured by a camera of a mobile robot when the camera is in a first pose; the image captures one or more specific components within the mobile robot's environment; processing the image using a machine learning (ML) model to generate a reduced-dimensional output indicating, for each of the N regions of the image, a corresponding anomaly detection probability indicating whether an anomaly is present within the respective region; determining a number of anomaly detection probabilities each satisfying a threshold based on the reduced dimensionality output; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit; displaying an alert indicating that an anomaly exists for the one or more particular components; A method comprising:

2. 10. The method of claim 1, further comprising selecting the ML model from a plurality of candidate ML models to process the image, wherein selecting the ML model to process the image is responsive to the image capturing the one or more particular components, and the ML model is trained based on images corresponding to the one or more particular components.

3. 10. The method of claim 1, further comprising selecting the ML model from a plurality of candidate ML models to process the image, wherein selecting the ML model to process the image is responsive to the image being captured in the first pose and the ML model being trained based on an image corresponding to the first pose.

4. The method of claim 1, wherein N is greater than 9.

5. The method of claim 4 , wherein the lower limit is 20% or less of N.

6. The method of claim 4 , wherein the upper limit is equal to or greater than 50% of N.

7. The method of claim 1 , wherein the lower limit is determined based on one or more types of the one or more particular components.

8. The method of claim 1 , wherein the upper limit value is determined based on one or more types of the one or more particular components.

9. The method of claim 1 , wherein one or both of the lower limit value and the upper limit value are selected based on the time of day the image is captured and / or based on the light level of the environment.

10. The method of claim 1 , wherein the camera is a mono camera, a stereo camera, an ultraviolet camera, or a thermal camera.

11. The method of claim 1 , wherein the mobile robot is a wheeled robot, an unmanned aerial vehicle, or a quadruped robot.

12. In response to determining that the quantity is less than or equal to the lower limit, The method of claim 1 , further comprising determining that no anomalies are detected for the one or more particular components.

13. The method of claim 1 , wherein the one or more particular components include a liquid tank and / or a liquid carried by the liquid tank.

14. 2. The method of claim 1 , wherein the ML model is trained based on one or more training instances, each of which includes (1) a corresponding image and (2) N ground truth labels, each of which indicates whether an anomaly is present for a corresponding one of the N regions of the corresponding image.

15. 1. A system comprising: one or more computing devices including a camera and in at least selective network communication with a mobile robot deployed within the environment, said one or more computing devices: receiving from the mobile robot an image captured by the camera, the image capturing one or more particular components within the environment; processing the image using a trained machine learning (ML) model to generate a reduced-dimensional output indicating, for each of N regions of the image, a corresponding probability that an anomaly is present within the respective region; determining a corresponding probability quantity that each satisfies a threshold based on the reduced dimensionality outputs; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit; displaying an alert indicating that an anomaly exists for said one or more particular components; The system.

16. The one or more computing devices further include:

16. The system of claim 15, further comprising: selecting the trained ML model from a plurality of candidate ML models to process the image based on determining that the image is captured by the camera at a given pose and based on the trained ML model being trained based on images corresponding to the given pose.

17. The system of claim 15, wherein N is greater than 9.

18. The system of claim 15 , wherein the lower limit is less than or equal to 20% of N and the upper limit is greater than or equal to 50% of N.

19. A mobile robot, A camera and a memory for storing instructions and for storing a trained machine learning (ML) model; one or more processors, wherein the one or more processors execute the instructions to receiving an image from the camera capturing one or more particular components of an environment in which the mobile robot is deployed; and processing the image using the trained ML model to locally generate a reduced-dimensional output indicating, for each of the N regions of the image, a corresponding probability indicating whether an anomaly is present within the respective region; determining the corresponding probability quantities each satisfying a threshold based on the reduced dimensionality output; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit; communicating with one or more additional computing devices to cause an alert to be displayed indicating that an anomaly exists for the one or more particular components; A mobile robot operable to:

Images