Anomaly detection for component through machine-learning based image processing and considering upper bound value and lower bound value
A mobile robot with a camera and machine learning model addresses the challenge of extensive sensor deployment in industrial facilities by accurately detecting anomalies with reduced false positives and negatives, improving detection efficiency and reducing maintenance.
Patent Information
- Application Number
- JP2024069796
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-25
- Filing Date
- 2024-04-23
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2044-04-23
AI Technical Summary
Industrial facilities require numerous fixed sensors for anomaly detection, leading to extensive wiring and susceptibility to failure, making maintenance difficult and resource-intensive.
Utilize a mobile robot equipped with a camera to capture images, process them using a machine learning model like a convolutional neural network to detect anomalies, setting lower and upper bounds on anomaly detection probabilities to mitigate false positives and negatives.
Reduces sensor deployment and wiring needs while enhancing anomaly detection accuracy and robustness by minimizing false alarms and ensuring timely corrective actions.
Smart Images

Figure 2024157539000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to machine learning based image processing and anomaly detection of components by considering upper and lower limits. [Background technology]
[0002] Complex industrial facilities, such as petrochemical refineries, chemical plants, and the like, can include numerous components utilized in the processing of liquids, gases, and / or other substances involved in the industrial process of the industrial facility. It is important to ensure that the components involved in the industrial process are operating as intended and / or that the substances involved in the industrial process are in their intended state.
[0003] A variety of sensors have been utilized within industrial facilities to monitor for anomalies in such components and / or materials, including temperature sensors utilized to monitor for temperature anomalies in the components and / or materials, optical sensors utilized to monitor for anomalies in the composition of materials (e.g., based on exciting the materials with a light source), and the like.
[0004] Although various sensors may be utilized within an industrial facility to monitor various anomalies, each such sensor is typically deployed at a corresponding fixed location within the industrial facility. Thus, a large number of sensors may be required to effectively monitor the industrial facility. Furthermore, providing power and / or information to each of the sensors may require extensive wiring throughout the industrial facility. Still further, each of the large number of sensors is prone to failure or malfunction and must be repaired and / or replaced when they do occur, which may be difficult in a large industrial facility that includes a large number of fixed sensors. Summary of the Invention [Means for solving the problem]
[0005] Implementations of the present disclosure are directed to using a mobile robot to capture images of components and / or substances in an environment, such as in an industrial facility or other environment, at a given pose via a camera of the mobile robot. The implementations are further directed to processing the captured images using a machine learning (ML) model to generate an ML output. The machine learning model may be a neural network model, such as a convolutional neural network (CNN) model including one or more convolutional layers. The implementations are further directed to determining whether an anomaly exists with the components and / or substances captured by the image based on the generated ML output, and if so, causing one or more corrective actions to be performed. Executing a corrective action may include, for example, displaying an alarm, stopping a process, and / or performing other corrective actions.
[0006] In many implementations, the ML model utilized may be a model trained to be used to process an image to generate a reduced-dimension ML output that is of reduced dimension compared to the processed image. Each value in the reduced-dimension ML output may directly or indirectly reflect the probability that a corresponding region of the image contains an anomaly. For example, each value may be a corresponding value from 0 to 1, and each value may correspond to a corresponding portion of the image encompassing multiple pixels of the image (e.g., a corresponding 64×64 pixel portion of the image). In those implementations, determining whether an anomaly exists may be based on determining the quantity of values that satisfy a threshold (e.g., a threshold such as 0.7) and determining that an anomaly exists only when the quantity is (a) greater than a lower bound and (b) less than an upper bound. For example, assume that the ML output includes 16 values. An anomaly may be detected only when the quantity of values that satisfy the threshold is greater than 1 or 2 (or other lower bound) but less than 8 or 9 (or other upper bound).
[0007] The occurrence of false positive and / or false negative anomaly detections may be mitigated by determining that a quantity exists only when the quantity is (a) greater than a lower limit and (b) less than an upper limit. For example, the occurrence of false positives in detecting anomalies may be prevented by considering an upper limit in addition to a lower limit. For example, the occurrence of false positives may be prevented by not determining that an anomaly exists when the quantity of values satisfying a threshold is greater than an upper limit. For example, when the quantity of values satisfying a threshold is greater than an upper limit, implementations disclosed herein recognize that it may not be due to an anomaly but may be due to other external factors in the environment, such as excess light (e.g., from the sun and / or lighting devices), the presence of a person or other moving object in the image, and / or issues with the camera that captured the image. Mitigating the occurrence of false positives may prevent the utilization of network and / or computing device resources in erroneously performing corrective actions, such as, for example, displaying a false positive alarm. Mitigating the occurrence of false negatives may prevent the occurrence of unsafe conditions in an industrial facility and / or damage to components and / or materials in the industrial facility.
[0008] In some implementations, the same lower limit value and / or the same upper limit value may be used in determining whether an anomaly exists for each of a plurality of distinct components and / or substances in an environment. For example, the same lower limit value (e.g., 2) and the same upper limit value (e.g., 11) may be utilized in analyzing both (a) a first ML output generated based on processing a first image of a tank (in a first area of the environment) with the ML model and (b) a second ML output generated based on processing a second image of a pipe (in a second area of the environment).
[0009] In some additional or alternative implementations, the lower and / or upper limit values used for the first component may be different than the lower and / or upper limit values used for the second component. For example, a first image determined (e.g., based on metadata of the first image) to capture a first component may be processed using an ML model to generate a first ML output, a first quantity of values of the first ML output that satisfy a threshold may be determined, and an anomaly for the first component is determined only if the first quantity is between 2 and 9. Continuing the example, a second image determined (e.g., based on metadata of the second image) to capture a second component may be processed using an ML model to generate a second ML output, a second quantity of values of the second ML output that satisfy a threshold may be determined, and an anomaly for the second component is determined only if the first quantity is between 3 and 8, which is a more restricted range than the range utilized for the first image determined to capture the first component. A more restricted range is less likely to lead to an anomaly being determined. In some of those implementations, the less restrictive range and / or the more restrictive range may optionally be explicitly manually specified based on user interface input. The manual specification may reflect the importance of each component (e.g., a first component may be more important than a second component) and / or may be based on additional or alternative considerations, such as how dynamic the area in which the components are located is expected to be. In some others of those implementations, the less restrictive range and / or the more restrictive range may optionally be set depending on implicit user interface input. For example, the range for the second component may initially be 2 to 9, but may be automatically adjusted to a more restrictive 3 to 8 based on a precedent of user feedback indicating that the anomaly determined based on the 2 to 9 range was not in fact an anomaly. For example, the precedent of user feedback may have been provided by an operator after reviewing an image of the first component and / or after the operator's own inspection of the first component (e.g., feedback indicating that the previous anomaly detection was a mistake).
[0010] In some implementations, the same lower limit value and / or the same upper limit value may always be used in determining whether an anomaly exists for a given component and / or a given substance in an environment.
[0011] In some alternative implementations, the lower and / or upper limits used for a given component and / or a given substance can change very dynamically over time. For example, a more restrictive range can be used during certain temporal conditions (e.g., day of the week, time of day, date of the year) and a less restrictive range can be used during certain other temporal conditions. As another example, a first range can be used for a first duration and then adjusted to a more or less restrictive range based on explicit or implicit user input as described above.
[0012] The captured image may be a high-resolution image having image dimensions (sometimes referred to as "pixel dimensions") of m x n x c pixels, where m is the width dimension, n is the height dimension, and c is the channel dimension. For example, c may be 3 when the image is a red, green, blue (RGB) image, and c may be 1 when the image is a grayscale image. The m x n value may be, for example, 1024 x 1024, 1024 x 768, 2048 x 2048, or other values. The reduced-dimensional ML output including the multiple probabilities may be arranged, for example, in a q1 x q2 matrix, where the q1 x q2 dimensions are smaller than the m x n dimensions. For example, the reduced dimensionality ML output may be a 2×2 matrix that defines four different probabilities (p1, p2, p3, p4), where p1 is a probability indicating that the anomaly is present in a first region of the image, p2 is a probability indicating that the anomaly is present in a second region of the image, p3 is a probability indicating that the anomaly is present in a third region of the image, and p4 is a probability indicating that the anomaly is present in a fourth region of the image. The first, second, third, and fourth regions may jointly correspond to all regions of the image. The first, second, third, and fourth regions may correspond to entirely unique regions of the image, or one or more of the regions may overlap partially with other of the regions.
[0013] The mobile robot utilized for capturing images to be processed in monitoring for anomalies according to the techniques disclosed herein may be a quadruped robot, a wheeled robot, an unmanned aerial vehicle, a track-guided robot, or any other robot that moves on its own in an environment. The images are captured via a vision component of the mobile robot. Each image is captured when the vision component is in a corresponding pose (i.e., a given position and orientation). The pose of the vision component when the image is captured at a given time is a function of the pose of the robot at the given time. When the vision component is in a fixed pose relative to the robot, the pose of the vision component is entirely a function of the pose of the robot. When the pose of the vision component is independently adjustable relative to the robot, the pose of the vision component is a function of the pose of the robot and the pose of the vision component relative to the robot. An image as utilized herein may include, for example, an RGB image including a red channel, a green channel, and a blue channel and captured by a monograph RGB camera, an RGB-D image including a depth channel in addition to red, green, and blue channels and captured by a stereo camera, an ultraviolet (UV) image including a UV channel and captured by a UV camera, or a thermal image including one or more thermal channels and captured by a thermal camera.
[0014] By utilizing a mobile robot that includes a vision component and moves around the industrial environment to capture images of components and / or materials of interest for anomaly monitoring / detection, the number of sensors monitoring the industrial environment may be reduced and / or the extensive wiring required for such sensors may be avoided. Moreover, the accuracy of anomaly detection may be improved by processing the captured images using an ML model to generate an ML output having multiple values, and determining that an anomaly exists only when the quantity of those values that satisfy a threshold is (a) greater than a lower limit and (b) less than an upper limit. This allows anomaly detection to be more robust and / or more accurate.
[0015] It should be appreciated that all combinations of the foregoing and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 illustrates a schematic of an environment in which selected aspects of the present disclosure may be implemented, according to various implementations. [Diagram 2] FIG. 1 illustrates a schematic diagram of an example of how the techniques described herein may be implemented, according to various implementations. [Diagram 3] FIG. 1 illustrates an exemplary method for carrying out selected aspects of the present disclosure. [Figure 4] FIG. 1 illustrates generally an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Implementations described herein relate to monitoring and detecting anomalies associated with components (e.g., liquid tanks, piping, wiring) in an environment (e.g., an industrial automation facility) based on ML outputs (e.g., reduced-dimensional outputs) generated by processing images captured of the components using a trained ML model (e.g., a trained neural network model). In various implementations, images of the components are captured by a vision component (e.g., a camera) carried by or integrated into a mobile robot capable of moving within the environment. The reduced-dimensional ML output can indicate, for each of N regions of the image, a corresponding probability that an anomaly is present in the respective region of the image. Determining whether an anomaly is present can be based on analyzing the reduced-dimensional ML output. For example, each value in the reduced-dimensional ML output can reflect a probability that a corresponding region of the image contains an anomaly, and an anomaly can be determined to be present only when a quantity of values satisfying a threshold is greater than a lower limit value and less than an upper limit value. Implementations further relate to performing one or more corrective actions in response to detecting that an anomaly is present in the environment. As noted above, each value in the reduced dimensionality output can reflect a probability that the corresponding region of the image contains an anomaly. In some implementations, the values can directly reflect the probability. For example, each value can be a corresponding value from 0 to 1 and can directly reflect the probability. For example, a value of 0.18 can reflect an 18% probability. In some implementations, the values can indirectly reflect the probability. For example, each value can be a corresponding value from 0 to 8 and can indirectly reflect the probability. For example, a value of 2 can reflect a 25 percent probability.
[0018] Referring now to FIG. 1, an example environment 100 in which various aspects of the disclosure may be implemented is illustrated generally. In some implementations, the example environment 100 may be or may include an industrial automation facility. The industrial facility may take many forms and may be designed to implement any number of at least partially automated processes. The industrial automation facility may take the form of a chemical processing plant, an oil or natural gas refinery, a catalyst factory, a manufacturing facility, an offshore oil platform, etc.
[0019] The exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operatively coupled with a process automation network 106 in an industrial automation facility. The client devices 103-A or 103-B may be implemented as a computer (e.g., laptop, desktop, notebook), tablet, robot, smart appliance (e.g., smartphone), messaging device, wearable device (e.g., watch), or any other applicable device. The process automation network 106 may be implemented using various wired and / or wireless communication technologies, including, but not limited to, the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), 3GPP® Long Term Evolution ("LTE") or other wireless protocols described as 3G, 4G, 5G and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).
[0020] The exemplary environment 100 may further include a mobile robot 101 having or carrying a vision component 1011. The mobile robot 101 may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, a robot that moves along elevated and / or non-elevated tracks in the environment, or any other applicable robot that can move in an industrial facility. The vision component 1011 may be a monographic camera, a stereo camera, a thermal camera, or any other applicable visual sensor for capturing one or more images of one or more specific components of the industrial automation facility (e.g., a liquid tank T or tube 102 that stores or transports a liquid substance). The vision component 1011 may be removably coupled to the mobile robot 101 or may be integrated as a non-removable coherent part of the mobile robot 101. In some implementations, the vision component 1011 may independently change location and / or orientation relative to the robot frame of the mobile robot 101, for example, by rotation or other movement via an actuator that solely controls the vision component 1011. In addition to the vision component 1011, the mobile robot 101 may include one or more additional vision components for navigating through an industrial facility, sensing static or dynamic objects, and / or capturing images.
[0021] The exemplary environment 100 may further include a server computing device 105. The server computing device 105 may include a machine learning (ML) engine 1051 and an anomaly detection engine 1052. The server computing device 105 may further include or otherwise have access to one or more trained machine learning (ML) models 1053, each trained to generate an output indicative of a plurality of probabilities, with each probability indicative of the likelihood that an anomaly exists for a respective region of the N regions of the processed image. The server computing device 105 may be connected to a plurality of client devices. The server computing device 105 may communicate with one or more local client devices (e.g., 122-A and 122-B) and / or with one or more remote client devices (not shown). The local client devices 122-A or 122-B may be connected to the server computing device 105 via one or more local area networks (e.g., the process automation network 106), and the remote client devices may be connected to the server computing device 105 via one or more wide area networks (e.g., the Internet). The local and remote client devices may be operable by personnel, such as system integrators, to configure and / or interact with various aspects of the example environment 100.
[0022] In some implementations, the server computing device 105, in addition to the ML engine 1051 and the anomaly detection engine 1052, may include a database (not shown) that stores information used by the ML engine 1051 and / or the anomaly detection engine 1052 to perform selected aspects of the disclosure. In some implementations, the server computing device 105 may include, in addition to the ML engine 1051 and the anomaly detection engine 1052, an image pre-processing engine 1055 that processes different images to have the same image dimensions. Various aspects of the server computing device 105, such as the ML engine 1051, the anomaly detection engine 1052, and / or the image pre-processing engine 1055, may be implemented using any combination of hardware and software. In some implementations, the ML engine 1051, the anomaly detection engine 1052, the image pre-processing engine 1055, or the trained ML model 1053 may be implemented across multiple computer systems as part of what is often referred to as a “cloud infrastructure” or simply “the cloud.” 1, for example, the ML engine 1051 is implemented in an industrial facility, for example, in a single building, or across a single campus of multiple buildings or other industrial infrastructure. In such an implementation, the ML engine 1051 may be implemented on one or more local computing systems, such as one or more local server computers.
[0023] In some implementations, the mobile robot 101 can navigate through an industrial facility and arrive at one or more designated spots or waypoints. The vision component 1011 of the mobile robot 101 can be (but need not necessarily be) configured at a given pose to capture an image of the liquid tube 102 at the given pose. The vision component 1011 can be configured at a given pose as a result of the mobile robot 101 being at a corresponding pose and / or as a result of the vision component 1011 being independently adjusted (when the vision component 1011 is independently adjustable relative to the robot frame of the mobile robot 101). The image captured by the vision component 1011 can include pixels or other data to capture and can correspond to an area or component of interest, such as the liquid tank T and the liquid substance contained therein (shown in FIG. 1).
[0024] The captured image may be processed by the ML engine 1051 as an input to the trained ML model to generate an ML output. The ML output may be a reduced-dimensional output indicating for each of the N regions of the image a corresponding probability that an anomaly exists within the respective region of the N regions. The anomaly detection engine 1052 may process the ML output in determining whether an anomaly exists for the component, and if an anomaly is determined to exist, may cause one or more corrective actions to be executed to address the detected anomaly. For example, the anomaly detection engine 1052 may cause an audio and / or visual alert to be displayed at the computing device, such as a visual alert 107 that is visually displayed at the client device 103-A.
[0025] Although only a single mobile robot 101 is shown in FIG. 1, it is understood that multiple mobile robots may be deployed in an industrial environment and utilized in the implementations disclosed herein. For example, each of the multiple mobile robots may include a corresponding vision component used to capture images, and images from the multiple mobile robots are transmitted to the server device 105 for processing by the ML engine 1051 and the anomaly detection engine 1052. Also, while the ML engine 1051, the ML models 1053, the anomaly detection engine 1052, and the image pre-processing engine 1055 are shown in FIG. 1 as being implemented separately from the mobile robot 101, in some implementations, all aspects may be implemented by the mobile robot 101. For example, the mobile robot 101 may include the ML engine 1051, the anomaly detection engine 1052, and at least a subset of the ML models 1053. For example, the mobile robot 101 may include a subset of the ML model 1053 at a given time based on that subset's correspondence to a component, and the mobile robot 101 captures an image of that component with a task to be performed by the mobile robot 101 at the given time.
[0026] FIG. 2 illustrates an example of how the techniques described herein may be implemented according to various implementations. As illustrated in FIG. 2, a vision component 1011 of a mobile robot 101 may capture an image 201. The image 201 captures a particular component (e.g., a liquid tank T and a liquid carried by the liquid tank) in an industrial automation facility. A vision sensor captures the image 201 at a given pose. The image 201 may have an image dimension of m×n pixels (and c channels) and may be processed using a trained ML model 211, which may be one (e.g., only one, or one of multiple) of the ML models 1053 of FIG. 1. The trained ML model 211 may be trained for use in processing the image to generate an ML output indicating, for each of a plurality of regions of the image, a corresponding probability that an anomaly exists in the region.
[0027] In some implementations, the trained ML model 211 is selected from multiple candidates of the ML model 1053 to process the image 201. In some of those implementations, the trained ML model 211 is selected to process the image 201 based on the image 201 capturing a particular component (e.g., liquid tank T), and the ML model 211 is trained based on training instances including training instance inputs of images also capturing the particular component (e.g., liquid tank T) and training instance outputs of ground truth probabilities. For example, the trained ML model 211 may be fine-tuned based on such training instances. In some of those implementations, metadata embedded or included in the image 201 may directly or indirectly indicate a particular component, and such metadata may be used in selecting the trained ML model 211. For example, the metadata may indicate a location where the image was captured, and such location may be correlated with a particular component and / or the trained ML model 211.
[0028] In some additional or alternative implementations, the trained ML model 211 is a model trained based on training instances, each including a training instance input of an image that captures or nearly captures a specific component (e.g., the liquid tank T) using corresponding visual components at a given pose of the image 201, and a training instance output of a ground truth probability. For example, the trained ML model 211 can be fine-tuned based on such training instances. In some of those implementations, the trained ML model 211 is selected based on determining that the image 201 captures a specific component at a given pose and that the trained ML model 211 is a model fine-tuned based on training instances for the specific component and the given pose. For example, metadata incorporated in or included in the image 201 can directly or indirectly indicate the given pose, and such metadata can be used in selecting the trained ML model 211. For example, the metadata can indicate a given pose of the visual components when the image was captured, and the given pose can be correlated with the trained ML model 211.
[0029] In some additional or alternative implementations, the trained ML model 211 can be selected from a plurality of trained ML models based on the image dimensions of the image 201.
[0030] The image 201 can be processed using the ML model 211 to generate an ML output 221 of the ML model 211. The ML output 221 can be a reduced-dimension output indicating, for each of N regions of the image 201, a corresponding probability (also referred to herein as an “anomaly detection probability”) that an anomaly exists within each region of the N regions. The ML output 221 can indicate or include a plurality of anomaly detection probabilities arrayed within a matrix (e.g., a vector or a multi-dimensional matrix), and the position of the i-th anomaly detection probability within the matrix corresponds to the location of the i-th (0 < i < N) region within the image 201.
[0031] 2, the reduced dimensional output may include first, second, ..., and sixteenth probabilities arranged in a matrix M1, also shown below: In matrix M1, the location of the probability (e.g., p13) may correspond to the location of a respective region (e.g., the thirteenth region shown in FIG. 2) in image 201, for which a probability (e.g., p13) is predicted.
[0032]
number
[0033] Based on the ML output 221 (i.e., the output of reduced dimension), it may be determined whether an anomaly is detected for a particular component. For example, based on the multiple anomaly detection probabilities indicated by the ML output 221, it may be determined whether an anomaly is detected for a particular component. In some implementations, a quantity of anomaly detection probabilities that satisfy a threshold (e.g., 0.7 or other threshold) may be determined from the N anomaly detection probabilities. With reference to FIG. 2, p1, p2, p4, p5, p8, and p9 may be identified as satisfying a threshold (e.g., 0.7), such that the quantity of anomaly detection probabilities that satisfy the threshold may be determined to be “6”. In those implementations, if the quantity (e.g., 6) of anomaly detection probabilities that satisfy a threshold (e.g., 0.7) is greater than a lower limit value (e.g., “1”) and less than an upper limit value (e.g., “11”), an image 201 of the particular component may be determined to capture an anomaly associated with the particular component. As described herein, the lower and / or upper limits may optionally depend on the particular component captured (e.g., as indicated by metadata of the image 201), temporal conditions, and / or other factors. Stated differently, in some implementations, the lower and / or upper limits may be dynamic for a given component and / or may be component-based.
[0034] In response to determining that the image 201 captures an anomaly, one or more corrective actions may be performed. The one or more corrective actions may include causing an alarm to be displayed visually and / or audibly. Alternatively or additionally, the one or more corrective actions may include pausing one or more processes, such as one or more processes involving the particular component. Alternatively or additionally, the one or more corrective actions may include notifying one or more operators to inspect the particular component.
[0035] 3 is a flowchart illustrating an example method 300 of performing selected aspects of the disclosure, according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computing devices, such as one or more components of the server computing device 105 (and / or additional computing devices, such as the mobile robot 101 or the client device 103-A), including the ML engine 1051 and / or the anomaly detection engine 1052. Moreover, although the operations of the method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.
[0036] In block 302, the system identifies an image captured by a vision component (e.g., a camera) of the mobile robot, the image capturing one or more specific components of an industrial automation facility. The industrial automation facility may be a chemical processing plant, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable facility implementing one or more at least partially automated processes. The one or more specific components may include, by way of non-limiting example, a liquid tank and / or a liquid that the liquid tank carries. The mobile robot may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot capable of moving within the industrial automation facility.
[0037] The vision component may be a monographic RGB camera, a stereo camera, a thermal camera, a UV camera, or any other applicable vision component. The image captured by the vision component may accordingly be an RGB image, an RGB-D image, a UV image, a thermal image, or any other applicable image. In some implementations, the vision sensor may be integrated with the mobile robot or may be removably coupled to the mobile robot.
[0038] At block 304, the system processes the image using an ML model (e.g., a CNN or other neural network model) to generate a reduced-dimensional output that indicates or predicts an anomaly detection probability (sometimes simply referred to as a "probability") for each of the N regions of the image, indicating whether an anomaly is present within the respective region of the N regions. The value of "N" may be, for example, 9 or greater, but this is not intended to be limiting. For example, the value N may be equal to 4, 6, 25, 36, or any other applicable positive integer.
[0039] The ML model used by the system in processing the images at block 304 may be trained based on one or more training instances, each training instance including a training instance input of a corresponding image and a training instance output of N corresponding ground truth labels, each ground truth label indicating whether an anomaly is present in a respective one of the N regions of the respective image. For example, each of the ground truth labels may be either an "anomaly present" label (e.g., "1") or an "anomaly absent" label (e.g., 0). The "anomaly present" label, if present, may be applied to a label corresponding to a region of the corresponding image that includes an anomaly, and the "anomaly absent" label, if present, may be applied to a label corresponding to a region of the corresponding image that does not include an anomaly. For example, if the anomaly is corrosion and corrosion is present in only two of the 16 regions of the image, the ground truth labels may include two "anomaly present" labels corresponding to the two anomalous regions of the image and 14 "anomaly absent" labels corresponding to the 14 non-anomalous regions of the image.
[0040] In some implementations, the system optionally selects an ML model for processing the image from the multiple trained ML models at block 3041. In some of those implementations, the system selects an ML model for processing the image based on an image capturing one or more particular components, and the ML model is trained based on an image corresponding to the one or more particular components. In some additional or alternative implementations of those implementations, the system selects an ML model from the multiple trained ML models based on the image being captured at a given pose (e.g., a first pose corresponding to a first position and / or a first orientation of a visual component), and the ML model is trained based on an image corresponding to the given pose. In some additional or alternative implementations of those implementations, the system selects an ML model from the multiple trained ML models based on the image based on a type of image (e.g., thermal, RGB, RGB-D) and / or image dimensions (e.g., 1024×768 pixels).
[0041] As a non-limiting example, the plurality of trained ML models may include a first ML model trained using an image captured by a visual sensor at a first pose, where the first ML model is trained to process the first image captured by the visual sensor at the first pose to output a first quantity ("N_1") of anomaly detection probability for a total number of N_1 regions in the first image captured at the first pose. The plurality of trained ML models may include a second ML model trained using an image captured by the visual sensor at a second pose, where the second ML model is trained to process the second image captured by the visual sensor at the second pose to output a second quantity ("N_2") of anomaly detection probability for a total number of N_2 regions in the second image captured at the second pose. The second quantity ("N_2") may optionally be different from the first quantity ("N_1"), where the first pose is different from the second pose. The multiple trained ML models can include additional ML models, such as a third model trained based on images captured in a third pose.
[0042] In block 306, the system can determine a quantity of anomaly detection probabilities that each satisfy a threshold based on the reduced dimension output. In some implementations, the system identifies anomaly detection probabilities that satisfy a threshold from the reduced dimension output in block 3061. The system can determine a quantity based on a count of the identified anomaly detection probabilities that satisfy the threshold in block 3063. For example, assuming the threshold is 0.7, the reduced dimension output shows a total of 16 anomaly detection probabilities: 0.65, 0.72, 0.65, 0.45, 0.55, 0.78, 0.85, 0.8, 0.85, 0.55, 0.45, 0.82, 0.78, 0.48, 0.58, 0.63. In such an example, the anomaly detection probabilities that satisfy the threshold (0.7) may be identified to include: 0.72, 0.78, 0.85, 0.8, 0.85, 0.82, 0.78. In this example, the quantity of anomaly detection probabilities identified as satisfying the threshold may be determined to be "7."
[0043] In block 308, the system determines whether an anomaly exists based on comparing the quantity against both the lower limit and the upper limit. For example, the system may determine that an anomaly exists only when the quantity is higher than the lower limit and lower than the upper limit. In some implementations, the lower limit and / or upper limit are static. In some implementations, the lower limit and upper limit may be determined based on the value of "N" (i.e., the quantity of regions reflected in the reduced dimensional output). For example, the lower limit may be less than or equal to 20% of N, such as about 10% of the value of "N". Also, for example, the upper limit may be greater than or equal to 50 percent of N, such as about 90 percent of the value of "N".
[0044] Alternatively or additionally, in some implementations, the lower limit value and / or the upper limit value may be dynamic. For example, one or both may be determined based on the type or other characteristics of the component captured by the image, current temporal conditions, detected or inferred light levels within the industrial automation facility, and / or other factors. For example, a lower limit value utilized during daylight hours and / or periods of higher light levels may differ from a lower limit value utilized during nighttime hours and / or periods of lower light levels.
[0045] As a non-limiting example, "N" can have 16 values. In this example, the lower limit may be predefined to be "1" or "2" and the upper limit may be predefined to be "7" or "8." When the quantity of anomaly detection probability identified as satisfying the threshold is determined to be "5," which is greater than the lower limit and less than the upper limit, an image with a predicted anomaly detection probability may be determined to capture anomalies for one or more particular components.
[0046] Additionally, in some implementations, the system may display an alert indicating that an anomaly exists for one or more particular components in response to determining that the quantity is greater than the lower limit and less than the upper limit. Alternatively or additionally, the system may pause one or more processes in response to determining that the quantity is greater than the lower limit and less than the upper limit.
[0047] In some implementations, the system can determine that the quantity is less than or equal to the lower limit, in which the system can determine that no anomaly is detected for one or more particular components in response to determining that the quantity is less than or equal to the lower limit.
[0048] In some implementations, the system can determine that the quantity is equal to or greater than the upper limit value. In these implementations, the system can generate a notification recommending manual inspection of the image to determine whether the image is suitable for detecting an anomaly for one or more specific components. For example, if an image is captured while the visual sensor is exposed to excessive light, or if the image captures a moving object instead of one or more specific components, the image can be identified as not containing an anomaly for one or more specific components. Such images, optionally with supervised labeled output that does not show an anomaly for any of the N regions, can be used to further train the ML model. Optionally, the system can further control the mobile robot to capture additional images via the visual sensor. Optionally, the additional images can be captured at different poses than the images processed for anomaly detection, and a different ML model can be selected to process the additional images to see if an anomaly is detected.
[0049] Instead of or in addition to determining to cause an alarm to be displayed, the system may perform one or more other corrective actions. The one or more other corrective actions may include pausing or stopping one or more industrial processes (e.g., automated processes), such as an industrial process involving one or more particular components in which an anomaly is detected. For example, the system may determine to pause a process involving one or more particular components in an industrial automation facility based on both the first output and the second output.
[0050] In some implementations, at least some aspects of the methods described herein are performed by one or more processors of one or more computing devices that are separate from and not attached to the mobile robot, in some of those implementations, the images are transmitted by the mobile robot to the computing device and identified by the computing device after being transmitted by the mobile robot.
[0051] 4 is a block diagram of an exemplary computing device 410 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 410 generally includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computing device 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.
[0052] The user interface output devices 422 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 410 or onto a communications network.
[0053] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 410 to a user or to another machine or computing device.
[0054] Storage subsystem 424 stores program and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may contain logic for implementing the various components shown in Figures 1 and 2, as well as for performing selected aspects of the method of Figure 3.
[0055] These software modules are generally executed by the processor 414 alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 426 within the storage subsystem 424 or in other machines accessible by the processor 414.
[0056] The bus subsystem 412 provides a mechanism for allowing the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus system 412 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.
[0057] The computing device 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 410 shown in FIG. 4 is intended only as a specific example to illustrate some implementations. Many other configurations of the computing device 410 can have more or fewer components than the computing device shown in FIG.
[0058] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications are deemed to be within the scope of the implementations described herein. More generally, it is intended that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific implementations described herein. It is therefore to be understood that the foregoing implementations are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be performed differently from those specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, item, material, tool, and / or method described herein. In addition, any combination of two or more of such features, systems, articles, materials, tools, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, tools, and / or methods are not mutually inconsistent.
[0059] In various implementations, the method may be implemented using one or more processors and may include identifying an image captured by a camera of the mobile robot, the image capturing one or more specific components of the environment. The environment may optionally be an industrial automation facility implementing any number of at least partially automated processes, such as a chemical processing plant, an oil or natural gas refinery, a catalyst factory, a manufacturing facility, an offshore oil platform, or any other applicable industrial environment. The mobile robot may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot capable of moving within the industrial automation facility. The camera may be a Monograph RGB camera, a stereo camera, a thermal camera, or any other applicable visual sensor. Accordingly, the image may be an RGB image, an RGB-D image, a thermal image, or any other applicable image having image dimensions of m×n×c (m, n, c are positive integers equal to or greater than 1).
[0060] In various implementations, the method may further include processing the image using the trained machine learning (ML) model to generate a reduced-dimension output indicating for each of N regions of the image a corresponding probability that an anomaly exists within the region, where N is a positive integer less than (m×n). As a non-limiting example, the image may be a 1024×768 RGB image, and the reduced-dimension output may indicate for each of 3×3 (=9) regions of the image a corresponding probability that an anomaly exists within each of the 9 regions. In this non-limiting example, the reduced-dimension output may indicate, for example, a first probability p1 for one or more anomalies to be detected for a first region of the image, a second probability p2 for one or more anomalies to be detected for a second region of the image, ..., and a sixteenth probability p16 for one or more anomalies to be detected for a sixteenth region of the image, where the first, second, ..., sixteenth regions together form the image.
[0061] As a non-limiting example, the reduced dimensional output may include first, second, ..., and sixteenth probabilities arranged in a matrix. The location of the probability (e.g., p2) in the matrix may correspond to the location of the respective region (e.g., second region) in the image for which the probability (e.g., p2) is predicted.
[0062] In various implementations, the method may further include determining, based on the reduced dimension output, a quantity of corresponding probabilities that each satisfy a threshold. In various implementations, the method may further include determining whether the quantity is greater than a lower bound and less than an upper bound. Continuing with the non-limiting example above, the threshold may be approximately 0.7, and the reduced dimension output may indicate p1=0.6, p2=0.8, p3=0.5, p4=0.75, p5=0.77, p6=0.82, p7=0.85, p8=0.5, p9=0.9, p10=0.65, p11=0.77, p12=0.58, p13=0.83, p14=0.3, p15=0.48, and p16=0.28. In this example, the quantity of probabilities each satisfying a threshold value (e.g., 0.7) may be determined as 8 (p2=0.8, p4=0.75, p5=0.77, p6=0.82, p7=0.85, p9=0.9, p11=0.77, and p13=0.83). If the lower limit is predefined to be "2" and the upper limit is predefined to be "10", the quantity of probabilities satisfying the thresholds (e.g., 8) may be determined to be greater than the lower limit (e.g., 2) but also less than the upper limit (e.g., 10).
[0063] In various implementations, the method may further include causing an alert to be displayed in response to determining that the quantity is greater than the lower limit and less than the upper limit, indicating that an anomaly exists for one or more specific components. Continuing with the non-limiting example above, in response to determining that 8 out of 16 probabilities, 2 ("lower limit") < 8 < 10 ("upper limit"), satisfy the threshold, an alert may be displayed to warn that one or more specific components have an anomaly. The alert may be a text message (e.g., email, text, etc.) indicating the one or more specific components experiencing an anomaly, a confidence level of the presence / detection of the anomaly for the one or more specific components, and / or the time the anomaly was detected, etc. The alert may also be an audio message announcing the presence of a possible anomaly for one or more specific components. If the quantity is greater than the upper limit or less than the lower limit, no alert may be generated. For example, if the quantity is greater than the upper limit value, it is highly likely that the image does not accurately or completely reflect the one or more particular components due to the presence of a moving object (e.g., a person) blocking the one or more particular components when the image is being captured by the camera, exposure to excessive light when the image is captured, etc. If the quantity is less than the lower limit value, it may be determined that no anomaly exists for the one or more particular components.
[0064] In some implementations, instead of or in addition to determining to display an alert, the method may include pausing one or more processes involving one or more particular components in the environment in response to determining that the quantity is greater than the lower limit value and less than the upper limit value.
[0065] Additionally, some implementations include one or more processors of the mobile robot and / or one or more computing devices operable to execute instructions stored in associated memory, the instructions configured to cause any of the methods disclosed herein to be performed. Some implementations additionally or alternatively include one or more transitory or non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the methods disclosed herein.
[0066] In some implementations, a system is provided that includes one or more processors and a memory that stores instructions that, in response to execution by the one or more processors, cause the one or more processors to identify images captured by a camera of a mobile robot when the camera is in a first pose, where the images capture one or more particular components within an environment of the mobile robot; process the images using a machine learning (ML) model to generate a reduced-dimensional output indicating, for each of N regions of the image, a corresponding probability that an anomaly is present within the region; determine corresponding probability quantities, each of which satisfies a threshold, based on the reduced-dimensional output; and determine whether the quantities are greater than a lower limit value and less than an upper limit value.
[0067] In various implementations, the system can further include instructions for causing an alert to be displayed indicating that an anomaly exists for one or more particular components in response to determining that the quantity is greater than the lower limit and less than the upper limit. In various implementations, the system can further include instructions for pausing one or more processes utilizing one or more particular components in response to determining that the quantity is greater than the lower limit and less than the upper limit.
[0068] In some implementations, a mobile robot is provided that includes a camera, a memory storing instructions and storing a trained machine learning (ML) model, and one or more processors. The processor is operable to execute instructions to receive from the camera an image capturing one or more specific components of an environment in which the mobile robot is deployed, process the image using the trained ML model to locally generate a reduced-dimensional output indicating for each of the N regions of the image a corresponding probability indicating whether an anomaly exists within the respective region of the N regions of the image, determine corresponding probability quantities each satisfying a threshold based on the reduced-dimensional output, determine whether the quantities are greater than a lower bound and less than an upper bound, and in response to determining that the quantities are greater than the lower bound and less than the upper bound, communicate with one or more additional computing devices to cause an alert to be displayed indicating that an anomaly exists for the one or more specific components. [Explanation of symbols]
[0069] 100 Example Environment 101 Mobile Robot 102 Tube 103-A Client Device, Local Client Device 103-B Client Device, Local Client Device 105 Server Computing Devices 106 Process Automation Network 107 Visual alarm 122-A Local Client Device 122-B Local client device 201 images 211 pre-trained ML models 221ML output 410 Computing Devices 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Memory Subsystem 425 Memory Subsystem, Memory 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM) 1011 Visual Components 1051 Machine Learning (ML) Engine 1052 Anomaly Detection Engine 1053 Trained Machine Learning (ML) Models, ML Models 1055 Image Pre-processing Engine
Claims
1. 1. A method implemented by one or more processors, comprising: identifying an image captured by a camera of a mobile robot when the camera is in a first pose, the image captures one or more particular components within an environment of the mobile robot; processing the image using a machine learning (ML) model to generate a reduced dimensional output indicating for each of the N regions of the image a corresponding anomaly detection probability indicating whether an anomaly is present within each of the N regions of the image; determining a quantity of anomaly detection probabilities each satisfying a threshold based on the reduced dimensionality output; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit, causing an alert to be displayed indicating that an anomaly exists for the one or more particular components; A method comprising:
2. 10. The method of claim 1, further comprising: selecting the ML model from a plurality of candidate ML models to process the image, wherein selecting the ML model to process the image is responsive to the image capturing the one or more particular components, and the ML model is trained based on images corresponding to the one or more particular components.
3. 2. The method of claim 1 , further comprising: selecting the ML model from a plurality of candidate ML models to process the image, wherein selecting the ML model to process the image is responsive to the image being captured in the first pose and the ML model being trained based on an image corresponding to the first pose.
4. The method of claim 1 , wherein N is greater than 9.
5. The method of claim 4 , wherein the lower limit is 20% or less of N.
6. The method of claim 4 , wherein the upper limit is equal to or greater than 50% of N.
7. The method of claim 1 , wherein the lower limit is determined based on one or more types of the one or more particular components.
8. The method of claim 1 , wherein the upper limit value is determined based on one or more types of the one or more particular components.
9. The method of claim 1 , wherein one or both of the lower limit value and the upper limit value are selected based on the time of day the image is captured and / or based on the light level of the environment.
10. The method of claim 1 , wherein the camera is a monographic camera, a stereo camera, an ultraviolet camera, or a thermal camera.
11. The method of claim 1 , wherein the mobile robot is a wheeled robot, an unmanned aerial vehicle, or a quadruped robot.
12. in response to determining that the quantity is less than or equal to the lower limit, The method of claim 1 , further comprising determining that no anomaly is detected for the one or more particular components.
13. 13. The method of claim 12, further comprising the step of causing a selectable element to be displayed via a display indicating the absence of anomalies, the selectable element, when selected, providing access to the image and / or access to metadata associated with the image.
14. The method of claim 1 , wherein the one or more particular components include a liquid tank and / or a liquid carried by the liquid tank.
15. 2. The method of claim 1 , wherein the ML model is trained based on one or more training instances, each of which includes (1) a corresponding image and (2) N ground truth labels, each of which indicates whether an anomaly is present for a corresponding one of the N regions of the corresponding image.
16. 1. A system comprising: one or more computing devices in at least selective network communication with the mobile robot deployed in the environment, the computing devices including: receiving from the mobile robot an image captured by the camera, the image capturing one or more particular components within the environment; processing the image using a trained machine learning (ML) model to generate a reduced dimensional output indicating, for each of the N regions of the image, a corresponding probability that an anomaly is present within each of the N regions of the image; determining a corresponding probability quantity that each satisfies a threshold based on the reduced dimensionality outputs; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit, displaying an alarm indicating that an anomaly exists for said one or more particular components; A system that performs the above.
17. The one or more computing devices further include:
17. The system of claim 16, further comprising: selecting the trained ML model from a plurality of candidate ML models to process the image based on determining that the image is captured by the camera at a given pose and based on the trained ML model being trained based on images corresponding to the given pose.
18. The system of claim 16 , wherein N is greater than 9.
19. 17. The system of claim 16, wherein the lower limit is less than or equal to 20% of N and the upper limit is greater than or equal to 50% of N.
20. A mobile robot, Camera and a memory for storing instructions and for storing a trained machine learning (ML) model; and one or more processors, the one or more processors executing the instructions to receiving an image from the camera capturing one or more particular components of an environment in which the mobile robot is deployed; processing the image using the trained ML model to locally generate a reduced-dimensional output indicating for each of the N regions of the image a corresponding probability indicating whether an anomaly is present within the respective region; determining the corresponding probability quantities, each of which satisfies a threshold, based on the reduced dimensionality outputs; determining whether the quantity is greater than a lower limit and less than an upper limit; in response to determining that the quantity is greater than the lower limit and less than the upper limit, communicating with one or more additional computing devices to cause an alert to be displayed indicating that an anomaly exists for the one or more particular components; The mobile robot is operable to:
Citation Information
Patent Citations
Plant-monitoring method
JP2001117633A
Estimating device, learning device, estimating method, and program
JP2021165984A
Abnormality determination device, abnormality determination method, and abnormality determination program
JP2022128651A
