Method for training and / or verifying anomaly detection model

Abstract

The invention relates to a method for training and / or verifying an anomaly detection model. Embodiments are described herein for monitoring and detecting anomalies of industrial facility components based on ML image processing. In various implementations, a plurality of text strings are determined that each describe an exception within an industrial facility setting, and for each of the plurality of text strings, a plurality of iterative processes are performed on the text strings using a text-to-image model to generate a corresponding composite image at each iteration. The generated composite image can be used to train an anomaly detection machine learning (ML) model, or to trim or validate a trained anomaly detection ML model.

Description

Technical Field

[0001] The present invention relates to a method for training and / or validating anomaly detection models, in particular to generating synthetic images using a text-to-image model for training and / or validating anomaly detection models. Background Art

[0002] Complex industrial facilities, such as petrochemical refineries, chemical plants, etc., can include many components for the processing of liquids and / or other substances involved in the industrial process of the industrial facility. To ensure that the components involved in the industrial process operate as expected and / or to ensure that the substances involved in the industrial process are in the expected state, it is important to monitor anomalies in the industrial facility (e.g., the presence of oil or other liquids on the ground, corrosion of pipelines, etc.) so that one or more remedial actions (e.g., alarms, stopping automated processes, etc.) can be executed in a timely manner when an anomaly is detected. Summary of the invention

[0003] Various techniques have been proposed for processing images using machine learning (ML) models to generate outputs indicating whether a condition exists in the image. For example, techniques have been proposed for processing images capturing one or more components of an industrial facility using an anomaly detection ML model (sometimes referred to herein as an "anomaly detection model") to generate outputs indicating whether an anomaly exists in the one or more components captured by the image. However, in order to be effective and have satisfactory predictive accuracy, the anomaly detection ML model must be trained based on a large number of different "positive" ground-truth images, each of which reflects the presence of a corresponding anomaly.

[0004] In addition, text-to-image models have been released that can provide text prompts, and generate detailed and realistic synthetic images conditioned on the text prompts. For example, a text prompt of "an orange catriding a donkey" can be processed to generate a realistic synthetic image including an orange cat riding a donkey. Some of these models also implement image-to-image conversion conditioned on text prompts. For example, a base image including a black cat and a prompt of "make the cat orange" can be provided, and the generated converted synthetic image includes an orange cat instead of a black cat, but generally conforms to the base image. Non-limiting examples of text-to-image and image-to-image models (many models are able to do both operations at the same time) include "Stable Diffusion" and "DALL-E".

[0005] As mentioned above, to be effective, anomaly detection ML models must be trained on a large number of different "positive" ground-truth images, each reflecting the presence of a corresponding anomaly. Obtaining such a large number of different real-life images in an industrial facility setting can be difficult because the corresponding anomalies rarely occur and / or may be dangerous.

[0006] Furthermore, prior to using such anomaly detection ML models in a particular industrial facility, there is currently no method to (a) fine-tune the model to the particular industrial facility and / or (b) validate the model's performance in detecting anomalies within the industrial facility. For example, with respect to (a), it is not possible to fine-tune (i.e., limited additional training) an anomaly detection ML model for a particular industrial facility because it is difficult to obtain "positive" ground truth real images that capture anomalies and are specific to the industrial facility. For example, it may be unsafe or undesirable to create such anomalies in an industrial facility. Furthermore, for example, the industrial facility may have been built but not yet put into operation. As another example, with respect to (b), prior to using the anomaly detection ML model in a particular facility, it is similarly impossible to validate whether the anomaly detection ML model is effective for the particular industrial facility. This may be due to the fact that such validation requires determining that the ML model is able to detect the occurrence of anomalies in the particular facility. Similarly, it may be unsafe or undesirable to actually create anomalies in the particular facility.

[0007] Embodiments disclosed herein are capable of generating a large number of synthetically diverse "positive" ground-truth images, each of which reflects the presence of a corresponding anomaly within an industrial facility. These embodiments use a text-to-image model to generate synthetic images. More specifically, they use prompts that describe the anomaly and that describe the industrial facility setting to prompt the text-to-image model, thereby generating images that include anomalies and are in an industrial facility setting. For example, a prompt that "there is an oil pit outside the chemical plant" and / or a prompt that "there is oil on the floor inside the chemical plant." It is worth noting that these embodiments are capable of generating multiple different synthetic images by processing the same prompt multiple times. Although the same prompt is used in each iteration, different synthetic images will be generated because the model randomly uses different "seeds" in each iteration.

[0008] Embodiments additionally or alternatively disclosed herein can generate synthetically diverse "positive" ground truth images, each of which reflects the presence of a corresponding anomaly within an industrial facility, and each of which is specific to a specific industrial facility. This enables these specific images to be used to (a) fine-tune the ML model before using it for a specific industrial facility (which can improve accuracy and / or robustness) and / or (b) validate the ML model before using it for a specific industrial facility (to ensure that it is effective for the specific industrial facility).

[0009] These embodiments can capture real images of a particular industrial facility and then process these real images using an image-to-image conversion model conditioned on a text prompt describing an anomaly. For example, processing a real image and a text prompt of "add oil pit" can generate a synthetic image that substantially conforms to the real image, but includes the oil pit on the ground. As a specific example, hundreds or thousands of real images of a particular industrial facility can be captured (e.g., by a robot) without the industrial facility having an anomaly. These real images can then be processed using an image-to-image model conditioned on a text prompt describing the anomaly to generate corresponding synthetic images that include the anomaly. These corresponding synthetic images can then be used for fine-tuning and / or verification as described herein.

[0010] Embodiments use a text-to-image model to generate a large number of synthetically diverse "positive" ground-truth images, each of which reflects the presence of a corresponding anomaly within an industrial facility. More specifically, they prompt the text-to-image model with prompts describing the anomaly and being in an industrial facility setting, resulting in images that include anomalies and are in an industrial facility setting. For example, a prompt of "a chemical plant room has pipes that include corrosion." Some of the embodiments are also capable of generating additional synthetic "positive" ground-truth images using real images without anomalies from an industrial facility and an image-to-image conversion model. For example, a real image without anomalies and a prompt of "add some corrosion" can be processed to generate a synthetic image that conforms to the real image, but includes some corrosion on objects in the image.

[0011] The generated synthetic ground-truth images can then be selectively labeled (e.g., by a human reviewer). For example, each label can indicate whether an anomaly is present in the image, and optionally the region in which the anomaly occurs and / or the type of anomaly. The synthetic ground-truth images and their labels can then be used to train an anomaly detection ML model. The anomaly detection ML model can then be used for anomaly detection in industrial facilities.

[0012] Some embodiments additionally or alternatively generate transformed synthetic diverse "positive" ground truth images, each image reflecting the presence of a corresponding anomaly within the industrial facility, and each image being specific to a specific industrial facility. This enables such specific images to be used to (a) fine-tune the anomaly detection ML model before using it for a specific industrial facility (which can improve accuracy and / or robustness) and / or (b) validate the anomaly detection ML model before using it for a specific industrial facility (to ensure that it will be effective for the specific industrial facility).

[0013] These embodiments can capture real images of a particular industrial facility and then process these real images using an image-to-image transformation model conditioned on a text prompt describing an anomaly. For example, processing a real image and a text prompt of "add corrosion to a component" can generate a synthetic image that substantially conforms to the real image, but includes corrosion on the components of the image. As a specific example, hundreds or thousands of real images of a particular industrial facility can be captured (e.g., by a robot) without the industrial facility having anomalies. These real images can then be processed using an image-to-image model conditioned on a text prompt describing anomalies to generate corresponding transformed synthetic images that include the anomalies. These corresponding transformed synthetic images can then be used for fine-tuning and / or verification.

[0014] For example, when validating an anomaly detection ML model for a particular industrial facility, the transformed synthetic images can be processed separately using the anomaly detection ML model to generate corresponding outputs indicating whether anomalies are present. These corresponding outputs can then be compared with the true value outputs (which indicate whether anomalies are actually present in the corresponding transformed synthetic images) to generate an accuracy metric for the output (e.g., if 900 out of 1000 predictions are correct, then 90%). Optionally, if the accuracy metric and / or other validation metrics meet a threshold, the anomaly detection ML model will only be deployed for use within the particular industrial facility.

[0015] In various embodiments, a method implemented by a processor is provided, the method comprising, for each of a plurality of text strings describing an anomaly and a corresponding industrial facility setting, respectively: performing multiple iterations of processing the text string using a text-to-image model to generate a corresponding composite image at each iteration (e.g., conditioned on the text string).

[0016] An industrial facility setting can include an industrial automation facility that implements any number of at least partially automated processes. For example, an industrial automation facility can take the form of a chemical processing plant, an oil or gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable industrial environment.

[0017] A plurality of text strings can each describe an anomaly present in one of one or more components within an industrial facility setting. For example, different text strings can describe different anomalies of the same component within or around an industrial facility setting. As another example, different text strings can describe different anomalies of different components within or around an industrial facility setting. As yet another example, different text strings can describe different anomalies of different components within different industrial facility settings. The composite image can be a detailed and realistic image capturing the presence of a corresponding anomaly in one or more components within or around an industrial facility setting.

[0018] In various embodiments, the method can further include training an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images. For example, for a synthetic image M generated in a first iteration using a first text string (which describes or indicates a first anomaly associated with a component within or around a corresponding industrial facility setting), the method can further include training an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images. 1-1 , can generate label L 1-1 . Label L 1-1 The label can be automatically generated based on the first text string, or can be manually generated (or reviewed) by a trained human technician or expert to identify an object within an industrial facility setting. 1-1 can be used as training instance input and processed using an anomaly detection ML model to generate a corresponding output, which, for example, indicates a corresponding anomaly A 1-1 The type, name and / or location of (which can be highlighted using a bounding box). The corresponding output can then be associated with the label L 1-1 (It is considered to describe anomaly A 1-1 The actual value type, name and / or location of the synthetic image M generated in the second iteration is compared to determine the difference, and one or more parameters / weights of the anomaly detection ML model can be adjusted based on the difference. 1-2 , can generate label L 1-2 . Synthetic image M 1-2 can be used as training instance input and processed using an anomaly detection ML model to generate a corresponding output, which, for example, indicates a corresponding anomaly A 1-2 The type, name and / or location of (which can be highlighted using a bounding box). The corresponding output can then be associated with the label L 1-2 The comparison is performed to determine additional differences, and one or more parameters of the anomaly detection ML model can be adjusted based on the additional differences.

[0019] Alternatively or additionally, for the composite image M generated in the first iteration using a second text string different from the first text string 2-1 , can generate label L 2-1 . Synthetic image M 2-1 can be used as training instance input and processed using an anomaly detection ML model to generate a corresponding output, which, for example, indicates a corresponding anomaly A 2-1 The type, name and / or location of (which can be highlighted using a bounding box). The corresponding output can then be associated with the label L 2-1 The comparison is performed to determine further differences, and one or more parameters of the anomaly detection ML model can be adjusted based on the further differences.

[0020] In some embodiments, the second text string and the first text string can describe different types of anomalies. In some embodiments, the second text string and the first text string can describe the same type of anomaly but for different components within the same or different industrial facilities. The anomaly detection ML model can be trained using different synthetic images and corresponding supervised labels until the difference between the model output and the corresponding label is minimized or a difference threshold is met.

[0021] In various implementations, the method can further include providing a trained anomaly detection ML model for anomaly detection within a particular industrial facility.

[0022] In various embodiments, the method can further include, prior to providing the trained anomaly detection ML model for anomaly detection within a specific industrial facility, and for each of a plurality of real images of the specific industrial facility: processing the real image and the prompt describing the anomaly using an image-to-image translation model to generate a corresponding translated synthetic image; and fine-tuning the trained anomaly detection ML model by further training the anomaly detection ML model based on the translated synthetic image.

[0023] In some embodiments, multiple real images can be captured using one or more visual sensors carried by a mobile robot. The mobile robot can be a quadruped robot (e.g., a robot dog), a wheeled robot, a drone, or any other applicable robot that can move within an industrial automation facility. The one or more visual sensors can be a monochrome graphic RGB camera, a stereo camera, a thermal imager, or any other applicable visual sensor. Accordingly, the real images can be RGB images, RGB-D images, thermal images, or any other applicable images.

[0024] In some embodiments, the method can further include using the trained anomaly detection ML model when processing real images, wherein using the trained anomaly detection ML model when processing real images includes: using the trained anomaly detection ML model to process real images of a plurality of real images to generate a model output indicating whether there are any anomalies in one or more components; and causing one or more remedial actions to be performed in response to the model output indicating the presence of anomalies in one or more components.

[0025] In various embodiments, the method can further include, prior to providing the trained anomaly detection ML model for anomaly detection within the specific industrial facility, and for each real image in a plurality of real images of the specific industrial facility: processing the real image and the prompt describing the anomaly using the image-to-image conversion model to generate a corresponding converted synthetic image; and validating the trained anomaly detection ML model based on the converted synthetic image. Providing the trained anomaly detection ML model for anomaly detection within the specific industrial facility is in response to determining that the validation satisfies one or more conditions.

[0026] In various embodiments, validating a trained anomaly detection ML model based on a transformed synthetic image is achieved by determining an accuracy metric for anomaly predictions made based on output from the anomaly detection ML model based on processing the transformed synthetic image. In these embodiments, determining that the validation satisfies one or more conditions can include determining that the accuracy metric satisfies a threshold accuracy metric.

[0027] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to perform any of the aforementioned methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0028] In some embodiments, a system is provided, the system being equipped with one or more processors and a memory storing instructions, the instructions, in response to execution of the one or more processors, causing the one or more processors to: for each of a plurality of text strings describing anomalies and corresponding industrial facility settings, respectively: perform multiple iterations of processing the text string using a text-to-image model to generate a corresponding synthetic image at each iteration; train an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images; and provide the trained anomaly detection ML model for anomaly detection within a specific industrial facility.

[0029] In various embodiments, the system can further include the following instructions: before providing the trained anomaly detection ML model for anomaly detection within a specific industrial facility and for each real image of a plurality of real images of the specific industrial facility: processing the real image and the prompt describing the anomaly using an image-to-image conversion model to generate a corresponding converted synthetic image; and fine-tuning the trained anomaly detection ML model by further training the anomaly detection ML model based on the converted synthetic image.

[0030] In various embodiments, the system can further include the following instructions: before providing the trained anomaly detection ML model for anomaly detection within a specific industrial facility and for each real image in a plurality of real images of the specific industrial facility: using an image-to-image conversion model to process the real image and the prompt describing the anomaly to generate a corresponding converted synthetic image; and verifying the trained anomaly detection ML model based on the converted synthetic image. In these embodiments, providing the trained anomaly detection ML model for anomaly detection within the specific industrial facility is in response to determining that the verification satisfies one or more conditions. In some embodiments, verifying the trained anomaly detection ML model based on the converted synthetic image includes: determining an accuracy metric for anomaly predictions made based on output from the anomaly detection ML model based on processing the converted synthetic image. For example, one or more conditions can include a threshold accuracy metric.

[0031] In some embodiments, providing a trained anomaly detection ML model for anomaly detection within a specific industrial facility includes: causing the trained anomaly detection ML model to process real images captured via a visual sensor within or outside the specific industrial facility. In some embodiments, the visual sensor can be carried by a mobile robot that moves around the specific industrial facility. In some of these embodiments, causing the trained anomaly detection ML model to process real images can include: causing the trained anomaly detection ML model to be locally downloaded to a mobile robot that moves around the specific industrial facility for monitoring and detecting anomalies. By setting the trained anomaly detection ML model locally at the mobile robot, the trained anomaly detection ML model can be used to process images captured using the visual sensor of the mobile robot for timely anomaly detection and / or notification.

[0032] In some embodiments, causing the trained anomaly detection ML model to be used to process real images includes causing the trained anomaly detection ML model to be downloaded to one or more on-site computers (e.g., servers) located at the industrial facility. In some of these embodiments, images captured by the visual sensor are provided to one or more on-site computers within a particular industrial facility and processed using the trained anomaly detection ML model downloaded to the servers to perform anomaly detection.

[0033] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1AAn example environment is schematically depicted in which selected aspects of the present disclosure can be implemented according to various embodiments.

[0035] Figure 1B Depicted are example flow charts for performing selected aspects of the present disclosure, according to various embodiments.

[0036] Figure 1C Depicted are examples of machine learning (ML) models for performing selected aspects of the present disclosure, according to various embodiments.

[0037] Figure 2A Examples of images generated using the techniques described herein are schematically depicted, according to various embodiments.

[0038] Figure 2B An example of another image generated using the techniques described herein is schematically depicted, in accordance with various embodiments.

[0039] Figure 2C Examples of further images generated using the techniques described herein are schematically depicted according to various embodiments.

[0040] Figure 2D Examples of additional images generated using the techniques described herein are schematically depicted, according to various embodiments.

[0041] Figure 3 Example methods for performing selected aspects of the present disclosure are shown.

[0042] Figure 4 An example computer architecture is schematically shown upon which selected aspects of the present disclosure can be implemented. DETAILED DESCRIPTION

[0043] Embodiments of the present disclosure utilize one or more image generation models to generate a large number of images for training, fine-tuning, and / or validating anomaly detection ML models for detecting one or more anomalies in components and / or materials within an industrial environment (e.g., an industrial automation facility). The one or more image generation models can include, for example, a text-to-image model that can be used to process text cues and generate realistic synthetic images conditioned on the text cues.

[0044] For example, the text-to-image model can be used to process a text prompt (sometimes referred to herein as a "prompt" or "text string") that indicates or describes an anomaly and generate a detailed and realistic synthetic image conditioned on the text prompt. In some embodiments, the same text prompt can be processed multiple times / iteratively using the text-to-image model to generate multiple different synthetic images, each of which reflects the anomaly.

[0045] In some embodiments, different textual prompts can describe different anomalies within or around a particular industrial automation facility or a particular type of industrial automation facility (sometimes referred to herein as an "industrial facility"), and accordingly, different synthetic images generated using a text-to-image model can capture different anomalies within or around a particular (or particular type of) industrial automation facility. In some embodiments, different textual prompts can describe different anomalies within or around different industrial facilities, and accordingly, different synthetic images generated using a text-to-image model can capture different anomalies within or around different industrial facilities. Different synthetic images generated for multiple iterations based on the same textual prompt and / or different synthetic images generated based on different textual prompts can be applied to training, fine-tuning, and / or validating anomaly detection ML models.

[0046] In various embodiments, one or more image generation models can additionally or alternatively include an image-to-image model that can be used to process a base image and a text prompt to generate a transformed / modified composite image that generally conforms to the base image, but includes modifications consistent with the text prompt. For example, an image capturing one or more components of an industrial facility (e.g., a real-world image that does not reflect an anomaly, or a realistic composite image) and a text prompt describing / indicating an anomaly present in one or more components can be provided to the image-to-image model. In this case, the image and text prompt are processed using the image-to-image model to generate a modified image that reflects the anomaly described or indicated in the text prompt, but generally conforms to the image. For example, if the image captures a floor without oil, two tanks, and a pipe, and the text prompt is "add oil on the ground", the modified image can also include the floor, two tanks, and a pipe, but will include oil on the ground. Accordingly, after processing the base image and the text prompt, a modified image can be generated based on the model output of the image-to-image model, wherein the modified image is a modified version of the real-world picture that reflects the anomaly indicated in the text prompt.

[0047] In some embodiments, using the same image capturing the same component of the industrial facility and different textual prompts describing different anomalies of the component, an image-to-image model can be used to generate various modified images. Each of the various modified images can reflect a corresponding anomaly described or indicated by a corresponding one of the different textual prompts, and the various modified images (or portions thereof) can be applied as diverse "positive" ground truth images to train, fine-tune, and / or validate anomaly detection ML models in detecting anomalies of certain specific components within the industrial facility.

[0048] In some embodiments, as a working example, one or more real visual sensors can be used to acquire different images that capture different components of an industrial facility (or more than one industrial facility) in its normal state (i.e., without abnormalities). For example, one or more visual sensors can be attached to a robot located within the industrial facility, or can be located at different locations within the industrial facility. The one or more visual sensors can be a monochrome graphic RGB camera, a stereo camera, a thermal imager, and / or any other applicable visual sensor. For example, the robot can be a mobile robot, such as a robot dog that walks around and through the industrial facility. It is worth noting that images captured using real visual sensors or other real sensor devices are generally referred to as "real images", "real life images", or "real world images", while images generated using the above-mentioned image generation models (e.g., text-to-image models or image-to-image models) are generally referred to as "synthetic images".

[0049] Continuing with the above working example, for example, different text prompts describing different anomalies can be received from user input or from a collected anomaly text description file. Each different image and corresponding text prompt can be processed using an image-to-image model so that the image-to-image model outputs a modified image reflecting the anomaly described in the corresponding text prompt. In this way, different modified images can be generated to describe different anomalies of different components and / or different industrial facilities.

[0050] Different modified images (or portions thereof) can be applied as diverse "positive" true value images for use in training, fine-tuning and / or validating anomaly detection ML models in detecting anomalies of different components within the same or different industrial facilities. Additionally or alternatively, one or more real-world images can be selected from different images acquired using one or more visual sensors, and a "no anomaly" supervision label (by capturing components in a normal state) can be assigned to each image. These one or more real-world images can be applied as "negative" true value images for training, fine-tuning and / or validating anomaly detection ML models. It is worth noting that the term "positive" used here indicates the presence of an anomaly, while the term "negative" used indicates the absence of an anomaly. It should also be noted that when different modified images (or portions thereof) are used to train, fine-tune and / or validate the same ML model (e.g., anomaly detection ML model), the different modified images can be optionally divided into different sets (e.g., a first set for training, a second set for fine-tuning, and / or a third set for validation), wherein the modified image is not included in more than one different set.

[0051] In some embodiments, in addition to or in lieu of using real-world images captured by sensors, an image-to-image model can be used to process as input a realistic synthetic image capturing one or more components (within an industrial facility) showing no anomalies, and a textual prompt describing anomalies that are foreseeable in the industrial facility, to generate a modified image for training, fine-tuning, and / or validation purposes. For example, a realistic synthetic image can be generated based on a text string that simply describes an industrial facility (e.g., a "chemical plant") using an ML model that performs text-to-image conversion (e.g., the text-to-image model described above).

[0052] By utilizing one or more image generation models to generate synthetic images that reflect anomalies within one or more industrial facilities (or modifying real-life images that capture one or more industrial facilities and do not have anomalies to reflect anomalies, or modifying realistic synthetic images that do not have anomalies), a large number of diverse synthetic images can be generated, each of which reflects a corresponding anomaly within one or more industrial facilities, without the need to manually create (and subsequently clean up and eliminate) dangerous anomalies. This can also save labor and time in capturing or waiting to capture images when certain types of anomalies occur (e.g., anomalies that are not easy to find). In addition, by using diverse and realistic synthetic images to train anomaly detection ML models, the accuracy of the anomaly detection ML models in detecting anomalies can be improved.

[0053] Additionally or alternatively, a large number of synthetic images (or a portion thereof) can also be applied to fine-tune the anomaly detection ML model, so that the fine-tuned anomaly detection ML model can be applied to detect different anomalies of a specific component or a specific industrial facility. Additionally or alternatively, a portion of the large number of synthetic images can also be applied to verify the anomaly detection ML model before deployment, so that the quality of the anomaly detection ML module in detecting anomalies can be ensured.

[0054] In some embodiments, given a total of N (where N>1) different synthetic images generated using a text-to-image model and / or an image-to-image model, a first set of different synthetic images can be applied to initially train an anomaly detection ML model in order to train the first anomaly detection ML model for anomaly detection within or around an industrial facility. Further, for example, a second set of different synthetic images can be additionally or alternatively applied to fine-tune the trained anomaly detection ML model to enhance anomaly detection for a specific type of anomaly, a specific industrial setting, and / or a specific type of industrial setting. Further, for example, a third set of different synthetic images can be additionally or alternatively applied to validate the trained, optionally fine-tuned anomaly detection machine learning model before deploying the trained anomaly detection ML model for detecting anomalies within a specific industrial setting.

[0055] As described herein, in various embodiments, the second set and / or the third set of synthetic images can include (e.g., be limited to) modified synthetic images generated using an image-to-image conversion model and based on corresponding real images of a particular industrial setting. For example, the second set of synthetic images can include those synthetic images generated based on processing corresponding real images of a particular industrial setting and corresponding text prompts using an image-to-image conversion model. In these and other ways, fine-tuning and / or validation can be specific to a particular industrial setting. Fine-tuning an anomaly detection ML model based on such images can improve the accuracy and / or robustness of the anomaly detection ML model for a particular industrial setting. Validating the anomaly detection ML model based on these images can ensure that it is effective for a particular industrial setting.

[0056] Accordingly, embodiments described herein relate to training, fine-tuning, and / or validating anomaly detection machine learning (ML) models to monitor and detect anomalies of one or more components (e.g., liquid tanks) within an industrial facility (e.g., an industrial automation facility) based on textual prompts and / or based on images (realistic synthetic images and / or real-world images captured using one or more visual sensors (such as cameras)) capturing the one or more components (or portions thereof).

[0057] In various embodiments, a plurality of synthetic images that respectively reflect anomalies present in one or more components can be used to train, fine-tune, and / or validate an anomaly detection ML model. For example, the plurality of synthetic images can include a first set of synthetic images generated using a text-to-image ML model (sometimes referred to as a "text-to-image model") based on one or more text strings that respectively describe anomalies and industrial facility settings in which the anomalies exist. Alternatively or additionally, the plurality of synthetic images can include a second set of synthetic images generated using an image-to-image ML model (sometimes referred to as an "image-to-image model") based on real-world images in which each image is conditioned on a text string (e.g., an anomaly is introduced / added in a corresponding image of the real-world image).

[0058] It is worth noting that in some embodiments, the image generation model can be used as both a text-to-image model and an image-to-image model. In other words, the image generation model can perform text-to-image conversion and image-to-image conversion. For example, one or more visual sensors carried by a mobile robot (e.g., a robot dog) can be used to capture real-world images. By using a wide range of (or selected) synthetic images (e.g., realistic synthetic images) generated by a text-to-image model and / or an image-to-image model and reflecting the corresponding anomalies in the industrial facility setting, respectively, to train, fine-tune or verify the anomaly detection ML model, there is no need to wait and discover the occurrence of anomalies (e.g., corrosion of pipelines or other components) that take some time. There is no need to force dangerous anomalies to occur, such as oil leaks or other liquids on the ground of an industrial facility setting. In addition, it is also possible to save or reduce the cost and labor associated with capturing appropriate images of the occurrence of anomalies.

[0059] Now turning to the attached figure, Figure 1A An example environment is schematically depicted in which selected aspects of the present disclosure can be implemented according to various embodiments. Figure 1B Depicted are example flow charts for performing selected aspects of the present disclosure, according to various embodiments. Figure 1C Depicted are examples of machine learning (ML) models for performing selected aspects of the present disclosure according to various embodiments. Figure 1A , schematically depicts an example environment 100 in which various aspects of the present disclosure can be implemented. The example environment 100 can be or can include an industrial automation facility, which can take a variety of forms. For example, the example environment 100 can be designed to implement any number of at least partially automated processes. The industrial automation facility can take the form of a chemical processing plant, an oil or gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, and the like.

[0060] The example environment 100 can include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled to a process automation network 106 in an industrial automation facility. The client device 103-A or 103-B can be implemented as a computer (e.g., a laptop, a desktop computer, a notebook computer), a tablet computer, a robot, a smart appliance (e.g., a smartphone), a messaging device, a wearable device (e.g., a watch), or any other applicable device. The process automation network 106 can be implemented using various wired and / or wireless communication technologies, including but not limited to the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), cellular networks such as 3GPP Long Term Evolution ("LTE") or other wireless protocols designated as 3G, 4G, 5G and above, and / or other types of communication networks of various topologies (e.g., mesh).

[0061] The example environment 100 can further include a mobile robot 101 having or carrying a vision component 1011. The mobile robot 101 can be a quadruped robot (e.g., a robot dog), a wheeled robot, a drone, or any other applicable robot that can move within an industrial facility. The vision component 1011 can be a monochrome graphic camera, a stereo camera, a thermal imager, or any other applicable visual sensor to capture one or more images of one or more specific components of an industrial automation facility (e.g., a liquid tank T or a pipe 102 that stores or transports liquid substances). The vision component 1011 can be detachably coupled to or integrated into the mobile robot 101. In some embodiments, the vision component 1011 can change the position and / or orientation relative to the mobile robot 101, for example, by rotating or otherwise moving. In addition to the vision component 1011, the mobile robot 101 can also include one or more additional vision components for navigating in an industrial facility, sensing static or dynamic objects, and / or capturing images.

[0062] The example environment 100 can further include a server computing device 105 (can be referred to simply as a “server device”). The server computing device 105 can include a machine learning (ML) engine 1051, an anomaly detection engine 1052, and an image generation engine 1057. The server computing device 105 can further include or otherwise access one or more machine learning (ML) models 1053. The one or more ML models 1053 can include an anomaly detection ML model (e.g., Figure 1B Alternatively or additionally, the one or more ML models 1053 can include an image generation model that performs text-to-image conversion and / or image-to-image conversion (e.g., Figure 1BIn some implementations, the one or more ML models 1053 can include a first model for anomaly detection at a first site and a second model for anomaly detection at a second site, where the first site is different from the second site.

[0063] The server computing device 105 can be connected to multiple client devices. The server computing device 105 can communicate with one or more local client devices (e.g., 103-A and 103-B), and / or communicate with one or more remote client devices (not shown). The local client device 103-A or 103-B can be connected to the server computing device 105 via one or more local area networks (e.g., process automation network 106), and the remote client device can be connected to the server computing device 105 via one or more wide area networks (e.g., the Internet). The local client devices and the remote client devices can be operated by personnel such as system integrators to configure and / or interact with various aspects of the example environment 100.

[0064] In some embodiments, in addition to the ML engine 1051 and the anomaly detection engine 1052, the server computing device 105 may also include a database (not shown) that stores information used by the ML engine 1051 and / or the anomaly detection engine 1052 and / or the image generation model 1057 to practice selected aspects of the present disclosure. In some embodiments, in addition to the ML engine 1051 and the anomaly detection engine 105, the server computing device 105 may also include an image pre-processing engine 1055 that processes different images (e.g., images captured using different visual sensors) to have the same image dimensions, for example. Various aspects of the server computing device 105, such as the ML engine 1051, the anomaly detection engine 1052, the image generation engine 1057, and / or the image pre-processing engine 1055, can be implemented using any combination of hardware and software.

[0065] In some embodiments, the ML engine 1051, anomaly detection engine 1052, image pre-processing engine 1055, or trained ML model 1053 can be implemented across multiple computer systems as part of what is often referred to as a "cloud infrastructure" or simply "the cloud." However, this is not required; Figure 1A In the embodiment, the ML engine 1051 is implemented within an industrial facility, for example, in a single building or across a single building campus or other industrial infrastructure. In such an embodiment, the ML engine 1051 can be implemented on one or more local computing systems, such as on one or more server computers.

[0066] In some embodiments, the mobile robot 101 can navigate in an industrial facility and capture images at one or more locations (e.g., designated locations). Figure 1A As shown, the visual component 1011 of the mobile robot 101 can (but does not necessarily need to) be configured to be in a given posture to capture an image of the liquid tube 102 in a given posture. In some embodiments, the image captured by the visual component 1011 can be processed as an input using an anomaly detection model (after training and / or validation) to generate a model output indicating whether there is an anomaly in the component captured in the image.

[0067] For example, the model output can include a classification result that predicts the probability of each of a predetermined number of anomalies existing within the image. As a non-limiting example, for example, the classification result can be: (oil leak, 0.7), (no oil leak, 0.3). In this non-limiting example, it can be determined that an anomaly corresponding to "oil leak" is detected from the image with a confidence score of approximately "0.7" (which satisfies a predefined confidence score threshold, such as 0.6). In response to detecting the anomaly "oil leak", one or more mitigation / remediation measures can be performed, including but not limited to: generating a warning message (e.g., Figure 1A 107, presented via the client device 103-A) or emits a warning sound to notify the responsible personnel of the oil spill.

[0068] As another non-limiting example, for example, the classification results can be: (oil leak, 0.1); (corrosion, 0.8); (fire, 0.1). In this non-limiting example, it can be determined that an anomaly corresponding to "corrosion" is detected from the image, and its confidence score is about "0.8" (which meets a predefined confidence score threshold, such as 0.6). In response to the detected anomaly "oil leak", one or more mitigation / remediation measures can be performed, including but not limited to: issuing a warning message or emitting a warning sound to notify the responsible personnel of the corrosion situation.

[0069] In some embodiments, an image captured by the vision component 1011 can be processed as an input along with a text prompt (or text string) that introduces an anomaly into the image using an image-to-image model to generate a modified image that reflects the anomaly described or indicated in the text prompt. As a non-limiting example, the image captured by the vision component 1011 can be a real-world image capturing one or more oil tanks within a storage room of an industrial facility. This real-world image and the text prompt (e.g., "oil leak added on the ground") can be provided to the above-mentioned image generation model to perform an image-to-image transformation conditioned on the text prompt, wherein the output of the image generation model (e.g., the image-to-image model) corresponds to a modified image (which is a realistic synthetic image) that modifies the real-world image to show an oil leak on the ground of the storage room where the one or more oil tanks are located.

[0070] In the above non-limiting example, a supervisory label can be generated based on the modified image and / or based on a textual prompt (e.g., "oil leak added on the ground"). A human operator trained in identifying anomalies in an industrial facility setting can scan the modified image to determine the supervisory label of the anomaly (e.g., "oil leak"). Alternatively, the supervisory label (e.g., "oil leak") can be automatically extracted from the textual prompt (e.g., "oil leak added on the ground") and / or verified by a human operator by inspecting the modified image. The modified image and supervisory label can then be applied to generate training instances (here, there are "positive" training instances that are anomalies), or instances for fine-tuning or verification.

[0071] For example, a modified image (e.g., a realistic synthetic image showing an oil leak on the floor of a storage room storing one or more oil tanks) can be applied and saved as a training instance input (a "positive" ground-truth image) for a training instance, and a supervisory label can be applied and saved as a ground-truth label for the training instance. The training instance can be applied to train an anomaly detection ML model, where the modified image can be processed using the anomaly detection ML model to generate a model output (e.g., a confidence score indicating whether an anomaly of "oil leak" was detected within the modified picture and / or a confidence score that an anomaly of "oil leak" was detected within the modified image). The model output can be compared to the ground-truth label of "oil leak", based on which one or more weights / parameters of the anomaly detection ML model can be fitted (or fine-tuned in the case where the modified image and ground-truth label are saved for fine-tuning the model).

[0072] Additionally or alternatively, the real-world images of one or more oil tanks captured in a storage room of an industrial facility can be assigned a supervisory label of "no anomaly". The real-world images of one or more oil tanks captured in a storage room of an industrial facility and the supervisory label of "no anomaly" can be saved as "negative" training instances to train an anomaly detection ML model. Additionally or alternatively, the real-world images of one or more oil tanks captured in a storage room of an industrial facility and the supervisory label of "no anomaly" can be applied to generate instances for fine-tuning and / or validating an additional anomaly detection ML model that is different from the anomaly detection ML model.

[0073] Optionally, more than one real-world image can be captured using the vision component 1011 (and / or using additional sensors). As a non-limiting example, a total of 1000 real-world images can be captured using the vision component 1011. The total of 1000 real-world images (or a portion thereof) can be conditioned on a first text prompt describing a first anomaly within a first class of facilities, and processed 100 times using the image generation model (image-to-image model 111B) to generate 100,000 synthetic images reflecting the first anomaly within the first class of facilities. The total of 1000 real-world images (or a portion thereof) can be conditioned on a second text prompt describing a second anomaly within a first class of facilities (or a second class of facilities different from the first class), and processed 50 times using the image generation model (image-to-image model 111B) to generate 50,000 additional synthetic images reflecting the second anomaly within the first (or second) class of facilities. Further synthetic images can be generated similarly, and repeated descriptions are omitted herein.

[0074] Notably, the number of iterations performed for each text prompt can depend on the type of anomaly described or indicated in the corresponding text prompt, and / or can depend on the functionality / usage of the anomaly detection ML model to be trained (or fine-tuned or validated). In some implementations, the number of iterations performed on a particular text prompt can be increased by identifying anomalies described by a particular text prompt of particular interest to train, fine-tune, or validate the anomaly detection ML model.

[0075] Continuing with the above non-limiting example, in some embodiments, 100,000 synthetic images reflecting a first anomaly within a first type of facility can be assigned a first supervised label identifying the first anomaly (e.g., type, name, location, etc.), and 50,000 additional synthetic images reflecting a second anomaly within the first (or second) type of facility can be assigned a second supervised label identifying the second anomaly (e.g., type, name, location, etc.). The 100,000 synthetic images and the 50,000 additional synthetic images can be divided into three groups, one for training, fine-tuning, and validation purposes.

[0076] For example, 100,000 synthetic images and 50,000 additional synthetic images can be divided into: a first training set of synthetic images, including 80,000 of the 100,000 synthetic images and 40,000 of the 50,000 additional synthetic images; a second fine-tuning set of synthetic images, including 15,000 of the 100,000 synthetic images and 8,000 of the 50,000 additional synthetic images; and a third validation set of synthetic images, including 5,000 of the 100,000 synthetic images and 2,000 of the 50,000 additional synthetic images. For example, the first training set of synthetic images can be applied to generate "positive" training examples to train an anomaly detection model, the second fine-tuning set of synthetic images can be applied to generate "positive" examples to fine-tune the same anomaly detection model, and the third validation set of synthetic images can be applied to generate "positive" examples to validate the anomaly detection model.

[0077] Notably, if training and fine-tuning (or validation) are applied to different anomaly detection models, 100,000 synthetic images (or a portion thereof) and 50,000 additional synthetic images (or a portion thereof) can be applied to train a first anomaly detection model, and the same 100,000 synthetic images (or a portion thereof) and the same 50,000 additional synthetic images can be applied to fine-tune a second anomaly detection model that is different or separate from the first anomaly detection model.

[0078] Continuing with the above non-limiting example, the above total number of 1000 real-world images (and / or additional real-world images) can be assigned a supervised label of "no anomaly" respectively. For example, the 1000 real-world images (and / or additional real-world images) can be divided into: a first training group of real-world images, comprising 7,500 real-world images; a second fine-tuning group of real-world images, comprising 2,000 real-world images; and a third validation group of real-world images, comprising 500 real-world images. The first training group can be applied to generate "negative" training examples to train an anomaly detection model, the second fine-tuning group of real-world images can be applied to generate "negative" fine-tuning examples to fine-tune the same anomaly detection model, and the third validation group of real-world images can be applied to generate "negative" validation examples to validate the same anomaly detection model. Notably, if training and fine-tuning (or validation) are applied to different anomaly detection models, images from the 1000 real-world images (and / or additional real-world images) can be applied to training (fine-tuning or validating) the first model, while also being applied to training (fine-tuning or validating) a second model that is different from the first model.

[0079] Reference now Figure 1C, the ML models 1053 stored in the ML model database can include a text-to-image model 111A that performs text-to-image conversion, an image-to-image model 111B that performs image-to-image conversion, an anomaly detection ML model M1 for detecting a first specific type of anomaly, an anomaly detection ML model M2 for detecting a second specific type of anomaly different from the first specific type, and / or an anomaly detection ML model M3 that can detect a predefined number of anomalies (e.g., including first and second specific types of anomalies), and / or other ML models. In some embodiments, in addition to storing the text-to-image model 111A and the image-to-image model 111B that is separate or independent from the text-to-image model 111A, the ML model database can include or otherwise access an image generation model that performs text-to-image conversion and image-to-image conversion. It is noteworthy that the image-to-image conversion can be conditioned on a text prompt that describes a corresponding anomaly that is unique to the industrial facility. The anomaly described in the text prompt can vary depending on the type and location of the facility (or other factors). In addition, different text prompts can describe different anomalies.

[0080] Reference now Figure 1B, a text string 108 (sometimes referred to as a "hint" or "text hint") can be provided to a text-to-image model 111A. For example, the text string 108 can be "Inside a chemical plant with oil on the floor". The text string 108 can be processed multiple times / iteratively using the text-to-image model 111A. For example, at a first iteration, the text string 108 can be processed as a single input using the text-to-image model 111A to generate an image 120_1 of a first chemical plant showing oil on the floor of the first chemical plant. At a second iteration, the text string 108 can be processed as a single input using the text-to-image model 111A to generate an additional image 120_2 of a second chemical plant showing oil on the floor. The second chemical plant can be the same as or different from the first chemical plant. Alternatively, the oil on the floor of the first chemical plant can have a different size, position, or color (or other feature) than the oil on the floor of the second chemical plant. Similarly, at the Nth iteration, the text string 108 can be processed as a single input using the text-to-image model 111A to generate an image 120_N of the Nth chemical plant showing oil on the floor. Optionally, the images 120_1, 120_2, ..., 120_N can be stored in the image database 140. The images 120_1, 120_2, ..., 120_N can be inspected (e.g., by a technician, a human operator, etc.) to determine the corresponding supervisory labels (e.g., "oil on the ground", "oil leaking", "oil leaking on the left", etc.). Alternatively, the supervisory labels (e.g., "oil on the ground") can be determined or extracted from the text string 108 (e.g., "inside a chemical plant with oil on the floor") and / or can be verified by a human operator.

[0081] Once the supervised labels are determined for each of the images 120_1-120_N, a plurality of training instances can be generated. For example, a first training instance can be generated to include the image 120_1 as a first training instance input and include a first supervised label (e.g., "oil spill") as a true value label for use with an anomaly detection ML model (e.g., Figure 1A Similarly, a second training instance can be generated to include image 120_2 as a second training instance input and include a second supervised label (e.g., “oil leak”) as a true value label for use with an anomaly detection ML model (e.g., Figure 1A The Nth training instance can be generated to include image 120_N as the Nth training instance input and include the Nth supervised label (e.g., “oil spill”) as the true value label for use with an anomaly detection ML model (e.g., Figure 1ABased on the comparison, one or more weights / parameters of the anomaly detection ML model M1 can be fitted / modified / updated.

[0082] In some embodiments, in addition to being used to generate training instances, images 120_1~120_N (or a portion thereof) and their corresponding supervisory labels can also be used to generate one or more fine-tuning instances. The generated one or more fine-tuning instances (if they do not contain images that have been used to train the model M1) can be used to fine-tune the model M1, that is, to fine-tune or adjust one or more weights / parameters of one or more layers of the anomaly detection ML model M1. The generated one or more fine-tuning instances (if they contain images that have been used to train the model M1) can be used to fine-tune a trained anomaly detection machine learning model that is different from the anomaly detection ML model M1 (for example, Figure 1B M2 in ). For example, image 120_1 can be selected as a first fine-tuning instance input for a first fine-tuning instance, and a supervised label corresponding to image 120_1 can be selected as a first true value label for the first fine-tuning example. Image 120_1 can be processed using the trained anomaly detection ML model to generate a model output indicating, for example, the name and location of the anomaly. The model output can be compared with the supervised label corresponding to image 120_1 to determine a difference, and based on the difference, one or more weights of the trained anomaly detection ML model are fine-tuned.

[0083] In some embodiments, in addition to or instead of being used to generate training instances, images 120_1-120_N (or a portion thereof) and their corresponding supervisory labels can be applied to generate one or more validation instances (sometimes also referred to as "test instances"). For example, image 120_1 can be selected as a first validation instance input for a first validation instance, and a supervisory label corresponding to image 120_1 can be selected as a first true value label for the first validation instance. The first validation instance can be applied to validate the anomaly detection ML model (e.g., a test instance) by determining whether the difference between the first true value label and the model output of the anomaly detection ML model corresponding to image 120_1 satisfies a difference threshold. Figure 1B If images 120_1~120_N (or a portion thereof) and their corresponding supervised labels can be used to generate test instances, anomaly detection ML models (e.g., Figure 1B The performance of M3 in the model M3 is determined, for example, based on whether the output of the model M3 satisfies one or more conditions (for example, the accuracy metrics described in other aspects of the present disclosure).

[0084] In some embodiments, still refer to Figure 1B, an image 110 (thus image 110 is a real-life image) of an industrial facility component in a normal state (e.g., without anomalies) can be captured using a visual sensor. Image 110 can be provided to an image-to-image model 111B along with a text string 109 (e.g., "Add oil pit") that introduces an anomaly of "oil pit" to the industrial facility. Image 110 and text string 109 (e.g., "Add oil pit") can be processed using image-to-image model 111B as input to generate a modified image (e.g., 130_1, which is realistic and synthetic) that modifies the real-life image 110 to show the oil pit relative to the industrial facility component.

[0085] In some embodiments, the image 110 and the text string 109 (e.g., "add oil pit" or "add oil pit near oil tank") can be processed multiple times / iteratively as input using the image-to-image model 111B, where at a first iteration, a modified image 130_1 is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109, at a second iteration, a modified image 130_2 is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109, and at an Nth iteration, a modified image 130_N is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109. The modified images 130_1, 130_2, ..., 130_N (or a portion thereof) and the supervisory labels (e.g., extracted or determined from the text string 109) can be respectively applied to train, fine-tune, and / or validate one or more anomaly detection ML models.

[0086] Additionally or alternatively, a text string different from the text string 109 (e.g., "add corrosion to the oil tank") can be provided to the image-to-image model 111B along with the image 110. Such a text string and the image 110 can be processed multiple times iteratively using the image-to-image model 111B to generate multiple modified images. Similarly, a supervisory label can be created and verified for each of the multiple modified images, where, for example, such a supervisory label can be determined from the text string (e.g., "add corrosion to the oil tank", or "add corrosion to the left oil tank").

[0087] It should be noted that in some embodiments, the image 110 need not be a real image. For example, the image 110 can be a realistic synthetic image generated by, for example, the text-to-image model 111A based on a text string describing an industrial facility setting. In these embodiments, the text string processed by the text-to-image model 111A does not describe an anomaly, and therefore, the generated realistic synthetic image does not display or reflect an anomaly other than the industrial facility setting.

[0088] In some embodiments, additionally or alternatively, the image database 140 can further store one or more real images 150 captured using a physical sensor, and each image reflects a corresponding anomaly. In some embodiments, additionally or alternatively, the image database 140 can further store one or more real images 160 captured using a physical sensor, and each image does not reflect an anomaly. One or more real images 150 can be stored in the image database 140 in association with corresponding supervisory labels that identify or indicate corresponding anomalies. One or more real images 160 can each be stored in the image database 140 in association with supervisory labels, such as "no anomaly". One or more real images 150 (or selected portions thereof) and corresponding supervisory labels can be applied as "positive" ML instances to train, fine-tune, and / or validate an anomaly detection model such as model M1 (or other anomaly detection ML models such as M2 or M3) for anomaly detection. One or more real images 150 (or selected portions thereof) and corresponding supervisory labels can be applied as “negative” training examples to train, fine-tune and / or validate an anomaly detection model such as model M1 (or other anomaly detection ML models such as M2 or M3) for anomaly detection.

[0089] In some embodiments, one or more images 113A can be selected from the image database 140 to train an anomaly detection ML model (e.g., M1), wherein the one or more images 113A can include images from synthetic images (e.g., images 120_1-120_N) generated using the text-to-image model 111A, images from synthetic images (e.g., 130_1-130_N) generated using the image-to-image model 111B, images from real images 150 that capture anomalies, images from real images 160 that do not capture anomalies, and / or other applicable images. For example, the image database 140 may also include synthetic images showing anomalies, wherein such synthetic images without anomalies are generated using the image-to-image model 111B based on an initial synthetic image (without anomalies), which is generated using the text-to-image model 111A (which receives text input that simply describes a scene or industrial facility). The images included in the image database 140 are not limited to the description herein.

[0090] In some embodiments, one or more images 113B can be selected from the image database 140 to fine-tune the anomaly detection ML model (e.g., M2), wherein the one or more images 113B can include: images from synthetic images generated using the text-to-image model 111A (e.g., images 120_1~120_N), images from synthetic images generated using the image-to-image model 111B (e.g., 130_1~130_N), images from real images 150 that capture anomalies, images from real images that do not capture anomalies, and / or other applicable images (e.g., as described above).

[0091] In some embodiments, one or more images 113C can be selected from the image database 140 to validate the anomaly detection ML model (e.g., M3), wherein the one or more images 113C can include: images from synthetic images generated using the text-to-image model 111A (e.g., images 120_1~120_N), images from synthetic images generated using the image-to-image model 111B (e.g., 130_1~130_N), images from real images 150 that capture anomalies, images from real images that do not capture anomalies, and / or other applicable images (e.g., as described above).

[0092] Figure 2A Examples of images generated using the techniques described herein are schematically depicted, according to various embodiments. Figure 2B An example of another image generated using the techniques described herein is schematically depicted, in accordance with various embodiments. Figure 2C Schematically depicts examples of further images generated using the techniques described herein according to various embodiments. Figure 2A , a mobile robot 101 carrying a visual sensor is used to capture images of an industrial facility 200 over a period of time (e.g., weeks, months, or even longer). From the images captured during the period of time, a collection of images is selected and labeled, each of which captures anomalies within the industrial facility, wherein the collection of labeled images is applied to train anomaly detection ML model 221. An additional collection of images that do not capture any anomalies can be selected from the images captured during the period of time. For example, the additional collection of images can include an image 201 (which is a real-world image) that captures multiple oil tanks (e.g., oil tank A, oil tank B, oil tank C, and a portion of oil tank D, as shown by the area of ​​the industrial facility 200 outlined by the dashed line). The image 201 can be processed one or more times / iteratively together with a text string A (e.g., "add oil pit on the ground") as input using an image-to-image model 211.

[0093] For example, Figure 2AAs shown, at the first iteration, a realistic synthetic image 213_A1 can be derived from the model output of the image-to-image model 211 corresponding to the image 201 and the text string A. The realistic synthetic image 213_A1 can be a modified version of the real-world image 201 by adding an oil pit "O1" on the ground next to the oil tank A. A supervisory label L_A1 (e.g., "oil leak" or "oil pit") can be determined for the realistic synthetic image 213_A1, and a training instance A1 is generated to include the realistic synthetic image 213_A1 as a training instance input and the supervisory label L_A1 as a true value label. The training instance A1 can be used to train (or if already trained, validate or fine-tune) the anomaly detection ML model 221. For example, the realistic synthetic image 213_A1 can be processed as input using the anomaly detection ML model 221 to generate a model output 223_A1 for comparison with the supervisory label L_A1 to determine the difference between the model output 223_A1 and the supervisory label L_A1. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be fitted (or fine-tuned).

[0094] In the second iteration, refer to Figure 2B , the same image 201 and the same text string A can be processed as input using an image-to-image model 211 to generate a model output from which a realistic synthetic image 213_A2 can be generated / derived. The realistic synthetic image 213_A2 can be a modified version of the real-world image 201 by adding an oil puddle "O2" on the ground next to the oil tank A. The oil puddle "O2" in the realistic synthetic image 213_A2 can have a different size, shape, and / or position than the oil puddle "O1" in the realistic synthetic image 213_A1. A supervisory label L_A2 (e.g., "oil leak" or "oil puddle") can be determined for the realistic synthetic image 213_A2, and a training instance A2 can be generated to include the realistic synthetic image 213_A2 as a training instance input and the supervisory label L_A2 as a true value label. The training instance A2 can be used to train (or validate or fine-tune) an anomaly detection ML model 221. For example, the realistic synthetic image 213_A2 can be processed as input using the anomaly detection ML model 221 to generate a model output 223_A2 for comparison with the supervisory label L_A2 to determine the difference between the model output 223_A2 and the supervisory label L_A2. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be modified / fitted (or fine-tuned).

[0095] In the third iteration, reference Figure 2C, the same image 201 and the same text string A can be processed as input using an image-to-image model 211 to generate a model output from which a realistic synthetic image 213_A3 can be generated / derived. The realistic synthetic image 213_A3 can be a modified version of the real-world image 201 by adding an oil puddle "O3" on the ground next to the oil tank D. The oil puddle "O3" in the realistic synthetic image 213_A3 can have a different size, shape, and / or position than the oil puddle "O1" in the realistic synthetic image 213_A1 (and / or the oil puddle "O2" in the realistic synthetic image 213_A2). A supervisory label L_A3 (e.g., "oil leak" or "oil puddle") can be determined for the realistic synthetic image 213_A3, and a training instance A3 can be generated to include the realistic synthetic image 213_A3 as a training instance input and the supervisory label L_A3 as a true value label. The training instance A3 can be used to train (or validate or fine-tune) the anomaly detection ML model 221. For example, the realistic synthetic image 213_A3 can be processed as input using the anomaly detection ML model 221 to generate a model output 223_A3 for comparison with the supervisory label L_A3 to determine the difference between the model output 223_A3 and the supervisory label L_A3. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be fitted (or fine-tuned if fine-tuning is performed). It is worth noting that if additional iterations are performed, more realistic synthetic images can be generated, and repeated descriptions are omitted herein.

[0096] In some embodiments, the anomaly detection ML model 221 can be an ML model trained (or validated, or fine-tuned) based on real-world images (if any captured anomalies exist) and / or realistic synthetic images (e.g., image 201) generated / derived by processing real-world images (not capturing any anomalies) conditioned on text strings describing anomalies in an industrial facility. Figure 2D , an image 201 (e.g., a real-world image) can be processed together with a text string B (e.g., "add leak") as input using an image-to-image model 211 to generate a model output from which a realistic synthetic image 213_B1 can be generated / derived. By adding a leak (in Figure 2D denoted by “L1” in FIG. 2 ), the realistic synthetic image 213_B1 can be a modified version of the real-world image 201 .

[0097] A supervisory label L_B1 (e.g., "oil leak") can be determined for the realistic synthetic image 213_B1, and a training instance B1 can be generated to include the realistic synthetic image 213_B1 as a training instance input and the supervisory label L_B1 as a true value label. The training instance B1 can be used to train the anomaly detection ML model 221. For example, the realistic synthetic image 213_B1 can be processed as an input using the anomaly detection ML model 221 to generate a model output 223_B1 for comparison with the supervisory label L_B1, thereby determining the difference between the model output 223_B1 and the supervisory label L_B1. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be modified / fitted. It is worth noting that if additional iterations are performed, a more realistic synthetic image can be generated, and the repeated description is omitted herein.

[0098] Figure 3 300 for practicing selected aspects of the present disclosure according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system can include various components of various computer systems, such as one or more components of a server computing device 105 (and / or an additional computing device such as a client device 103-A), including an ML engine 1051, anomaly detection engine 1052, and / or a database. In addition, while the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted, or added.

[0099] At block 302, the system may, for example, via a server such as the server computing device 105, for each of a plurality of text strings that respectively describe (or indicate) an anomaly and a corresponding industrial facility setting: perform a plurality of iterations of processing the text string using a text-to-image model to generate a corresponding composite image at each iteration. The industrial facility setting can include, but is not limited to, including, for example, one or more of a chemical processing plant, an oil or gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable facility (which can implement one or more at least partially automated processes).

[0100] In some embodiments, the system can determine a plurality of text strings describing anomalies and corresponding industrial facility settings, respectively, before performing a plurality of iterations of processing the text strings for each of the plurality of text strings. As non-limiting examples, the plurality of text strings can include a first text string of "oil leak" stored in association with a first chemical plant (or "oil leak in chemical plant"), a second text string of "pipeline corrosion" stored in association with a second chemical plant (which can be the same as or different from the first chemical plant), and / or a third text string of "sensor wire breakage" stored in association with a manufacturing plant.

[0101] The first text string, the second text string, the third text string, and / or the additional text string can be received from a user input, and can also be retrieved from a database or file storing / listing text descriptions collected for different anomalies. For example, the database (or file) can include a first entry storing the first text string, a second entry storing the second text string, and three entries storing the third text string. In some embodiments, the entries of the database can be grouped based on the type of different anomalies, the degree of danger of different anomalies, industrial facilities that can foresee different anomalies, and / or historical locations or regions that witness or record different anomalies.

[0102] Continuing with the above non-limiting example, given a first text string of "oil leak in chemical plant", the first text string can be processed as input using a text-to-image model at a first iteration to generate an image N_11 (which is a realistic synthetic image) showing a portion of the chemical plant reflecting the oil leak (e.g., on the internal ground of the chemical plant). In this way, using the text-to-image model, the first text string of "oil leak in chemical plant" can be repeatedly processed as input (e.g., m times / iteration) to generate a set (e.g., a total number of m) of images (e.g., image N_11, image N_12, ..., image N_1m) that respectively reflect the oil leak in the corresponding chemical plant. In some embodiments, the generated set of images can be stored in association with the first text string of "oil leak in chemical plant" (e.g., locally, on a server device, or in the cloud) for subsequent use.

[0103] For example, a first training instance can be generated and stored, and image N_11 is included as training instant input and a first text string of “oil leak in a chemical plant” (or a keyword of “oil leak” extracted from the first text string) is included as a supervisory label, a second training instance can be generated and stored to include image N_12 as training instant input and the first text string of “oil leak in a chemical plant” (or a keyword of “oil leak” extracted from the first text string) as a supervisory label, …, an mth training instance can be generated and stored, and image N_1m (where m is a positive integer greater than or equal to “1”) is used as training instant input and the first text string of “oil leak in a chemical plant” (or a keyword of “oil leak” extracted from the first text string) is used as a supervisory label.

[0104] Optionally, before storing the first, second, ..., and m-th training instances in, for example, a database, the first to m-th training instances can be checked (e.g., by an expert or technician trained to identify oil spills) to correct / modify supervisory labels or remove one or more generated training instances (in case the image in the corresponding training instance does not reflect "oil spill" well). In other words, the first text string can be processed to automatically extract supervisory labels for the image generated by the text-to-image model corresponding to the first text string, and before storing the corresponding training instances including the supervisory labels, the expert can verify or modify one or more supervisory labels (if necessary).

[0105] Continuing with the above non-limiting example, given a second text string of "pipeline corrosion" (or "pipeline corrosion in a chemical plant", or "pipeline corrosion in a manufacturing site", etc.), the second text string can be processed as input using a text-to-image model in a first iteration to generate an image N_21 (which is a realistic synthetic image) showing "pipeline corrosion" (e.g., on a pipeline). Optionally, the second text string of "pipeline corrosion" can be repeatedly processed as input using a text-to-image model (e.g., p times / iteration) to generate a set of images (e.g., image N_21, image N_22, ..., image N_2p) that respectively reflect pipeline corrosion of one or more pipelines (e.g., a total number of "p", where p is a positive integer greater than or equal to "1", and "p" can be the same as or different from "m"). Similarly, based on the set of generated images (e.g., image N_21, image N_22, ..., image N_2p) and the second text string of "pipeline corrosion", a total number of p training instances can be generated and / or stored in the above database. Repeated descriptions are omitted herein.

[0106] At block 304, the system (e.g., via a server such as server computing device 105) can train an anomaly detection machine learning (ML) model using the generated synthetic image and the corresponding supervised labels for the generated synthetic image. The anomaly detection ML model can be different from the text-to-image model described above. While the text-to-image model can perform text-to-image conversion, the anomaly detection ML model can be a classifier for detecting one or more anomalies (e.g., predicting the probability of presence of each predefined anomaly in a real-world image within an industrial facility setting) so that appropriate and timely remedial actions can be performed accordingly in response to detecting one of the predefined anomalies. In some embodiments, the anomaly detection ML model can be used to detect anomalies other than those reflected in the synthetic image (which is generated by the text-to-image model based on multiple text strings), and / or detect anomalies not described by the multiple text strings.

[0107] In some implementations, for each generated synthetic image, the system can train an anomaly detection ML model by processing the corresponding synthetic image as an input using the anomaly detection ML model to generate an output (e.g., a classification output) indicating a plurality of probabilities of each predefined number of anomalies present in the industrial facility setting captured in the corresponding synthetic image. The output can be compared to a supervised label corresponding to the corresponding synthetic image to determine a difference, and based on the determined difference, one or more weights / parameters of the anomaly detection ML model can be adjusted or modified.

[0108] It is worth noting that although the generated synthetic images described herein are used to train an anomaly detection ML model, the generated synthetic images (or a portion thereof) can be additionally or alternatively applied to fine-tune or verify an additional anomaly detection ML model. Alternatively, the generated synthetic images can be divided into two groups or three groups, one group for training an anomaly detection ML model, one group for fine-tuning anomaly detection ML module, and / or one group for verifying anomaly detection ML model, wherein the group of synthetic images used to train the anomaly detection ML module can have the largest number of synthetic images among the two groups or three groups divided from the generated synthetic images.

[0109] At block 306, the system, for example, through a server such as server computing device 105, can provide a trained anomaly detection ML model for anomaly detection within a specific industrial facility. For example, one or more visual sensors of a specific industrial facility can be used to monitor the specific industrial facility by capturing images or videos. Using the trained anomaly detection ML model, each image (or image frame in a video) or a selected image (or image frame) captured for the specific industrial facility can be processed as an input to generate an output indicating an anomaly name, a likelihood of an anomaly existing within the specific industrial facility, and / or a location of the anomaly relative to the captured image (e.g., highlighted by a bounding box). In response to the output of the trained anomaly detection ML model indicating the presence of an anomaly (e.g., based on a likelihood of satisfying a likelihood threshold), one or more remedial actions can be performed. For example, the one or more remedial actions can include a first remedial action of generating and delivering a warning message, a second remedial action of pausing or stopping one or more industrial processes, a third remedial action of cutting off power, and the like. It is noteworthy that the one or more remedial actions are not limited to the description herein, and different remedial actions can be performed based on the type (or name), degree of danger, and / or location (or size or quantity estimated based on the above output) of the detected anomaly.

[0110] In some embodiments, one or more vision sensors can be included in the mobile robot, and the one or more vision sensors can capture images of one or more specific components within the industrial facility setting. As a non-limiting example, the one or more specific components can include: a liquid tank and / or a liquid carried by the liquid tank. The mobile robot 101 can be a quadruped robot (e.g., a robot dog), a wheeled robot, a drone, or any other suitable robot that can move within an industrial facility setting.

[0111] The one or more visual sensors can include, for example, a monochrome graphic RGB camera, a stereo camera, a thermal imager, and / or any other applicable visual sensor. The image captured by the visual sensor can be an RGB image, an RGB-D image, a thermal image, or any other applicable image, respectively. In some embodiments, the image can be a high-resolution image. In some embodiments, the visual sensor can be integrated with the mobile robot or can be detachably connected to the mobile robot.

[0112] In some embodiments, providing a trained anomaly detection ML model for anomaly detection within a specific industrial facility includes: enabling the trained anomaly detection ML model to process real images captured via visual sensors inside or outside the specific industrial facility. In some embodiments, the visual sensor can be carried by a mobile robot (robot dog) that moves around the specific industrial facility. In these embodiments, enabling the trained anomaly detection ML model to process real images can include: enabling the trained anomaly detection ML model to be downloaded locally to the mobile robot that moves around the specific industrial facility for continuous or periodic anomaly monitoring and detection. By having a trained anomaly detection ML model locally at the mobile robot, the trained anomaly detection ML model can be used to process images captured using the visual sensors of the mobile robot for timely anomaly detection and / or notification.

[0113] Additionally or alternatively, enabling the trained anomaly detection ML model to process real images includes enabling the trained anomaly detection ML model to be downloaded to one or more on-site computers (e.g., servers). In these embodiments, images captured by the visual sensor are provided to one or more on-site computers within a particular industrial facility and processed using the trained anomaly detection ML model downloaded to the servers for anomaly detection.

[0114] In some embodiments, more than one anomaly detection ML model can be trained, for example, using one or more steps described throughout the present disclosure. The more than one anomaly detection ML model can be stored at a remote server (or can be distributed across one or more remote servers). As a non-limiting example, the more than one anomaly detection ML model can include: a first anomaly detection ML model fine-tuned or validated for a first site (e.g., a first industrial facility), and a second anomaly detection ML model fine-tuned or validated for a second site different from the first site (e.g., a second industrial facility). For example, a real image captured by a sensor at the first site can be forwarded / transmitted to the remote server along with an identifier of the first site. For example, additional real images captured by a sensor at the second site can be forwarded / transmitted to the remote server along with an identifier of the second site.

[0115] In the above non-limiting example, in response to receiving a real image with an identifier of a first site, a first anomaly detection ML model fine-tuned or validated for the first site can be accessed and used when processing the real image to determine whether an anomaly exists in the real image. Similarly, in response to receiving an additional real image with an identifier of a second site, a second anomaly detection ML model fine-tuned or validated for the second site can be accessed and used when processing the additional real image to determine whether an anomaly exists in the additional real image.

[0116] Additionally or alternatively, a trained anomaly detection ML model (and / or one or more additional trained anomaly detection ML models) can be accessed via the cloud. For example, being able to access and use for processing real images includes causing the trained anomaly detection ML model to be used.

[0117] In various embodiments, before providing the trained anomaly detection ML model for anomaly detection within a specific industrial facility and each of a plurality of real images of the specific industrial facility, the system may, at block 3051A, process the real image and the prompt describing the anomaly using an image-to-image translation model to generate a corresponding translated synthetic image (e.g., which includes an image representation of the anomaly described in the prompt); and, at block 3053A, fine-tune the trained anomaly detection ML model by further training the anomaly detection ML model based on the translated synthetic image.

[0118] By fine-tuning a trained anomaly detection ML model using a converted synthetic image generated from a real image of a specific industrial facility, one or more weights / parameters of the trained anomaly detection ML model can be fine-tuned, and the performance of the trained anomaly detection ML model in detecting anomalies of the specific industrial facility (and / or detecting certain anomalies) can be improved.

[0119] In some embodiments, as previously described, a trained anomaly detection ML model can be trained based on a large number of diverse synthetic images (e.g., synthetic images generated by a text-to-image model in multiple iterations at block 302), where each diverse synthetic image reflects a corresponding anomaly within an industrial facility setting. In these embodiments, anomalies artificially created / introduced in the converted synthetic images (which come from real images capturing one or more components or substances within a particular industrial facility) and anomalies described in prompts (e.g., at block 3051A) can have little or no overlap with anomalies described in multiple text strings (e.g., at block 302). As non-limiting examples, multiple text strings can describe anomalies such as "oil leak", "corroded pipeline", and "broken wire", while prompts used to adjust the real image can describe anomalies that are unique to a particular industrial facility (e.g., "water leak").

[0120] In various embodiments, before providing the trained anomaly detection ML model for anomaly detection within a specific industrial facility and each of a plurality of real images for the specific industrial facility, the system can process the real image and the prompt describing the anomaly using an image-to-image conversion model to generate a corresponding converted synthetic image; and verify (or test) the trained anomaly detection ML model based on the converted synthetic image. In these embodiments, providing the trained anomaly detection ML model for anomaly detection within the specific industrial facility is responsive to determining that the verification satisfies one or more conditions.

[0121] In some implementations, validating the trained anomaly detection ML model based on the transformed synthetic image includes: determining an accuracy metric for anomaly predictions made based on outputs from the anomaly detection ML model based on processing the transformed synthetic image. The one or more conditions can include, for example, a threshold accuracy metric.

[0122] As a non-limiting example, assume that fifty transformed synthetic images generated by the image-to-image transformation model are used, and the trained anomaly detection ML model is validated using the fifty transformed synthetic images. In this non-limiting example, if forty-two of the fifty outputs of the trained anomaly detection ML model (which correspond to the fifty transformed synthetic images) match the supervisory labels of the corresponding fifty transformed synthetic images, which results in an accuracy measure of approximately 0.84 (which satisfies a threshold accuracy measure, e.g., 0.8), then the trained anomaly detection ML model can be considered validated and ready to be deployed on a particular industrial facility for anomaly monitoring and detection.

[0123] The methods described herein or in other aspects of the present disclosure can be performed via one or more processors carried by a mobile robot or included in one or more computing devices separate from and not attached to the mobile robot. In this case, real-world images captured by the mobile robot for monitoring anomalies can be transmitted by the mobile robot and, after being transmitted by the mobile robot, identified by the one or more computing devices for subsequent use and processing.

[0124] Figure 44 is a block diagram of an example computing device 410 that can optionally be used to perform one or more aspects of the techniques described herein. The computing device 410 typically includes at least one processor 414 that communicates with a plurality of peripheral devices via a bus subsystem 412. These peripheral devices can include a storage subsystem 424 that includes, for example, a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow for user interaction with the computing device 410. The network interface subsystem 416 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0125] The user interface input device 422 can include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 410 or a communication network.

[0126] User interface output device 420 can include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visual images. The display subsystem can also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computing device 410 to a user or another machine or computing device.

[0127] The storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 424 can include an executable Figure 3 The logic for selected aspects of the methods, as well as the logic for implementing the various components shown in Figures 1-2.

[0128] These software modules are typically executed by the processor 414 alone or in conjunction with other processors. The memory 425 used in the storage subsystem 424 can include multiple memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular embodiment can be stored in the storage subsystem 424 by the file storage subsystem 426, or in other machines accessible to the processor(s) 414.

[0129] The bus subsystem 412 provides a mechanism for the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus subsystem 412 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0130] Computing device 410 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 4 The description of computing device 410 depicted in FIG. 4 is intended only as a specific example to illustrate some embodiments. Many other configurations of computing device 410 may have more Figure 4 The computing devices shown may have greater or fewer components.

Claims

1. A method implemented by one or more processors, the method comprising: For each of a plurality of text strings describing an anomaly and a corresponding industrial facility setting, respectively: Using a text-to-image model, performing multiple iterations of processing the text string to generate a corresponding composite image at each of the iterations; training an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervision labels for the generated synthetic images; and Provides trained anomaly detection ML models for anomaly detection within specific industrial facilities.

2. The method according to claim 1, further comprising: Prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility: For each real image of the plurality of real images of the particular industrial facility: Processing the real image and the hint describing the anomaly using an image-to-image translation model to generate a corresponding translated synthetic image; and The trained anomaly detection ML model is fine-tuned by further training the anomaly detection ML model based on the converted synthetic image.

3. The method according to claim 2, wherein: Fine-tuning the trained anomaly detection ML model involves: Fine-tune weights of one or more layers of the trained anomaly detection ML model.

4. The method according to claim 1, further comprising: Prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility: For each real image of the plurality of real images of the particular industrial facility: Processing the real image and the hint describing the anomaly using an image-to-image translation model to generate a corresponding translated synthetic image; and validating the trained anomaly detection ML model based on the converted synthetic image; Wherein providing the trained anomaly detection ML model for anomaly detection within the particular industrial facility is in response to determining that the verification satisfies one or more conditions.

5. The method according to claim 4, in, Validating the trained anomaly detection ML model based on the transformed synthetic image includes: determining a measure of accuracy of anomaly predictions made based on outputs from the anomaly detection ML model based on processing the transformed synthetic image; and Wherein, determining that the verification satisfies one or more conditions includes: determining that the accuracy metric satisfies a threshold accuracy metric.

6. The method according to claim 1, wherein: The plurality of text strings include: a first text string describing a first anomaly within a first industrial facility setting, and A second text string describing a second anomaly within the first industrial facility setting.

7. The method according to claim 1, wherein: The plurality of text strings include: A text string describing an anomaly within an industrial facility setting, and An additional text string describing the one anomaly within the additional industrial facility setting.

8. The method according to claim 1, wherein: The corresponding supervisory labels for the generated composite image are automatically determined based at least on the plurality of text strings describing the anomalies.

9. The method according to claim 1, wherein: Providing the trained anomaly detection ML model for anomaly detection within the specific industrial facility includes: The trained anomaly detection ML model is caused to be used to process real images captured via visual sensors within the specific industrial facility.

10. The method according to claim 9, further comprising: Using the trained anomaly detection ML model in processing the real image, wherein using the trained anomaly detection ML model in processing the real image comprises: processing real images of the real images using the trained anomaly detection ML model to generate a model output indicating whether any anomalies are present in one or more components; and One or more remedial actions are performed in response to the model output indicating the presence of an anomaly in one or more of the components.

11. The method according to claim 9, wherein: The visual sensor is carried by the mobile robot, and The real image is captured by the mobile robot at a designated location within the specific industrial facility.

12. The method according to claim 10, wherein: The one or more remedial actions include a warning message that alerts that the anomaly was detected in one or more of the components at the specified location.

13. A method implemented by one or more processors, the method comprising: For each real image of multiple real images of a specific industrial facility: processing the real image and the cue describing an anomaly associated with the particular industrial facility using an image-to-image translation model to generate a corresponding translated synthetic image; as well as The trained anomaly detection ML model is fine-tuned by further training the trained anomaly detection ML model based on the converted synthetic images.

14. The method according to claim 13, wherein: The trained anomaly detection ML model is pre-trained based on one or more real images that respectively capture corresponding anomalies.

15. The method according to claim 14, wherein: The corresponding anomaly is captured using a visual sensor within the specific industrial facility.

16. The method according to claim 13, wherein: The trained anomaly detection ML model is pre-trained based on a plurality of synthetic images, each of which depicts a realistic scene of a corresponding anomaly in an industrial facility setting, and the plurality of synthetic images are generated using a text-to-image model based on one or more text strings that respectively describe the anomaly.

17. A method implemented by one or more processors, the method comprising: For each real image of multiple real images of a specific industrial facility: processing the real image and the cue describing anomalies associated with the particular industrial facility using an image-to-image translation model to generate a corresponding translated synthetic image; and Based on the transformed synthetic images, the trained anomaly detection ML model is validated.

18. The method according to claim 17, further comprising: determining whether the verification satisfies one or more conditions; as well as In response to determining that the verification satisfies the one or more conditions, providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility.

19. The method according to claim 18, wherein: Validating the trained anomaly detection ML model based on the transformed synthetic image includes: determining a measure of accuracy of anomaly predictions made based on outputs from the anomaly detection ML model based on processing the transformed synthetic image, and, Wherein the one or more conditions include a threshold accuracy metric.

20. The method according to claim 17, wherein: pre-training the trained anomaly detection ML model based on a plurality of synthetic images, the plurality of synthetic images respectively depicting realistic scenes of corresponding anomalies in an industrial facility setting, The plurality of composite images are generated using a text-to-image model based on one or more text strings that respectively describe the anomalies.

21. The method according to claim 17, wherein: The trained anomaly detection ML model is pre-trained based on one or more real images that respectively capture corresponding anomalies within an industrial facility setting.

Images