Generation of composite image for training and / or verifying abnormality detection model using text-to-image model

Abstract

To provide an implementation for monitoring and detecting abnormality in a component of industrial facilities based on ML-based image processing.SOLUTION: In various embodiments, a plurality of text strings each describing abnormality in a surrounding environment of industrial facilities are determined, and for each of the plurality of text strings, a plurality of iterations of processing of the text string are executed using a text-to-image model in order to generate a corresponding composite image in each of the iterations. The generated composite image may be utilized for training an abnormality detection machine learning (ML) model or for fine-tuning or verifying a trained abnormality detection ML model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to techniques for processing images using machine learning (ML) models. [Background technology]

[0002] Complex industrial facilities, such as petrochemical refineries, chemical plants, etc., may include numerous components utilized in handling liquids and / or other substances involved in the industrial process of the industrial facility. To ensure that the components involved in the industrial process are operating as intended and / or to ensure that the substances involved in the industrial process are in their intended state, it is important to monitor anomalies within the industrial facility (e.g., the presence of oil or other liquids in the ground, corroded piping, etc.) so that one or more remedial actions (e.g., raising an alarm, stopping an automated process, etc.) can be taken in a timely manner when an anomaly is detected. Summary of the Invention [Means for solving the problem]

[0003] Various techniques have been proposed for processing images using machine learning (ML) models to generate an output indicating whether a condition exists in the image. For example, techniques have been proposed for processing images capturing one or more components of an industrial facility using an anomaly detection ML model (sometimes referred to herein as an "anomaly detection model") to generate an output indicating whether an anomaly exists in one or more components captured by the image. However, to be effective and have sufficient predictive accuracy, an anomaly detection ML model must be trained based on a large number of diverse "positive" ground truth images, each of which reflects the presence of a corresponding anomaly.

[0004] Separately, text-to-image models have been released that allow for the provision of text prompts and the generation of detailed and realistic synthetic images conditioned on the text prompt. For example, a text prompt of "an orange cat riding a donkey" can be processed to generate a realistic synthetic image containing an orange cat riding a donkey. Some of these models also allow image-to-image transformation conditioned on a text prompt. For example, a base image containing a black cat can be provided with a prompt of "make the cat orange" to generate a transformed synthetic image that contains an orange cat instead of a black cat but otherwise largely matches the base image. Non-limiting examples of text-to-image and image-to-image models (many models can do both) include "Stable Diffusion" and "DALL-E".

[0005] As noted above, to be effective, an anomaly detection ML model must be trained on a large number of diverse "positive" ground truth images, each of which reflects the presence of a corresponding anomaly. Obtaining such a large number of diverse real-world images can be difficult in industrial settings, where the corresponding anomalies may be rare and / or dangerous.

[0006] Furthermore, there is currently no way to (a) fine-tune the model for a particular industrial facility and / or (b) validate the model's performance in detecting anomalies in the industrial facility before utilizing such an anomaly detection ML model in the particular industrial facility. For example, with respect to (a), fine-tuning (i.e., limited additional training) of the anomaly detection ML model for a particular industrial facility may not be possible because it may be difficult to obtain actual images of "positive" ground truth that capture anomalies and are specific to the industrial facility. For example, it may not be safe or desirable to create such anomalies in the industrial facility. Also, for example, the industrial facility may have been built but not yet been actually used. As another example, with respect to (b), it may similarly not be possible to validate that the anomaly detection ML model is effective for a particular industrial facility before utilizing the anomaly detection ML model in the particular facility. This may be because such validation requires determining that the ML model is capable of detecting the occurrence of anomalies in the particular facility. Again, it may not be safe or desirable to actually create anomalies in the particular facility.

[0007] Implementations disclosed herein enable the generation of a large number of diverse synthetic "positive" ground truth images, each reflecting the presence of a corresponding anomaly in an industrial facility. The implementations use a text-to-image model to generate the synthetic images. More specifically, the text-to-image model is prompted with prompts describing the anomaly and describing the surroundings of the industrial facility, resulting in images of the surroundings of the industrial facility, including the anomaly. For example, a prompt "exterior of chemical plant with oil puddles" and / or a prompt "interior of a chemical plant with oil on floor." It is noted that the implementations can generate multiple different synthetic images by processing the same prompt multiple times. The same prompt is used in each iteration, but different "seeds" are randomly utilized by the model in each iteration, resulting in different synthetic images.

[0008] Implementations disclosed herein may additionally or alternatively enable the generation of multiple synthetic "positive" ground truth images, each reflecting the presence of a corresponding anomaly in the industrial facility, each for a particular industrial facility, such that such specialized images can be used to (a) fine-tune the ML model (which may improve accuracy and / or robustness) prior to use in the particular industrial facility, and / or (b) validate the ML model (to ensure it is valid for the particular industrial facility) prior to use in the particular industrial facility.

[0009] These implementations can capture actual images of a particular industrial facility and then process those actual images using the image-to-image transformation model, conditioned on text prompts describing the anomalies. For example, processing the actual images with a text prompt to "add an oil puddle" can result in a synthetic image that closely matches the actual image, but includes an oil puddle in the ground. As a particular example, hundreds or thousands of actual images of a particular industrial facility can be captured (e.g., by a robot) while the industrial facility is free of anomalies. Those actual images can then be processed using the image-to-image model, conditioned on text prompts describing the anomalies, to generate corresponding synthetic images that include the anomalies. Those corresponding synthetic images can then be used for fine-tuning and / or validation, as described herein.

[0010] Implementations use the text-to-image model to generate a large number of synthetic, diverse "positive" ground truth images, each reflecting the presence of a corresponding anomaly in the industrial facility. More specifically, the text-to-image model is prompted with prompts describing the anomaly and describing the surroundings of the industrial facility, resulting in an image that includes the anomaly and is within the surroundings of the industrial facility. For example, a prompt "chemical plant room with pipes that include corrosion." Some of these implementations can also generate additional synthetic "positive" ground truth images using actual images without anomalies from the industrial facility and the image-to-image transformation model. For example, an actual image without anomalies can be processed with a prompt "add some corrosion" to generate a synthetic image that matches the actual image but includes some corrosion on the objects in the image.

[0011] The generated synthetic ground truth images can then be optionally labeled (e.g., by a human reviewer). For example, each label can indicate whether an anomaly is present in the image, and optionally the area where the anomaly occurs and / or the type of anomaly. The synthetic ground truth images and their labels can then be used to train an anomaly detection ML model. The anomaly detection ML model can then be utilized for anomaly detection in industrial facilities.

[0012] Some implementations additionally or alternatively generate a variety of transformed, synthesized, "positive" ground truth images, each reflecting the presence of a corresponding anomaly in the industrial facility, each for a particular industrial facility, such that such specialized images can be used to (a) fine-tune the anomaly detection ML model prior to use in the particular industrial facility (which may improve accuracy and / or robustness) and / or (b) validate the anomaly detection ML model prior to use in the particular industrial facility (to ensure it is valid for the particular industrial facility).

[0013] These implementations can capture actual images of a particular industrial facility and then process those actual images using the image-to-image transformation model, conditioned on a text prompt describing the anomaly. For example, processing the actual images with a text prompt "add corrosion to one component" can result in a transformed synthetic image that closely matches the actual image, but includes corrosion in a component of the image. As a particular example, hundreds or thousands of actual images of a particular industrial facility can be captured (e.g., by a robot) while the industrial facility is free of anomalies. Those actual images can then be processed using the image-to-image model, conditioned on a text prompt describing the anomaly, to generate corresponding transformed synthetic images that include the anomaly. Those corresponding transformed synthetic images can then be used for fine-tuning and / or validation.

[0014] For example, when validating an anomaly detection ML model for a particular industrial facility, the transformed synthetic images can each be processed using the anomaly detection ML model to generate a corresponding output indicating whether an anomaly is present. Those corresponding outputs can then be compared to the ground truth output (indicating whether an anomaly is actually present in the corresponding transformed synthetic image) to generate an accuracy metric for the output (e.g., 90% if 900 out of 1000 predictions are correct). Optionally, the anomaly detection ML model is deployed for use within the particular industrial facility only if the accuracy metric and / or other validation metrics meet a threshold.

[0015] In various implementations, a processor-implemented method is provided that includes, for each of a plurality of text strings, each describing an anomaly and a corresponding ambient environment of an industrial facility, performing a plurality of iterations of processing the text string, in each iteration, using a text-to-image model to generate a corresponding synthetic image (e.g., conditioned on the text string).

[0016] The industrial facility environment may include an industrial automation facility implementing any number of at least partially automated processes For example, the industrial automation facility may take the form of a chemical processing plant, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable industrial environment.

[0017] Each of the multiple text strings can describe an anomaly present in one of the one or more components in the industrial facility's surroundings. For example, different text strings can describe different anomalies in the same component in or around the industrial facility's surroundings. As another example, different text strings can describe different anomalies in different components in or around the industrial facility's surroundings. As yet another example, different text strings can describe different anomalies in different components in different industrial facility's surroundings. The synthetic image can be a detailed, realistic image that captures the presence of each anomaly in one or more components in or around the industrial facility's surroundings.

[0018] In various implementations, the method may further include training an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images. For example, in a first iteration, a synthetic image M generated using a first text string (that describes or indicates a first anomaly associated with a component in or around the surrounding environment of the corresponding industrial facility) may be trained. 1-1 For label L 1-1 It is possible to generate a label L 1-1 can be automatically generated based on the first text string, or can be manually generated (or inspected) by a human technician or expert trained to identify labels of objects in the industrial facility's surroundings. In this example, for example, the corresponding anomaly A 1-1to generate a corresponding output showing the type, name, and / or location (which can be highlighted using a bounding box) of 1-1 can be applied as training instance inputs and processed using an anomaly detection ML model. The corresponding output is then denoted by the labels L 1-1 (Abnormal A 1-1 The first text string may be compared to the first iteration (which may be considered as a ground truth label describing the ground truth type, name, and / or location of the first text string) to determine a difference and adjust one or more parameters / weights of the anomaly detection ML model based on the difference. Similarly, the synthetic image M generated in the second iteration using the first text string may be compared to the first iteration (which may be considered as a ground truth label describing the ground truth type, name, and / or location of the first text string) to determine a difference and adjust one or more parameters / weights of the anomaly detection ML model based on the difference. 1-2 For label L 1-2 For example, the corresponding anomaly A 1-2 to generate a corresponding output showing the type, name, and / or location (which can be highlighted using a bounding box) of 1-2 can be applied as training instance inputs and processed using an anomaly detection ML model. The corresponding output is then denoted by the labels L 1-2 , to determine additional differences, and adjust one or more of the parameters of the anomaly detection ML model based on the additional differences.

[0019] Alternatively, or in addition, a synthetic image M generated in the first iteration using a second text string different from the first text string. 2-1 For label L 2-1 For example, the corresponding anomaly A 2-1 to generate a corresponding output showing the type, name, and / or location (which can be highlighted using a bounding box) of 2-1 can be applied as training instance inputs and processed using an anomaly detection ML model. The corresponding output is then denoted by the labels L 2-1to determine further differences and adjust one or more of the parameters of the anomaly detection ML model based on the further differences.

[0020] In some implementations, the second text string and the first text string can describe different types of anomalies. In some implementations, the second text string and the first text string can describe the same type of anomaly, but for different components in the same or different industrial facilities. The anomaly detection ML model can be trained using different synthetic images and corresponding supervised labels until the difference between the model output and the corresponding labels is minimized or meets a difference threshold.

[0021] In various implementations, the method may further include providing a trained anomaly detection ML model for use in anomaly detection within a particular industrial facility.

[0022] In various implementations, the method may further include processing the actual images and prompts describing the anomaly using the image-to-image transformation model to generate a corresponding transformed synthetic image for each of a plurality of actual images of the particular industrial facility prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, and fine-tuning the trained anomaly detection ML model through further training the anomaly detection ML model based on the transformed synthetic images.

[0023] In some implementations, one or more visual sensors mounted on a mobile robot can be used to capture multiple actual images. The mobile robot can be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot that can move within an industrial automation facility. The one or more visual sensors can be a monographic RGB camera, a stereographic camera, a thermal camera, or any other applicable visual sensor. Accordingly, the actual image can be an RGB image, an RGB-D image, a thermal image, or any other applicable image.

[0024] In some implementations, the method may further include using the trained anomaly detection ML model when processing the actual images, where using the trained anomaly detection ML model when processing the actual images includes processing one of the actual images with the trained anomaly detection ML model to generate a model output indicating whether an anomaly is present in one or more of the components, and causing one or more repair actions to be performed in response to the model output indicating an anomaly is present in one or more of the components.

[0025] In various implementations, the method may further include processing the actual images and prompts describing the anomaly using an image-to-image transformation model to generate a corresponding transformed synthetic image for each of a plurality of actual images of the particular industrial facility prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, and validating the trained anomaly detection ML model based on the transformed synthetic images. Providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility occurs in response to a determination that the validation satisfies one or more conditions.

[0026] In various implementations, validating the trained anomaly detection ML model based on the transformed synthetic image is implemented by determining an accuracy metric of the anomaly prediction made based on output from the anomaly detection ML model based on processing the transformed synthetic image, In those implementations, determining that the validation satisfies the one or more conditions can include determining that the accuracy metric meets a threshold accuracy metric.

[0027] Additionally, some implementations include one or more processors of one or more computing devices operable to execute instructions stored in associated memory, the instructions configured to cause any of the aforementioned methods to be performed. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0028] In some implementations, the system includes one or more processors and a memory that stores instructions that, upon execution by the one or more processors, cause the one or more processors to: for each of a plurality of text strings, each describing an anomaly and a corresponding ambient environment of an industrial facility, perform multiple iterations of processing the text strings using a text-to-image model to generate, in each iteration, a corresponding synthetic image; train an anomaly-detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images; and provide the trained anomaly-detection ML model for use in anomaly detection within a particular industrial facility.

[0029] In various implementations, the system may further include instructions for processing the actual images and prompts describing the anomaly using the image-to-image transformation model to generate a corresponding transformed synthetic image for each of a plurality of actual images of the particular industrial facility prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, and fine-tuning the trained anomaly detection ML model through further training the anomaly detection ML model based on the transformed synthetic images.

[0030] In various implementations, the system may further include instructions for processing the actual images and the prompts describing the anomaly using an image-to-image conversion model to generate a corresponding transformed synthetic image for each of a plurality of actual images of the particular industrial facility prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, and validating the trained anomaly detection ML model based on the transformed synthetic image. In those implementations, providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility occurs in response to determining that the validation satisfies one or more conditions. In some implementations, validating the trained anomaly detection ML model based on the transformed synthetic image includes determining an accuracy metric of anomaly predictions made based on output from the anomaly detection ML model based on processing of the transformed synthetic image. The one or more conditions may include, for example, a threshold accuracy metric.

[0031] In some implementations, providing a trained anomaly detection ML model for use in anomaly detection within a particular industrial facility includes having the trained anomaly detection ML model used in processing actual images captured via a visual sensor within or outside the particular industrial facility. In some implementations, the visual sensor can be mounted on a mobile robot that moves around the particular industrial facility. In some of those implementations, having the trained anomaly detection ML model used in processing actual images can include having the trained anomaly detection ML model downloaded locally to the mobile robot that moves around the particular industrial facility for monitoring and anomaly detection. By having the trained anomaly detection ML model locally on the mobile robot, images captured using the visual sensor of the mobile robot can be processed using the trained anomaly detection ML model for rapid anomaly detection and / or notification.

[0032] In some implementations, having the trained anomaly detection ML model used when processing actual images includes having the trained anomaly detection ML model downloaded to one or more on-site computers (e.g., servers) located at the industrial facility. In some of those implementations, images captured by the visual sensor are provided to one or more on-site computers located within a particular industrial facility and processed using the trained anomaly detection ML model downloaded to those servers for anomaly detection.

[0033] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter listed at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0034] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary environment in which selected aspects of the present disclosure can be implemented, according to various embodiments. [Figure 1B] 1 is an illustrative flowchart for carrying out selected aspects of the present disclosure, according to various embodiments. [Figure 1C] FIG. 1 illustrates an example of a machine learning (ML) model for carrying out selected aspects of the present disclosure, according to various embodiments. [Figure 2A] 1 is a schematic diagram of an example of an image generated utilizing techniques described herein, according to various embodiments. [Figure 2B] 4 is a schematic diagram of another example image generated utilizing the techniques described herein, in accordance with various embodiments. [Figure 2C] 4A-4C are schematic diagrams of examples of further images generated utilizing the techniques described herein, according to various embodiments. [Figure 2D] 4A-4C are schematic diagrams of examples of additional images generated utilizing the techniques described herein, according to various embodiments. [Diagram 3] FIG. 1 illustrates an exemplary method for carrying out selected aspects of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Implementations of the present disclosure utilize one or more image generation models to generate large volumes of images for training, fine-tuning, and / or validation of anomaly detection ML models that are utilized to detect one or more anomalies in components and / or materials in an industrial environment (e.g., an industrial automation facility). The one or more image generation models can include, for example, a text-to-image model that can be utilized to process text prompts and generate realistic synthetic images conditioned on the text prompts.

[0036] For example, a text-to-image model can be utilized to process a text prompt (sometimes referred to herein as a "prompt" or "text string") that indicates or describes an anomaly, and generate a detailed, realistic synthetic image conditioned on the text prompt. In some implementations, the same text prompt can be processed multiple times / repetitions using the text-to-image model to generate multiple different synthetic images, each reflecting an anomaly.

[0037] In some implementations, the different text prompts can describe different anomalies in or around a particular industrial automation facility or a particular type of industrial automation facility (sometimes referred to herein as an “industrial facility”), and correspondingly, the different synthetic images generated using the text-to-image model can capture different anomalies in or around the particular (or particular type) industrial automation facility. In some implementations, the different text prompts can describe different anomalies in or around the different industrial facilities, and correspondingly, the different synthetic images generated using the text-to-image model can capture different anomalies in or around the different industrial facilities. The different synthetic images generated based on the same text prompt for multiple iterations and / or the different synthetic images generated based on different text prompts can be applied to train, fine-tune, and / or validate an anomaly detection ML model.

[0038] In various implementations, the one or more image generation models can additionally or alternatively include an image-to-image model that can be utilized to process the base image and the text prompt to generate a transformed / modified synthetic image that generally matches the base image but includes modifications consistent with the text prompt. For example, an image capturing one or more components of an industrial facility (e.g., a real-world image that does not reflect anomalies or a realistic synthetic image) and a text prompt that describes / indicates an anomaly present in one or more of the components can be provided to the image-to-image model. In this example, the image-to-image model is utilized to process the image and the text prompt to generate a modified image that reflects the anomaly described or indicated in the text prompt but otherwise generally matches the image. For example, if an image captured a floor with no oil, two tanks, and piping, and the text prompt was "add oil to the ground," the modified image could similarly include the floor, two tanks, and piping, but would include oil on the floor. Thus, after processing the base image and the text prompt, a modified image can be generated based on the model output of the image-to-image model, where the modified image is a modified version of the real-world image that reflects the anomaly indicated in the text prompt.

[0039] In some implementations, the same image captured of the same component of an industrial facility and different text prompts describing different anomalies of the component can be used to generate different corrected images using the image-to-image model, where each of the different corrected images can reflect a corresponding anomaly described or indicated by a respective one of the different text prompts, and the different corrected images (or portions thereof) can be applied as various "positive" ground truth images for training, fine-tuning, and / or validating an anomaly detection ML model in detecting anomalies of a particular component in an industrial facility.

[0040] In some implementations, as a practical example, one or more real visual sensors may be used to obtain various images capturing various components of an industrial facility (or multiple industrial facilities) under normal conditions (i.e., no abnormalities). For example, the one or more visual sensors may be attached to a robot located in the industrial facility or may be located at various locations in the industrial facility. The one or more visual sensors may be monographic RGB cameras, stereographic cameras, thermal cameras, and / or any other applicable visual sensors. The robot may be, for example, a mobile robot, such as a robot dog, that moves around or through the industrial facility. It should be noted that images captured using real visual sensors or other real sensor devices are often referred to as "real images," "real images," or "real-world images," while images generated using the aforementioned image generation models (e.g., text-to-image models or image-to-image models) are often referred to as "synthetic images."

[0041] Continuing with the practical example above, different text prompts describing different anomalies can be received, for example, from a user input or from a file of collected text descriptions of the anomalies. Each of the different images and the corresponding text prompts can be processed using the image-to-image model such that the image-to-image model outputs a modified image that reflects the anomaly described in the corresponding text prompt. In this manner, different modified images can be generated that describe different anomalies in different components and / or different industrial facilities.

[0042] The various corrected images (or portions thereof) can be applied as various "positive" ground truth images for training, fine-tuning, and / or validation of an anomaly detection ML model in detecting anomalies of various components in the same or various industrial facilities. Additionally or alternatively, one or more real-world images can be selected from the various images acquired using one or more of the visual sensors, and each can be assigned a supervised label of "no anomaly" (by capturing the component in a normal state). These one or more real-world images can be applied as "negative" ground truth images for training, fine-tuning, and / or validation of an anomaly detection ML model. It is noted that the term "positive" is applied herein to indicate the presence of an anomaly, and the term "negative" is applied herein to indicate the absence of an anomaly. It should also be noted that when different corrected images (or portions thereof) are used to train, fine-tune, and / or validate the same ML model (e.g., an anomaly detection ML model), the different corrected images can be optionally divided into different sets (e.g., a first set for training, a second set for fine-tuning, and / or a third set for validation), where no corrected image is included in more than one of the different sets.

[0043] In some implementations, instead of or in addition to real-world images captured using sensors, realistic synthetic images captured of one or more components (within the industrial facility) that do not display anomalies can also be processed as inputs using the image-to-image model, along with text prompts describing foreseeable anomalies in the industrial facility, to generate corrected images for training, fine-tuning, and / or validation purposes. The realistic synthetic images can be generated using an ML model (e.g., the text-to-image model described above) that performs text-to-image conversion, for example, based on a text string that simply describes the industrial facility (e.g., a "chemical plant").

[0044] By utilizing one or more image generation models to generate synthetic images reflecting anomalies in one or more industrial facilities (or modifying real captured images of one or more industrial facilities without anomalies to reflect anomalies or modifying realistic synthetic images without anomalies), a large number of diverse synthetic images can be generated, each reflecting a corresponding anomaly in one or more industrial facilities, without a human having to create (and later clean up and remove) dangerous anomalies. This can also save effort and time in capturing images or waiting for images to be captured when certain types of anomalies (e.g., anomalies that are not easily spotted) occur. Additionally, training an anomaly detection ML model with diverse and realistic synthetic images can improve the accuracy of the anomaly detection ML model in detecting anomalies.

[0045] Additionally or alternatively, the large number of synthetic images (or a portion thereof) can be applied to fine-tune the anomaly detection ML model, such that the fine-tuned anomaly detection ML model can be applied to detect various anomalies in a particular component or in a particular industrial facility. Additionally or alternatively, a portion of the large number of synthetic images can be applied to validate the anomaly detection ML model before deployment, such that the quality of the anomaly detection ML model in detecting anomalies can be ensured.

[0046] In some implementations, given a total of N (N greater than 1) different synthetic images generated using a text-to-image model and / or an image-to-image model, a first set of different synthetic images may be applied to initially train a first anomaly detection ML model for use in anomaly detection in or around an industrial facility. A second set of different synthetic images may also be applied additionally or alternatively to fine-tune the trained anomaly detection ML model in enhancing anomaly detection for a particular type of anomaly, a particular industrial surrounding environment, and / or a particular type of industrial surrounding environment. A third set of different synthetic images may also be applied additionally or alternatively to validate the trained, and optionally fine-tuned, anomaly detection ML model, for example, before the trained anomaly detection ML model is deployed to detect anomalies in a particular industrial surrounding environment.

[0047] As described herein, in various implementations, the second and / or third set of synthetic images can include (e.g., limited to) modified synthetic images generated using an image-to-image transformation model and generated based on corresponding actual images of the particular industrial surroundings. For example, the second set of synthetic images can include images generated based on processing using an image-to-image transformation model, corresponding actual images of the particular industrial surroundings, and corresponding text prompts. In these and other methods, the fine-tuning and / or validation can be specific to the particular industrial surroundings. Fine-tuning the anomaly detection ML model based on such images can improve the accuracy and / or robustness of the anomaly detection ML model for the particular industrial surroundings. Validating the anomaly detection ML model based on such images ensures that it is valid for the particular industrial surroundings.

[0048] Accordingly, implementations described herein relate to training, fine-tuning, and / or validating anomaly detection machine learning (ML) models to monitor and detect anomalies in one or more components (e.g., liquid tanks) within an industrial facility (e.g., an industrial automation facility) based on text prompts and / or based on images (realistic synthetic images and / or real-world images captured using one or more visual sensors such as cameras) capturing one or more of the components (or portions thereof).

[0049] In various implementations, the anomaly detection ML model can be trained, fine-tuned, and / or validated using a plurality of synthetic images, each reflecting an anomaly present in one or more of the components. For example, the plurality of synthetic images can include a first set of synthetic images generated using a text-to-image ML model (sometimes referred to as a "text-to-image model") based on one or more text strings that each describe an anomaly and the surrounding environment of the industrial facility in which the anomaly is present. Alternatively, or additionally, the plurality of synthetic images can include a second set of synthetic images generated using an image-to-image ML model (sometimes referred to as an "image-to-image model") based on real-world images, each conditioned on a text string (e.g., introducing / adding an anomaly to a respective one of the real-world images).

[0050] It should be noted that in some implementations, the image generation model can function as both a text-to-image model and an image-to-image model. In other words, the image generation model can perform both text-to-image and image-to-image conversion. For example, one or more visual sensors mounted on a mobile robot (e.g., a robot dog) can be used to capture real-world images. By training, fine-tuning, or validating an anomaly detection ML model using a wide range (or selected) of synthetic images (e.g., realistic synthetic images) generated by the text-to-image model and / or the image-to-image model and reflecting corresponding anomalies in the surrounding environment of the industrial facility, respectively, there is no need to wait for and discover time-consuming anomalies (e.g., corrosion of piping or other components) to occur. There is also no need to force dangerous anomalies to occur, such as, for example, oil or other liquids spilling on the ground in the surrounding environment of the industrial facility. Furthermore, the cost and effort associated with capturing appropriate images at the time of anomaly occurrence can be saved or reduced.

[0051] Referring now to the drawings, FIG. 1A illustrates an exemplary environment in which selected aspects of the present disclosure may be implemented, according to various embodiments. FIG. 1B illustrates an exemplary flow chart for performing selected aspects of the present disclosure, according to various embodiments. FIG. 1C illustrates an example of a machine learning (ML) model for performing selected aspects of the present disclosure, according to various embodiments. Referring now to FIG. 1A, an exemplary environment 100 in which various aspects of the present disclosure may be implemented is illustrated. The exemplary environment 100 may be or include an industrial automation facility and may take a variety of forms. For example, the exemplary environment 100 may be designed to implement any number of at least partially automated processes. The industrial automation facility may take the form of a chemical processing plant, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, and the like.

[0052] The exemplary environment 100 can include one or more client devices (e.g., local client devices 103-A and 103-B) operatively coupled with a process automation network 106 in an industrial automation facility. The client devices 103-A and 103-B can be implemented as computers (e.g., laptops, desktops, notebooks), tablets, robots, smart appliances (e.g., smartphones), messaging devices, wearable devices (e.g., watches), or any other applicable devices. The process automation network 106 can be implemented using various wired and / or wireless communication technologies, including, but not limited to, cellular networks such as the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), 3GPP Long Term Evolution ("LTE"), or other wireless protocols designated as 3G, 4G, 5G and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).

[0053] The exemplary environment 100 may further include a mobile robot 101 having or carrying a vision component 1011. The mobile robot 101 may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot that can move within an industrial facility. The vision component 1011 may be a monographic camera, a stereographic camera, a thermal camera, or any other applicable visual sensor for capturing one or more images of one or more specific components of the industrial automation facility (e.g., a liquid tank T or tube 102 that stores or transports a liquid substance). The vision component 1011 may be removably coupled to the mobile robot 101 or may be integrated into the mobile robot 101. In some implementations, the vision component 1011 may change location and / or orientation relative to the mobile robot 101, for example, by rotation or other movement. In addition to the vision component 1011, the mobile robot 101 may include one or more additional vision components for sensing static or dynamic objects moving through the industrial facility and / or capturing images.

[0054] The exemplary environment 100 may further include a server computing device 105 (sometimes simply referred to as a “server device”). The server computing device 105 may include a machine learning (ML) engine 1051, an anomaly detection engine 1052, and an image generation engine 1057. The server computing device 105 may further include or otherwise have access to one or more machine learning (ML) models 1053. The one or more ML models 1053 may include anomaly detection ML models (e.g., M1, M2, and / or M3 in FIG. 1B). Alternatively or additionally, the one or more ML models 1053 may include image generation models (e.g., text-to-image model 111A or image-to-image model 111B in FIG. 1B) that perform text-to-image and / or image-to-image conversion. In some implementations, the one or more ML models 1053 may include a first model for anomaly detection at a first site and a second model for anomaly detection at a second site, where the first site is different from the second site.

[0055] The server computing device 105 can be connected to multiple client devices. The server computing device 105 can communicate with one or more local client devices (e.g., 103-A and 103-B) and / or with one or more remote client devices (not shown). The local client devices 103-A or 103-B can be connected to the server computing device 105 via one or more local area networks (e.g., process automation network 106), and the remote client devices can be connected to the server computing device 105 via one or more wide area networks (e.g., the Internet). The local and remote client devices can be operable by personnel, such as a system integrator, to configure and / or interact with various aspects of the exemplary environment 100.

[0056] In some implementations, the server computing device 105 may include, in addition to the ML engine 1051 and the anomaly detection engine 1052, a database (not shown) that stores information used by the ML engine 1051 and / or the anomaly detection engine 1052 and / or the image generation engine 1057 to implement selected aspects of the present disclosure. In some implementations, the server computing device 105 may include, in addition to the ML engine 1051 and the anomaly detection engine 1052, an image pre-processing engine 1055 that, for example, processes different images (e.g., images captured using different visual sensors) to have the same image dimensions. Various aspects of the server computing device 105, such as the ML engine 1051, the anomaly detection engine 1052, the image generation engine 1057, and / or the image pre-processing engine 1055, may be implemented using any combination of hardware and software.

[0057] In some implementations, the ML engine 1051, the anomaly detection engine 1052, the image preprocessing engine 1055, or the trained ML model 1053 may be implemented across multiple computer systems as part of what is often referred to as a "cloud infrastructure" or simply "cloud." However, this is not required, and for example, in FIG. 1A, the ML engine 1051 is implemented within an industrial facility, e.g., within a single building, or across a single premises of a building or other industrial infrastructure. In such implementations, the ML engine 1051 may be implemented on one or more local computing systems, such as one or more server computers.

[0058] In some implementations, the mobile robot 101 can move through an industrial facility and capture images at one or more spots (e.g., designated spots). For example, as shown in FIG. 1A, the vision component 1011 of the mobile robot 101 can be (but is not required to be) configured at a given pose to capture an image of the liquid tube 102 at the given pose. In some implementations, the images captured by the vision component 1011 can be processed as inputs using an anomaly detection model (after being trained and / or validated) to generate a model output indicating whether an anomaly is present in the component captured in the image.

[0059] For example, the model output may include a classification result predicting the probability that each of a predefined number of anomalies is present in the image. As a non-limiting example, the classification result may be, for example, (oil spill, 0.7), (no oil spill, 0.3). In this non-limiting example, it may be determined that an anomaly corresponding to "oil spill" has been detected from an image having a confidence score of about "0.7" (meeting a predefined confidence score threshold, e.g., 0.6). In response to detection of the anomaly "oil spill", one or more mitigation / remediation actions may be performed, including, but not limited to, generating a warning message (e.g., rendered via client device 103-A, such as 107 in FIG. 1A) or sounding an alarm to notify a person responsible for the oil spill.

[0060] As another non-limiting example, the classification result may be, for example, (oil spill, 0.1), (corrosion, 0.8), (fire, 0.1). In this non-limiting example, it may be determined that an anomaly corresponding to "corrosion" has been detected from an image having a confidence score of about "0.8" (meeting a predefined confidence score threshold, e.g., 0.6). In response to detecting the anomaly "oil spill", one or more mitigation / remediation actions may be performed, including, but not limited to, generating a warning message or emitting a warning sound to notify a person responsible for the corrosion.

[0061] In some implementations, the image captured by the vision component 1011 can be processed as input along with a text prompt (or text string) that introduces an anomaly into the image using an image-to-image model to generate a modified image that reflects the anomaly described or indicated in the text prompt. As a non-limiting example, the image captured by the vision component 1011 can be a real-world image capturing one or more oil tanks in a storage room of an industrial facility. Such a real-world image can be provided to the aforementioned image generation model along with a text prompt (e.g., "add oil spill in the ground") to perform an image-to-image transformation conditioned on the text prompt, where the output of the image generation model (e.g., image-to-image model) corresponds to a modified image (a realistic synthetic image) that modifies the real-world image to show oil spilling in the ground in the storage room in which the one or more oil tanks are located.

[0062] In the non-limiting example above, a supervised label can be generated based on the modified image and / or based on a text prompt (e.g., "add oil spill to the ground"). A human operator trained to recognize anomalies in the surrounding environment of an industrial facility can scan the modified image to determine the supervised label of the anomaly (e.g., "oil spill"). Alternatively, the supervised label (e.g., "oil spill") can be automatically extracted from the text prompt (e.g., "add oil spill to the ground") and / or verified by a human operator by inspecting the modified image. The modified image and supervised label can then be applied to generate training instances (here, "positive" training instances because an anomaly is present) or instances for fine-tuning or validation.

[0063] For example, a corrected image (e.g., a realistic synthetic image showing oil spilling on the ground in a storage room in which one or more oil tanks are stored) can be applied and saved as a training instance input (a "positive" ground truth image) for a training instance, and a supervised label can be applied and saved as a ground truth label for the training instance. The training instance can be applied to train an anomaly detection ML model, where the corrected image can be processed using the anomaly detection ML model to generate a model output (e.g., indicating whether an "oil spill" anomaly was detected in the corrected image and / or a confidence score that an "oil spill" anomaly was detected in the corrected image). The model output can be compared to the ground truth label for "oil spill", and based thereon, one or more weights / parameters of the anomaly detection ML model can be adapted (or fine-tuned, if the corrected image and ground truth label are saved for purposes of fine-tuning the model).

[0064] Additionally or alternatively, the aforementioned real-world image capturing one or more oil tanks in a storage room of an industrial facility can be assigned a supervised label of "no anomaly". The real-world image capturing one or more oil tanks in a storage room of an industrial facility and the supervised label of "no anomaly" can be saved as a "negative" training instance to train an anomaly detection ML model. Additionally or alternatively, the real-world image capturing one or more oil tanks in a storage room of an industrial facility and the supervised label of "no anomaly" can be applied to generate instances for fine-tuning and / or validating additional anomaly detection ML models that are different from the anomaly detection ML model.

[0065] Optionally, more than one real-world image can be captured using the vision component 1011 (and / or using additional sensors). As a non-limiting example, the vision component 1011 can be used to capture a total of 1,000 real-world images. The total of 1,000 real-world images (or a portion thereof) can each be conditioned based on a first text prompt describing the first anomaly in the first type of facility and processed in 100 iterations using the image generation model (image-to-image model 111B) to generate 100,000 synthetic images reflecting the first anomaly in the first type of facility. The total of 1,000 real-world images (or a portion thereof) may each be conditioned based on a second text prompt describing a second anomaly in the first type of facility (or a facility of a second type different from the first type) and processed in 50 iterations using the image generation model (image-to-image model 111B) to generate 50,000 additional synthetic images reflecting a second anomaly in the first (or second) type of facility. Additional synthetic images may be generated in a similar manner. No duplicate description will be repeated herein.

[0066] Note that the number of iterations performed for each text prompt may vary depending on the type of anomaly described or indicated in the corresponding text prompt and / or may also vary depending on the capabilities / use of the anomaly detection ML model being trained (or fine-tuned or validated). In some implementations, the number of iterations performed for a particular text prompt may be increased by identifying that the anomaly described by the particular text prompt is particularly important for training, fine-tuning, or validating the anomaly detection ML model.

[0067] Continuing with the above non-limiting example, in some implementations, 100,000 synthetic images reflecting a first anomaly in a first type of facility can be assigned a first supervised label (e.g., type, name, location, etc.) that identifies the first anomaly, and 50,000 additional synthetic images reflecting a second anomaly in the first (or second) type of facility can be assigned a second supervised label (e.g., type, name, location, etc.) that identifies the second anomaly. The 100,000 synthetic images and the 50,000 additional synthetic images can be split into three groups for training, fine-tuning, and validation purposes.

[0068] For example, the 100,000 synthetic images and the 50,000 additional synthetic images can be divided into a first training group of synthetic images including 80,000 of the 100,000 synthetic images and 40,000 of the 50,000 additional synthetic images, a second fine-tuning group of synthetic images including 15,000 of the 100,000 synthetic images and 8,000 of the 50,000 additional synthetic images, and a third validation group of synthetic images including 5,000 of the 100,000 synthetic images and 2,000 of the 50,000 additional synthetic images. For example, a first training group of synthetic images can be applied to generate "positive" training instances to train an anomaly detection model, a second fine-tuning group of synthetic images can be applied to generate "positive" instances to fine-tune the same anomaly detection model, and a third validation group of synthetic images can be applied to generate "positive" instances to validate the same anomaly detection model.

[0069] It should be noted that when training and fine-tuning (or validation) are applied to different anomaly detection models, the 100,000 synthetic images (or a portion thereof) and the 50,000 additional synthetic images (or a portion thereof) may be applied to train a first anomaly detection model, and the same 100,000 synthetic images (or a portion thereof) and the same 50,000 additional synthetic images may be applied to fine-tune a second anomaly detection model that is different from or separate from the first anomaly detection model.

[0070] Continuing with the above non-limiting example, each of the aforementioned total 1,000 real-world images (and / or additional real-world images) can be assigned a supervised label of "no anomaly". For example, the 1,000 real-world images (and / or additional real-world images) can be divided into a first training group of real-world images including 7,500 real-world images, a second fine-tuning group of real-world images including 2,000 real-world images, and a third validation group of real-world images including 500 real-world images. The first training group can be applied to generate "negative" training instances to train an anomaly detection model, the second fine-tuning group of real-world images can be applied to generate "negative" fine-tuning instances to fine-tune the same anomaly detection model, and the third validation group of real-world images can be applied to generate "negative" validation instances to validate the same anomaly detection model. It should be noted that when training and fine-tuning (or validation) are applied to different anomaly detection models, one image out of the 1,000 real-world images (and / or additional real-world images) may be applied to train (fine-tune or validate) a first model and simultaneously be applied to train (fine-tune or validate) a second model that is different from the first model.

[0071] 1C, the ML models 1053 stored in the ML model database may include a text-to-image model 111A performing text-to-image conversion, an image-to-image model 111B performing image-to-image conversion, an anomaly detection ML model M1 for detecting a first specific type of anomaly, an anomaly detection ML model M2 for detecting a second specific type of anomaly different from the first specific type, and / or an anomaly detection ML model M3 capable of detecting a predefined number of anomalies (e.g., including the first and second specific types of anomalies), and / or other ML models. In some implementations, instead of storing the text-to-image model 111A and the image-to-image model 111B separate or independent from the text-to-image model 111A, the ML model database may include or otherwise access an image generation model performing both text-to-image conversion and image-to-image conversion. It should be noted that the image-to-image conversion may be conditional on a text prompt describing a corresponding anomaly specific to the industrial facility. The anomaly described in the text prompt may vary depending on the type and location of the facility (or other factors). Additionally, different text prompts may describe different anomalies.

[0072] 1B, a text string 108 (sometimes referred to as a "prompt" or "text prompt") may be provided to the text-to-image model 111A. The text string 108 may be, for example, "interior of a chemical plant with oil on the floor." The text string 108 may be processed multiple times / iterations using the text-to-image model 111A. For example, in a first iteration, the text string 108 may be processed as a single input using the text-to-image model 111A to generate an image 120_1 showing a first chemical plant with oil on the floor of the first chemical plant. In a second iteration, the text string 108 may be processed as a single input using the text-to-image model 111A to generate an additional image 120_2 showing a second chemical plant with oil on the floor. The second chemical plant may be the same or different from the first chemical plant. Alternatively, the oil on the floor of the first chemical plant may have a different size, location, or color (or other characteristics) than that of the second chemical plant. Similarly, in the Nth iteration, the text string 108 can be processed as a single input using the text-to-image model 111A to generate an image 120_N showing an Nth chemical plant with oil on the floor. Optionally, the images 120_1, 120_2, ..., 120_N can be stored in an image database 140. The images 120_1, 120_2, ..., 120_N can be inspected (e.g., by a technician, a human operator, etc.) to determine a corresponding supervised label (e.g., "oil on the ground", "oil spill", "oil spill on the left", etc.). Alternatively, the supervised label (e.g., "oil on the ground") can be determined or extracted from the text string 108 (e.g., "interior of a chemical plant with oil on the floor") and / or verified by a human operator.

[0073] Once a supervised label is determined for each of the images 120_1-120_N, multiple training instances can be generated. For example, a first training instance can be generated to include image 120_1 as a first training instance input and include a first supervised label (e.g., "oil spill") as a ground truth label for comparison with a model output of an anomaly detection ML model (e.g., M1 in FIG. 1A) that processes the first training instance as an input. Similarly, a second training instance can be generated to include image 120_2 as a second training instance input and include a second supervised label (e.g., "oil spill") as a ground truth label for comparison with a model output of an anomaly detection ML model (e.g., M1 in FIG. 1A) that processes the second training instance as an input. The Nth training instance can be generated to include image 120_N as the Nth training instance input and include the Nth supervised label (e.g., “oil spill”) as a ground truth label for comparison with a model output of an anomaly detection ML model (e.g., M1 in FIG. 1A ) that processes the Nth training instance as input. Based on the comparison, one or more weights / parameters of the anomaly detection ML model M1 can be adapted / modified / updated.

[0074] In some implementations, instead of or in addition to being applied to generate training instances, the images 120_1-120_N (or a portion thereof) and their corresponding supervised labels may be applied to generate one or more fine-tuning instances. The generated one or more fine-tuning instances (if they do not include images that were used to train the model M1) may be applied to fine-tune the model M1, i.e., fine-tune or adjust one or more of the weights / parameters of one or more layers of the anomaly detection ML model M1. The generated one or more fine-tuning instances (if they include images that were used to train the model M1) may be applied to fine-tune a trained anomaly detection ML model (e.g., M2 in FIG. 1B) different from the anomaly detection ML model M1. For example, the image 120_1 may be selected as a first fine-tuning instance input for the first fine-tuning instance, and the supervised label corresponding to the image 120_1 may be selected as a first ground truth label for the first fine-tuning instance. Image 120_1 can be processed using the trained anomaly detection ML model to generate a model output that indicates, for example, the name and location of the anomaly. The model output can be compared to the supervised labels corresponding to image 120_1 to determine differences, based on which one or more weights of the trained anomaly detection ML model are fine-tuned.

[0075] In some implementations, instead of or in addition to being applied to generate training instances, images 120_1-120_N (or a portion thereof) and their corresponding supervised labels may be applied to generate one or more validation instances (sometimes referred to as "test instances"). For example, image 120_1 may be selected as a first validation instance input for the first validation instance, and the supervised label corresponding to image 120_1 may be selected as a first ground truth label for the first validation instance. The first validation instance may be applied to validate an anomaly detection ML model (e.g., M3 in FIG. 1B) by determining whether a difference between the first ground truth label and a model output of the anomaly detection ML model corresponding to image 120_1 satisfies a difference threshold. If images 120_1-120_N (or a portion thereof) and their corresponding supervised labels can be applied to generate test instances, the performance of an anomaly detection ML model (e.g., M3 in FIG. 1B ) can be evaluated based, for example, on whether the output of model M3 satisfies one or more conditions (e.g., accuracy metrics described in other aspects of this disclosure).

[0076] 1B , in some implementations, a visual sensor may be used to capture an image 110 capturing components of an industrial facility in a normal state (e.g., free of anomalies) (thus, image 110 is a real image). Image 110 may be provided to image-to-image model 111B along with a text string 109 (e.g., “add an oil basin”) that introduces an “oil basin” anomaly to the industrial facility. Image 110 and text string 109 (e.g., “add an oil basin”) may be processed as inputs using image-to-image model 111B to generate a modified image (e.g., real and synthetic, 130_1) that modifies real image 110 to show an oil basin with respect to the components of the industrial facility.

[0077] In some implementations, the image 110 and the text string 109 (e.g., “add an oil reservoir” or “add an oil reservoir near an oil tank”) can be processed as input multiple times / iterations using the image-to-image model 111B, where in a first iteration, a modified image 130_1 is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109, in a second iteration, a modified image 130_2 is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109, and in an Nth iteration, a modified image 130_N is generated by the image-to-image model 111B based on the image 110 conditioned on the text string 109. The corrected images 130_1, 130_2, ..., 130_N (or portions thereof), along with supervised labels (e.g., extracted or determined from the text strings 109), may each be applied to train, fine-tune, and / or validate one or more anomaly detection ML models.

[0078] Additionally or alternatively, a text string different from text string 109 (e.g., "add corrosion to oil tank") can be provided to image-to-image model 111B along with image 110. Such text string and image 110 can be processed multiple iterations using image-to-image model 111B to generate multiple corrected images. Similarly, a supervised label can be created and verified for each of the multiple corrected images, where such supervised labels can be determined, for example, from the text string (e.g., "add corrosion to oil tank" or "add corrosion to left tank").

[0079] It should be noted that in some implementations, image 110 need not be an actual image. For example, image 110 can be a realistic synthetic image, such as text-to-image model 111A, generated based on text strings describing the surroundings of the industrial facility. In these implementations, the text strings processed by text-to-image model 111A do not describe anomalies, and as a result, the generated realistic synthetic image does not show or reflect anomalies other than the surroundings of the industrial facility.

[0080] In some implementations, additionally or alternatively, the image database 140 may further store one or more actual images 150 captured using a physical sensor, each reflecting a corresponding anomaly. In some implementations, additionally or alternatively, the image database 140 may further store one or more actual images 160 captured using a physical sensor, each not reflecting an anomaly. The one or more actual images 150 may be stored in the image database 140 in association with a corresponding supervised label that identifies or indicates the corresponding anomaly. The one or more actual images 160 may each be stored in the image database 140 in association with a supervised label, such as, for example, "no anomaly." The one or more actual images 150 (or selected portions thereof) and the corresponding supervised labels may be applied as "positive" ML instances to train, fine-tune, and / or validate an anomaly detection model, such as model M1 for anomaly detection (or other anomaly detection ML models, such as M2 or M3). One or more actual images 150 (or selected portions thereof) and corresponding supervised labels can be applied as “negative” training instances to train, fine-tune, and / or validate an anomaly detection model, such as model M1 for anomaly detection (or other anomaly detection ML models, such as M2 or M3).

[0081] In some implementations, one or more images 113A can be selected from the image database 140 to train the anomaly detection ML model (e.g., M1), where the one or more images 113A can include images from synthetic images (e.g., images 120_1-120_N) generated using the text-to-image model 111A, images from synthetic images (e.g., images 130_1-130_N) generated using the image-to-image model 111B, images from actual images 150 capturing anomalies, images from actual images 160 not capturing anomalies, and / or other applicable images. For example, the image database 140 can also include synthetic images showing anomalies, where such anomaly-free synthetic images are generated using the image-to-image model 111B based on initial synthetic images (without anomalies) generated using the text-to-image model 111A (receiving text input that simply describes a scenario or industrial facility). The images included in the image database 140 are not limited to those described herein.

[0082] In some implementations, one or more images 113B can be selected from the image database 140 to fine-tune the anomaly detection ML model (e.g., M2), where the one or more images 113B can include images from synthetic images (e.g., images 120_1-120_N) generated using the text-to-image model 111A, images from synthetic images (e.g., images 130_1-130_N) generated using the image-to-image model 111B, images from actual images 150 capturing an anomaly, images from actual images not capturing an anomaly, and / or other applicable images (e.g., images described above).

[0083] In some implementations, one or more images 113C can be selected from the image database 140 to validate the anomaly detection ML model (e.g., M3), where the one or more images 113C can include images from synthetic images (e.g., images 120_1-120_N) generated using the text-to-image model 111A, images from synthetic images (e.g., images 130_1-130_N) generated using the image-to-image model 111B, images from actual images 150 capturing an anomaly, images from actual images not capturing an anomaly, and / or other applicable images (e.g., images described above).

[0084] FIG. 2A illustrates a schematic example of an image generated utilizing the techniques described herein, according to various embodiments. FIG. 2B illustrates a schematic example of another image generated utilizing the techniques described herein, according to various embodiments. FIG. 2C illustrates a schematic example of a further image generated utilizing the techniques described herein, according to various embodiments. Referring now to FIG. 2A, a mobile robot 101 carrying a visual sensor is used to capture images of an industrial facility 200 over a period of time (e.g., weeks, months, or more). From the images captured during the period, a set of images that each capture an anomaly in the industrial facility is selected and labeled, and the set of labeled images is applied to train an anomaly detection ML model 221. From the images captured during the period, an additional set of images that do not capture anomalies can be selected. For example, the additional set of images can include an image 201 (a real-world image) that captures multiple oil tanks (e.g., a portion of Tank A, Tank B, Tank C, and Tank D visible within the area of ​​the industrial facility 200 surrounded by the dashed line). An image 201, along with a text string A (e.g., "adding oil puddles to the ground"), can be processed as input using the image-to-image model 211 one or more times / iterations.

[0085] For example, as shown in FIG. 2A, in a first iteration, a realistic synthetic image 213_A1 can be derived from the model output of the image-to-image model 211 corresponding to the image 201 and the text string A. The realistic synthetic image 213_A1 can be a modified version of the real-world image 201 by adding an oil puddle "O1" in the ground next to the tank A. A supervised label L_A1 (e.g., "oil spill" or "oil puddle") can be determined for the realistic synthetic image 213_A1, and a training instance A1 can be generated to include the realistic synthetic image 213_A1 as a training instance input and the supervised label L_A1 as a ground truth label. An anomaly detection ML model 221 can be trained (or, if trained, validated or fine-tuned) using the training instance A1. For example, a realistic synthetic image 213_A1 can be processed as an input using anomaly detection ML model 221 to generate a model output 223_A1 for comparison with a supervised label L_A1 and to determine a difference between the model output 223_A1 and the supervised label L_A1. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be adapted (or fine-tuned).

[0086] In a second iteration, referring to FIG. 2B, the same image 201 and the same text string A can be processed as input using the image-to-image model 211 to generate a model output from which a realistic synthetic image 213_A2 can be generated / derived. The realistic synthetic image 213_A2 can be a modified version of the real-world image 201 by adding an oil puddle "O2" in the ground next to the tank A. The oil puddle "O2" in the realistic synthetic image 213_A2 may differ in size, shape, and / or location compared to the oil puddle "O1" in the realistic synthetic image 213_A1. A supervised label L_A2 (e.g., "oil spill" or "oil puddle") can be determined for the realistic synthetic image 213_A2, and a training instance A2 can be generated to include the realistic synthetic image 213_A2 as a training instance input and the supervised label L_A2 as a ground truth label. The anomaly detection ML model 221 can be trained (or validated or fine-tuned) using the training instances A2. For example, a realistic synthetic image 213_A2 can be processed as an input using the anomaly detection ML model 221 to generate a model output 223_A2 for comparison with a supervised label L_A2 and to determine a difference between the model output 223_A2 and the supervised label L_A2. Based on the difference, one or more weights / parameters of the anomaly detection ML model 221 can be modified / adapted (or fine-tuned).

[0087] In a third iteration, referring to FIG. 2C , the same image 201 and the same text string A can be processed as input using the image-to-image model 211 to generate a model output from which a realistic synthetic image 213_A3 can be generated / derived. The realistic synthetic image 213_A3 can be a modified version of the real-world image 201 by adding an oil pool "O3" in the ground next to the tank D. The oil pool "O3" in the realistic synthetic image 213_A3 can be of a different size, shape, and / or location compared to the oil pool "O1" in the realistic synthetic image 213_A1 (and / or the oil pool "O2" in the realistic synthetic image 213_A2). A supervised label L_A3 (e.g., "oil spill" or "oil pool") can be determined for the realistic synthetic image 213_A3, and a training instance A3 can be generated to include the realistic synthetic image 213_A3 as a training instance input and the supervised label L_A3 as a ground truth label. The anomaly detection ML model 221 can be trained (or validated or fine-tuned) using the training instance A3. For example, the realistic synthetic image 213_A3 can be processed as an input using the anomaly detection ML model 221 to generate a model output 223_A3 for comparison with the supervised label L_A3 and to determine a difference between the model output 223_A3 and the supervised label L_A3. Based on this difference, one or more weights / parameters of the anomaly detection ML model 221 can be adapted (or fine-tuned, if fine-tuning is performed). It should be noted that as additional iterations are performed, a more realistic synthetic image can be generated. Repetitive descriptions are omitted in this specification.

[0088] In some implementations, the anomaly detection ML model 221 may be an ML model that is trained (or validated or fine-tuned) based on real-world images (if they capture any anomalies) conditioned on text strings describing anomalies in an industrial facility, and / or realistic synthetic images (e.g., image 201) generated / derived from processing real-world images (not capturing anomalies). For example, referring to FIG. 2D, the image-to-image model 211 may process image 201 (e.g., a real-world image) as input along with text string B (e.g., “oil spill”) to generate a model output that may generate / derive realistic synthetic image 213_B1. The realistic synthetic image 213_B1 may be a modified version of real-world image 201 by adding an oil leak (indicated by “L1” in FIG. 2D) in tank B (or other tank).

[0089] A supervised label L_B1 (e.g., "oil spill") can be determined for the realistic synthetic image 213_B1, and a training instance B1 can be generated to include the realistic synthetic image 213_B1 as a training instance input and the supervised label L_B1 as a ground truth label. The anomaly detection ML model 221 can be trained using the training instance B1. For example, the realistic synthetic image 213_B1 can be processed as an input using the anomaly detection ML model 221 to generate a model output 223_B1 for comparison with the supervised label L_B1, thereby determining the difference between the model output 223_B1 and the supervised label L_B1. Based on this difference, one or more weights / parameters of the anomaly detection ML model 221 can be modified / adapted. It should be noted that if additional iterations are performed, a more realistic synthetic image can be generated. Repetitive descriptions are omitted in this specification.

[0090] 3 is a flowchart illustrating an example method 300 for implementing selected aspects of the disclosure, according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a server computing device 105 (and / or additional computing devices, such as client device 103-A), including an ML engine 1051, an anomaly detection engine 1052, and / or a database. Additionally, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0091] In block 302, the system, e.g., via a server such as server computing device 105, may perform, for each of a plurality of text strings each describing (or indicative of) an anomaly and a corresponding industrial facility surroundings, multiple iterations of processing the text strings using the text-to-image model to generate, in each iteration, a corresponding synthetic image. The industrial facility surroundings may include, for example, but are not limited to, one or more of a chemical processing plant, an oil or natural gas refinery, a catalyst plant, a manufacturing facility, an offshore oil platform, or any other applicable facility (one in which one or more at least partially automated processes may be implemented).

[0092] In some implementations, the system may determine a plurality of text strings, each describing an anomaly and a corresponding ambient environment of the industrial facility, prior to performing multiple iterations of text string processing for each of the plurality of text strings. As a non-limiting example, the plurality of text strings may include a first text string "oil spill" stored in association with a first chemical plant (or "oil spill at chemical plant"), a second text string "pipe corrosion" stored in association with a second chemical plant (which may be the same as or different from the first chemical plant), and / or a third text string "broken sensor wire" stored in association with a manufacturing plant.

[0093] The first text string, the second text string, the third text string, and / or the additional text strings may be received from a user input or may be retrieved from a database or file that stores / lists collected text descriptions of various anomalies. For example, the database (or file) may include a first entry that stores the first text string, a second entry that stores the second text string, and a third entry that stores the third text string. In some implementations, the entries in the database may be grouped based on the types of various anomalies, the danger levels of the various anomalies, the industrial facilities in which the various anomalies are respectively foreseeable, and / or historical locations or areas where the various anomalies have been sighted or recorded, etc.

[0094] Continuing with the non-limiting example above, given a first text string of "oil spill at chemical plant", the text-to-image model may be used to process the first text string as an input in a first iteration to generate an image N_11 (a realistic synthetic image) showing a portion of the chemical plant (e.g., on the ground inside the chemical plant) reflecting an oil spill. In this manner, the text-to-image model may be used to repeatedly (e.g., m times / iterations) process the first text string of "oil spill at chemical plant" as an input to generate a set of (e.g., a total of m) images (e.g., image N_11, image N_12, ..., image N_1m), each of which reflects an oil spill within a corresponding chemical plant. In some implementations, the generated set of images may be stored (e.g., locally, at a server device, or on the cloud) in association with the first text string of "oil spill at chemical plant" for subsequent use.

[0095] For example, a first training instance can be generated and stored to include an image N_11 as a training instant input and a first text string "oil spill at chemical plant" (or a keyword "oil spill" extracted from the first text string) as a supervised label, a second training instance can be generated and stored to include an image N_12 as a training instant input and the first text string "oil spill at chemical plant" (or a keyword "oil spill" extracted from the first text string) as a supervised label, and an mth training instance can be generated and stored to include an image N_1m (m is a positive integer equal to or greater than 1) as a training instant input and the first text string "oil spill at chemical plant" (or a keyword "oil spill" extracted from the first text string) as a supervised label.

[0096] Optionally, the first through mth training instances can be inspected (e.g., by an expert or technician trained to identify oil spills) to correct / modify the supervised labels or to remove one or more of the generated training instances (if the image in the corresponding training instance does not adequately reflect an "oil spill") before the first, second, ..., and mth training instances are stored, e.g., in a database. In other words, the first text string can be processed to automatically extract supervised labels for images generated by the text-to-image model that correspond to the first text string, and one or more supervised labels (if necessary) can be verified or modified by an expert before the corresponding training instance including the supervised labels is stored.

[0097] Continuing with the non-limiting example above, given a second text string of "pipe corrosion" (or, alternatively, "pipe corrosion in chemical plant" or "pipe corrosion at manufacturing site", etc.), the second text string can be processed as an input using the text-to-image model in a second iteration to generate an image N_21 (a realistic synthetic image) showing "pipe corrosion" (e.g., on a single pipe). Optionally, the second text string of "pipe corrosion" can be processed as an input repeatedly (e.g., p times / iterations) using the text-to-image model to generate a set of images (e.g., image N_21, image N_22, ..., image N_2p) each reflecting pipe corrosion on one or more pipes (e.g., a total of "p", where p is a positive integer equal to or greater than "1", and "p" may be the same as or different from "m"). Similarly, a total of p training instances may be generated and / or stored in the aforementioned database based on the generated set of images (e.g., image N_21, image N_22, ..., image N_2p) and the second text string "pipe corrosion", and no duplicate description will be given herein.

[0098] In block 304, the system, e.g., via a server such as server computing device 105, may train an anomaly detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images. The anomaly detection ML model may be different from the text-to-image model described above. While the text-to-image model may perform text-to-image conversion, the anomaly detection ML model may be a classifier utilized to detect one or more anomalies (e.g., to predict the probability of each of the predefined anomalies being present in a real-world image within the industrial facility's surrounding environment), and thus, upon detection of one of the predefined anomalies, appropriate and prompt remedial actions may be performed accordingly. In some implementations, the anomaly detection ML model may be utilized in detecting anomalies that are not among the anomalies reflected in the synthetic image (generated by the text-to-image model based on the multiple text strings) and / or in detecting anomalies that are not described by the multiple text strings.

[0099] In some implementations, for each of the generated synthetic images, the system can train the anomaly detection ML model by processing the respective synthetic image as an input with the anomaly detection ML model to generate an output (e.g., a classification output) indicative of a plurality of possibilities that each of a predefined number of anomalies is present in the ambient environment of the industrial facility captured in each synthetic image. The output can be compared to the supervised labels corresponding to each synthetic image to determine differences, and one or more weights / parameters of the anomaly detection ML model can be adjusted or modified based on the determined differences.

[0100] It should be noted that while the generated synthetic images herein are described for use in training an anomaly detection ML model, the generated synthetic images (or a portion thereof) may additionally or alternatively be applied to fine-tune or validate an additional anomaly detection ML model. Alternatively, the generated synthetic images may be split into two or three groups, one used to train the anomaly detection ML model, one used to fine-tune the anomaly detection ML model, and / or one used to validate the anomaly detection ML model, and the group of synthetic images used to train the anomaly detection ML model may have the largest number of synthetic images of the two or three groups split from the generated synthetic images.

[0101] In block 306, the system can provide, for example, via a server such as the server computing device 105, the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility. For example, one or more visual sensors of the particular industrial facility can be utilized to monitor the particular industrial facility through capturing images or videos. Each captured image (or image frame from a video) or selected images (or image frames) of the particular industrial facility can be processed as input using the trained anomaly detection ML model to generate an output indicating the name of the anomaly, the likelihood of the anomaly's presence within the particular industrial facility, and / or the location of the anomaly relative to the captured image (e.g., highlighted by a bounding box). Depending on the output of the trained anomaly detection ML model indicating the presence of an anomaly (e.g., based on the likelihood of meeting a likelihood threshold), one or more remedial actions can be executed. The one or more remedial actions can include, for example, a first remedial action of generating and delivering a warning message, a second remedial action of pausing or stopping one or more industrial processes, a third remedial action of disconnecting power, etc. It should be noted that the one or more remediation actions are not limited to those described herein, and different remediation actions may be performed based on the type (or name), risk level, and / or location (or estimated size or amount based on the aforementioned output) of the detected anomaly.

[0102] In some implementations, the mobile robot may include one or more visual sensors that may capture images of one or more specific components within the surrounding environment of the industrial facility. The one or more specific components may include, by way of non-limiting example, a liquid tank and / or a liquid that the liquid tank carries. The mobile robot may be a quadruped robot (e.g., a robot dog), a wheeled robot, an unmanned aerial vehicle, or any other applicable robot that is mobile within the surrounding environment of the industrial facility.

[0103] The one or more visual sensors can include, for example, a monographic RGB camera, a stereographic camera, a thermal camera, and / or any other applicable visual sensor. Similarly, the images captured by the visual sensor can be RGB images, RGB-D images, thermal images, or any other applicable images. In some implementations, the images can be high resolution images. In some implementations, the visual sensor can be integrated with the mobile robot or can be removably coupled to the mobile robot.

[0104] In some implementations, providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility includes having the trained anomaly detection ML model used in processing real images captured via a visual sensor within or outside the particular industrial facility. In some implementations, the visual sensor can be mounted on a mobile robot (e.g., a robot dog) that moves within the particular industrial facility. In these implementations, having the trained anomaly detection ML model used in processing real images can include having the trained anomaly detection ML model downloaded locally to the mobile robot moving around the particular industrial facility for constant or periodic monitoring and detection of anomalies. By having the trained anomaly detection ML model locally on the mobile robot, images captured using the visual sensor of the mobile robot can be processed using the trained anomaly detection ML model for rapid anomaly detection and / or notification.

[0105] Additionally or alternatively, having the trained anomaly detection ML model used when processing actual images includes having the trained anomaly detection ML model downloaded to one or more on-site computers (e.g., servers). In these implementations, images captured by the visual sensor are provided to one or more on-site computers within a particular industrial facility and processed using the trained anomaly detection ML models downloaded to those servers for anomaly detection.

[0106] In some implementations, for example, two or more anomaly detection ML models can be trained using one or more steps described throughout this disclosure. The two or more anomaly detection ML models can be stored in a remote server (or distributed across one or more remote servers). As a non-limiting example, the two or more anomaly detection ML models can include a first anomaly detection ML model fine-tuned or validated for a first site (e.g., a first industrial facility) and a second anomaly detection ML model fine-tuned or validated for a second site (e.g., a second industrial facility) different from the first site. An actual image captured by a sensor at the first site can be forwarded / sent to the remote server, for example, along with an identifier of the first site. An additional actual image captured by a sensor at the second site can be forwarded / sent to the remote server, for example, along with an identifier of the second site.

[0107] In the above non-limiting example, in response to receiving an actual image having an identifier for a first site, a first anomaly detection ML model fine-tuned or validated for the first site may be accessed and utilized in processing the actual image to thereby determine whether an anomaly is present in the actual image. Similarly, in response to receiving an additional actual image having an identifier for a second site, a second anomaly detection ML model fine-tuned or validated for the second site may be accessed and utilized in processing the additional actual image to thereby determine whether an anomaly is present in the additional actual image. Additionally or alternatively, the trained anomaly detection ML model (and / or one or more additional trained anomaly detection ML models) may be accessed on the cloud, e.g., for use in processing actual images, including putting the trained anomaly detection ML model to use.

[0108] In various implementations, prior to providing a trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, for each of a plurality of actual images of the particular industrial facility, the system processes the actual image and the prompt describing the anomaly using an image-to-image transformation model to generate a corresponding transformed synthetic image (e.g., one that includes an image representation of the anomaly described in the prompt) in block 3051A. In block 3051B, the trained anomaly detection ML model is fine-tuned through further training the anomaly detection ML model based on the transformed synthetic image.

[0109] By fine-tuning the trained anomaly detection ML model using the transformed synthetic images generated from actual images of the particular industrial facility, one or more weights / parameters of the trained anomaly detection ML model may be fine-tuned to improve the performance of the trained anomaly detection ML model in detecting anomalies in the particular industrial facility (and / or in detecting the particular anomaly).

[0110] In some implementations, as described above, the trained anomaly detection ML model can be trained based on a large number of diverse synthetic images (e.g., synthetic images generated by the text-to-image model in multiple iterations in block 302), each reflecting a corresponding anomaly in the surrounding environment of the industrial facility. In these implementations, the anomalies artificially created / introduced in the transformed synthetic images (derived from actual images capturing one or more components or substances in the particular industrial facility) and described in the prompts (e.g., in block 3051A) may have little or no overlap with respect to the anomalies described in the multiple text strings (e.g., in block 302). As a non-limiting example, the multiple text strings can describe anomalies such as "oil spill," "pipe corrosion," and "wire breakage," and the prompts used to align the actual images can describe anomalies specific to the particular industrial facility (e.g., "water leak").

[0111] In various implementations, for each of a plurality of actual images of the particular industrial facility, the system may process the actual image and the prompt describing the anomaly using an image-to-image transformation model to generate a corresponding transformed synthetic image, prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, and validate (or test) the trained anomaly detection ML model based on the transformed synthetic image. In these implementations, providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility is responsive to determining that the validation satisfies one or more conditions.

[0112] In some implementations, validating the trained anomaly detection ML model based on the transformed synthetic image includes determining an accuracy metric of an anomaly prediction made based on an output from the anomaly detection ML model based on processing the transformed synthetic image. The one or more conditions may include, for example, a threshold accuracy metric.

[0113] As a non-limiting example, assume that 50 transformed synthetic images are generated using an image-to-image transformation model, and the 50 transformed synthetic images are used to validate a trained anomaly detection ML model. In this non-limiting example, if 42 of the 50 outputs of the trained anomaly detection ML model (corresponding to the 50 transformed synthetic images) match the supervised labels of the corresponding 50 transformed synthetic images, resulting in an accuracy index of approximately 0.84 (meeting a threshold accuracy index, e.g., 0.8), the trained anomaly detection ML model can be considered validated and is ready to be deployed for anomaly monitoring and detection in a particular industrial facility.

[0114] The methods described herein or in other aspects of the present disclosure may be executed via one or more processors on-board the mobile robot or may be included in one or more computing devices that are separate from and not attached to the mobile robot, in which case images of the real world captured by the mobile robot to monitor for anomalies may be transmitted by the mobile robot and, after being transmitted by the mobile robot, identified by the one or more computing devices for subsequent use and processing.

[0115] 4 is a block diagram of an exemplary computing device 410 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 410 typically includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with the computing device 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0116] The user interface input devices 422 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 410 or a communications network.

[0117] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 410 to a user or to another machine or computing device.

[0118] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic to perform selected aspects of the method of FIG. 3, as well as logic to implement the various components shown in FIGS. 1-2.

[0119] These software modules are typically executed by the processor 414 alone or in combination with other processors. The memory 425 used in the storage subsystem 424 can include multiple memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 can provide persistent storage for program and data files and can include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation can be stored by the file storage subsystem 426 in the storage subsystem 424 or on other machines accessible by the processor 414.

[0120] The bus subsystem 412 provides a mechanism for allowing the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus subsystem 412 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0121] The computing device 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 410 shown in Figure 4 is intended only as a specific example to illustrate some implementations. Many other configurations of the computing device 410 are possible, having more or fewer components than the computing device shown in Figure 4. [Explanation of symbols]

[0122] 100 Example Environment 101 Mobile Robot 102 Liquid tube, tube 103-A Local client device 103-B Local client device 105 Server Computing Devices 106 Process Automation Network 108 Text Strings 109 Text Strings 110 images 111A Text to Image Model 111B Image-to-Image Model 113A Images 113B Images 113C Images 120_1 Image 120_2 Image 120_N Image 130_1 Image 130_2 Image 130_N Image 140 Image Database 150 Actual Images 160 Actual Images 200 Industrial Facilities 201 images 211 Image-to-Image Model 213_A1 Realistic composite image 213_A2 Realistic composite image 213_A3 Realistic composite image 213_B1 Realistic composite image 221 Anomaly Detection ML Model 223_A1 Model output 223_A2 Model output 223_A3 Model output 223_B1 Model output 300 ways 410 Computing Devices 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Storage Subsystem 425 Memory Subsystem 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM) 1011 Visual Components 1051 Machine Learning (ML) Engine 1052 Anomaly Detection Engine 1053 Machine Learning (ML) Models 1055 Image Pre-processing Engine 1057 Image Generation Engine

Claims

1. 1. A method implemented by one or more processors, comprising: For each of a plurality of text strings, each describing an anomaly and a corresponding ambient environment of the industrial facility, performing a number of iterations of processing the text string, in each iteration using a text-to-image model to generate a corresponding synthetic image; training an anomaly-detection machine learning (ML) model using the generated synthetic images and corresponding supervised labels of the generated synthetic images; providing the trained anomaly detection ML model for use in anomaly detection within a particular industrial facility; A method comprising:

2. Prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, For each of a plurality of actual images of the particular industrial facility, processing the actual image and the prompt describing the anomaly using an image-to-image transformation model to generate a corresponding transformed synthetic image; fine-tuning the trained anomaly detection ML model by further training the anomaly detection ML model based on the transformed synthetic image; The method of claim 1, further comprising:

3. 2. Fine-tuning the trained anomaly detection ML model, The method of claim 2 , comprising fine-tuning weights of one or more layers of the trained anomaly detection ML model.

4. Prior to providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, For each of a plurality of actual images of the particular industrial facility, processing the actual image and the prompt describing the anomaly using an image-to-image transformation model to generate a corresponding transformed synthetic image; validating the trained anomaly detection ML model based on the transformed synthetic image; Further comprising:

13. The method of claim 1 , wherein providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility occurs in response to a determination that the validation satisfies one or more conditions.

5. Validating the trained anomaly detection ML model based on the transformed synthetic image comprises determining an accuracy measure of anomaly predictions made based on output from the anomaly detection ML model based on processing the transformed synthetic image; The method of claim 4 , wherein determining that the validation satisfies one or more conditions comprises determining that the accuracy indicator satisfies a threshold accuracy indicator.

6. 2. The method of claim 1, wherein the plurality of text strings includes a first text string describing a first anomaly in an environment of a first industrial facility and a second text string describing a second anomaly in an environment of the first industrial facility.

7. 2. The method of claim 1, wherein the plurality of text strings includes one text string describing an anomaly in an environment of an industrial facility and an additional text string describing the anomaly in an environment of an additional industrial facility.

8. The method of claim 1 , wherein the corresponding supervised labels of the generated synthetic images are automatically determined based on at least the plurality of text strings describing the anomaly.

9. Providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility, 2. The method of claim 1, comprising: causing the trained anomaly detection ML model to be used in processing actual images captured via a vision sensor in the particular industrial facility.

10. The method further comprises using the trained anomaly detection ML model when processing the actual image, the step of using the trained anomaly detection ML model when processing the actual image comprising: processing one of the actual images using the trained anomaly detection ML model to generate a model output indicative of whether an anomaly is present in one or more of the components; responsive to the model output indicating an anomaly exists in one or more of the components, causing one or more remedial actions to be performed; 10. The method of claim 9, comprising:

11. 10. The method of claim 9, wherein the visual sensor is mounted on a mobile robot and the actual image is captured by the mobile robot at a designated location within the particular industrial facility.

12. The method of claim 10 , wherein the one or more remedial actions include a warning message that alerts to detection of the anomaly in one or more of the components at the specified location.

13. 1. A method implemented by one or more processors, comprising: For each of a plurality of actual images of a particular industrial facility, processing the actual images and prompts describing anomalies associated with the particular industrial facility using an image-to-image transformation model to generate a corresponding transformed synthetic image; fine-tuning the trained anomaly detection ML model by further training the trained anomaly detection ML model based on the transformed synthetic image; A method comprising:

14. 14. The method of claim 13, wherein the trained anomaly detection ML model is pre-trained based on one or more actual images, each capturing a corresponding anomaly.

15. The method of claim 14 , wherein the corresponding anomalies are captured using a visual sensor within the particular industrial facility.

16. 14. The method of claim 13, wherein the trained anomaly detection ML model is pre-trained based on a plurality of synthetic images each depicting a realistic scene of a corresponding anomaly in an industrial facility environment, and the plurality of synthetic images are generated using a text-to-image model based on one or more text strings each describing an anomaly.

17. 1. A method implemented by one or more processors, comprising: For each of a plurality of actual images of a particular industrial facility, processing the actual images and prompts describing anomalies associated with the particular industrial facility using an image-to-image transformation model to generate a corresponding transformed synthetic image; Validating a trained anomaly detection ML model based on the transformed synthetic image. A method comprising:

18. determining whether the verification satisfies one or more conditions; responsive to a determination that the validation satisfies the one or more conditions, providing the trained anomaly detection ML model for use in anomaly detection within the particular industrial facility.

20. The method of claim 17, further comprising:

19. 20. The method of claim 18, wherein validating the trained anomaly detection ML model based on the transformed synthetic image comprises determining an accuracy metric of anomaly predictions made based on output from the anomaly detection ML model based on processing the transformed synthetic image, and the one or more conditions include a threshold accuracy metric.

20. 20. The method of claim 17, wherein the trained anomaly detection ML model is pre-trained based on a plurality of synthetic images each depicting a realistic scene of a corresponding anomaly in an industrial facility environment, the plurality of synthetic images being generated using a text-to-image model based on one or more text strings each describing an anomaly.

21. 20. The method of claim 17, wherein the trained anomaly detection ML model is pre-trained based on one or more actual images, each capturing a corresponding anomaly in an industrial facility environment.

Images