System and method for validation and virtual commissioning of artificial intelligence-based automation systems

Abstract

In a virtual commissioning of an industrial automation system, an industrial system or process is simulated by a simulation engine (118) utilizing a digital twin (112) of the industrial system or process. A virtual data acquisition system (120) utilizes a simulation snapshot (128) to generate data communicated as a model input (140) to a virtual inference system (124) having an AI model (138) deployed on it. The virtual inference system (124) processes the model input (140) using the AI model (138) to generate a model output (142) communicated to a virtual automation controller (126). The virtual automation controller (126) processes the model output (142) using automation logic (116) running on it to generate control data (144) communicated to the simulation engine (118). A closed-loop behavior of the automation logic (116) responsive to interaction with the AI model (138) is thereby validated. Communications between the virtual system components (118, 120, 124, 126) are based on data models and / or runtime interfaces that match communications between corresponding physical system components (102, 104, 106, 108) of the industrial automation system.

Description

SYSTEM AND METHOD FOR VALIDATION AND VIRTUAL COMMISSIONING OF ARTIFICIAL INTELLIGENCE-BASED AUTOMATION SYSTEMSTECHNICAL FIELD

[0001] The present disclosure relates to the field of industrial automation, and in particular to automation systems including artificial intelligence (Al) applications. Specifically, the present disclosure relates to systems, methods, and computer program products for validation and virtual commissioning of Al-based industrial automation systems.BACKGROUND

[0002] Modern industrial automation can involve a wide variety of use-cases and solutions around Al and machine learning technologies. However, the design and deployment of Al-based solutions involves a significant effort on the part of the automation system engineer, integrator, and user to ensure the quality, accuracy, and reliability expected of such industrial applications. Moreover, deployment may often be the first step, since an automation system that includes an Al model that is trained on a dataset may not perform as expected in the real world. Multiple iterations of the system design, Al model, and automation logic may be needed as an industrial system or process (e.g., a production line) collects new data and the operators discover operational issues or poor accuracy of the Al application.SUMMARY

[0003] Aspects of the present disclosure provide a methodology for validation and virtual commissioning of components of an Al-based industrial automation system that can significantly reduce the overall upfront effort needed to develop and deploy an industrial Al application and ensure superior performance of the Al application at a first iteration of deployment.

[0004] A first aspect of the disclosure provides a computer-implemented method for virtual commissioning of an industrial automation system. The method comprises simulating an industrial system or process by a simulation engine utilizing a digital twin of the industrial system or process togenerate a simulation snapshot representing a simulated state of the industrial system or process. The method further comprises utilizing the simulation snapshot by a virtual data acquisition system to generate data communicated as a model input to a virtual inference system having deployed thereon an Al model. The virtual inference system includes a virtualization of a physical inference system of the industrial automation system. The method further comprises processing the model input by the virtual inference system using the Al model to generate a model output communicated to a virtual automation controller. The virtual automation controller includes a virtualization of a physical automation controller of the industrial automation system. The method further comprises processing the model output using automation logic running on the virtual automation controller to generate control data communicated to the simulation engine. A closed-loop behavior of the automation logic responsive to interaction with the Al model is thereby validated. According to the method, virtual system components are integrated into a virtual commissioning dataflow such that communications therebetween are based on data models and / or runtime interfaces that match communications between corresponding physical system components of the industrial automation system.

[0005] Further aspects of the disclosure are directed to a computing system and to a computer program product including instructions executable by a processor to carry out such a method.

[0006] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0008] FIG. 1 is a schematic diagram illustrating a system for virtual commissioning of an industrial automation system, according to disclosed embodiments.

[0009] FIG. 2 is a flowchart illustrating a process flow for virtual commissioning of an industrialautomation system for a first example use-case of visual quality inspection in a production line.

[0010] FIG. 3 is a flowchart illustrating a simplified process flow for virtual commissioning of an industrial automation system for a second example use-case of part classification in a flexible production line.

[0011] FIG. 4 is a flowchart illustrating a simplified process flow for virtual commissioning of an industrial automation system for a third example use-case of anomaly detection in an industrial process.DETAILED DESCRIPTION

[0012] Automation system components, such as controllers, edge devices, etc., can be virtually commissioned using software tools provided by the vendor, which allow these system components to be virtually tested and validated before actual hardware is purchased. Such software tools alone, however, may be insufficient to test and validate the performance of an Al -based automation system, unless, for example, physical sensor integration is already finalized, such that real-world data can be used for virtual commissioning. Many commercial simulation or gaming engines can offer combined physics and rendering capabilities, which are often used to generate datasets for Al model training as well as for virtual testing of robot kinematics. However, these engines are much more focused on physics interactions and rendering, and do not readily integrate with industrial automation and commissioning systems. Additionally, these engines do not typically generate commissioning artifacts, such as robot code, and do not simulate any deployment steps that would be required for commissioning of an Al-based automation system.

[0013] The disclosed methodology realizes an improvement over traditional methods of virtual commissioning by providing an integration of virtual system components, which include virtual representations of the physical system components such as the industrial system or process machinery, the data acquisition system, the Al inference system, and the automation controller, into a virtual commissioning dataflow that mirrors the dataflow between the physical system components. This enables Al-in-the-loop testing, whereby the behavior or interaction of the automation system with the Al application can be validated in a virtualized environment. The disclosed methodology can allow an automation system user to quickly plan, test, and iterate on the integration of new components (hardware, software, and Al model) into an industrial system or process before any physical systemcomponents are modified.

[0014] According to the disclosed methodology for virtual commissioning, an industrial system or process is simulated by a simulation engine utilizing a digital twin of the industrial system or process. A virtual data acquisition system utilizes a simulation snapshot to generate data communicated as a model input to a virtual inference system having an Al model, such as a deep neural network, deployed on it. The virtual data acquisition system may simulate a data acquisition process to generate data with real-world data properties, such as random noise or artifacts, from the simulation snapshots. The Al model may be previously trained and tested on a training dataset that is created, at least in part, using data generated by the virtual data acquisition system utilizing simulation snapshots generated by the simulation engine during a dataset generation process. The virtual inference system processes the model input using the Al model to generate a model output communicated to a virtual automation controller. The virtual automation controller processes the model output using automation logic running on it to generate control data communicated to the simulation engine. A closed-loop behavior of the automation logic responsive to interaction with the Al model is thereby validated. When integrating components in the virtual space, it is ensured that communications between the virtual system components are based on data models and / or runtime software interfaces that match communications between corresponding physical system components of the industrial automation system. This provides the ability to seamlessly migrate from using virtual system components in simulation to their real-world counterparts.

[0015] Through the above-described steps, components of an Al-based automation system, such as the data acquisition system, the Al inference system, the automation controller, etc., can be virtually commissioned, i.e., verified ahead of time before any modifications of the physical industrial system or process are performed. Once deployed, the system can be continuously improved by augmenting the model training dataset with real- world data. Future changes to any part of the system (e.g., automation logic, data acquisition system, Al model, etc.) can also be planned and verified virtually.

[0016] Turning now to the drawings, FIG. 1 illustrates a system 100 for virtual commissioning of an industrial automation system according to one or more embodiments. The shown system 100 comprises a number of physical system components, engineering components and virtual system components, which define a commissioning workflow. The components are primarily described in connection with a specific industrial use-case, namely visual quality inspection, which involves thetask of defect detection and localization in parts manufactured on a production line (see FIG. 2). However, aspects of the disclosed methodology can be applied to a variety of other industrial usecases, non-limiting examples of which have been subsequently outlined (see FIG. 3-4).

[0017] Referring to FIG. 1, the physical system components may include field components of the industrial automation system, such as industrial system or process machinery 102, a data acquisition system 104, an inference system 106 and an automation controller 108. For example, in the visual quality inspection use-case, the industrial system or process machinery 102 may comprise an inspection cell for a production line, e.g., including a robot or a conveyor system. The data acquisition system 104 may comprise one or more cameras mounted on a robot or a fixed frame. The inference system 106 may comprise an edge device or other computing device running an Al inference engine configured to draw a conclusion or “inference” from the data acquired by the data acquisition system 104, utilizing a trained Al model. The automation controller 108 may comprise a programmable logic controller (PLC) or other type of real-time controller that can operate on automation logic (e.g., a PLC program) for generating real-time control data for the industrial system or process machinery 102 based on the inference result. The automation logic in the controller 108 may also be responsible for triggering or time- synchronizing the data acquisition by the data acquisition system 104.

[0018] The engineering components may include components that can be used for generating and modifying software artifacts such as digital twins, automation logic, Al model, training dataset, etc. In the shown embodiment, the engineering components include a digital twin creator 110, an automation engineering system 114 and a model training server 134.

[0019] The digital twin creator 110 may be used to create a digital twin 112 of the industrial system or process. For example, in the use-case of visual quality inspection, the digital twin 112 may include a digital twin of the production line (e.g., plant model), or specifically, of the inspection cell of the production line (e.g., robot model), and may additionally include a digital twin of a part (e.g., a CAD model) being manufactured on the production line. The digital twin creator 110 may also be used to create a digital twin of the data acquisition system based on information such as number, placement and orientation of cameras, design of fixture to mount cameras, and camera settings such as exposure, color mode, resolution, analog gain, field of view, depth of focus, etc.

[0020] The automation engineering system 114 may be used to generate the automation logic 116(e.g., PLC program) for the automation controller 108. For example, in a robot assisted visual quality inspection, the automation logic 116 may be generated by computing an optimal trajectory and PLC triggers based on defined waypoints. The automation logic 116 may be iteratively tested and modified to fix errors, e.g., incorrect path or collision, data flow or connectivity issues, triggering or synchronization issues, etc., during virtual commissioning phase before deployment on the physical automation controller 108.

[0021] The model training server 134 may be utilized to train an Al model 138 (e.g., a deep neural network) for the inference system 106 utilizing a training dataset 136. For visual quality inspection, the training dataset 136 may include images of both nominal and defective parts. According to disclosed embodiments, the training dataset 136 can be generated completely or at least substantially synthetically based on an integration of virtual system components as described below. The trained Al model 138 may be iteratively optimized to achieve a desired performance during the virtual commissioning phase before deployment on the physical inference system 106.

[0022] The virtual system components may include components that define representations of corresponding physical system components in a virtual environment. In the shown embodiment, the virtual system components include an industrial system or process simulation engine 118, a virtual data acquisition system 120, a virtual inference system 124 and a virtual automation controller 126, which define virtual representations of the physical system components 102, 104, 106 and 108 respectively.

[0023] The industrial system or process simulation engine 118 may execute a simulation model of the industrial system or process utilizing the digital twin 112 to generate simulation snapshots 128, where each simulation snapshot 128 represents a simulated state of the industrial system or process. The implementation of the simulation engine 118 may depend on the specific use-case. For example, in the visual quality inspection use-case, the simulation engine 118 may comprise a commercially available manufacturing simulation software, such as Process Simulate™ developed by Siemens Industry Software, Inc., among others. The simulation snapshots 128 in this case may represent images of parts on the production line, for example, based on a CAD model of the parts, simulated camera position, field of view, etc. In other embodiments, the simulation engine 118 may include a physicsbased or process-based simulation engine and the simulation snapshots 128 may represent time series data, e.g., including process sensor data, vibration sensor data, etc. In various examples, the simulationsnapshots 128 may be synchronous, e.g., based on a clock-cycle, or asynchronous, e.g., based on events.

[0024] The virtual data acquisition system (hereinafter, “vDAQ system”) 120 may be configured to utilize simulation snapshots 128 to generate data suitable for use in downstream processes, such as training and inference. Often, the simulation snapshots generated via commercial simulation engines may not represent realistic data suitable for Al applications. The creation and incorporation of the vDAQ system 120 can enable simulation of the data acquisition process to generate data with real- world data properties, such as random noise or artifacts, from the simulation snapshots 128. In one embodiment, the vDAQ system 120 may utilize a digital twin of the physical data acquisition system 104 that includes one or more sensors (e.g., cameras, microphones, etc.) for transforming simulation snapshots 128 to generate virtual sensor data simulating physical sensor data. For visual quality inspection, the vDAQ system 120 may incorporate ray-tracing capabilities to generate photo-realistic images from the simulation snapshots 128. In an example implementation, the vDAQ system 120 may include a rendering engine such as Omniverse™ developed by Nvidia Corporation, which can be used to generate a realistic representation of lighting and camera conditions in the inspection cell.

[0025] The vDAQ system 120 may be integrated with the virtual automation controller 126 in the virtual commissioning dataflow such that it is triggered or time-synchronized by automation logic 116 running on the virtual automaton controller 126. The virtual automation controller 126 may include a virtualization of the physical automation controller 108. The virtual automation controller 126 may include the automation logic 116 (generated by the automation engineering system 114) running within a virtual machine that virtualizes the hardware of the physical automation controller 108. The virtual machine may be managed, for example, by a real-time hypervisor in a server that can support the execution of deterministic real-time control loops. In a non-limiting example, the virtual automation controller 126 may include a SIMATIC S7-1500™ virtual PLC (vPLC), developed by Siemens AG. In certain use-cases, the vDAQ system functionality may be implemented by the virtual automation controller or vPLC 126, which may gather data from virtual sensors, such as process sensors (e.g., thermocouples, pressure sensors, etc.). Virtualizing the data acquisition process including the vPLC integration can ensure a realistic synthetic data result and allow for data acquisition to be performed in the virtual world before commissioning of the automation system.

[0026] In some embodiments, for creating the training dataset 136, prior to data generation by thevDAQ system 120, random modifications may be introduced to create synthetic data 132 from the simulation snapshots 128. The random modifications may represent, for example, randomized defects or anomalies in the training dataset 136. In one embodiment, as shown, a defect randomizer 122 may be used to generate modified snapshots 130 by injecting randomized defects into the simulation snapshots 128. The defects may be generated using a data-driven approach, by randomizing parameterized defect data from an existing or expected defect dataset (e.g., using generative algorithms), and programmatically injected into the simulation snapshots 128 to create the modified snapshots 130. The generated dataset may roughly match the distribution of the source dataset but may be sufficiently different so as to provide additional data for model training. An example of the above-mentioned approach for visual quality inspection is disclosed in the International Patent Application No. PCT / US2023 / 027309 filed by the present Applicant, which is incorporated by reference herein in its entirety. Briefly described, this approach involves generating synthetic defects by randomizing parameterized geometries (e.g., polylines) of existing defects obtained from a digitized defect library and carrying out a 3D superimposition of the synthetic defects at randomized positions on a CAD model of the part, to generate modified snapshots that include virtual representation of defective parts. The modified snapshots 130 may be communicated to the vDAQ system 120 to generate synthetic data 132 for the training dataset 136.

[0027] In some embodiments, such as when the simulation engine 118 provides a physics-based or Monte Carlo simulation, a certain degree of randomization, including defects or anomalies, may be already present in the simulation snapshots 128. For example, in the case of visual quality inspection on painted parts, a physics-based simulation of the paint process may be used to simulate paint coverage and quality across the part. Inputs to this process can then be varied using a randomization engine to introduce quality issues in the simulated part. This approach can be computationally expensive, but for some processes, may result in the production of more realistic training data. In such cases, the defect randomizer 122 may not be necessary. Furthermore, certain Al models, for example, Al models for some anomaly detection use-cases, can be trained purely on nominal data (i.e., without defects), obviating the need for the defect randomizer 122.

[0028] In some embodiments, the random modifications introduced during model training may also represent randomized environmental and / or operational parameters of the industrial system or process, to provide sufficient variance in the training dataset 136 for ensuring robustness of the Al model 138. For example, for visual quality inspection, environmental parameters can include background color,ambient lighting, dimensions of cell / conveyor, etc., while operational parameters can include part orientation, part translation, conveyor speed, camera vibration (e.g., due to robot movement), etc. In an example implementation, a custom tool built on the Omniverse™ Kit and Replicator can be used to randomize the environmental and / or operational parameters for each defective part and render photorealistic images as captured by cameras in the inspection cell. Distinct from traditional domain randomization, the proposed environmental and / or operational randomization may be carried out while keeping camera parameters (e.g., placement, angle, field of view, etc.) constrained using the digital twin of the data acquisition system.

[0029] Synthetic data 132 generated by the vDAQ system 120 may be used to create a training dataset 136 to train an initial version of the Al model 138. In some embodiments, the Al model 138 may be trained in a supervised learning process based on labels attached to nominal and defective data samples in the training dataset 136. For training an Al model for defect detection, synthetic images of defective parts may be labeled, automatically or manually, by attaching annotations (e.g., bounding boxes) to localize defects. The Al model 138 may comprise a deep neural network (DNN), e.g., including a convolutional neural network architecture. The supervised learning process may include inputting the synthetic images in the training dataset 136 (which may define a DNN input layer) and comparing a predicted output of the model (which may define a DNN output layer) against the annotations as ground truth to compute a loss function. The learning process may comprise iteratively adjusting learnable model parameters (e.g., DNN weights and biases), for example, using a method of gradient descent, over a set of training images, such that the loss function is minimized. The trained Al model 138 may be tested for accuracy on testing data samples in the training dataset 136. In some embodiments, such as in anomaly detection, the Al model 138 may be trained in an unsupervised learning process using a training dataset 136 with only nominal data.

[0030] The trained Al model 138 may be initially deployed on the virtual inference system 124, which includes a virtualization of the physical inference system 106. The virtual inference system 124 may comprise an Al inference engine running within a virtual machine that virtualizes the hardware of the physical inference system 106. In a non-limiting example, the virtual inference system 106 may include an Industrial Edge Virtual Device (IEVD), which includes a virtualization of an Industrial Edge Device (IED), such as SIMATIC IPC™, developed by Siemens AG.

[0031] The virtual commissioning dataflow may be utilized to execute control loops as follows. Thesimulation engine 118 may generate a simulation snapshot 128 representing a simulated state of the industrial system or process. The vDAQ system 120 may utilize the simulation snapshot 128 to generate data that defines a model input 140 for the virtual inference system 124. The vDAQ system 120 may be triggered to carry out data acquisition, in a synchronous or asynchronous manner, by the automation logic 116 running on the virtual automaton controller 126. The vDAQ system 120 may communicate the model input 140 to the virtual inference system 124 having deployed thereon the trained Al model 138. The virtual inference system 124 may process the model input 140 using the trained Al model 138 to generate a model output 142 and communicate the model output 142 to the virtual automation controller 126. The virtual automation controller 126 may process the model output 142 using the automation logic 116 to generate control data / automation data 144, which may be then communicated to the simulation engine 118.

[0032] The control loops may be executed in real-time. In some embodiments, the control loops may be executed faster or slower than real-time according to computational resources available. By executing one or multiple control loops, a closed-loop behavior of the automation logic 116 responsive to interaction with the Al model 138 can be validated. For validating performance, integration of the virtual system components, such as the simulation engine 118, the vDAQ system 120, the virtual inference system 124 and the virtual automation controller 126, is an essential consideration. According to the disclosed methodology, the virtual system components may be integrated into the virtual commissioning dataflow such that communications between them are based on data models (e.g., protocols, PLC tag tables, etc.) and / or runtime interfaces (e.g., software interfaces such as APIs, dataflow connectors, network interfaces, etc.) that match communications between corresponding physical system components of the industrial automation system.

[0033] The above-described integration technique provides the ability to seamlessly migrate from using virtual system components in simulation to their real-world counterparts. This technique can enable a vPLC that executes automation logic with a specific protocol and tag tables to be directly replaced by the real-world PLC executing the same program. A similar technique can be used to replace the virtual inference system (e.g., Siemens IEVD) in the test system with a physical inference system (e.g., Siemens IED) to verify system performance on real hardware by transferring the applications and configurations used on the virtual inference system to the physical inference system. Additionally, the vDAQ system can be seamlessly replaced with the real sensor or vision system, as the vDAQ should utilize the same communication protocol as the real data acquisition system.

[0034] For an illustration of the above-described integration technique, consider the non-limiting example of an Al based vision system performing visual quality inspection, in which: the virtual automation controller is a SIMATIC S7-1500™ PLCSim Advanced™ instance, running in the Siemens PLCSim Advanced™ virtualization environment; the production simulation engine is provided by Process Simulate™ running in Cyclic Event Evaluation mode; the vDAQ system includes a virtualized camera running in Omniverse™; and the Al inference engine is running on a Siemens IEVD. Integration between PLCSim Advanced™ and Process Simulate™ may be provided over existing protocols already integrated in software (either the PLCSim Advanced™ API directly from Process Simulate™, or by connection over TCP). This communication can utilize PLC tag tables and addresses, such that no changes are required to PLC communication when transferring to the physical system. The integration between Process Simulate™ and Omniverse™ may be provided utilizing both, the respective vendor specific Tecnomatix™ API, and the Omniverse™ Kit API. When the simulated camera is triggered by the vPLC, the corresponding signal change may be read by Process Simulate™ and the current scene may be exported over the Tecnomatix™ API to Omniverse™. This scene can then be read and rendered in Omniverse™ using the Omniverse™ Kit API. The output from Omniverse™ may be an image or camera feed that can then be streamed to the IEvD over standard vision protocols like RTSP. The IEvD may then receive the simulated image as if from a real camera and communicates results back to the vPLC over a virtualized SIMATIC S7™ connection, completing the control loop.

[0035] The above-described integration technique can enable validation of closed-loop and realtime performance of the virtualized automation system components based on the satisfaction of defined performance criteria. For example, in visual quality inspection, performance criteria for the automation logic 116 may include generating correct triggers or time-synchronization signals for the vDAQ system 120, generating optimum robot trajectories avoiding collisions, generating correct control actions in response to the model output 142 of virtual inference system 124, e.g., actuating appropriate servo motors to divert a defective part on the production line to a reject bin, and so on. The validation process may include iteratively modifying or debugging the automation logic 116 by the automation engineering system 114 and testing it within the virtual commissioning workflow until a control performance criterion, such as mentioned above, is met.

[0036] Additionally, performance of the virtual inference system 124 can also be validated against defined Al model performance criteria, such as model accuracy (e.g., Fl score). The validation processmay include iteratively optimizing the Al model 138 (e.g., by tuning model architecture or hyperparameters) and / or augmenting the training dataset 136 with additional synthetic (or real) data and testing it within the virtual commissioning workflow, until an Al model performance criterion is met.

[0037] Furthermore, the virtual commissioning workflow can include iteratively adjusting sensor configuration parameters of the vDAQ system 120, until an optimum sensor configuration is reached based on a data acquisition performance objective. For example, in visual quality inspection, the placement of a virtual camera may be iteratively adjusted and optimized in an automated manner based on an objective defined by visible part surface coverage (to minimize blind spots) in the acquired synthetic images, as disclosed in the International Patent Application No. PCT / US2022 / 039894 filed by the present Applicant, which is incorporated by reference herein in its entirety.

[0038] After the validation process, the automation logic 116 may be deployed on the physical automation controller 108 and the Al model 138 may be deployed on the physical inference system 106. In some embodiments, after the initial deployment, the system can be continuously improved by augmenting the model training dataset with real-world data, and any future changes to the inspection system can be planned and verified virtually.

[0039] For example, subsequent to deployment, control loops may be executed on the real -world system as follows. The automation controller 108 may use the deployed automation logic 116 to generate trigger or synchronization signals 146 for the data acquisition system 104. Based on the trigger or synchronization signals 146, the data acquisition system 104 may acquire real -world data, e.g., via physical sensors, which may be communicated as model input 148 to the inference system 106. The inference system 106 may process the model input 148 using the deployed Al model 138 to generate a real-world inference, which may be communicated as model output 150 to the automation controller 108. The automation controller 108 may process the model output 150 using the deployed automation logic 116 to generate control data / automation data 152, which may be communicated to the industrial system or process machinery 102.

[0040] Based on real-world performance of the system, changes to any of the system components may be needed, e.g., modifications to thePLC program. In such cases, the changes to software artifactscan be directly re-imported to the virtual commissioning dataflow system and the modified virtual system can be again evaluated and virtually commissioned in simulation.

[0041] In one embodiment, as shown, the training dataset 136 may be augmented based on real- world data 154 acquired by the data acquisition system 104 and real -world inference results 156 generated by the inference system 106 from the real-world data 154. For example, data may be augmented around inaccurate real -world inference results 156 based on operator / expert-determined ground truths. The data augmentation may also include, for example, injection of random modifications, as described above. The Al model 138 may then be retrained on the augmented training dataset 136. The retrained Al model 138 may be deployed once again on the virtual inference system 124 to validate a closed-loop performance in a virtualized environment, as described above, before deployment on the physical inference system 106.

[0042] Referring now to FIG. 2, an exemplary virtual commissioning process flow 200 for a visual inspection use-case is illustrated. The shown process flow 200 broadly includes the following phases, namely, setup 210, dataset generation 220, model training 230, deployment on virtualized environment (virtual deployment) 240 and deployment on physical system (physical deployment) 250. The different phases and the activity blocks within these phases are depicted in an example sequence for the purpose of illustration and not meant to be construed in a limiting sense. Furthermore, one or more of the phases may be executed over multiple iterations in the process flow 200.

[0043] The setup phase 210 may include the following activity blocks. Block 212 may include creation of a digital twin of the production line. This may involve, for example, creating a layout or plant model, importing a robot model, and attaching a virtual camera to the robot, among others. The digital twin may also include respective CAD models of the parts for which quality inspection is to be carried out. Block 214 may include integrating the vPLC with simulation engine and the vDAQ system, such that dataflow can be established using data models (e.g., protocols, PLC tag tables) and runtime interfaces (e.g., APIs, network interfaces) that mirror dataflow between the corresponding physical system components. Block 216 may include generating a PLC program for the robot. This may include, for example, setting waypoints or targets for robot motion and using it to calculate an optimal robot trajectory and PLC triggers. The generated PLC program may then be deployed on the vPLC. In some embodiments, the setup may be modified one or more times subsequent to the physical deployment 250 based on an evaluation of the Al model performance in the physical system.

[0044] The dataset generation phase 220 may include the following activity blocks. Block 222 may include running a simulation model using the digital twin and the PLC program. At this stage, simulation snapshots may be generated based on trigger signals from the vPLC. Furthermore, at this stage, the robot path may be checked (e.g., for collisions) and the dataflow / connectivity between the virtual system components verified. Block 224 may include rendering virtual camera images of parts in the inspection cell by the vDAQ system utilizing the simulation snapshots, to generate a dataset of nominal parts. In some embodiments, the dataset generation may include performing environmental and / or operational randomization by the vDAQ system. At block 226A, randomized defects may be injected into the simulation snapshots, for example, via a data-driven defect randomizer, to generate modified snapshots. Block 226B may include rendering virtual camera images of defective parts by the vDAQ system utilizing the modified snapshots (e.g., carrying out environmental / operational randomization), to generate a labeled dataset of defective parts. At block 228, the datasets of nominal and defective parts may be combined to create the training dataset.

[0045] The model training phase 230 may include the following activity blocks. Block 232 may include training an Al model for defect detection, i.e., detection and localization of a defect in an image of a part. At this stage, one or more Al models (e.g., DNNs) may be trained in a supervised learning process using labeled training data samples from the training dataset. For this purpose, the training dataset may be split into training data samples and testing data samples. At block 234, the one or more trained Al models may be tested using the testing data samples and evaluated on metrics, such as model accuracy. Block 236 may include fine-tuning the one or more Al models as necessary, e.g., by updating the Al model architecture and / or hyperparameters. Multiple Al models may be evaluated and compared in the initial model training phase 230.

[0046] The virtual deployment phase 240 may include the following activity blocks. Block 242 may include deploying a trained Al model on the virtual inference system, such as an IEVD. At block 244, the virtual inference system may be integrated into the virtual commissioning dataflow (using described integration techniques) and a performance is validated, e.g., against defined performance criteria, based on interaction between vPLC and the virtual inference system in a closed (feedback) control loop. Block 246 may include optimizing the Al model and / or modifying the PLC program based on a satisfaction of the defined performance criteria.

[0047] The physical deployment phase 250 may include the following activity blocks. Block 252may include deploying the software artifacts, which are generated or modified by the virtual commissioning workflow, on the physical system. This may include uploading the PLC program on the physical automation controller and the trained Al model on the physical inference system. At block 254, real-world camera images captured by the vision system may be saved, to create a dataset of real parts. At block 256, the performance of the Al model may be evaluated based on a comparison between real-world inference results generated by the physical inference system and ground truth information (e.g., determined by an operator / expert) associated with the real -world camera images. Based on the evaluation, the model training phase 230 may be repeated by augmenting the training dataset with the real-world camera images labeled using the ground truth information. The dataset augmentation may be carried out particularly around inaccurate real-world inference results based on comparison with the ground truth information. Additionally, or alternately, the setup phase 210 may be repeated by modifying the digital twin and / or the PLC program.

[0048] FIG. 3 illustrates a virtual commissioning process flow 300 for a second example use-case of part classification for a flexible production line that can manufacture multiple different parts on the same line. The shown process flow 300 is simplified. For example, it omits many of the detailed activities pertaining to virtual system integration, dataset generation, model training that were previously described. Activity blocks and their sequence depicted in FIG. 3 are for the purpose of illustration and not meant to be construed in a limiting sense.

[0049] Block 302 may include creation of a digital twin of the production line. The digital twin may include a plant model of the production line. The digital twin may further incorporate a model of the vision system including one or more virtual cameras. Block 304 may include picking out CAD models of a random set of parts, e.g., from an existing CAD folder, which may be used to generate a number of simulation snapshots by a simulation engine. At block 306, photorealistic images may be rendered by a vDAQ system from the simulation snapshots using camera settings specified in the digital twin of the vision system. The vDAQ system may be triggered by automation logic running on a vPLC integrated with the vDAQ system / simulation engine. The rendered images may depict nominal part images of a number of different parts, which may be used to create a synthetic training dataset. The images may be rendered based on environmental and / or operational randomization by the vDAQ system to create a large synthetic dataset. Block 308 may include training a classification Al model using the training dataset. The Al model may be trained in a supervised learning process using classification labels attached to data samples in the training dataset.

[0050] At block 310, the trained Al model may be deployed on a virtual inference system. Block 312 may involve validation of a system performance, e.g., against defined performance criteria, based on execution of closed (feedback) control loops in a virtualized environment. The performance criteria can include, for example, generating correct triggers, generating correct control signals for actuating appropriate servo motors for sorting parts according to classification, accuracy of classification, etc. The validation process may include iteratively modifying the PLC program and / or optimizing the Al model until the defined performance criteria are met. At block 314, as an additional measure for improving model accuracy, the training dataset may be augmented by adding new part models to the CAD folder.

[0051] Subsequently, at block 316, the software artifacts generated or modified by the virtual commissioning workflow, such as the PLC program and the trained Al model, may be deployed on the respective physical system components, namely the physical automation controller and the physical inference system. Block 318 may include capturing real- world camera images by the vision system and generating real-world inference results on classification by the physical inference system using the deployed Al model. Block 320 may involve generating a labeled dataset based on ground truth information (e.g., determined by an operator / expert) associated with the real- world camera images. The labeled dataset may be utilized to retrain the classification Al model. Retraining using real-world camera images may be performed, for example, dependent on the evaluation of an accuracy of the Al model during physical deployment based on the ground truth information.

[0052] FIG. 4 illustrates a virtual commissioning process flow 400 for a third example use-case of anomaly detection in an industrial process. The shown process flow 400 is simplified. For example, it omits many of the detailed activities pertaining to virtual system integration, dataset generation, model training that were previously described. Activity blocks and their sequence depicted in FIG. 4 are for the purpose of illustration and not meant to be construed in a limiting sense.

[0053] Block 402 may include creation of a digital twin of an industrial motor. The digital twin may include a CAD model of the industrial motor. The digital twin may also incorporate virtual process sensors, e.g., thermocouples, pressure sensors, etc., for measuring simulated process parameters. Block 404 may involve running a physics-based simulation model of the industrial motor by a simulation engine utilizing the digital twin. The simulation model may include, for example, a high- fidelity mathematical model or a reduced order model thereof capturing a behavior of the industrialmotor. Block 406 may include integration of a vPLC running a PLC program with the simulation engine. The vPLC may be connected to the virtual process sensors to generate time series process data. The time series process data may be generated based on a fixed acquisition interval implemented by PLC triggers / sync. signals. The time series process data may be used to create a training dataset. Block 408 may involve training an Al model for anomaly detection using the training dataset. The Al model may include, for example, a recurrent neural network (RNN) architecture. The Al model may be trained in an unsupervised learning process using unlabeled time series process data in the training dataset to learn a normal or nominal behavior of the industrial motor.

[0054] At block 410, the trained Al model may be deployed on a virtual inference system. Block 412 may involve validation of a system performance, e.g., against defined performance criteria, based on execution of closed (feedback) control loops in a virtualized environment. The performance criteria can include, for example, implementing appropriate drive train control in case an anomaly is detected, accuracy of anomaly inference, etc. The validation process may include iteratively modifying the PLC program and / or optimizing the Al model until the defined performance criteria are met. Additional steps may be carried out for improving model accuracy by modifying the training dataset. For example, at block 414, the data acquisition characteristics, e.g., arrangement of the virtual sensors, acquisition interval, etc., may be modified. Additionally, or alternately, at block 416, the physicsbased simulation model may be modified, for example, to introduce anomalies in the behavior of the industrial motor.

[0055] Subsequently, at block 418, the software artifacts generated or modified by the virtual commissioning workflow, such as the PLC program and the trained Al model, may be deployed on the respective physical system components, namely the physical automation controller and the physical inference system. Block 420 may include capturing real- world process data from the physical process sensors and generating real-world inference results on anomalies by the physical inference system using the deployed Al model. The real-world process data may be utilized to retrain the anomaly detection Al model. Retraining using real-world process data may be performed, for example, dependent on the evaluation of an accuracy of the Al model during physical deployment based on ground truth information (e.g., determined by an operator / expert) associated with the real- world process data.

[0056] The embodiments of the present disclosure may be implemented with any combination ofhardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non- transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0057] The computer readable storage medium can include a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0058] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the disclosure to accomplish the same objectives. Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for virtual commissioning of an industrial automation system, comprising: simulating an industrial system or process by a simulation engine (118) utilizing a digital twin (112) of the industrial system or process to generate a simulation snapshot (128) representing a simulated state of the industrial system or process, utilizing the simulation snapshot (128) by a virtual data acquisition system (120) to generate data communicated as a model input (140) to a virtual inference system (124) having deployed thereon an Al model (138), the virtual inference system (124) including a virtualization of a physical inference system (106) of the industrial automation system, processing the model input (140) by the virtual inference system (124) using the Al model (138) to generate a model output (142) communicated to a virtual automation controller (126) that includes a virtualization of a physical automation controller (108) of the industrial automation system, and processing the model output (142) using automation logic (116) running on the virtual automation controller (126) to generate control data (144) communicated to the simulation engine (118), to thereby validate a closed-loop behavior of the automation logic (116) responsive to interaction with the Al model (138), wherein virtual system components (118, 120, 124, 126) are integrated into a virtual commissioning dataflow such that communications therebetween are based on data models and / or runtime interfaces that match communications between corresponding physical system components (102, 104, 106, 108) of the industrial automation system.

2. The method according to claim 1, comprising training and testing the Al model (138) prior to deployment on a training dataset (136) that is created, at least in part, using synthetic data (132) generated by the virtual data acquisition system (120) utilizing simulation snapshots (128) generated by the simulation engine (118) during a dataset generation process.

3. The method according to claim 2, wherein the training dataset (136) is created, at least in part, by introducing random modifications to generate the synthetic data (132) from the simulation snapshots (128) during the dataset generation process.

4. The method according to claim 3, wherein the random modifications represent randomized defects or anomalies in the training dataset.

5. The method according to claim 4, wherein the randomized defects are generated by randomizing parameterized defect data from an existing or expected defect dataset and injected into the simulation snapshots (128).

6. The method according to any of claims 3 to 5, wherein the random modifications represent randomized environmental and / or operational parameters of the industrial system or process in the training dataset.

7. The method according to any of claims 1 to 6, wherein the virtual data acquisition system (120) is integrated into the virtual commissioning dataflow so as to be triggered or time-synchronized by the automation logic (116) running on the virtual automation controller (126).

8. The method according to any of claims 1 to 7, wherein the virtual data acquisition system (120) utilizes a digital twin of a physical data acquisition system (104) comprising one or more sensors for transforming simulation snapshots (128) obtained from the simulation engine (118) to generate virtual sensor data simulating physical sensor data.

9. The method according to claim 8, comprising iteratively adjusting sensor configuration parameters of the virtual data acquisition system (120) until an optimum sensor configuration is reached based on a data acquisition performance objective.

10. The method according to any of claims 1 to 9, wherein the validation comprises iteratively modifying the automation logic (116) until a control performance criterion is met.

11. The method according to any of claims 1 to 10, wherein the validation comprises optimizing the Al model (138) and / or augmenting a training dataset (136) until a model performance criterion is met.

12. The method according to any of claims 1 to 11, comprising, subsequent to the validation, deploying the automation logic (116) on the physical automation controller (108) and deploying the Al model (138) on the physical inference system (106).

13. The method according to claim 12, further comprising, in a subsequent step: augmenting the training dataset (136) based on real-world data (154) acquired by a physical data acquisition system (104) and inference results (156) generated by the physical inference system (106) from the real-world data (154), retraining the Al model (138) on the augmented training dataset (136), and deploying the retrained Al model (138) on the virtual inference system (124) integrated into the virtual commissioning dataflow, to validate the closed-loop behavior of the automation logic responsive (116) to interaction with the retrained Al model (138).

14. A non-transitory computer-readable storage medium encoded with instructions that, when processed by a computing system, configure the computing system to perform the method according to any of claims 1 to 13.

15. A computing system comprising: one or more processors, and non-transitory memory storing instructions executable by the one or more processors to carry out a method according to any of claims 1 to 13.

Images