System and method for providing assurance of artificial intelligence (AI)-driven automation system

Abstract

A method (600) and a system (102) for providing assurance of AI-Driven Automation system are disclosed. One or more processors (110), of the system (102), receive, an AI model intended to solve a real-time problem, in a pre-deployment state of the AI model and verify (202) that the AI model solves the real-time problem, by performing at least one of a plurality of analyses on the AI model. The one or more processors (110) validates (302) that the AI model solves the real-time problem, based on one or more predefined techniques and perform, a safety and risk operation on the AI model to determine a health status of the AI model, while deploying the AI model and assess (502) one or more operations of the AI model to solve the real-time problem, after performing the safety and risk operation, in a post-deployment of the AI model in the system.

Description

[0001] Siemens Aktiengesellschaft

[0002] 1

[0003] SYSTEM AND METHOD FOR PROVIDING ASSURANCE OF ARTIFICIAL INTELLIGENCE (AI)-DRIVEN AUTOMATION SYSTEM

[0004] DESCRIPTION

[0005] The present invention generally relates to Artificial Intelligence (AI)-Driven automation systems and more particularly relates to a system and a method for providing assurance of the ALDriven automation systems.

[0006] With the extensive adoption of state-of-the-art sensing, communication, information management, and Al technologies, the adoption of data-driven automation and Al in the industry is more accelerated today than ever. With the evolving automation, there is a need for different, more expansive definition of assurance in terms of verification, validation, safety and risk analysis, governance and regulatory constraints, and ethical considerations, especially for safety critical and mission critical industrial systems. Specifically, Al assurance refers to the process of ensuring reliability, safety, and performance of artificial intelligence (Al) systems. Apart from the structural and architectural aspects of traditional software, AL based systems have multiple additional aspects that make it challenging to verify and validate them, and in consequence quantify their safety, risk, biases, etc.

[0007] There is a lack of a comprehensive framework for assessing the robustness and conducting thorough dataset analysis in the current market. Existing solutions lack the depth or breadth needed to fully evaluate the robustness of models or datasets, leaving gaps in the analysis process.

[0008] Additionally, the increasing complexity of Al models, such as deep neural networks, has made it challenging to understand the decision-making process behind their predictions. This is crucial in especially high impact use cases, such as healthcare, where there is a need to be able to trust the decisions of the model. One of the current solutions include Explainable Al (XAI) methods which address this issue by providing clear and interpretable explanations for model decisions. However, existing XAI solutions often lack user-friendliness and compatibility with popular machine learning model architectures.

[0009] Accordingly, there is a need for providing a holistic Al assurance framework along with interpretability mechanisms which address at least the aforementioned problems.

[0010] According to one embodiment of the present invention, a system for providing assurance of Artificial Intelligence (AI)-Driven Automation system is disclosed. The system comprises a memory and one or more processors communicatively coupled to the memory. The memory comprises programmable instructions which, when executed by the one or more processors, cause the one or more processors to perform one or more of to receive an Al model intended Siemens Aktiengesellschaft

[0011] 2 to solve a real-time problem, in a pre-deployment state of the Al model. The real-time problem is defined as challenge(s) that have an impact on business operation or decision making, which the Al model is intended resolve when deployed in the AI-Driven automation system. Further, the one or more processors verify that the Al model solves the real-time problem, based on a plurality of analyses. Furthermore, the one or more processors validate, upon successful verification, that the Al model solves the real-time problem, based on one or more predefined techniques. Furthermore, the one or more processors perform, upon successful validation, a safety and risk operation on the Al model to determine a health status of the Al model while deploying the Al model in the system. The safety and risk operation is defined as an operation in which different measures are taken to ensure the reliable and controlled operation of the Al model. Additionally, the one or more processors assess one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation, in a post-deployment state of the Al model in the system.

[0012] Advantageously, the aforementioned system is a modular, expandable, generalizable, and comprehensive Al assurance framework which may be used to assess and assure industrial Al-based automation solutions with respect to domain, application, and / or technology specific requirements.

[0013] Further, for validating that the Al model solves the real-time problem, the one or more processors are configured to initially conceptually validate that the Al model is intended to solve the real-time problem, based on a plurality of factors, wherein the plurality of factors include at least one of a theoretical alignment factor, a clarity and coherence factor, an internal consistency factor, and an external consistency factor. Further, the one or more processors are configured to subsequently operationally validate that the Al model solves the real-time problem, when deployed in the system, based on a plurality of factors, wherein the plurality of factors includes at least one of a measurement tools and metrics factor, a statistical validation factor, an empirical correspondence factor, and an inter-disciplinary factor.

[0014] Further, for verifying and validating the Al model, the one or more processors are further configured to quantify a plurality of degrees of testing performed on the Al model, to verify and validate that the Al model solves the real-time problem, by a pre-defined technique, wherein the pre-defined technique is a coverage calculation module technique.

[0015] Furthermore, for performing the safety and risk operation, the one or more processors are configured to receive a plurality of training and test data, by the one or more processors, during the deployment of the Al model in the system. Additionally, the one or more processors are configured to perform a technical risk operation to determine the health status of the Al model. Siemens Aktiengesellschaft

[0016] 3

[0017] Furthermore, for performing the safety and risk operation, the one or more processors are further configured to receive field data, during the deployment of the Al model in the system. Additionally, the one or more processors are further configured to compare the field data with the plurality of training and test data to perform an operational risk operation to determine the health status of the Al model.

[0018] Furthermore, for assessing the one or more operations of the Al model, the one or more processors are configured to compare a plurality of training and test data with field data, when the Al model is deployed in the system. Further, the one or more processors are configured to assess the one or more operations of the Al model to determine at least one optimal feature in the Al model, a fairness of the Al model, a problem in the Al model, and solving the problem in the Al model, based on the comparison. The one or more operations comprise an explainability and interpretability operation to determine the optimal feature in the Al model, a bias detection and mitigation operation to determine the fairness of the Al model, and a model diagnosis and accountability operation to determine the problem in the Al model and solving the determined problem in Al model. The explainability and interpretability operation support plurality of data, for example, XGBoost, SVM, Linear Regression, Logistic Regression, LightGBM, Decision Tree, Random Forest, Catboost, Autikeras, YOLOv8.

[0019] According to another embodiment of the present invention, a cloud computing environment is disclosed. The cloud computing environment includes a system, as discussed throughout the present invention, and a client device communicatively coupled to the system via a network. The system is configured to perform one or more methods, as discussed throughout the present invention.

[0020] According to another embodiment of the present invention, a computer-program product is disclosed. The computer-program product has machine-readable instructions stored therein, that when executed by one or more processors, cause the one or more processors to perform one or more methods, as discussed throughout the present invention.

[0021] According to another embodiment of the present invention, a non-transitory computer- readable medium is disclosed. The non-transitory computer-readable medium is encoded with executable instructions, that when executed by one or more processors, cause the one or more processors to perform one or more methods, as discussed throughout the present invention.

[0022] According to another embodiment of the present invention, a computer-implemented method for providing assurance of AI-Driven Automation system is disclosed. The method comprises receiving, by one or more processors, an Al model intended to solve a real-time problem, in a pre-deployment state of the Al model. Further, the method comprises verifying, by the one or more processors, that the Al model solves the real-time problem, by performing at least one Siemens Aktiengesellschaft

[0023] 4 of a plurality of analyses on the Al model. Furthermore, the method comprises validating, by the one or more processors, upon successful verification, that the Al model solves the realtime problem, based on one or more predefined techniques. Furthermore, the method comprises performing, by the one or more processors, upon successful validation, a safety and risk operation on the Al model to determine a health status of the Al model, while deploying the Al model in the system. Furthermore, the method comprises assessing, by the one or more processors, one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation, in a post-deployment of the Al model in the system.

[0024] To further clarify the advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail with the accompanying drawings.

[0025] These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0026] FIG. 1 illustrates a block diagram of an environment comprising a system for providing assurance of Artificial Intelligence (AI)-Driven Automation system, according to an embodiment of the present invention;

[0027] FIG. 2 illustrates a block diagram for a verification of the Al model, according to an embodiment of the present disclosure;

[0028] FIG. 3 illustrates a block diagram for a validation of the Al model, according to an embodiment of the present disclosure;

[0029] FIG. 4 illustrates a block diagram for a safety and risk operation of the Al model, according to an embodiment of the present disclosure;

[0030] FIG. 5 illustrates a block diagram for assessment of one or more operations of the Al model, according to an embodiment of the present disclosure; and

[0031] FIG. 6 illustrates an exemplary process flow depicting a method for providing the assurance of the Artificial Intelligence (AI)-Driven automation system, according to an embodiment of the present invention. Siemens Aktiengesellschaft

[0032] 5

[0033] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present invention. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0034] For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates.

[0035] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the invention and are not intended to be restrictive thereof.

[0036] Reference throughout this specification to “an aspect”, “another aspect” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrase “in an embodiment”, “in another embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0037] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by “comprises... a” does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.

[0038] FIG. 1 illustrates a block diagram of an environment 100 comprising a system 102 for providing assurance of Artificial Intelligence (AI)-Driven Automation system, according to Siemens Aktiengesellschaft

[0039] 6 an embodiment of the present invention. The system 102 is communicatively coupled to a client device 104 via a network 106.

[0040] The client device 104 comprises a user interface 108. In an embodiment, the user interface 108 enables a user associated with the client device 104 to transmit and receive data to the system 102. In an embodiment, the user interface 108 is associated with a display unit that enables a user to view data associated with the system 102. In an embodiment, the user interface 108 is a graphical user interface (GUI).

[0041] In an exemplary embodiment, the client device 104 includes a laptop computer, a desktop computer, a smartphone, and the like. Further, the network 106 includes a wireless network or a wired network. For example, the network 106 corresponds to cellular networks or mobile networks, such as 3G, 4G, 5G, pre-5G, 6G network, or any other wireless communication network.

[0042] The system 102 for providing assurance of the Artificial Intelligence (AI)-Driven automation system includes one or more processors 110 and a memory 112. As a non-limiting example, the one or more processors 110 are a single processing unit or a set of units each including multiple computing units. The one or more processors 110 are implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions (computer-readable instructions) stored in the memory 112. Among other capabilities, the one or more processors 110 are configured to fetch and execute computer-readable instructions and data stored in the memory 112. The one or more processors 110 include one or a plurality of processors. The plurality of processors is further implemented as a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The plurality of processors controls the processing of the input data in accordance with a predefined operating rule or an artificial intelligence (Al) model stored in the memory 112. The predefined operating rule or the Al model is provided through training or learning.

[0043] The one or more processors 110 are disposed in communication with one or more input / output (VO) devices via an Input / Output (I / O) interface 114. The I / O interface 114 employs communication code-division multiple access (CDMA), high-speed packet access (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMax, or the like, etc. In another embodiment of the present invention, the I / O interface 114 employs ethemet, industrial wireless Local Area Network (LAN), Process Field Bus (PROFIBUS), Actuator Sensor (AS) Interface, and the like. Siemens Aktiengesellschaft

[0044] 7

[0045] In some embodiments, the memory 112 is communicatively coupled to the one or more processors 110. The memory 112 is configured to store programmable-instructions executable by the one or more processors 110. In one embodiment, the memory 112 communicates via a bus within the system 102. The memory 112 includes, but is not limited to, a non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like. In one example, the memory includes a cache or random-access memory (RAM) for the one or more processors 110. In alternative examples, the memory 112 is separate from the one or more processors 110 such as a cache memory of a processor, the system memory, or other memory. The memory 112 is an external storage device or a database for storing data. The memory 112 is operable to store programmable instructions executable by the one or more processors 110 which cause the one or more processors 110 to perform one or more of method(s) as discussed later in the subsequent paragraphs. The functions, acts, or tasks illustrated in the figures or described are performed by the programmed processor for executing the instructions stored in the memory 112. The functions, acts, or tasks are independent of the particular type of instructions set, storage media, processor, or processing strategy and may be performed by software, hardware, integrated circuits, firmware, microcode and the like, operating alone or in combination. Likewise, processing strategies include multiprocessing, multitasking, parallel processing, and the like.

[0046] The memory 112 further includes a database 116 to store data. Further, the memory 112 includes an operating system 118 for performing one or more tasks of the system 102, as performed by a generic operating system in the communications domain. In one embodiment, the database 116 is configured to store the information as required by the one or more processors 110 to perform one or more functions for providing assurance of the Artificial Intelligence (AI)-Driven automation system.

[0047] In an embodiment, the system 102 is a cloud-based system, such as a cloud server. In such an embodiment, the functionality of the system 102 may be provided by the cloud server, and the details provided with respect to the system 102 would be applicable for the cloud server. In another embodiment, the system 102 is hosted on the client device 104. In an embodiment, the system 102 is provided in a distributed manner, in that, one or more components and / or functionalities of the system 102 are provided through the client device 104, and one or more components and / or functionalities of the system 102 are provided through a cloud-based unit, such as, a cloud storage or a cloud-based server. In a non-limiting example, the database 116 may be provided through the cloud storage and the one or more processors 110 may be integrated with an electronic device (such as the client device 104). Siemens Aktiengesellschaft

[0048] 8

[0049] Further, the present invention also contemplates a computer-program product that includes instructions or receives and executes instructions responsive to a propagated signal. Further, the instructions may be transmitted or received over the network 106 via a communication port or interface or using a bus (not shown). The communication port or interface may be a part of the one or more processors 110 or may be a separate component. The communication port may be created in software or may be a physical connection in hardware. The communication port may be configured to connect with the network 106, external media, the display, or any other components in the system 102. The connection with the network 106 may be a physical connection, such as a wired ethernet connection, or may be established wirelessly. Likewise, the additional connections with other components of the system 102 may be physical or may be established wirelessly. The network 106 may alternatively be directly connected to the bus. For the sake of brevity, the architecture, and standard operations of the memory 112 and the one or more processors 110 are not discussed in detail.

[0050] Therefore, the present invention contemplates a cloud-computing environment, having the system 102 and the client device 104. The client device 104 is communicatively coupled with the system 102 via the network 106. The system 102 is configured to perform a method for providing assurance of the Artificial Intelligence (AI)-Driven Automation system.

[0051] In an embodiment, the computer-program product, having machine-readable instructions stored therein, when executed by one or more processors 110, cause the one or more processors 110 to perform a method for providing assurance of the Artificial Intelligence (AI)-Driven Automation system. The details on the method(s) performed by the one or more processors 110 have been elaborated in subsequent paragraphs at least with reference to FIG. 2 to 5.

[0052] Further, the present invention also contemplates a non-transitory computer-readable medium encoded with executable instructions. The executable instructions, when executed by the one or more processors 110, cause the one or more processors 110 to perform the method for providing assurance of the Artificial Intelligence (AI)-Driven Automation system. The details on the method(s) performed by the one or more processors 110 have been elaborated in subsequent paragraphs at least with reference to FIG. 2 to 5. Examples of computer-readable mediums include nonvolatile, hard-coded type mediums such as read-only memories (ROMs) or erasable, electrically programmable read-only memories (EEPROMs), and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read-only memories (CD-ROMs) or digital versatile disks (DVDs).

[0053] FIG. 2 illustrates a block diagram for a verification 202 of the Al model, according to an embodiment of the present disclosure. FIG. 3 illustrates a block diagram for a validation 302 Siemens Aktiengesellschaft

[0054] 9 of the Al model, according to an embodiment of the present disclosure. FIG. 4 illustrates a block diagram for a safety and risk operation 402 of the Al model, according to an embodiment of the present disclosure. FIG. 5 illustrates a block diagram for assessment of one or more operations of the Al model, according to an embodiment of the present disclosure.

[0055] In an embodiment, FIGs. 2 to 5 are explained in conjunction with FIG. 1 for the sake of brevity.

[0056] The one or more processors 110 are configured to receive an Al model intended to solve a real-time problem including challenges that have an impact on business operation or decision making, which the Al model is intended to resolve when deployed in the AI-Driven automation system. The one or more processors 110 are configured to receive the Al model in the pre deployment state of the Al model.

[0057] The one or more processors 110 are configured to verify 202 that the Al model solves the real-time problem, based on a plurality of analyses. In an embodiment, the plurality of analyses are stored in the database 116. The plurality of analyses include at least one of a functional analysis and testing 204, a plurality of standard requirements analysis and testing 206, a repeatability analysis 208, a behaviour analysis / version comparison 210, a cross- validation analysis 212, and a continuous monitoring analysis 214.

[0058] The functional analysis and testing 204 is performed to verify examination of a behavior of the Al model, identification of components of the Al model, understanding interaction of the identified components, and evaluation performance of the Al model, etc. The plurality of standard requirement analysis and testing 206 is performed to verify that the output of the Al model complies with the regulatory compliance. The repeatability analysis 208 is performed to verify different behaviors and failure modes related to the Al model. Further, the version comparison 210 is performed on the Al model to verify that an updated version of the Al model depicts exact performance / behaviour expected during the updation of the Al model when deployed in the system. The cross-validation analysis 212 ensures verification of the performance of the Al model based on different subsets of data. The continuous monitoring analysis 214 verifies the performance, reliability of the Al model in the pre-deployment state by tracking and analyzing various metrics, such as, an accuracy of the Al model. Further, at least one analysis from the plurality of analyses is selectively chosen based on at least one of a plurality of parameters, by a user. In an embodiment, the plurality of parameters includes a type of problem to be solved by the Al model, a type of data, a type of decision frequency, and a label of safety criticality of an application by the Al model. The type of problems includes classification problems, regression problems, optimization problems, insight mining problems, etc. The type of data includes tabular data, image data, natural language data, Siemens Aktiengesellschaft

[0059] 10 signals (time series), etc. The type of decision frequency includes low frequency decisions and high frequency decisions. The level of safety criticality of the application includes low criticality, medium criticality, high criticality, etc. In another embodiment, the at least one analysis is selectively chosen in a predetermined manner by the one or more processors 110.

[0060] The one or more processors 110 are configured to validate 302, upon successful verification, that the Al model solves the real-time problem, based on one or more predefined techniques. The one or more predefined techniques include at least one of an empirical validation technique, a black box validation technique, a gray box validation technique, and a statistical validation technique. In the empirical validation technique through the black box validation technique inputs, and outputs of the Al model are validated with respect to an expectation of solving the real-time problem. The gray box validation is performed to validate any intermediate processed data as well as implicit, unstored training updates such as the firing of the neurons in a neural network of the Al model through visualization and analysis. The system is flexible and modular which ensures onboarding of any domain and application specific evaluation metrics. Further, the statistical validation is also performed where statistical consistency checks, distribution shift checks, and anomaly checks are performed on the Al model.

[0061] Further, at least one predefined technique from the one or more predefined techniques is selectively chosen based on at least one of a plurality of factors. In an embodiment, the plurality of factors includes the type of problem to be solved by the Al model, the type of data, the type of decision frequency, and the label of safety criticality of an application by the Al model. The type of problems includes classification problems, regression problems, optimization problems, insight mining problems, etc. The type of data includes tabular data, image data, natural language data, signals (time series), etc. The type of decision frequency includes low frequency decisions and high frequency decisions. The level of safety criticality of the application includes low criticality, medium criticality, high criticality, etc. In another embodiment, the at least one predefined technique is selectively chosen in the predetermined manner by the one or more processors 110.

[0062] The one or more processors 110 are configured to validate, initially, conceptually 304 that the Al model is intended to solve the real-time problem, based on a plurality of factors. The plurality of factors includes at least one of a theoretical alignment factor, a clarity and coherence factor, an internal consistency factor, and an external consistency factor. The theoretical alignment factor refers to a coefficient (hypothetical) that measures / validates the degree to which an action of the Al model is aligned with the solving of the real-time problem. The clarity and coherence factor refers to an assessment / validation of the quality of the language generated by the Al model by evaluating different parameters, for example, sentence structure, vocabulary, etc. The internal consistency factor validates that the internal Siemens Aktiengesellschaft

[0063] 11 representation and reasoning processes of Al model are aligned to solve the real-time problem. The external consistency factor refers to a validation of a regulatory compliance, societal norms, etc., by the Al model while solving the real-time problem.

[0064] The conceptual 304 validation assesses if the Al model, as verified, is in line with the realtime problem which the Al model is expected to solve. The conceptual validation includes a manual or automated theoretical validation of the Al model with a state-of-the-art / exi sting literature, (manual / automated) comparison with competitive offerings. Further, different parameters, for example, an internal logical consistency and a validation of hypotheses with respect to domain specific practices and robustness definitions are considered while conceptually 304 validating the Al model.

[0065] Further, the one or more processors 110 are configured to validate, subsequently, operationally 306, that the Al model solves the real-time problem, when deployed in the system, based on the plurality of factors. The plurality of factors includes at least one of a measurement tools and metrics factor, a statistical validation factor, an empirical correspondence factor, and an inter disciplinary factor. In an embodiment, the empirical correspondence factor refers to the validation of degree to which the output of the Al model reflects the output from solving the real-time problem by the Al model. The interdisciplinary validation verifies the efficacy and accuracy of the Al model through collaborations and exchanges between subject matter experts from multiple fields related to, but is not limited to, the Al. The statistical validation factor validates statistical consistency checks, distribution shift checks, and anomaly checks of the Al model. The measurement tools and metrics factor validate the performance and effectiveness of the Al mode based on factors such as predicted positives. Further, the system 102 is configured to perform the operational validation to check if the Al model, to be implemented, solves the real-time problem which is meant to solved by the Al model.

[0066] Further, for verifying and validating the Al model, the one or more processors 110 are configured to quantify a plurality of degree of testing performed on the Al model, to verify 202 and validate 302 that the Al model solves real-time problem, by a pre-defined technique. The pre-defined technique is a coverage calculation module technique. The coverage module technique quantifies the various degrees of testing of the Al model. The coverage module technique includes data coverage computation, domain coverage computation, functional / use- case coverage computation, model coverage computation, and adversarial coverage computation.

[0067] The one or more processors 110 are configured to perform, upon successful validation, the safety and risk operation 402 on the Al model to determine a health status of the Al model while deploying the Al model in the system. The one or more processors 110 perform Siemens Aktiengesellschaft

[0068] 12 adaptiveness, robustness and adversarial tests for determining the safety of the Al model. The safety and risk operation 402 also provides a functional safety analysis module which allows safety analysis with respect to the requirement definition, solution design and the implemented solution. The one or more processors 110 perform health awareness checks with respect to the state of the Al model, the health of a hardware on which the AI-Driven automation system is implemented, human input divergence from specification / designed ranges, drift awareness, etc. A sim2real based analysis is performed to quantify the gap between the operational specification in the design of the Al model and the specification in the design faced in real world and recorded during operation. The one or more processors 110 offers risk assessment and mitigation with respect to technical risk operation 404, such as, model efficiency, model size, data quality, data imbalance, model underfitting / overfitting, hypersensitivity, and adversarial vulnerability which are explained in detail in subsequent paragraphs. The operational risk operation 404 analyses, quantifies, and prescribes solutions for countering operational risks such as concept / label / covariate / adversarial drifts, data poisoning, fallback mode and version control / traceability analysis which are explained in detail in subsequent paragraphs.

[0069] The one or more processors 110 are configured to receive a plurality of training and test data, during the deployment of the Al model in the system. Further, the one or more processors 110 are configured to perform a technical risk operation 404 to determine the health status of the Al model. In such an embodiment, the technical risk operation 404 includes modal efficiency operation, model overfitting operation, data imbalance operation, adversarial vulnerability operation, hypersensitive model operation, model size operation, model underfitting operation, data quality operation, low volume / variance in data operation. The model efficiency operation, the model overfitting operation, the model size operation, the model underfitting operation form part of a risk assessment from the safety and risk operation 402. Further, the data imbalance, the adversarial vulnerability, the hypersensitive models, the data quality, the low volume / variance in data form part of a risk mitigation from the safety and risk operation 402.

[0070] The model efficiency operation checks the delivery of accurate and reliable results based on the received plurality of training data and test data while minimizing computational resources. The modal underfitting operation and the modal overfitting operation receive the plurality of training and test data as an input. Further, based on the input, a report is generated for the user, where the report discloses whether the Al model underfits or overfits. The model size operation analyses the amount of memory required to run / operate the Al model.

[0071] The data imbalance operation diagnoses a class in the data set having more instances than other classes of the Al model based on the plurality of training and test data to determine the performance of the Al model. The adversarial vulnerability operation diagnoses the Siemens Aktiengesellschaft

[0072] 13 susceptibility of the Al model to exploitation by various means, such as, modified inputs, noise, etc. The hypersensitive model / model sensitivity operation diagnoses the cause / reason of the sensitivity of the Al model based on the plurality of training and test data. The data quality operation diagnoses the quality of each of the plurality of test data and training data to determine the performance of the Al model. The low volume / variance in data operation diagnoses the limited size of each of the plurality of test and training data to determine the efficiency of the Al model.

[0073] The technical risk operation 404 as disclosed above is selectively chosen by the user as per the requirement. In another embodiment, the technical risk operation 404 is selectively chosen in a predetermined manner by the one or more processors 110.

[0074] Further, the one or more processors 110 are configured to receive field data, during the deployment of the Al model in the system. The one or more processors 110 are configured to compare the field data with the plurality of training and test data to perform the operational risk operation 406 to determine the health status of the Al model. In such an embodiment, the operational risk operation 404 includes concept drift operation, label drift operation, fall back and version control operation, data poisoning operation, security risks operation, covariate drift operation, adversarial drift operation, data privacy operation, and model stealing operation.

[0075] The concept drift operation, the label drift operation, the covariate drift operation, and the adversarial drift operation form part of the risk assessment from the safety and risk operation 402. Further, the fall back and version control operation, the data poisoning operation, the security risks operation, the data privacy operation, the model stealing operation, form part of the risk mitigation from the safety and risk operation 402.

[0076] The concept drift operation diagnoses changes in patterns or relationship between variables of the Al model, based on the comparison of the field data with the plurality of training and test data to determine the performance of the Al model. The label drift operation diagnoses changes in the relationship of the input data and labels of the input data, based on the comparison of the field data with the plurality of training and test data to determine the performance of the Al model. The covariate drift operation diagnoses changes in a feature of the inputs, based on the comparison of the field data with the plurality of training and test data to determine the performance of the Al model. The adversarial drift operation diagnoses the vulnerability of the Al model because of various factors, such as, external attacks, noisy data, etc.

[0077] The fall back and version control operation diagnose and handle unexpected failures of the Al model. The data poisoning operation diagnoses the alteration of each of the plurality of Siemens Aktiengesellschaft

[0078] 14 training data, test data, field data. The security risk operation diagnoses any security breach by the Al model. The data privacy operation diagnoses breaches of the data by the Al model. The model stealing operation diagnoses if the Al model is replicated to avoid the model stealing and thus maintain the performance of the Al model.

[0079] The operational risk operation 406 as disclosed above is selectively chosen by the user as per the requirement. In another embodiment, the operation risk operation 406 is selectively chosen in a predetermined manner by the one or more processors 110.

[0080] When the Al model is deployed in the system of the client device 102, then in that case, there is the possibility of having plurality of data, for example, tabular data and image data. The tabular data includes data quality checks operation, adversarial robustness check operation, and outlier detection check operation. In such an embodiment, the outlier detection check operation forms part of the operational risk operation 406. Further, the data quality check operation, and the adversarial robustness check operation form part of the technical risk operation 404.

[0081] The tabular data also includes feature map generator, feature label correlation, conflicting label identifier, and class imbalance.

[0082] In feature correlation analysis of the tabular data, the feature mapping includes the creation of a visual representation of the relationships between different features (also known as variables or columns) in a tabular dataset by the feature map generator. The feature mapping captures relevant information on relation / connection of different features with each other and helps identify patterns or correlations that exist within the data. Further, based on the feature mapping method, the feature label correlation check computes the correlation between each feature and the label, potentially spotting features highly correlated with the label. The conflicting labels identifier check searches for identical samples with different labels. This occurs due to either mislabeled data, or when the data as collected is missing features necessary to separate the labels. The mislabeled data confuses the Al model and thus impacts the performance of the Al model.

[0083] The class imbalance check produces a distribution of the target variable. An indication of an imbalanced dataset is an uneven distribution in label classes which impacts the performance of the Al model. An imbalanced dataset has limitations, such as learning the characteristics of the minority label, scarce minority instances to train on (or test for), and defining the right evaluation metric which is thus checked by the class imbalance check. Siemens Aktiengesellschaft

[0084] 15

[0085] The adversarial sample generator helps in generating adversarial samples designed which is to be misclassified. This method minimizes the perturbation made to the data while ensuring misclassification.

[0086] The outlier detection feature includes a plurality of methods, i.e., probabilistic based feature and distance based feature which are used as per the requirement. The outlier detection feature is used to identify columns which are identified as “outliers” based on a new definition of an outlier.

[0087] Similarly, the image data includes data quality check operation, no reference image quality check operation, corruption robustness check operation, and adversarial robustness check operation. In such an embodiment, the no reference image quality check operation, the corruption robustness check operation form part of the operational risk operation 406. Further, the data quality check operation, and the adversarial robustness check operation form part of the technical risk operation 404.

[0088] The image data also includes dataset drift calculator, heatmap generation, Mean Average Precision (mAP) report, confusion matrix report, corruption robustness calculator, weak segments performance, adversarial image generator, and NR image quality assessment.

[0089] In the dataset drift calculator, the feature mapping method is used to capture relevant information from images. The relevant information is used to calculate drifts on various image properties between training and test datasets.

[0090] The heatmap generation and comparison is a method of detecting data drifting in the image data. Data drift is a change in the distribution of data over time or between several distinct cases which degrades the performance of the Al model gradually, or when applied to new scenarios. The heatmap comparison check computes an average image for all images in each dataset, trains and tests, and visualizes the average images. Thus, this configuration ensures a visual comparison of the difference between the dataset’s brightness distribution. For example, if training data contains significantly more images with sky, in that case, the average train image is brighter in the upper half of the heatmap.

[0091] The Mean Average Precision Report evaluates the mAP metric on the given Al model and data, plots the AP on graph, and returns the mAP values per bounding box size category (small, medium, large). This check only works on the Object Detection task.

[0092] The confusion matrix check outputs a confusion matrix for both classification problems and object detection problems. In object detection problems, some predictions do not overlap on any label and can be classified as not found in the confusion matrix. Siemens Aktiengesellschaft

[0093] 16

[0094] The corruption robustness calculator quantifies the corruption robustness of an object detection AI / ML model and uses various types of corruption which are most relevant to image datasets.

[0095] The weak segments performance check is designed to easily identify the model’ s weakest segments in the data provided. In addition, the check enables the user to provide a sublist of the Dataset’s features, thus limiting the check to search in interesting subspaces.

[0096] The adversarial sample generator helps in generating adversarial patches designed to help misclassify objects. This method maximizes the chance of misclassification during the creation of adversarial patches.

[0097] The NR Image quality assessment uses a technique to provide a quantitative metric of the quality of an image without any other reference image. This check assists the user in understanding the amount of useful information available in the Dataset.

[0098] Further, the safety and risk operation 402 also includes the system safety analysis 408. The system safety analysis 408 includes defining a safe input space through benchmarking, physics constraints, real world data, standards and industry specifications. The system safety analysis 408 also includes functional safety analysis, safe input space definition analysis, robustness and adversarial tests analysis, adaptiveness tests analysis, state awareness and safe mode analysis, multi-mode models analysis, hardware health awareness analysis, Sim2Real deterioration over time analysis, covariate drift awareness analysis, human error awareness analysis, and off-override switch analysis.

[0099] Thus, these configurations as disclosed above ensure the safety and risk operation 402 of the Al model, during the deployment state.

[0100] The one or more processors 110 are configured to assess 502 one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation 402, in a post deployment state of the Al model in the system. The one or more processors 110 are configured to compare the plurality of training and test data with the field data, when the Al model is deployed in the system. The one or more processors 110 are configured to assess the one or more operations of the Al model to determine at least one optimal feature in the Al model, a fairness of the Al model, a problem in the Al model, and a solving the problem in the Al model, based on the comparison. The system 102 ensures the one or more operations for testing, diagnosing, and solving the problem in the deployed Al model. The one or more operations are also used to assess the fairness and other ethical considerations in the Al model, when deployed. The one or more operations include an explainability and Siemens Aktiengesellschaft

[0101] 17 interpretability operation 504 to determine the optimal feature, a bias detection and mitigation operation 506 to determine the fairness of the Al model, and a model diagnosis and accountability operation 508 to determine the problem in the Al model and solving the determined problem in the Al model.

[0102] The explainability and interpretability operation 504 includes sensitivity analysis, counterfactual analysis, factorial analysis, contrastive explanation analysis, local interpretable model agnostic explanation analysis, and other explainable Al techniques. The sensitivity analysis analyses changes in the output of the Al model based on changes in the input of the Al model. The counterfactual analysis analyses the operation of the Al model in the alternative scenarios or conditions. The factorial analysis helps the user to analyze and identify different parameters influencing the performance of the Al model. The contrastive explanation analysis helps the user in analyzing the difference between predictions of the Al model with another Al model. The local interpretable model agnostic explanation analysis and other explainable Al techniques help in explaining the predictions of the Al model to the user. Further, the explainability and interpretability operation 504 also includes operations for tabular data-regression and classification, and an image data-object detection. The operation for tabular data regression and classification includes local explanation generator analysis 510, local global explanation contrast analysis 512, feature impact analysis 514, model agnostic interpretation suite analysis 516. The local explanation generator analysis 510 and the local global explanation contrast analysis 512 form part of a tool of the tabular data- regression and classification. Further, the feature impact analysis 514, and the model agnostic interpretation suite analysis 516 form part of a method of the tabular data-regression and classification. Similarly, the operation for the image data-object detection includes a layered relevance analysis 518 and semantic attention mapping module analysis 520. The layered relevance analysis 518 form part of a method of the image data-object detection. The semantic attention mapping module analysis 520 form part of a tool of the image data-object detection.

[0103] The local explanation generator analysis 510 includes the feature importance analysis. Further, the local global explanation contrast 512 includes the feature importance and weight importance, where the weights are learnable parameters in the Al model. The feature impact analysis 514 includes a bar plot, a contribution spectrum, a waterfall plot, and a summary plot which are extracted and interpreted by the user as per the requirement. The model agnostic interpretation suite analysis 516 includes variable importance, PCP and ALE plots, ceteris- Paribus plots, residual diagnostic which are extracted and interpreted by the user as per the requirement. The variable importance includes the sensitivity analysis, the counterfactual analysis, the factorial analysis, etc. The residual diagnostic enables to diagnose the remaining error in the Al model, in the post deployment state. Siemens Aktiengesellschaft

[0104] 18

[0105] Further, the layered relevance analysis includes a class-specific relevance mapping and a contrastive-multi class division. The layered relevance analysis ensures the interpretation of data at each layer of a neural network of the Al model by the Al model. The semantic attention mapping module analysis ensures the generation of an image similar to an image considered as an input by the Al model. For example, the generation and detection of TV, monitors, etc. The semantic attention mapping module includes eigenvalue-based analysis, gradient-based analysis, multi-order gradient-based analysis, and sensitivity conservation gradient-optimization based analysis.

[0106] The operations as disclosed under the explainability and interpretability operation 504 support plurality of data, for example, XGBoost, SVM, Linear Regression, Logistic Regression, LightGBM, Decision Tree, Random Forest, Catboost, Autikeras, YOLOv8.

[0107] The operations as discussed above enable the users to directly upload data files, set model inputs, and select the explainability and interpretability operations 504. The data is processed in the backend using the selected operation and the results are displayed back to the user.

[0108] The operation as disclosed ensures integration of interpretability operations, user-friendly interface, model compatibility, backend processing, and result display. For the integration of interpretability operations, the one or more processors 110 integrate a variety of interpretability operations providing the local and global explanation analysis, the model agnostic interpretation suite analysis, the feature impact analysis, the semantic attention mapping module, etc. customized to be compatible with different architecture of each Al model. The interface for the explainability and interpretability operation 504 as developed is a user-friendly interface allowing the users to directly upload data files, specify model inputs, and select the explainability and interpretability operation 504 through intuitive options and descriptions. The platform of the system 102 is designed to support a wide range of machine learning model / AI model architectures commonly used in various domains, such as linear regression, logistic regression, decision trees, and ensemble methods like Random Forest and XGBoost. Further, the uploaded data and model inputs are processed by the Al model backend using the selected operation and the selected machine learning model architecture to generate interpretable explanations for model decisions. The generated explanations from the algorithm operation are displayed back to the user in the form of a result in a clear and understandable format, allowing them to interpret and analyze the model's decision-making process.

[0109] The bias detection and mitigation operation 506 includes detection and mitigation of data input bias, modelling bias, a label bias, an ethical evaluation, bias correction methodology, and legal evaluations. The detection and mitigation of the data input bias refers to detection and mitigation of a behavior of the Al model favoring a particular type of data. The detection Siemens Aktiengesellschaft

[0110] 19 and mitigation of the modelling bias refers to the detection and mitigation of unfairly preprogrammed preferences of the Al model. The detection and mitigation of the label bias refers detection and mitigation of problems that occur due to training of the Al model on an imbalanced and biased data set. The ethical evaluation ensures the assessment of the Al model for transparency, accountability, etc. The bias correction methodology ensures trust in a decision-making process of the Al model. The legal evaluation ensures that the output generated by the Al model complies with the rules and regulations of a jurisdiction.

[0111] The model diagnosis and accountability operation 508 includes feature correlation analysis, a data imbalance and quality analysis, feature importance analysis, data noise, and model uncertainty analysis, performance root case analysis, and data imbalance and quality analysis. The feature correlation analysis indicates the selection of relevant features that help in the development of the accurate Al model. The data imbalance and quality analysis indicate analysis of the uneven distribution of data in the Al model for ensuring accurate performance of the Al model. The feature importance analysis indicates analysis of the important feature of the input which assists majorly in prediction by the Al model. The data noise and model uncertainty analysis indicate analysis of the presence of incorrect, incomplete, or noisy data that can affect the accuracy of Al model, and model uncertainty represents the Al model’s limitation in capturing the underlying relationships between variables. The performance root case analysis indicates the analysis of the cause of the problem in the Al model. The data imbalance and quality analyses indicate analysis a data set with a larger number of instances than other data set which affects the performance of the Al model, the quality analysis indicates analysis of the data for training the Al model, where the data is reliable, relevant, and free from errors.

[0112] FIG. 6 illustrates an exemplary process flow depicting a method 600 for providing the assurance of the Artificial Intelligence (AI)-Driven automation system. The method 600 is performed by the system 102, as shown in FIGS. 1-5. In an embodiment, the method 600 is performed by one or more processors 110 of the system 102.

[0113] At step 602, the method 600 includes receiving by the one or more processors 110 the Al model intended to solve the real time problem, in the pre-deployment state of the Al model.

[0114] At step 604, the method 600 includes verifying, by the one or more processors 110, that the Al model solves the real-time problem, by performing at least one of the plurality of analyses on the Al model. In an embodiment, the plurality of analyses comprises at least one of the plurality of functional analysis and testing 204, the plurality of standard requirements analysis and testing 206, the repeatability analysis 208, and the behaviour analysis 210. Further, at least one analysis from the plurality of analyses is selectively chosen based on at least one of the plurality of parameters. Siemens Aktiengesellschaft

[0115] 20

[0116] At step 606, the method 600 includes validating, by the one or more processors 110, upon successful verification, that the Al model solves the real-time problem, based on the one or more predefined techniques. In an embodiment, the one or more predefined techniques comprises at least one of the empirical validation technique, the block box validation technique, the gray box validation technique, and the statistical validation technique. Further, at least one predefined technique from the one or more predefined techniques is selectively chosen based on at least one of the plurality of factors.

[0117] In such an embodiment, the method 600 includes validating, initially, conceptually 304 that the Al model is intended to solve the real-time problem, based on the plurality of factors. The plurality of factors includes at least one of the theoretical alignment factor, the clarity and coherence factor, the internal consistency factor, and the external consistency factor. Further, the method 600 includes validating, subsequently, operationally 306, that the Al model solves the real-time problem, when deployed in the system, based on the plurality of factors. The plurality of factors includes at least one of the measurement tools and metrics factor, the statistical validation factor, the empirical correspondence factor, and the inter-disciplinary factor.

[0118] In an embodiment, for verifying and validating the Al model, the method 600 further includes quantifying, by the one or more processors 110, the plurality of degrees of testing performed on the Al model, to verify 202 and validate 302 that the Al model solves the real-time problem, by the pre-defined technique. The pre-defined technique is a coverage calculation module technique.

[0119] At step 608, the method 600 includes performing, by the one or more processors 110, upon successful validation, the safety and risk operation 402 on the Al model to determine the health status of the Al model, while deploying the Al model in the system.

[0120] In such an embodiment, the method 600 includes receiving the plurality of training and test data, by the one or more processors 110, during the deployment of the Al model in the system. The method 600 includes performing, by the one or more processors 100, the technical risk operation 404 to determine the health status of the Al model.

[0121] The method 600 includes receiving, by the one or more processors 110, the field data, during the deployment of the Al model in the system. The method 600 includes comparing, by the one or more processors 110, the field data with the plurality of training and test data to perform the operational risk operation 406 to determine the health status of the Al model. Siemens Aktiengesellschaft

[0122] 21

[0123] At step 610, the method 600 includes assessing, by the one or more processors 110, one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation, in the post-deployment of the Al model in the system.

[0124] The method 600 includes comparing, by the one or more processors 110, the plurality of training and test data with the field data, when the Al model is deployed in the system. The method 600 includes assessing, by the one or more processors 100, the one or more operations of the Al model to determine at least one of optimal features in the Al model, the fairness of the Al model, the problem in the Al model, and solving the problem in the Al model, based on the comparison. The one or more operations comprises the explainability and interpretability operation 504 to determine the optimal feature in the Al model, the bias detection and mitigation operation 506 to determine the fairness of the Al model, and the model diagnosis and accountability operation 508 to determine the problem in the Al model and solving the determined problem in Al model.

[0125] While the above steps shown in FIG. 6 are described in a particular sequence, the steps may occur in variations to the sequence in accordance with various embodiments of the present invention. Further, the details related to various steps of FIG. 6, which are already covered in the description related to FIGS. 1-5 are not discussed again in detail here for the sake of brevity.

[0126] The present invention provides for various technical advancements based on the key features discussed above. The system 102 and the method 600 as disclosed assure a safe, reliable, functional AI-Driven Automation system, where the system also is compatible with / complies with risk management, ethical requirements, technical requirements, accountability, fairness, robustness, explainability, etc. It can also be used to diagnose any performance issues, identify root causes, and improve an existing solution. The system 102 and the method 600 as disclosed ensure flexibility to use individual operations for assurance aspects based on the industry and / or domain specific requirements, flexibility to plug-in geographical regulatory requirements and industry-specific standards into individual modules, flexibility to select suboperations of the assurance under the various assurance aspects based on industry / company / geographical regulatory requirements. Particularly, the system 102 integrates multiple analysis / techniques / factors / operations, etc., tailored to be compatible with performing various tasks. This comprehensive approach allows the users to choose the most suitable analysis / techniques / factors / operations for their specific requirements. This configuration ensures flexibility and accuracy in generating interpretable explanations for model decisions. Further, the variety of analysis / techniques / factors / operations as provided are compatible with state-of-the-art (SOTA) models. Siemens Aktiengesellschaft

[0127] 22

[0128] Particularly, the verification and validation check and confirm that the Al model is aligned with the solving of the real-time problem, in the pre-deployment state.

[0129] The safety and risk operation 402 as disclosed ensures protection of the Al model from possible risk threat.

[0130] The one or more operations for the assessment of the Al model at the post deployment state ensure the fairness of the Al model. Further, the comprehensive integration of interpretability operation, the user-friendly interface, and compatibility with popular machine learning model architectures provides several advantages:

[0131] Comprehensive Integration of the operation from the explainability and interpretability operation 504: The system 102 includes a broad range of the explainability and interpretability operation 504 that is customized to fit into different types of architecture of machine learning model architectures / Al model. For instance, users can choose the operation based on their specific needs such as object detection, classification, or regression tasks. This allows flexibility for users to generate and compare interpretability results for different models depending on their requirements all in one single platform.

[0132] User-Friendly Interface: Unlike most existing solutions, the user-friendly interface is provided that does not require coding or extensive technical knowledge to implement interpretability algorithms. This enables users to upload their data files thereby specifying the inputs of the model and selecting the appropriate operation using an intuitive interface without difficulty. This level of accessibility lowers the entry barrier for users (both technical and non-technical), enabling a wider range of individuals to leverage the benefits of the explainability and interpretability operation 504.

[0133] Compatibility with architectures of the Machine Learning Model / AI model: The explainability and interpretability operation 504 as disclosed is compliant and consistent with the widely used standard machine learning models in different fields. Some of these include Y0I0V8 for object detection, linear regression for simple regression tasks or other complex models such as XGBoost and LightGBM for complex classification problems. Further, the explainability and interpretability operation 504 supports a large range of Machine Learning models. This compatibility means that their current models have XAI techniques applied on them without requiring many changes or adjustments.

[0134] While specific language has been used to describe the present subject matter, any limitations arising on account thereto, are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein. The drawings and the foregoing description give examples Siemens Aktiengesellschaft

[0135] 23 of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment.

[0136] Siemens Aktiengesellschaft

[0137] 24

[0138] List of Reference Numerals

[0139] 100 Environment

[0140] 102 System

[0141] 104 Client device

[0142] 106 Network

[0143] 108 User interface

[0144] 110 One or more processors

[0145] 112 Memory

[0146] 114 Input / output (I / O) interface

[0147] 116 Database

[0148] 118 Operating system

[0149] 202 Verify / verification

[0150] 204 Functional analysis and testing

[0151] 206 Standard requirement analysis and testing

[0152] 208 Repeatability analysis

[0153] 210 Behaviour analysis

[0154] 212 Cross-validation analysis

[0155] 214 Continuous monitoring analysis

[0156] 302 Validate

[0157] 304 Conceptually

[0158] 306 Operationally

[0159] 402 Safety and risk operation

[0160] 404 Technical risk operation

[0161] 406 Operational risk operation

[0162] 408 System safety analysis

[0163] 502 Assess

[0164] 504 Explainability and interp ret ability operation

[0165] 506 Bias detection and mitigation operation

[0166] 508 Model diagnosis and accountability operation

[0167] 600 (602-610) Methods

Claims

Siemens Aktiengesellschaft25CLAIMS1. A system (102) for providing assurance of Artificial Intelligence (AI)-Driven Automation system, the system (102) comprising: a memory (112); and one or more processors (110) communicatively coupled to the memory (112), wherein the memory (112) comprises programmable instructions which, when executed by the one or more processors (110), cause the one or more processors (110) to perform: receive an Al model intended to solve a real-time problem, in a pre-deployment state of the Al model; verify (202) that the Al model solves the real-time problem, based on a plurality of analyses; validate (302), upon successful verification, that the Al model solves the real-time problem, based on one or more predefined techniques; perform, upon successful validation, a safety and risk operation (402) on the Al model to determine a health status of the Al model while deploying the Al model in the system; and assess (502) one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation (402), in a post-deployment state of the Al model in the system.

2. The system (102) according to claim 1, wherein the plurality of analyses comprises at least one of a plurality of functional analysis and testing (204), a plurality of standard requirements analysis and testing (206), a repeatability analysis (208), and a behaviour analysis (210), wherein at least one analysis from the plurality of analyses is selectively chosen based on at least one of a plurality of parameters.

3. The system (102) according to any of the preceding claims, wherein the one or more predefined techniques comprises at least one of an empirical validation technique, a black box validation technique, a gray box validation technique, and a statistical validation technique, wherein at least one predefined technique from the one or more predefined techniques is selectively chosen based on at least one of a plurality of factors.

4. The system (102) according to any of the preceding claims, wherein for validating that the Al model solves the real-time problem, the one or more processors (110) are configured to: validate, initially, conceptually (304) that the Al model is intended to solve the realtime problem, based on a plurality of factors, wherein the plurality of factors include at least one of a theoretical alignment factor, a clarity and coherence factor, an internal consistency factor, and an external consistency factor; andSiemens Aktiengesellschaft26 validate, subsequently, operationally (306), that the Al model solves the real-time problem, when deployed in the system, based on a plurality of factors, wherein the plurality of factors includes at least one of a measurement tools and metrics factor, a statistical validation factor, an empirical correspondence factor, and an inter-disciplinary factor.

5. The system (102) according to any of the preceding claims, wherein for verifying and validating the Al model, the one or more processors (110) are further configured to: quantify a plurality of degrees of testing performed on the Al model, to verify (202) and validate (302) that the Al model solves the real-time problem, by a pre-defined technique, wherein the pre-defined technique is a coverage calculation module technique.

6. The system (102) according to any of the preceding claims, wherein for performing the safety and risk operation (402), the one or more processors (110) are configured to: receive a plurality of training and test data, by the one or more processors (110), during the deployment of the Al model in the system; and perform a technical risk operation (404) to determine the health status of the Al model.

7. The system (102) according to any of the preceding claims, wherein for performing the safety and risk operation (402), the one or more processors (110) are further configured to: receive field data, during the deployment of the Al model in the system; and compare the field data with the plurality of training and test data to perform an operational risk operation (406) to determine the health status of the Al model.

8. The system (102) according to any of the preceding claims, wherein for assessing the one or more operations of the Al model, the one or more processors (110) are configured to: compare a plurality of training and test data with field data, when the Al model is deployed in the system; and assess the one or more operations of the Al model to determine at least one optimal feature in the Al model, a fairness of the Al model, a problem in the Al model, and solving the problem in the Al model, based on the comparison, wherein the one or more operations comprise: an explainability and interpretability operation (504) to determine the optimal feature in the Al model; a bias detection and mitigation operation (506) to determine the fairness of the Al model; and a model diagnosis and accountability operation (508) to determine the problem in the Al model and solving the determined problem in Al model.Siemens Aktiengesellschaft279. A computer-program product, having machine-readable instructions stored therein, which when executed by the one or more processors (110), cause the one or more processors (110) to perform according to any of the claims 1 to 8.

10. A computer-implemented method (600) for providing assurance of Artificial Intelligence (AI)-Driven Automation system, the method (600) comprising: receiving (602), by one or more processors (110), an Al model intended to solve a real-time problem, in a pre-deployment state of the Al model; verifying (604), by the one or more processors (110), that the Al model solves the real-time problem, by performing at least one of a plurality of analyses on the Al model; validating (606), by the one or more processors (110), upon successful verification, that the Al model solves the real-time problem, based on one or more predefined techniques; performing (608), by the one or more processors (110), upon successful validation, a safety and risk operation (402) on the Al model to determine a health status of the Al model, while deploying the Al model in the system; and assessing (610), by the one or more processors (110), one or more operations of the Al model to solve the real-time problem, after performing the safety and risk operation (402), in a post-deployment of the Al model in the system.

11. The computer-implemented method (600) according to claim 10, wherein the plurality of analyses comprises at least one of a plurality of functional analysis and testing (204), a plurality of standard requirements analysis and testing (206), a repeatability analysis (208), and a behaviour analysis (210), wherein at least one analysis from the plurality of analyses is selectively chosen based on at least one of a plurality of parameters.

12. The computer-implemented method (600) according to any of the preceding claims, wherein the one or more predefined techniques comprises at least one of an empirical validation technique, a black box validation technique, a gray box validation technique, and a statistical validation technique, wherein at least one predefined technique from the one or more predefined techniques is selectively chosen based on at least one of a plurality of factors.

13. The computer-implemented method (600) according to any of the preceding claims, wherein validating, by the one or more processors (110), that the Al model solves the realtime problem comprises:V alidating, initially, conceptually (304) that the Al model is intended to solve the realtime problem, based on a plurality of factors, wherein the plurality of factors includes at least one of a theoretical alignment factor, a clarity and coherence factor, an internal consistency factor, and an external consistency factor; andSiemens Aktiengesellschaft28 validating, subsequently, operationally (306), that the Al model solves the real-time problem, when deployed in the system, based on a plurality of factors, wherein the plurality of factors includes at least one of a measurement tools and metrics factor, a statistical validation factor, an empirical correspondence factor, and an inter-disciplinary factor.

14. The computer-implemented method (600) according to any of the preceding claims, wherein verifying and validating the Al model further comprises: quantifying, by the one or more processors (110), a plurality of degrees of testing performed on the Al model, to verify (202) and validate (302) that the Al model solves the real-time problem, by a pre-defined technique, wherein the pre-defined technique is a coverage calculation module technique.

15. The computer-implemented method (600) according to any of the preceding claims, wherein performing the safety and risk operation (402) comprises: receiving a plurality of training and test data, by the one or more processors (110), during the deployment of the Al model in the system; and performing, by the one or more processors (110), a technical risk operation (404) to determine the health status of the Al model.

16. The computer-implemented method (600) according to any of the preceding claims, wherein performing the safety and risk operation (402) further comprises: receiving, by the one or more processors (110), a field data, during the deployment of the Al model in the system; and comparing, by the one or more processors (110), the field data with the plurality of training and test data to perform an operational risk operation (406) to determine the health status of the Al model.

17. The computer-implemented method (600) according to any of the preceding claims, wherein assessing the one or more operations of the Al model comprises: comparing, by the one or more processors (110), a plurality of training and test data with a field data, when the Al model is deployed in the system; and assessing, by the one or more processors (110), the one or more operations of the Al model to determine at least one of optimal features in the Al model, a fairness of the Al model, a problem in the Al model, and solving the problem in the Al model, based on the comparison, wherein the one or more operations comprises: an explainability and interpretability operation (504) to determine the optimal feature in the Al model; a bias detection and mitigation operation (506) to determine the fairness of the Al model; andSiemens Aktiengesellschaft29 a model diagnosis and accountability operation (508) to determine the problem in the Al model and solving the determined problem in Al model.

Images