System and method for configuring test parameter in machine learning evaluation

Abstract

To provide a method.SOLUTION: There are provided a method, system, and device for configuring test parameters used in evaluating a machine learning (ML) model. The method may include: acquiring a user input including at least one test parameter; storing the acquired user input as a requirement file (RaC file) as a code; interpreting, by a requirement management layer, at least one test parameter from the RaC file; and evaluating the ML model on the basis of at least one test parameter interpreted from the RaC file.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] Systems and methods consistent with embodiments of the present disclosure relate to configuring test parameters for machine learning evaluation. [Background technology]

[0002] Machine learning (ML) models can be used to automate various tasks. When developing an ML model, a developer may have certain criteria or parameters that need to be met by the ML model. For example, if the ML model is intended to automate a safety-critical task, the ML model may need to achieve a certain reliability rating. Therefore, it may be necessary for the developer to test and evaluate the ML model by performing an ML evaluation to ensure that the ML model can meet the specifications.

[0003] In the prior art, simple metrics can be used to automatically determine if a model is good or not. For example, the mean average precision (mAP) may be used as a simple number that indicates whether an ML model is good, bad, or excellent.

[0004] Prior art systems and methods may only provide a limited degree of evaluation of ML models, and may not match the exact criteria or parameters required for the ML model to achieve. In particular, prior art techniques may typically only use simple metrics, and improving one metric may require tradeoffs / penalties with other metrics, making it difficult to optimize the performance of the ML model. Prior art methods and systems may not have a means to establish quantitative and unambiguous criteria for determining which ML model is optimal, especially in scenarios where there are many candidate ML models.

[0005] Furthermore, even if a "good" ML model is found that passes acceptance tests and has a good metric model, it may still be deemed suboptimal and may have unforeseen flaws that are only discovered after it is deployed.

[0006] Therefore, there is a need for improved ways of configuring how the ML evaluation process is performed, particularly with respect to test parameters (which may include requirements, acceptance criteria, test conditions, etc.). Summary of the Invention

[0007] According to one or more embodiments, an apparatus and method are provided for obtaining test parameters (which may include requirements, pass criteria, test conditions, etc.) from a user, storing such test parameters in a Requirements as Code (RaC) file, and interpreting the test parameters from the RaC file using a layer (such as a requirements management layer). Thus, a user can simply configure a RaC file to match the requirements needed to meet a standard, and such RaC files can be used independently, sequentially, or in parallel with each other when performing an ML evaluation process. Thus, ML evaluation can be performed to ensure with a high degree of accuracy that the ML model being evaluated is optimal with respect to the desired properties of the ML model.

[0008] According to an embodiment, the method may include acquiring user input including at least one test parameter, saving the acquired user input as a Requirements-as-Code (RaC) file, interpreting, by a requirements management layer, the at least one test parameter from the RaC file, and evaluating the ML model based on the interpreted at least one test parameter from the RaC file.

[0009] According to an embodiment, the at least one test parameter may include one or more acceptance requirements, acceptance criteria, or test conditions. User input may be obtained via a graphical user interface. Obtaining user input may further include selecting the at least one test parameter from a predefined library of test parameters.

[0010] According to an embodiment, evaluating the ML model may be performed by an evaluation layer including an inference unit and a unit testing unit, the inference unit may be configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit may be configured to evaluate the ML model based on the output from the ML model to obtain metrics.

[0011] The output from the ML model may include an inference log, and upon completion of the inference process, the inference unit may be configured to transfer the inference log to the unit testing unit, and evaluating the ML model may include evaluating metrics from the inference log, and the evaluated metrics may be displayed in a graphical user interface.

[0012] The method may further include receiving, by the requirements management layer, an instruction to add or update at least one test parameter in the storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

[0013] According to an embodiment, an apparatus may be provided for configuring test parameters for use in evaluating a machine learning (ML) model, the apparatus may include at least one memory storing computer-executable instructions, and at least one processor configured to execute the computer-executable instructions to obtain user input including the at least one test parameter, store the obtained user input as a Requirements-as-Code (RaC) file, interpret, via a requirements management layer, the at least one test parameter from the RaC file, and evaluate the ML model based on the interpreted at least one test parameter from the RaC file.

[0014] According to an embodiment, the at least one processor may be further configured to execute computer-executable instructions to obtain user input by selecting at least one test parameter from a predefined library of test parameters.

[0015] Additional aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of presented embodiments of the present disclosure. [Brief description of the drawings]

[0016] Features, aspects, and advantages of certain preferred embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1] FIG. 1 is a diagram of example components of an apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram of a system architecture according to one or more embodiments; and [Diagram 3] FIG. 3 is a flow chart diagram illustrating a method for constructing a RaC file containing test parameters and evaluating an ML model based on the interpreted test parameters according to one or more embodiments. [Figure 4]FIG. 4 is a flow chart diagram illustrating a method for evaluating an ML model according to one or more embodiments; and [Diagram 5] FIG. 5 is a flow chart diagram illustrating a method for evaluating an ML model including adding or updating requirements according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The following detailed description of the embodiments refers to the accompanying drawings. The present disclosure provides illustration and description, but is not intended to be exhaustive or to limit one or more embodiments to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from the practice of one or more embodiments. Moreover, one or more features or components of one embodiment may be incorporated or combined with other embodiments (or one or more features of the other embodiments). Furthermore, in the flowcharts and descriptions of operations provided herein, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be rearranged.

[0018] It will be apparent that embodiments of the systems and / or methods and / or computer-readable non-transitory storage media described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit one or more embodiments. Thus, the operation and behavior of the systems and / or methods and / or computer-readable non-transitory storage media are described herein without reference to any specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0019] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes each dependent claim in combination with all other claims in the claim set.

[0020] No element, act, or instruction used herein should be construed as critical or required unless expressly recited as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar terms are used. Also, as used herein, "has," "have," "having," "include," "including," or the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of 'A' and 'B'" or "at least one of 'A' or 'B'" should be understood as including only A, only B, or both A and B.

[0021] 1 is a diagram of example components of a machine learning (ML) evaluation device 100. As shown in FIG. 1, the ML evaluation device 100 may include a bus 110, a processor 120, a memory 130, a storage unit 140, an input unit 150, an output unit 160, and a communication interface 170.

[0022] The bus 110 includes components that enable communication between the components of the ML evaluator 100. The processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing element. In one or more embodiments, the processor 120 includes one or more processors that are programmable to perform functions. The memory 130 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores instructions and / or information for use by the processor 120.

[0023] The storage unit 140 stores information and / or software related to the operation and use of the ML evaluation device 100. For example, the storage unit 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a semiconductor disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or a corresponding drive, as well as another type of computer-readable non-transitory medium. The input unit 150 includes components (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone) that enable the ML evaluation device 100 to receive information, for example, via user input. Additionally or alternatively, the input unit 150 may include sensors (e.g., a global positioning system (GPS) element, an accelerometer, a gyroscope, and / or an actuator) for detecting information. The output unit 160 includes components (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)) that provide output information from the ML evaluation device 100.

[0024] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the ML assessment device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 170 can enable the ML assessment device 100 to receive information from and / or provide information to another device. For example, the communication interface 170 can include, but is not limited to, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a WiFi interface, a cellular network interface, or the like.

[0025] The ML evaluation device 100 may perform one or more instances of the processes described herein. According to one or more embodiments, the ML evaluation device 100 may perform these processes in response to the processor 120 executing software instructions stored in a computer-readable non-transitory medium, such as the memory 130 and / or the storage unit 140. A computer-readable medium is defined herein as a non-transitory storage device. A storage device includes memory space within a single physical storage device or memory space across multiple physical storage devices.

[0026] The software instructions may be read into memory 130 and / or storage 140 from another computer readable medium or from another device via communications interface 170. When executed, the software instructions stored in memory 130 and / or storage 140 may cause processor 120 to perform one or more of the processes described herein.

[0027] Additionally, or instead, hardwired circuitry may be used in place of, or in combination with, software instructions to implement one or more of the processes described herein. Thus, one or more embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0028] The number and arrangement of components shown in Figure 1 are provided as an example. In practice, ML evaluator 100 may include additional, fewer, different, or differently arranged components than those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of ML evaluator 100 may perform one or more functions that are described as being performed by another set of components of ML evaluator 100.

[0029] FIG. 2 is a diagram of a system architecture according to one or more embodiments.

[0030] According to an embodiment, an evaluation layer 200 may be provided. The evaluation layer 200 may be responsible for performing ML evaluation processes (e.g., inference and device testing) on ​​the ML model 230. It should be understood that the evaluation layer 200 may implement any suitable means for performing ML evaluation. The evaluation layer 200 may be configured to receive instructions from the requirements management layer 210. It should be understood that according to some embodiments, the evaluation layer 200 may include a UI (not shown) to output the results of the evaluation to a user. In particular, the evaluation layer 200 may send an output signal including the results of the evaluation upon completing the evaluation, and the UI may receive the output signal and display the results of the evaluation. The UI may have a tabular format that lists the results of the evaluation tests and may highlight certain parts of the test results based on whether they passed or failed. Nevertheless, it should be understood that the inclusion of a UI for the evaluation layer 200 is up to the particular implementation, and the configuration may also be determined by one of ordinary skill in the art.

[0031] A requirements management layer 210 may be provided. The requirements management layer 210 may be responsible for interpreting files (e.g., requirements as code (RaC) files 1, 2...N (221-1, 221-2, 221-N...)) stored in the storage layer. Based on the interpretation of the RaC files, the requirements management layer 210 may send instructions to the evaluation layer 200 to perform the ML evaluation.

[0032] The storage device 220 may include all data used by the evaluation layer 200 and the requirements management layer 210. The storage device 220 may be implemented by any suitable storage means (e.g., a database, a cloud storage device, etc.). The storage device 220 may also include any number of RaC files 221-1, 221-2, 221-N, and a test parameter library 222. It should be understood that each RaC file and the test parameter library may be stored in the same storage medium or different storage media.

[0033] The requirements as code (RaC) files 221-1, 221-2, 221-N may serve to store the expected behavior of the ML model 230. The expected behavior may include test parameters. In particular, the test parameters may specify the performance metrics of the ML model 230 that need to be tested, for example, based on a specific number of metrics that need to be achieved, a standard that needs to be achieved, the type of test, etc. Thus, the RaC files may include test goals, file paths of test data, requirements, pass criteria, test conditions, acceptance requirements, and acceptance criteria as test parameters, which will be used during the ML evaluation process in the evaluation layer 200. Thus, such requirements, test goals, file paths of test data, pass criteria, and test conditions may be readily interpreted by the requirements management layer 210 to determine how ML testing and evaluation of the ML model 230 should be performed by the evaluation layer 200. For example, the RaC files may include criteria in the form of code, and the requirements management layer 210 may interpret the code into instructions on how to perform ML evaluation of the ML model 230 using the evaluation layer 200.

[0034] A RaC file may be computer-readable code in a format that is easily interpreted by humans, such as in a YAML format or a domain-specific language (DSL) format. This may allow someone who is not familiar with coding syntax to easily configure the ML evaluation and verify that the RaC file matches their quantitative requirements. Furthermore, this may allow a high degree of customization, for example allowing end users to adapt the ML evaluation in more unique environments. Furthermore, because the RaC file is separate from the code used to actually perform the ML evaluation, it may eliminate the need to "hard-code" requirements into the ML evaluation process.

[0035] According to an embodiment, the test parameters stored in the RaC file may be related to each other. For example, some parameters may be related to a particular subject and belong to the same group. Or, some parameters may be in a hierarchical relationship, for example, where one criterion is defined by the combination of the results of several other criteria. Nevertheless, the test parameters may be independent of each other.

[0036] Because the entire requirement is in the form of code rather than a set of numbers, there is greater precision that can be achieved by ensuring that the test requirements match the actual requirements, which can improve accuracy in ensuring that the ML model being evaluated is optimal.

[0037] If multiple RaC files 221-1, 221-2, 221-N are used in the ML evaluation, it is contemplated that each RaC file 221-1, 221-2, 221-N can be used to check each of the multiple sets of requirements (either sequentially or in parallel). In any case, since the RaC files can separate different sets of requirements, in some embodiments it may be simpler to check each set individually (one at a time).

[0038] According to some embodiments, the RaC files 221-1, 221-2, 221-N may be editable by a user input UI 240. In particular, the user input UI 240 may be a general command line interface (command terminal) used to edit source files, or a graphical user interface (GUI) with interactive elements that allow the user to drag and drop or select predefined configurations or select from a list. Such predefined configurations may be retrieved from a test parameter library 222 from the storage device 220, and may be, for example, one or more commonly used test parameter configurations. Nevertheless, it should be understood that any suitable user interface may be implemented for the user UI 240.

[0039] The test parameter library 222 may include predefined test parameters. The test parameter library 222 may be stored separately from the RaC file according to some embodiments. The specific format of the test parameters in the test parameter library 222 may be any suitable format.

[0040] It should be appreciated that the above-described UI (including user input UI 240) may be implemented in any suitable environment, for example, in a web interface or only locally on the user device. It should also be appreciated that, according to some embodiments, user input UI 240 may be displayed separately or simultaneously with another UI.

[0041] FIG. 3 is a flow chart diagram illustrating a method 300 for constructing a RaC file containing test parameters and evaluating an ML model based on the interpreted test parameters according to one or more embodiments.

[0042] 3, in operation S310, user input including at least one or more test parameters may be obtained, for example, by using a UI (which may be a GUI or a command line terminal) such as user input UI 240. According to an embodiment, the particular format of the user input may be a human readable format, and one or more test parameters may be entered simultaneously. Nevertheless, it should be understood that the user input may include information other than one or more test parameters.

[0043] At operation S320, the user input obtained from operation S310 may be stored in one or more RaC files 221-1, 221-2, 221-N in storage device 220. It should be appreciated that depending on the format of the user input, the user input may need to be converted into a different format or data type before being stored in RaC file 221-1, 221-2, or 221-N.

[0044] At act S330, after the test parameters are stored at act S320, the test parameters may be interpreted by requirements management layer 210 to determine how to instruct evaluation layer 200 to perform ML evaluation on ML model 230. This may include interpreting model tuning parameters, such as requirements, pass criteria, test conditions, acceptance requirements, acceptance criteria, test goals, test data paths, confidence thresholds, etc., from the test parameters, and determining the appropriate tests and test parameters that need to be performed using evaluation layer 200. Thus, the interpreted test parameters may be obtained by requirements management layer 210.

[0045] At act S340, evaluation layer 200 may perform an ML evaluation on ML model 230 based on the interpreted test parameters. For example, requirements management layer 210 may send instructions to evaluation layer 200 based on the interpreted test parameters from act S330 to cause evaluation layer 200 to perform an ML evaluation on ML model 230. According to some embodiments, once evaluation layer 200 has completed performing the ML evaluation, it may send an output signal including a result of the evaluation. According to some embodiments, the output signal may be sent to a UI to display the results.

[0046] 4 is a flow chart diagram illustrating a method 400 of evaluating an ML model using the evaluation layer 200, according to one or more embodiments. Similar layers and components shown in FIG. 2 may be used to implement the method 400, and a full description of similar steps from the method 300 as shown in FIG. 3 may be omitted for ease of reading.

[0047] According to the embodiment shown in FIG. 4, the evaluation layer 200 may include an inference unit and a unit test unit. The inference unit may be responsible for receiving test data, inputting the data into the ML model 230, and calculating outputs using the ML model 230 (i.e., inference). The unit test unit may be responsible for performing device testing on the ML model 230. In particular, the device testing may evaluate the results of outputs from the ML model 230 obtained during an inference process performed on the ML model 230 using the inference unit. Metrics on the performance of the ML model 230 may be obtained using the unit test unit.

[0048] 4, at operation S410, an instruction to perform an ML evaluation may be sent from requirements management layer 210 and received by evaluation layer 200 to cause evaluation layer 200 to perform an ML evaluation on ML model 230. The instruction received at operation S410 may be similar to that sent at operation S340 described in FIG. 3 above.

[0049] At operation S420, the evaluation layer 200 may send instructions to the inference unit to perform an inference operation using the ML model 230 with the test data. The output of this operation may include an inference log that includes data obtained from performing the inference operation on the ML model 230 with the test data.

[0050] At operation S430, the evaluation layer 200 may transmit an inference log from the inference component to the unit test component.

[0051] In operation S440, the evaluation layer 200 may send instructions to the unit test portion to evaluate the metric (eg, by calculating the metric and comparing it to a test criterion from the test parameters).

[0052] At operation S450, the evaluation layer 200 may output the evaluated metrics (which may include the comparison results). According to some embodiments, this may include sending the metrics and the comparison results to a UI, such as a GUI.

[0053] 5 is a flow chart diagram illustrating a method for evaluating an ML model including adding or updating requirements, according to one or more embodiments. Similar layers and components shown in FIG. 2 may be used to implement method 500, and a full description of similar steps from method 300 as shown in FIG. 3 and method 400 as shown in FIG. 4 above may be omitted for ease of reading.

[0054] Referring to FIG. 5, in operation S510, test parameters are received by requirements management layer 210 from storage device 220.

[0055] At operation S520, after receiving the requirements from storage 220, the requirements may be interpreted by requirements management layer 210 to determine how to instruct evaluation layer 200 to perform an ML evaluation on ML model 230. This may be similar to operation S330 described in FIG. 3 above.

[0056] At operation S530, requirements management layer 210 sends instructions to evaluation layer 200 based on the interpreted test parameters from operation S520 to cause evaluation layer 200 to perform ML evaluation on ML model 240. This may be similar to operation S330 described in FIG. 3 above.

[0057] At operation S540, an instruction to perform an ML evaluation may be received by evaluation layer 200 to cause evaluation layer 200 to perform an ML evaluation on ML model 230. This may be similar to operation S410 described in FIG. 4 above.

[0058] At operation S550, the evaluation layer 200 may send instructions to the unit test section to evaluate the metric (e.g., by calculating the metric and comparing it to a test criterion from the test parameters). This may be similar to operation S440 described in FIG. 4 above. Although not explicitly shown, it should be understood that, according to some embodiments, operations similar to S420-S430 may be included before operation S550.

[0059] At operation S560, the evaluation layer 200 may output the evaluated metrics (which may include the comparison results) and send the metrics and the comparison results to the UI, which may be similar to operation S450 described in FIG.

[0060] At operation S570, evaluation layer 200 may send instructions to requirements management layer 210 to add test parameters to storage layer 200 (e.g., by adding a new RaC file to storage 220) or to update current requirements (e.g., by editing an existing RaC file in storage 220). Instructions may be sent to update ML model tuning parameters, such as, but not limited to, confidence thresholds and test parameters to optimize the overall performance of a system that includes ML model 230.

[0061] In operation S580, requirements management layer 210 receives the instruction sent in operation S570 and adds or updates requirements in storage device 220 accordingly. Steps may then be repeated from S510. According to this embodiment, an iterative process may be implemented to search for requirements and parameters that may not yet be discovered.

[0062] According to the above embodiment, the requirements for the test parameters can be defined in a requirements-as-code (RaC) file, thus allowing a more complex yet precise range of test parameters that can be configured to evaluate machine learning (ML) models. Thus, ML evaluation may be performed to ensure with a high degree of accuracy that the ML model being evaluated is optimal with respect to the desired properties of the ML model.

[0063] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the one or more embodiments to the precise forms disclosed. Modifications and variations are possible from consideration of the disclosure or may be acquired from practice of one or more embodiments.

[0064] One or more embodiments may relate to a system, method, and / or computer-readable medium at any possible level of technical detail of integration. Furthermore, one or more of the above components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions for causing a processor to perform operations.

[0065] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media as used herein should not be interpreted as signals that are transitory in nature, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless communications, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0067] The computer readable program code / instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source code or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++ or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may run entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server, as a standalone software package. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In one or more embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects or operations using state information of the computer readable program instructions to personalize the electronic circuit.

[0068] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions operating on the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in the blocks or blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium having instructions stored therein includes an article of manufacture including instructions implementing aspects of the functions / operations specified in the blocks or blocks of the flowcharts and / or block diagrams.

[0069] Computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device and cause the computer, other programmable device, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions operating on the computer, other programmable device, or other device perform the functions / operations specified in the blocks or sections of the flowcharts and / or block diagrams.

[0070] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer-readable media according to one or more embodiments. In this regard, each block in the flowcharts or block diagrams may represent a microservice, a module, a segment, or a portion of instructions that include one or more executable instructions for implementing the specified logical function. The method, computer system, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those depicted in the figures. In one or more alternative implementations, the functions shown in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may in fact be executed in parallel or substantially in parallel, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that executes the specified functions or operations, or executes a combination of special-purpose hardware and computer instructions.

[0071] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit one or more embodiments. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for configuring test parameters for use in evaluating a machine learning model (ML model), comprising: obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; evaluating the ML model based on the interpreted at least one test parameter from the RaC file.

2. The method of claim 1 , wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

3. The method of claim 1 or 2, wherein the user input is obtained via a graphical user interface.

4. The method of claim 3 , wherein obtaining the user input further comprises selecting the at least one test parameter from a predefined library of test parameters.

5. 3. The method of claim 1 or 2, wherein evaluating the ML model is performed by an evaluation layer including an inference unit and a unit testing unit, the inference unit is configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a metric value.

6. 6. The method of claim 5, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

7. 7. The method of claim 6, further comprising receiving, by the requirements management layer, an instruction to add or update at least one test parameter in a storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

8. 1. An apparatus for configuring test parameters for use in evaluating a machine learning model (ML model), comprising: at least one memory storing computer executable instructions; at least one processor; The at least one processor is configured to execute the computer-executable instructions, obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; and evaluating the ML model based on the interpreted at least one test parameter from the RaC file; An apparatus configured to perform the steps of:

9. The apparatus of claim 8 , wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

10. 10. Apparatus according to claim 8 or 9, wherein the user input is obtained via a graphical user interface.

11. 11. The apparatus of claim 10, wherein the at least one processor is further configured to execute the computer-executable instructions to obtain the user input by selecting the at least one test parameter from a predefined library of test parameters.

12. 10. The apparatus of claim 8 or 9, wherein the at least one processor is further configured to evaluate the ML model using an evaluation layer comprising an inference unit and a unit testing unit, the inference unit configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit configured to evaluate the ML model based on the output from the ML model and obtain a metric value.

13. 13. The apparatus of claim 12, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

14. 14. The apparatus of claim 13, wherein the at least one processor is further configured to receive instructions from the requirements management layer to add or update at least one test parameter in a storage layer, the instructions being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

15. 1. A non-transitory computer-readable recording medium having stored thereon instructions executable by at least one processor to cause a method to be performed, the method comprising: obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; and evaluating a machine learning model (ML model) based on the interpreted at least one test parameter from the RaC file.

16. 20. The non-transitory computer-readable storage medium of claim 15, wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

17. 17. The non-transitory computer-readable storage medium of claim 15 or 16, wherein the user input is obtained via a graphical user interface.

18. 17. The non-transitory computer-readable recording medium of claim 15 or 16, wherein evaluating the ML model is performed by an evaluation layer including an inference unit and a unit testing unit, the inference unit configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit configured to evaluate the ML model based on the output from the ML model and obtain a metric value.

19. 20. The non-transitory computer-readable storage medium of claim 18, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

20. 20. The non-transitory computer-readable storage medium of claim 19, wherein the method further includes receiving, by the requirements management layer, an instruction to add or update at least one test parameter in a storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

Images