Systems and methods for configuring test parameters in machine learning evaluations

Abstract

To provide a method.SOLUTION: There are provided a method, system, and device for configuring test parameters used in evaluating a machine learning (ML) model. The method may include: acquiring a user input including at least one test parameter; storing the acquired user input as a requirement file (RaC file) as a code; interpreting, by a requirement management layer, at least one test parameter from the RaC file; and evaluating the ML model on the basis of at least one test parameter interpreted from the RaC file.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Systems and methods consistent with embodiments of the present disclosure relate to configuring test parameters for machine learning evaluations. [Background technology]

[0002] Machine learning (ML) models can be used to automate various tasks. When developing an ML model, a developer may have specific criteria or parameters that need to be met by the ML model. For example, if the ML model is intended to automate a safety-critical task, the ML model may need to achieve a certain reliability rating. Therefore, it may be necessary for a developer to test and evaluate the ML model by performing an ML evaluation to ensure that the ML model can meet the specifications.

[0003] In the prior art, simple metrics can be used to automatically determine whether a model is good or not. For example, the mean average precision (mAP) may be used as a simple number that indicates whether an ML model is good, bad, or excellent.

[0004] Prior art systems and methods can only evaluate ML models to a limited extent and may not match the exact criteria or parameters required for the ML model. In particular, prior art methods typically use only simple metrics, and improving one metric may result in tradeoffs / penalties with other metrics, making it difficult to optimize the performance of the ML model. Prior art methods and systems may lack a means to establish quantitative and unambiguous criteria for determining which ML model is optimal, especially in scenarios where there are many candidate ML models.

[0005] Furthermore, even if a "good" ML model is found that passes acceptance tests and has a good metric model, it may still be considered suboptimal and may have unforeseen flaws that are only discovered after it is deployed.

[0006] Therefore, there is a need for improved ways to configure how the ML evaluation process is performed, particularly with respect to test parameters (which may include requirements, acceptance criteria, test conditions, etc.). Summary of the Invention

[0007] According to one or more embodiments, an apparatus and method are provided for obtaining test parameters (which may include requirements, pass criteria, test conditions, etc.) from a user, saving such test parameters in a Requirements as Code (RaC) file, and interpreting the test parameters from the RaC file using a layer (such as a requirements management layer). Thus, a user can easily configure an RaC file to match the requirements needed to meet a standard, and such RaC files can be used independently, sequentially, or in parallel with each other when performing an ML evaluation process. Thus, ML evaluation can be performed to confirm with a high degree of accuracy that the ML model being evaluated is optimal with respect to the desired characteristics of the ML model.

[0008] According to an embodiment, the method may include acquiring user input including at least one test parameter; saving the acquired user input as a requirements-as-code (RaC) file; interpreting, by a requirements management layer, the at least one test parameter from the RaC file; and evaluating the ML model based on the interpreted at least one test parameter from the RaC file.

[0009] According to an embodiment, the at least one test parameter may include one or more acceptance requirements, acceptance criteria, or test conditions. User input may be obtained via a graphical user interface. Obtaining user input may further include selecting the at least one test parameter from a predefined library of test parameters.

[0010] According to an embodiment, evaluating the ML model may be performed by an evaluation layer including an inference unit and a unit testing unit, wherein the inference unit is configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit is configured to evaluate the ML model based on the output from the ML model to obtain metrics.

[0011] The output from the ML model may include an inference log, and upon completion of the inference process, the inference unit may be configured to transfer the inference log to the unit testing unit, and evaluating the ML model may include evaluating metrics from the inference log, and the evaluated metrics may be displayed in a graphical user interface.

[0012] The method may further include receiving, by the requirements management layer, an instruction to add or update at least one test parameter in the storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

[0013] According to an embodiment, an apparatus for configuring test parameters for use in evaluating a machine learning (ML) model may be provided, the apparatus may include at least one memory storing computer-executable instructions, and at least one processor configured to execute the computer-executable instructions to acquire user input including the at least one test parameter, store the acquired user input as a requirements-as-code (RaC) file, interpret the at least one test parameter from the RaC file through a requirements management layer, and evaluate the ML model based on the interpreted at least one test parameter from the RaC file.

[0014] According to an embodiment, the at least one processor may be further configured to execute computer-executable instructions to obtain user input by selecting at least one test parameter from a predefined library of test parameters.

[0015] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of presented embodiments of the present disclosure. [Brief explanation of the drawings]

[0016] Features, aspects, and advantages of certain preferred embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1] FIG. 1 is a diagram of example components of an apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram of a system architecture according to one or more embodiments; and [Figure 3] FIG. 3 is a flow chart diagram illustrating a method for constructing an RaC file containing test parameters and evaluating an ML model based on the interpreted test parameters, according to one or more embodiments. [Figure 4]FIG. 4 is a flow chart diagram illustrating a method for evaluating an ML model, according to one or more embodiments; and [Figure 5] FIG. 5 is a flow chart diagram illustrating a method for evaluating an ML model, including adding or updating requirements, according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following detailed description of the embodiments refers to the accompanying drawings. The present disclosure provides illustration and description, but is not intended to be exhaustive or to limit one or more embodiments to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from the practice of one or more embodiments. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with other embodiments (or one or more features of other embodiments). Furthermore, in the flowcharts and descriptions of operations provided herein, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) concurrently, and the order of one or more operations may be rearranged.

[0018] It will be apparent that embodiments of the systems and / or methods and / or computer-readable non-transitory storage media described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods does not limit one or more embodiments. Thus, the operation and behavior of the systems and / or methods and / or computer-readable non-transitory storage media are described herein without reference to specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0019] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes each dependent claim in combination with all other claims in the claim set.

[0020] No element, act, or instruction used herein should be construed as critical or required unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar terms are used. Also, as used herein, the terms "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of 'A' and 'B'" or "at least one of 'A' or 'B'" should be understood to include A only, B only, or both A and B.

[0021] 1 is a diagram of example components of a machine learning (ML) evaluation device 100. As shown in FIG. 1, the ML evaluation device 100 may include a bus 110, a processor 120, a memory 130, a storage unit 140, an input unit 150, an output unit 160, and a communication interface 170.

[0022] Bus 110 includes components that enable communication between components of ML evaluator 100. Processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 120 may be a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or another type of processing element. In one or more embodiments, processor 120 includes one or more processors that are programmable to perform functions. Memory 130 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores instructions and / or information for use by processor 120.

[0023] The memory unit 140 stores information and / or software related to the operation and use of the ML evaluation device 100. For example, the memory unit 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or a corresponding drive, as well as other types of computer-readable non-transitory media. The input unit 150 includes components (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone) that enable the ML evaluation device 100 to receive information, for example, via user input. Additionally or alternatively, the input unit 150 may include sensors (e.g., a global positioning system (GPS) element, an accelerometer, a gyroscope, and / or an actuator) for detecting information. The output unit 160 includes components (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)) that provide output information from the ML evaluation device 100.

[0024] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the ML evaluation device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 170 can enable the ML evaluation device 100 to receive information from and / or provide information to other devices. For example, the communication interface 170 can include, but is not limited to, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a WiFi interface, a cellular network interface, or the like.

[0025] The ML evaluation device 100 may perform one or more instances of the processes described herein. According to one or more embodiments, the ML evaluation device 100 may perform these processes in response to the processor 120 executing software instructions stored on a computer-readable, non-transitory medium, such as the memory 130 and / or the storage unit 140. A computer-readable medium is defined herein as a non-transitory storage device. A storage device may include memory space within a single physical storage device or memory space spanning multiple physical storage devices.

[0026] The software instructions may be loaded into memory 130 and / or storage 140 from another computer-readable medium or from another device via communications interface 170. When executed, the software instructions stored in memory 130 and / or storage 140 may cause processor 120 to perform one or more of the processes described herein.

[0027] Additionally, or instead, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more of the processes described herein. Thus, one or more embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0028] The number and arrangement of components shown in Figure 1 are provided as an example. In practice, ML evaluation apparatus 100 may include additional, fewer, different, or differently arranged components than those shown in Figure 1. Additionally or alternatively, one set of components (e.g., one or more components) of ML evaluation apparatus 100 may perform one or more functions that are described as being performed by another set of components of ML evaluation apparatus 100.

[0029] FIG. 2 is a diagram of a system architecture according to one or more embodiments.

[0030] According to an embodiment, an evaluation layer 200 may be provided. The evaluation layer 200 may be responsible for performing ML evaluation processes (e.g., inference and device testing) on ​​the ML model 230. It should be understood that the evaluation layer 200 may implement any suitable means for performing ML evaluation. The evaluation layer 200 may be configured to receive instructions from the requirements management layer 210. It should be understood that, according to some embodiments, the evaluation layer 200 may include a UI (not shown) to output the results of the evaluation to a user. In particular, the evaluation layer 200 may send an output signal including the results of the evaluation upon completing the evaluation, and the UI may receive the output signal and display the results of the evaluation. The UI may have a table format that lists the results of the evaluation tests and may highlight certain portions of the test results based on whether they passed or failed. Nevertheless, it should be understood that whether a UI is included in the evaluation layer 200 depends on the particular implementation, and its configuration may also be determined by one of ordinary skill in the art.

[0031] A requirements management layer 210 may be provided. The requirements management layer 210 may be responsible for interpreting files (e.g., requirements as code (RaC) files 1, 2...N (221-1, 221-2, 221-N...)) stored in the storage layer. Based on the interpretation of the RaC files, the requirements management layer 210 may send instructions to the evaluation layer 200 to perform ML evaluations.

[0032] The storage device 220 may include all data used by the evaluation layer 200 and the requirements management layer 210. The storage device 220 may be implemented by any suitable storage means (e.g., a database, cloud storage, etc.). The storage device 220 may also include any number of RaC files 221-1, 221-2, 221-N, and a test parameter library 222. It should be understood that each RaC file and test parameter library may be stored on the same storage medium or different storage media.

[0033] The requirements-as-code (RaC) files 221-1, 221-2, 221-N may serve to store the expected behavior of the ML model 230. The expected behavior may include test parameters. In particular, the test parameters may specify performance metrics of the ML model 230 that need to be tested, for example, based on a specific number of metrics that need to be achieved, a standard that needs to be achieved, the type of test, etc. Thus, the RaC files may include test goals, test data file paths, requirements, pass criteria, test conditions, acceptance requirements, and acceptance criteria as test parameters, which will be used during the ML evaluation process in the evaluation layer 200. Accordingly, such requirements, test goals, test data file paths, pass criteria, and test conditions may be readily interpreted by the requirements management layer 210 to determine how ML testing and evaluation of the ML model 230 should be performed by the evaluation layer 200. For example, the RaC files may include standards in the form of code, and the requirements management layer 210 can interpret the code into instructions on how to perform ML evaluation of the ML model 230 using the evaluation layer 200.

[0034] RaC files may be computer-readable code in a format that is easily interpreted by humans, such as YAML or a domain-specific language (DSL) format. This potentially allows those unfamiliar with coding syntax to easily configure the ML evaluation and verify that the RaC file matches their quantitative requirements. Furthermore, this potentially allows for a high degree of customization, for example, allowing end users to adapt the ML evaluation in more specific environments. Furthermore, because the RaC file is separate from the code used to actually perform the ML evaluation, it may eliminate the need to "hard-code" requirements into the ML evaluation process.

[0035] According to an embodiment, test parameters stored in an RaC file may be related to one another. For example, certain parameters may be related to a particular subject and belong to the same group. Alternatively, certain parameters may be in a hierarchical relationship, for example, where one criterion is defined by the combination of the results of several other criteria. Nevertheless, test parameters may be independent of one another.

[0036] Because the entire requirement is in the form of code rather than a set of numbers, there is greater precision that can be achieved by ensuring that the test requirements match the actual requirements, which can improve accuracy in ensuring that the ML model being evaluated is optimal.

[0037] If multiple RaC files 221-1, 221-2, 221-N are used in an ML evaluation, it is contemplated that each RaC file 221-1, 221-2, 221-N can be used to check each of the multiple sets of requirements (sequentially or in parallel). In any event, because the RaC files can separate different sets of requirements, in some embodiments it may be easier to check each set individually (one at a time).

[0038] According to some embodiments, the RaC files 221-1, 221-2, 221-N may be editable by a user input UI 240. In particular, the user input UI 240 may be a typical command line interface (command terminal) used for editing source files, or a graphical user interface (GUI) with interactive elements that allow a user to drag and drop, select predefined configurations, or select from a list. Such predefined configurations may be retrieved from a test parameter library 222 from the storage device 220 and may be, for example, one or more commonly used test parameter configurations. Nevertheless, it should be understood that any suitable user interface may be implemented for the user UI 240.

[0039] The test parameter library 222 may include predefined test parameters. The test parameter library 222 may be stored separately from the RaC file according to some embodiments. The specific format of the test parameters in the test parameter library 222 may be any suitable format.

[0040] It should be understood that the above-described UI (including user input UI 240) can be implemented in any suitable environment, for example, in a web interface or solely on a local user device. It should also be understood that, according to some embodiments, user input UI 240 may be displayed separately from or simultaneously with another UI.

[0041] FIG. 3 is a flow chart diagram illustrating a method 300 for constructing an RaC file containing test parameters and evaluating an ML model based on the interpreted test parameters, according to one or more embodiments.

[0042] 3, in operation S310, user input including at least one or more test parameters may be obtained, for example, by using a UI (which may be a GUI or a command line terminal) such as user input UI 240. According to an embodiment, the particular format of the user input may be a human-readable format, and one or more test parameters may be entered simultaneously. Nevertheless, it should be understood that the user input may include information other than one or more test parameters.

[0043] At operation S320, the user input obtained from operation S310 may be saved in one or more RaC files 221-1, 221-2, 221-N in storage device 220. It should be understood that depending on the format of the user input, the user input may need to be converted into a different format or data type before being saved in RaC file 221-1, 221-2, or 221-N.

[0044] At act S330, after the test parameters are saved at act S320, the test parameters may be interpreted by requirements management layer 210 to determine how to instruct evaluation layer 200 to perform ML evaluation on ML model 230. This may include interpreting model tuning parameters, such as requirements, pass criteria, test conditions, acceptance requirements, acceptance criteria, test goals, test data paths, and confidence thresholds, from the test parameters, and determining the appropriate tests and test parameters that need to be executed using evaluation layer 200. Thus, the interpreted test parameters may be obtained by requirements management layer 210.

[0045] At operation S340, evaluation layer 200 may perform an ML evaluation on ML model 230 based on the interpreted test parameters. For example, requirements management layer 210 may send instructions to evaluation layer 200 based on the interpreted test parameters from operation S330 to cause evaluation layer 200 to perform an ML evaluation on ML model 230. According to some embodiments, once evaluation layer 200 completes performing the ML evaluation, it may send an output signal including the results of the evaluation. According to some embodiments, this output signal may be sent to a UI to display the results.

[0046] 4 is a flowchart diagram illustrating a method 400 for evaluating an ML model using the evaluation layer 200, according to one or more embodiments. Similar layers and components shown in FIG. 2 may be used to implement the method 400, and a full description of similar steps from the method 300 as shown in FIG. 3 may be omitted for readability.

[0047] According to the embodiment shown in FIG. 4, the evaluation layer 200 may include an inference unit and a unit test unit. The inference unit may be responsible for receiving test data, inputting the data into the ML model 230, and calculating outputs using the ML model 230 (i.e., inference). The unit test unit may be responsible for performing device tests on the ML model 230. Specifically, the device tests may evaluate output results from the ML model 230 obtained during an inference process performed on the ML model 230 using the inference unit. Metrics regarding the performance of the ML model 230 may be obtained using the unit test unit.

[0048] 4, at operation S410, an instruction to perform an ML evaluation may be sent from requirements management layer 210 and received by evaluation layer 200 to cause evaluation layer 200 to perform an ML evaluation on ML model 230. The instruction received at operation S410 may be similar to that sent at operation S340 described in FIG. 3 above.

[0049] At operation S420, evaluation layer 200 may send instructions to the inference unit to perform an inference operation using ML model 230 with the test data. The output of this operation may include an inference log that includes data obtained from performing the inference operation on ML model 230 with the test data.

[0050] In operation S430, the evaluation layer 200 may send an inference log from the inference unit to the unit test unit.

[0051] In operation S440, the evaluation layer 200 may send instructions to the unit test section to evaluate the metric (eg, by calculating the metric and comparing it to a test criterion from the test parameters).

[0052] At operation S450, the evaluation layer 200 may output the evaluated metrics (which may include the comparison results). According to some embodiments, this may include sending the metrics and the comparison results to a UI, such as a GUI.

[0053] 5 is a flowchart illustrating a method for evaluating an ML model, including adding or updating requirements, according to one or more embodiments. Similar layers and components shown in FIG. 2 may be used to implement method 500, and full descriptions of similar steps from method 300 as shown in FIG. 3 and method 400 as shown in FIG. 4 above may be omitted for readability.

[0054] Referring to FIG. 5, in operation S510, test parameters are received by requirements management layer 210 from storage device 220.

[0055] At operation S520, after receiving the requirements from storage 220, the requirements may be interpreted by requirements management layer 210 to determine how to instruct evaluation layer 200 to perform an ML evaluation on ML model 230. This may be similar to operation S330 described in FIG. 3 above.

[0056] At operation S530, requirements management layer 210 sends instructions to evaluation layer 200 based on the interpreted test parameters from operation S520 to cause evaluation layer 200 to perform ML evaluation on ML model 240. This may be similar to operation S330 described in FIG. 3 above.

[0057] At operation S540, an instruction to perform ML evaluation may be received by evaluation layer 200 to cause evaluation layer 200 to perform ML evaluation on ML model 230. This may be similar to operation S410 described in FIG. 4 above.

[0058] At operation S550, the evaluation layer 200 may send instructions to the unit test unit to evaluate the metric (e.g., by calculating the metric and comparing it to a test criterion from the test parameters). This may be similar to operation S440 described in FIG. 4 above. Although not explicitly shown, it should be understood that, according to some embodiments, operations similar to S420-S430 may be included before operation S550.

[0059] At operation S560, the evaluation layer 200 may output the evaluated metrics (which may include the comparison results) and send the metrics and the comparison results to the UI, which may be similar to operation S450 described in FIG. 4 above.

[0060] At operation S570, evaluation layer 200 may send instructions to requirements management layer 210 to add test parameters to storage layer 200 (e.g., by adding a new RaC file to storage device 220) or to update current requirements (e.g., by editing an existing RaC file in storage device 220). Instructions may be sent to update ML model tuning parameters, such as, but not limited to, confidence thresholds and test parameters to optimize the overall performance of a system that includes ML model 230.

[0061] In operation S580, requirements management layer 210 receives the instruction sent in operation S570 and adds or updates requirements in storage device 220 accordingly. Steps may then be repeated from S510. According to this embodiment, an iterative process may be implemented to search for requirements and parameters that may not yet be discovered.

[0062] According to the above embodiments, the requirements for the test parameters can be defined in a requirements-as-code (RaC) file, thus allowing for a more complex yet precise range of test parameters that can be configured to evaluate machine learning (ML) models. Thus, ML evaluation may be performed to ensure with high accuracy that the evaluated ML model is optimal with respect to the desired characteristics of the ML model.

[0063] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit one or more embodiments to the precise forms disclosed. Modifications and variations are possible from consideration of the disclosure or may be acquired from practice of one or more embodiments.

[0064] One or more embodiments may relate to a system, method, and / or computer-readable medium at any possible level of technical detail of integration. Furthermore, one or more of the above components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions for causing a processor to perform an operation.

[0065] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution apparatus. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or grooved ridge structures having recorded instructions thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as signals that are transitory in nature, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless communications, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0067] The computer-readable program code / instructions for performing operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may operate entirely on the user's computer, as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In one or more embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions to perform aspects or operations using state information of the computer-readable program instructions to personalize the electronic circuit.

[0068] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, operating on the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer-readable program instructions may be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium storing the instructions therein comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0069] Computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device and cause the computer, other programmable device, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions operating on the computer, other programmable device, or other device perform the functions / operations defined in the blocks or blocks of the flowcharts and / or block diagrams.

[0070] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to one or more embodiments. In this regard, each block in the flowcharts or block diagrams may represent a microservice, module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to those depicted in the figures. In one or more alternative implementations, the functions depicted in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may in fact be executed in parallel or substantially in parallel, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by special-purpose hardware-based systems that perform the specified functions or operations or execute a combination of special-purpose hardware and computer instructions.

[0071] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit one or more embodiments. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A processor-implemented method for configuring test parameters for use in evaluating a machine learning model (ML model), comprising: obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; evaluating the ML model based on the interpreted at least one test parameter from the RaC file; Including, A method in which evaluating the ML model is performed by an evaluation layer including an inference unit and a unit testing unit, the inference unit is configured to receive test data and evaluate the ML model by performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a metric value.

2. The method of claim 1 , wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

3. The method of claim 1 or 2, wherein the user input is obtained via a graphical user interface.

4. The method of claim 3 , wherein obtaining the user input further comprises selecting the at least one test parameter from a predefined library of test parameters.

5. 2. The method of claim 1, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

6. 6. The method of claim 5, further comprising receiving, by the requirements management layer, an instruction to add or update at least one test parameter in a storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

7. 1. An apparatus for configuring test parameters for use in evaluating a machine learning model (ML model), comprising: at least one memory storing computer-executable instructions; at least one processor; The at least one processor may further include the computer-executable instructions: obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; and evaluating the ML model based on the interpreted at least one test parameter from the RaC file; evaluating the ML model using an evaluation layer comprising an inference unit and a unit testing unit; configured to run The apparatus is configured such that the inference unit receives test data, performs an inference process based on the received test data, and evaluates the ML model by obtaining an output from the ML model, and the unit testing unit evaluates the ML model based on the output from the ML model and obtains a metric value.

8. The apparatus of claim 7 , wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

9. 9. Apparatus according to claim 7 or 8, wherein the user input is obtained via a graphical user interface.

10. 10. The apparatus of claim 9, wherein the at least one processor is further configured to execute the computer-executable instructions to obtain the user input by selecting the at least one test parameter from a predefined library of test parameters.

11. 11. The apparatus of claim 10, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

12. 12. The apparatus of claim 11, wherein the at least one processor is further configured to receive, by the requirements management layer, an instruction to add or update at least one test parameter in a storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

13. 1. A non-transitory computer-readable recording medium having stored thereon instructions executable by at least one processor to perform a method, the method comprising: obtaining user input including at least one test parameter; saving the captured user input as a Requirements as Code file (RaC file); interpreting, by a requirements management layer, at least one test parameter from the RaC file; evaluating a machine learning model (ML model) based on the interpreted at least one test parameter from the RaC file; Including, A non-transitory computer-readable recording medium, wherein evaluating the ML model is performed by an evaluation layer including an inference unit and a unit testing unit, the inference unit is configured to evaluate the ML model by receiving test data and performing an inference process based on the received test data to obtain output from the ML model, and the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a metric value.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the at least one test parameter comprises one or more of an acceptance requirement, an acceptance criterion, or a test condition.

15. 15. The non-transitory computer-readable storage medium of claim 13 or 14, wherein the user input is obtained via a graphical user interface.

16. 14. The non-transitory computer-readable storage medium of claim 13, wherein the output from the ML model includes an inference log, and when the inference process is completed, the inference unit is configured to transfer the inference log to the unit testing unit, and evaluating the ML model includes evaluating metrics from the inference log, and the evaluated metrics are displayed in a graphical user interface.

17. 17. The non-transitory computer-readable storage medium of claim 16, wherein the method further comprises receiving, by the requirements management layer, an instruction to add or update at least one test parameter in a storage layer, the instruction being sent from the evaluation layer upon completion of evaluation of the ML model based on the evaluated metrics.

Images