Test parameter forming method, test parameter forming device and computer program product

Abstract

The invention provides a test parameter forming method, a test parameter forming device and a computer program product. A method, system, and apparatus for configuring test parameters used in evaluation of a machine learning (ML) model are provided. Alternatively, the method comprises: obtaining a user input comprising at least one test parameter; saving the acquired user input as a key file (RaC file) as a code; interpreting at least one test parameter from the RaC file by the essential element management layer; the ML model is evaluated based on the at least one test parameter that has been interpreted from the RaC file.

Description

Technical Field

[0001] Systems and methods consistent with embodiments of the present disclosure relate to constructing test parameters for machine learning evaluation. Background Art

[0002] Machine learning (ML) models can be used to automate various tasks. When developing ML models, developers sometimes have specific benchmarks or parameters that need to be met by the ML model. For example, in the case where the ML model is intended to automate safety-critical tasks, the ML model may need to obtain a certain reliability evaluation. Therefore, in order to ensure that the ML model can meet the technical specifications, the developer may need to test and evaluate the ML model by performing an ML evaluation.

[0003] In the prior art, in order to automatically determine whether a model is good or not, a simple measurement value can be used. For example, mean Average Precision (mAP) can be used as a simple numerical value to indicate whether an ML model is good, poor, or excellent.

[0004] The degree of evaluation of ML models that can be achieved by prior art systems and methods is limited and may not be consistent with the accurate benchmarks or parameters that need to be achieved by the ML models. In particular, in the prior art, only simple metrics can often be used, and when trying to improve one metric, there may be a trade-off / disadvantage with other metrics, so it may be difficult to optimize the performance of the ML model. In the prior art methods and systems, especially in scenarios where there are many candidate ML models, there may be no means to establish a quantitative and clear benchmark for deciding which ML model is best.

[0005] Furthermore, even in the case of finding a “good” ML model that passes acceptance tests and has excellent metrology models, such an ML model may be considered suboptimal and may have unexpected flaws that are only discovered after being configured.

[0006] Therefore, there is a need for an improved method of executing the ML evaluation process, particularly with respect to test parameters (which may also include requirements, acceptance criteria, test conditions, etc.). Summary of the invention

[0007] According to one or more embodiments, an apparatus and method are provided for acquiring test parameters (which may also include requirements, qualification criteria, test conditions, etc.) from a user, saving such test parameters as a requirement as code (RaC) file, and interpreting the test parameters from the RaC file using layers (such as a requirement management layer). Therefore, the user can simply construct the RaC file in a manner consistent with the requirements required to meet a standard, and when executing the ML evaluation process, such RaC files can be used independently, sequentially, or in parallel. Therefore, ML evaluation can be performed in order to confirm with high accuracy that the evaluated ML model is optimal in terms of the desired characteristics of the ML model.

[0008] According to an embodiment, the method may also include: obtaining user input including at least one test parameter; saving the obtained user input as a requirement as code (RaC) file; interpreting at least one test parameter from the RaC file through the requirement management layer; and evaluating the ML model based on at least one interpreted test parameter from the RaC file.

[0009] According to an embodiment, at least one test parameter may include one or more acceptance requirements, acceptance criteria or test conditions. Alternatively, the user input may be obtained via a graphical user interface. Alternatively, obtaining the user input may further include: selecting at least one test parameter from a predefined library of test parameters.

[0010] According to an embodiment, the evaluation of the ML model may be performed by an evaluation layer having an inference unit and a unit test unit. Alternatively, the inference unit is configured to receive test data, perform an inference process based on the received test data, obtain output from the ML model, and thereby evaluate the ML model; and the unit test unit is configured to evaluate the ML model based on the output from the ML model and obtain a measurement value.

[0011] Alternatively, the output from the ML model includes an inference log, the inference unit is configured to transmit the inference log to the unit testing unit when the inference process is completed, and the evaluation of the ML model includes evaluating the measurement values ​​from the inference log, and the evaluated measurement values ​​are displayed on a graphical user interface.

[0012] Alternatively, the method further includes: receiving, through the requirement management layer, an instruction to add or update at least one test parameter in the storage layer, wherein the instruction is sent from the evaluation layer when the evaluation of the ML model is completed based on the evaluated measurement value.

[0013] According to an embodiment, a device for forming a test parameter used in evaluating a machine learning (ML) model may also be provided. Alternatively, the device includes: at least one memory storing computer-executable instructions; and at least one processor configured to execute the following computer-executable instructions: obtaining user input including at least one test parameter; saving the obtained user input as a requirement (RaC) file as code; interpreting at least one test parameter from the RaC file through the requirement management layer; and evaluating the ML model based on the at least one test parameter from the RaC file that has been interpreted.

[0014] According to an embodiment, at least one processor may be further configured to execute the following computer executable instructions: obtaining user input by selecting at least one test parameter from a predefined library of test parameters.

[0015] Regarding the additional aspects, some are described in the following description, and some are obvious from the description or can be understood by practicing the embodiments presented in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Hereinafter, features, aspects, and advantages of certain preferred embodiments of the present disclosure will be described with reference to the accompanying drawings, in which like reference numerals represent like elements.

[0017] Figure 1 It is a diagram showing an example of components of a device according to an embodiment.

[0018] Figure 2 is a diagram of a system architecture of one or more embodiments.

[0019] Figure 3 A flowchart representing a method of constructing a RaC file including test parameters and evaluating an ML model based on the interpreted test parameters, in accordance with one or more embodiments.

[0020] Figure 4 A flowchart illustrating a method of evaluating an ML model according to one or more embodiments.

[0021] Figure 5 A flowchart of a method for evaluating an ML model including adding or updating elements is provided, according to one or more embodiments. DETAILED DESCRIPTION

[0022] The following detailed description of the embodiments refers to the accompanying drawings. The present disclosure provides examples and descriptions, but is not intended to be all inclusive, nor is it intended to limit one or more embodiments to the strict form disclosed. Modifications and changes may be made in view of the present disclosure, or modifications and changes may be obtained based on the execution of one or more embodiments. Moreover, one or more features or constituent elements of an embodiment may also be combined or combined with other embodiments (or one or more features of other embodiments). Moreover, it is understood that in the flowcharts and descriptions of the actions provided in this specification, one or more actions may be omitted, one or more actions may be added, one or more actions may be performed simultaneously (at least partially), and the order of one or more actions may be interchanged.

[0023] It is apparent that the implementation of the system and / or method and / or computer-readable non-transitory storage medium (or computer program product) described in this specification can be implemented by different forms of hardware, firmware, or a combination of hardware and software. The actual special control hardware or software code used to implement the system and / or method does not limit one or more implementations. Therefore, the actions and behaviors of the system and / or method and / or computer-readable non-transitory storage medium and / or computer program product are described in this specification without reference to specific software codes. It is understood that software and hardware can also be designed to implement the system and / or method based on the description of this specification.

[0024] Even if specific combinations of features are recorded in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can also be combined by methods not specifically recorded in the claims and / or disclosed in the specification. The dependent claims listed below are sometimes directly dependent on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0025] Unless explicitly stated otherwise, any element, action, or instruction used in this specification should not be interpreted as being decisive or essential. In addition, when used in this specification, the articles "a" and "an" are intended to include more than one item and can be used interchangeably with "more than one". Figure 1In the case of multiple items, the term "one" or the like is used. In addition, when used in this specification, "has", "have", "having", "include", "including" or similar terms are intended to be open-ended terms. Moreover, unless otherwise clearly stated, the phrase "based on" is intended to mean "based at least in part on". Moreover, expressions such as "at least one of 'A' and 'B'" or "at least one of 'A' or 'B'" should be understood as only A, only B, or both A and B.

[0026] Figure 1 FIG. 1 is a diagram showing an example of components of the machine learning (ML) evaluation device 100. Figure 1 As shown, the ML evaluation device 100 may also include a bus 110 , a processor 120 , a memory 130 , a storage unit 140 , an input unit 150 , an output unit 160 , and a communication interface 170 .

[0027] The bus 110 includes components that enable communication between components of the ML evaluation device 100. The processor 120 may also be implemented by hardware, firmware, or a combination of hardware and software. The processor 120 may also be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing elements. In one or more embodiments, the processor 120 includes one or more processors that can be programmed to perform functions. The memory 130 includes a random access memory (RAM), a read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that store instructions and / or information for use by the processor 120.

[0028] The storage unit 140 stores information and / or software related to the operation and use of the ML evaluation device 100. For example, the storage unit 140 may include other types of computer-readable non-transitory media in addition to a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a semiconductor disk), a compact disc (CD: Compact Disc), a digital versatile disc (DVD: Digital Versatile Disc), a floppy disk, a cassette tape, a magnetic tape, and / or a corresponding drive device. The input unit 150 includes components that enable the ML evaluation device 100 to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone). In addition or instead, the input unit 150 may also include a sensor for detecting information (e.g., a global positioning system (GPS: Global Positioning System) element, an accelerometer, a gyroscope, and / or an actuator). The output unit 160 includes components that provide output information from the ML evaluation device 100 (e.g., a display, a speaker, and / or one or more light emitting diodes (LED: Light Emitting Diode)).

[0029] The communication interface 170 includes a transceiver-type component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the ML evaluation device 100 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 170 enables the ML evaluation device 100 to receive information from other devices and / or provide information to other devices. For example, the communication interface 170 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a WiFi interface, a cellular phone network interface, or a similar interface, but is not limited thereto.

[0030] The ML evaluation device 100 may also perform one or more examples of the processes described in this specification. According to one or more embodiments, the ML evaluation device 100 may also perform these processes in response to the processor 120 executing software instructions stored in a computer-readable non-transitory medium such as the memory 130 and / or the storage unit 140. A computer-readable medium is defined as a non-transitory storage device in this specification. A storage device includes storage space within a single physical storage device or storage space across multiple physical storage devices.

[0031] The software instructions may also be read into the memory 130 and / or the storage unit 140 from other computer-readable media, or read into the memory 130 and / or the storage unit 140 from other devices via the communication interface 170. Alternatively, when the software instructions stored in the memory 130 and / or the storage unit 140 are executed, the software instructions cause the processor 120 to perform one or more processes described in this specification.

[0032] In addition or in lieu thereof, hardwired circuits may be used in place of software instructions or in combination with software instructions to perform one or more of the processes described in this specification. Therefore, the one or more embodiments described in this specification are not limited to any specific combination of hardware circuits and software.

[0033] Figure 1 The number and configuration of the constituent elements shown are provided as examples. Figure 1 The ML evaluation device 100 may include additional components, fewer components, different components, or components with different configurations than the components shown. In addition or in lieu thereof, a group of components (e.g., one or more components) of the ML evaluation device 100 may also perform one or more functions described as being performed by another group of components of the ML evaluation device 100.

[0034] Figure 2 is a diagram of a system architecture of one or more embodiments.

[0035] According to an embodiment, an evaluation layer 200 may also be provided. The evaluation layer 200 may also play a role in executing the ML evaluation process (e.g., reasoning and device testing) of the ML model 230. It should be understood that the evaluation layer 200 may also implement any appropriate components for performing ML evaluation. The evaluation layer 200 may also be configured to receive instructions from the requirement management layer 210. According to some embodiments, it should be understood that the evaluation layer 200 may also include a UI (User Interface) (not shown) to output the results of the evaluation to the user. In particular, the evaluation layer 200 may also send an output signal including the results of the evaluation when the evaluation is completed, and the UI may also receive the output signal and display the results of the evaluation. The UI may also have a table format showing an overview of the results of the evaluation test, and may also emphasize specific parts of the test results based on whether it is qualified or failed. However, it should be understood that whether the UI of the evaluation layer 200 is included depends on the specific implementation, and its composition may also be determined by those skilled in the art.

[0036] A requirement management layer 210 may also be provided. The requirement management layer 210 may also play a role in interpreting files stored in the storage layer (e.g., requirement (RaC) files 1, 2, ... N (221-1, 221-2, ... 221-N) as codes). The requirement management layer 210 may also send instructions to the evaluation layer 200 based on the interpretation of the RaC files to perform ML evaluation.

[0037] The storage device 220 may also include all data used by the evaluation layer 200 and the requirement management layer 210. The storage device 220 may also be implemented by any appropriate storage component (e.g., a database, a cloud storage device, etc.). The storage device 220 may also include any number of RaC files 221-1, 221-2, ... 221-N and a test parameter library 222. It should be understood that each RaC file and test parameter library may also be stored in the same storage medium or different storage media.

[0038] The requirement (RaC) files 221-1, 221-2, ... 221-N as code may also function to store the expected behavior of the ML model 230. The expected behavior may also include test parameters. In particular, the test parameters may also specify the performance metric of the ML model 230 to be tested, for example, based on a specific metric to be achieved, a standard to be achieved, a type of test, etc. Therefore, the RaC file may also include test targets, file paths for test data, requirements, qualification criteria, test conditions, acceptance requirements, and acceptance criteria as test parameters, which are used during the ML evaluation process in the evaluation layer 200. Therefore, such requirements, test targets, file paths for test data, qualification criteria, and test conditions may also be immediately interpreted by the requirement management layer 210 to determine how the ML testing and evaluation of the ML model 230 should be performed by the evaluation layer 200. For example, the RaC file may include the benchmark in the form of code, and the requirement management layer 210 may interpret the code as a description of a method for performing the ML evaluation of the ML model 230 using the evaluation layer 200.

[0039] The RaC file may also be a computer-readable code in a format that is easily interpreted by humans. For example, it may be a YAML format, a domain-specific language (DSL) format, or the like. Thus, a person who is not proficient in syntactic coding may be able to easily construct an ML evaluation, or may be able to confirm that the RaC file is consistent with the quantitative requirements. Furthermore, this may enable advanced customization, for example, an end user may be able to adapt the ML evaluation to a more specific environment. Furthermore, the RaC file is separated from the code that is actually used to perform the ML evaluation, so it may no longer be necessary to "hard code" the requirements in the ML evaluation process.

[0040] According to an embodiment, the test parameters stored in the RaC file may also be related to each other. For example, some parameters may be associated with specific objects and belong to the same group. Alternatively, some parameters may be in a hierarchical relationship defined by a combination of the results of several other benchmarks, for example. However, the test parameters may also be independent of each other.

[0041] Since the entire requirement is in the form of a code rather than a set of numerical values, it has a higher accuracy that can be achieved by ensuring that the test requirement is consistent with the actual requirement. As a result, the accuracy for ensuring that the evaluated ML model is optimal may be improved.

[0042] It is considered that when multiple RaC files 221-1, 221-2, ... 221-N are used for ML evaluation, each RaC file 221-1, 221-2, ... 221-N can be used to check each of the multiple sets of requirements (sequentially or in parallel). In any case, RaC files can separate different sets of requirements, so in some embodiments, it is sometimes simpler to check each set separately (one by one).

[0043] According to some embodiments, the RaC files 221-1, 221-2, ... 221-N can be edited through the user input UI 240. In particular, the user input UI 240 can be a general command line interface (command terminal) used to edit source files, or a graphical user interface (GUI) having interactive elements that allow users to drag and drop, select predetermined structures, or select from a list. Such predetermined structures can also be obtained from the test parameter library 222 from the storage device 220, for example, it can also be one or more commonly used test parameter structures. However, it should be understood that any appropriate user interface can also be implemented for the user UI 240.

[0044] The test parameter library 222 may also include predetermined test parameters. According to some embodiments, the test parameter library 222 may also be stored separately from the RaC file. The specific format of the test parameters of the test parameter library 222 may be any appropriate format.

[0045] It should be understood that the above UI (including user input UI240) can be implemented in any appropriate environment, such as in a network interface or only in a local user device. It should also be understood that according to some embodiments, user input UI240 can also be displayed separately from other UIs or displayed simultaneously.

[0046] Figure 3is a flow chart representing a method 300 of constructing a RaC file including test parameters and evaluating an ML model based on the interpreted test parameters, in accordance with one or more embodiments.

[0047] Reference Figure 3 In action S310, for example, a user input including at least one test parameter may be obtained by using a UI such as user input UI 240 (which may also be a GUI or a command line terminal). According to an embodiment, the specific format of the user input may also be a format that can be understood by humans, and more than one test parameter may also be input at the same time. However, it should be understood that the user input may also include information other than one or more test parameters.

[0048] In action S320, the user input obtained in action S310 may be saved as one or more RaC files 221-1, 221-2, ... 221-N in the storage device 220. It should be understood that depending on the format of the user input, the user input may need to be converted into a different format or data type before being saved as RaC files 221-1, 221-2, ... 221-N.

[0049] In action S330, after the test parameters are saved in action S320, the test parameters may be interpreted by the requirement management layer 210 to determine how to instruct the evaluation layer 200 to perform the ML evaluation of the ML model 230. This action S330 may also include: interpreting model adjustment parameters such as requirements, qualification criteria, test conditions, acceptance requirements, acceptance criteria, test targets, test data paths, and confidence thresholds according to the test parameters; and determining appropriate tests and test parameters to be performed using the evaluation layer 200. Therefore, the interpreted test parameters may be obtained through the requirement management layer 210.

[0050] In action S340, the evaluation layer 200 may perform ML evaluation on the ML model 230 based on the interpreted test parameters. For example, the requirement management layer 210 may send an instruction to the evaluation layer 200 based on the interpreted test parameters obtained from action S330 in order to make the evaluation layer 200 perform ML evaluation on the ML model 230. According to some embodiments, once the evaluation layer 200 completes the execution of the ML evaluation, it may send an output signal including the evaluation result. According to some embodiments, the output signal may also be sent to the UI to display the result.

[0051] Figure 4 is a flowchart of a method 400 for evaluating an ML model using the evaluation layer 200, according to one or more embodiments. Figure 2Similar layers and components as shown in the figure are used to implement the method 400. For ease of reading, the same layers and components as shown in the figure may be omitted. Figure 3 A complete description of the steps of method 300 is shown.

[0052] according to Figure 4 In the illustrated embodiment, the evaluation layer 200 may include an inference unit and a unit test unit. The inference unit may also play the role of receiving test data, inputting the data into the ML model 230, and calculating the output (i.e., inference) using the ML model 230. The unit test unit may also play the role of executing device testing on the ML model 230. Specifically, the device test may also evaluate the result of the output from the ML model 230, wherein the result is obtained during the inference process performed on the ML model 230 using the inference unit. Metrics related to the performance of the ML model 230 may also be obtained using the unit test unit.

[0053] Reference Figure 4 In action S410, an instruction to execute ML evaluation may be sent from the requirement management layer 210, and the instruction is received by the evaluation layer 200 so that the evaluation layer 200 executes ML evaluation on the ML model 230. The instruction received in action S410 may also be the same as that in the above-mentioned Figure 3 The instruction sent in action S340 described in is similar.

[0054] In action S420, the evaluation layer 200 may send an instruction to the inference unit to perform an inference action based on the test data and using the ML model 230. The output of this action may also include an inference log, wherein the inference log includes data obtained by performing an inference action on the ML model 230 based on the test data.

[0055] In Act S430 , the evaluation layer 200 may send the inference log from the inference unit to the unit test unit.

[0056] In action S440 , the evaluation layer 200 may send an instruction to the unit test unit to evaluate the measurement value (for example, by calculating the measurement value and comparing it with a test benchmark obtained based on the test parameters).

[0057] In act S450, the evaluation layer 200 may output the evaluated measurement value (which may also include the comparison result). According to some embodiments, act S450 may also include sending the measurement value and the comparison result to a UI such as a GUI.

[0058] Figure 5 is a flowchart showing one or more embodiments of a method for evaluating an ML model including adding or updating elements. Figure 2Similar layers and components shown in the figure are used to implement the method 500. For ease of reading, the same layers and components as shown in the figure may be omitted. Figure 3 The method 300 shown and described above Figure 4 A complete description of the steps of method 400 is shown.

[0059] Reference Figure 5 In action S510 , the test parameters are received from the storage device 220 via the requirement management layer 210 .

[0060] In action S520, after receiving the requirement from the storage device 220, the requirement management layer 210 interprets the requirement to determine how to instruct the evaluation layer 200 to perform the ML evaluation on the ML model 230. This action S520 may also be combined with the above-mentioned Figure 3 The recorded action S330 is similar.

[0061] In action S530, the requirement management layer 210 sends instructions to the evaluation layer 200 based on the interpreted test parameters obtained from action S520, so that the evaluation layer 200 performs ML evaluation on the ML model 230. This action S530 can also be combined with the above-mentioned Figure 3 The recorded action S340 is similar.

[0062] In action S540, the evaluation layer 200 may receive an instruction to perform ML evaluation, so that the evaluation layer 200 performs ML evaluation on the ML model 230. This action S540 may also be combined with the above-mentioned Figure 4 The recorded action S410 is similar.

[0063] In action S550, the evaluation layer 200 may send an instruction to the unit test unit to evaluate the measurement value (for example, by calculating the measurement value and comparing it with the test benchmark obtained according to the test parameters). This action S550 may also be combined with the above-mentioned Figure 4 Although not explicitly shown, it should be understood that, according to some embodiments, actions similar to S420 to S430 may be included before action S550.

[0064] In action S560, the evaluation layer 200 may output the evaluated measurement value (which may also include the comparison result) and send the measurement value and the comparison result to the UI. Figure 4 The recorded action S450 is similar.

[0065] In action S570, the evaluation layer 200 may send an instruction to add the test parameters to the storage layer 200 (for example, by adding a new RaC file to the storage device 220) or an instruction to update the current requirements (for example, by editing the existing RaC file in the storage device 220) to the requirement management layer 210. The instruction is not limited, and for example, an instruction may be sent to update ML model adjustment parameters such as a confidence threshold and a test parameter for optimizing the overall performance of the system including the ML model 230.

[0066] In action S580, the requirement management layer 210 receives the instruction sent in action S570, and based on the instruction, adds or updates the requirements of the storage device 220. Thereafter, the steps from S510 may be repeatedly executed. According to this embodiment, an iterative process of searching for requirements and parameters that may not have been found may also be implemented.

[0067] According to the above-described embodiment, since the requirements of the test parameters can be defined by the requirements (RaC) file as code, a more complex but accurate range of test parameters can be implemented, which can be configured for evaluating the machine learning (ML) model. Therefore, the ML evaluation can also be performed in order to confirm with high accuracy that the evaluated ML model is optimal in terms of the desired characteristics of the ML model.

[0068] The above disclosure provides examples and illustrations, but is not intended to be all-inclusive, nor is it intended to limit one or more embodiments to the strict form disclosed. Modifications and variations may be made in light of the disclosure or may be acquired from the execution of one or more embodiments.

[0069] One or more embodiments may also be associated with a system, method, and / or computer-readable medium at any possible level of technical detail of the integration. Moreover, one or more of the above-mentioned constituent elements may also be implemented as instructions stored in a computer-readable medium and executable by at least one processor (and / or may also include at least one processor). The computer-readable medium may also include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions, wherein the computer-readable program instructions are used to cause the processor to perform an action.

[0070] The computer-readable storage medium may be a tangible device capable of holding and storing instructions for use with an instruction execution device. The computer-readable storage medium is not limited to the following, and may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination of the above devices. A non-inclusive list of more specific examples of computer-readable storage media includes the following. Portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device such as a punch card or a raised structure with a groove on which the instructions are recorded, and any appropriate combination of the above devices. The computer-readable storage medium used in this specification should not be interpreted as an essentially temporary signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating in a waveguide or other transmission medium (for example, a light pulse passing through an optical cable), or an electrical signal sent via a wire.

[0071] The computer-readable program instructions recorded in this specification can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may also include copper transmission cables, optical transmission fibers, wireless communications, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions to the computer-readable storage medium stored in each computing / processing device.

[0072] The computer-readable program code / instructions for performing actions may also be assembly instructions, instruction set architecture (ISA: instruction-set-architecture) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, wherein the one or more programming languages ​​include object-oriented programming languages ​​such as Smalltalk, C++ or similar languages, and procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed as a stand-alone software package on the user's computer as a whole, or may be executed partially on the user's computer, or may be executed partially on the user's computer and partially on a remote computer, or may be executed as a whole on a remote computer or server. In the latter scenario, the remote computer may also be connected to the user's computer via any type of network such as a local area network (LAN), a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In one or more embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize state information of computer-readable program instructions that personalize the electronic circuit to execute computer-readable program instructions to perform a scheme or action.

[0073] In order to generate instructions for performing actions via a processor of a computer or other programmable data processing device to implement the components of the functions / actions specified in the functional blocks or functional blocks of the flowchart and / or block diagram, these computer-readable program instructions can also be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to generate a machine. In order to make a computer-readable storage medium with instructions stored therein include a product containing instructions for implementing the scheme of the functions / actions specified in the functional blocks or functional blocks of the flowchart and / or block diagram, these computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device and / or other device to perform functions in a specific way.

[0074] In order to enable the instructions for performing actions in a computer, other programmable device or other device to implement the functions / actions specified in the functional blocks or functional blocks of the flowchart and / or block diagram, computer-readable program instructions can also be read into a computer, other programmable data processing device or other device to produce a computer implementation process in a manner that causes the computer, other programmable device or other device to perform a series of action steps.

[0075] The flowchart and block diagram of the figure show the composition, function and action of the possible implementation of the system, method and computer-readable medium of more than one implementation. In this regard, each functional block of the flowchart or block diagram can also represent a part of the instruction of microservice, module, segment or more than one executable instruction for implementing the specified logical function. Compared with the composition recorded in the figure, the present method, computer system and computer-readable medium can also include additional functional blocks, fewer functional blocks, different functional blocks or be configured as different functional blocks. In one or more alternative embodiments, the functions shown in the functional blocks are sometimes generated in an order that deviates from the order shown in the figure. For example, the two functional blocks shown in succession are actually sometimes executed in parallel or substantially in parallel, or the functional blocks are sometimes executed in the opposite order according to the associated functions. It should also be noted that each functional block of the block diagram and / or flowchart and the combination of the functional blocks of the block diagram and / or flowchart can be implemented by a system based on special-purpose hardware, wherein the system performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0076] It is apparent that the systems and / or methods described in this specification may be implemented by various forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement the system and / or method does not limit the system and / or method to one or more implementations. Therefore, the actions and behaviors of the system and / or method are described in this specification without reference to specific software code, but it should be understood that software and hardware may also be designed to implement the system and / or method based on the description of this specification.

Claims

1. A method for constructing test parameters, which is a method for constructing test parameters used in evaluating a machine learning (ML) model, wherein: The method comprises: obtaining user input including at least one test parameter; The obtained user input is saved as a code element file, namely, a RaC file; interpreting at least one test parameter from the RaC file through a component management layer; and The ML model is evaluated based on the interpreted at least one test parameter from the RaC file.

2. The test parameter formation method according to claim 1, wherein: The at least one test parameter includes one or more acceptance requirements, acceptance criteria or test conditions.

3. The test parameter formation method according to claim 1 or 2, wherein: The user input is obtained via a graphical user interface.

4. The test parameter formation method according to claim 3, wherein: Obtaining the user input further includes: selecting the at least one test parameter from a predefined library of test parameters.

5. The test parameter formation method according to any one of claims 1 to 4, wherein: The evaluation of the ML model is performed by an evaluation layer having an inference unit and a unit testing unit. The inference unit is configured to receive test data, perform an inference process based on the received test data, obtain output from the ML model, and thereby evaluate the ML model; the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a measurement value.

6. The test parameter formation method according to claim 5, wherein: The output from the ML model includes an inference log, the inference unit is configured to transmit the inference log to the unit testing unit when the inference process is completed, and evaluating the ML model includes evaluating the measurement values ​​from the inference log, and the evaluated measurement values ​​are displayed on a graphical user interface.

7. The test parameter formation method according to claim 6, wherein: The method further includes: receiving, through the requirement management layer, an instruction to add or update at least one test parameter in the storage layer, The instruction is sent from the evaluation layer when the evaluation of the ML model is completed based on the evaluated measurement value.

8. A test parameter forming device is a device for forming test parameters used in the evaluation of a machine learning (ML) model, wherein: The device comprises: at least one memory storing computer-executable instructions; and At least one processor, The at least one processor is configured to execute the following computer executable instructions: obtaining user input including at least one test parameter; The obtained user input is saved as a code element file, namely, a RaC file; interpreting at least one test parameter from the RaC file through a requirement management layer; as well as The ML model is evaluated based on the interpreted at least one test parameter from the RaC file.

9. The test parameter forming device according to claim 8, wherein: The at least one test parameter includes one or more acceptance requirements, acceptance criteria or test conditions.

10. The test parameter forming device according to claim 8 or 9, wherein: The user input is obtained via a graphical user interface.

11. The test parameter forming device according to claim 10, wherein: The at least one processor is further configured to execute the computer-executable instructions for obtaining the user input by selecting the at least one test parameter from a predefined library of test parameters.

12. The test parameter forming device according to any one of claims 8 to 11, wherein: The at least one processor is further configured to evaluate the ML model using an evaluation layer having an inference unit and a unit testing unit; the inference unit is configured to receive test data, perform an inference process based on the received test data, obtain output from the ML model, and thereby evaluate the ML model; the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a measurement value.

13. The test parameter forming device according to claim 12, wherein: The output from the ML model includes an inference log, the inference unit is configured to transmit the inference log to the unit testing unit when the inference process is completed, and evaluating the ML model includes evaluating the measurement values ​​from the inference log, and the evaluated measurement values ​​are displayed on a graphical user interface.

14. The test parameter forming device according to claim 13, wherein: The at least one processor is further configured to receive, through the requirement management layer, an instruction to add or update at least one test parameter in the storage layer, The instruction is sent from the evaluation layer when the evaluation of the ML model is completed based on the evaluated measurement value.

15. A computer program product having instructions recorded thereon, wherein the instructions can be executed by at least one processor to execute a test parameter formation method, wherein the test parameter formation method comprises: obtaining user input including at least one test parameter; The obtained user input is saved as a code element file, namely, a RaC file; interpreting at least one test parameter from the RaC file through a requirement management layer; as well as A machine learning (ML) model is evaluated based on the interpreted at least one test parameter from the RaC file.

16. The computer program product of claim 15, wherein: The at least one test parameter includes one or more acceptance requirements, acceptance criteria or test conditions.

17. A computer program product according to claim 15 or 16, wherein: The user input is obtained via a graphical user interface.

18. The computer program product according to any one of claims 15 to 17, wherein: The evaluation of the ML model is performed by an evaluation layer having an inference unit and a unit testing unit. The inference unit is configured to receive test data, perform an inference process based on the received test data, obtain output from the ML model, and thereby evaluate the ML model; the unit testing unit is configured to evaluate the ML model based on the output from the ML model and obtain a measurement value.

19. The computer program product of claim 18, wherein: The output from the ML model includes an inference log, the inference unit is configured to transmit the inference log to the unit testing unit when the inference process is completed, and evaluating the ML model includes evaluating the measurement values ​​from the inference log, and the evaluated measurement values ​​are displayed on a graphical user interface.

20. The computer program product of claim 19, wherein: The method further includes: receiving, through the requirement management layer, an instruction to add or update at least one test parameter in the storage layer, The instruction is sent from the evaluation layer when the evaluation of the ML model is completed based on the evaluated measurement value.

Images