Machine learning evaluation method, machine learning evaluation device, and readable recording medium

Abstract

The invention provides a machine learning evaluation method for evaluating a machine learning (ML) model, a machine learning evaluation device, and a readable recording medium. The method may include: receiving, by a requirement management layer, at least one requirement obtained from a storage layer; interpreting the at least one element through an element management layer; and transmitting, by the element management layer, a command for performing an ML evaluation process to an execution layer based on the interpreted element, where the execution layer transmits an output signal having a result of the ML evaluation process when the ML evaluation process is completed.

Description

Technical Field

[0001] Systems and methods integrated with exemplary embodiments of the present disclosure relate to providing a pipeline for evaluating machine learning models. Background Art

[0002] Machine Learning (ML) models can be used to automate a wide variety of tasks. When developing ML models, developers may have specific benchmarks or parameters that need to be met by the ML model. For example, where the ML model is intended to automate safety-critical tasks, the ML model may need to achieve a specific reliability rating. Therefore, in order to ensure that the ML model can meet the standards, developers may need to test and evaluate the ML model by performing an ML evaluation.

[0003] In related technologies, in order to automatically determine whether a model is good or not, a simple metric can be used. For example, the mean average precision (mAP) can be used as a simple numerical value to indicate whether an ML model is good, poor, or excellent.

[0004] The process of evaluating ML models used by systems and methods of the prior art may be limited and slow. In particular, typically, the components used in the ML evaluation process are independent. For example, the components for acquiring test data, the ML model itself, and the ML evaluation test unit may all be located in separate systems. Moreover, in the related art, each step in the evaluation process may require a human user to interpret the results of the ML evaluation and consider how to iterate the evaluation process to obtain a more optimized ML model.

[0005] Therefore, more rationalized and automated methods are needed for ML evaluation. Summary of the invention

[0006] According to one or more exemplary embodiments, an apparatus and method for evaluating a machine learning (ML) model are provided. In particular, the apparatus and method of the exemplary embodiment may be, in the requirement management layer, receiving at least one requirement obtained from the storage layer, interpreting the requirement in the requirement management layer, and sending a command for performing an ML evaluation process to the execution layer based on the interpreted test parameters through the requirement management layer. It may be that when the ML evaluation process is completed, an output signal having the result of the ML evaluation process is sent through the execution layer. It may be that, based on the output signal, information on the result is displayed on a user interface (UI). Therefore, the entire process of constituting and executing the ML evaluation process can be rationalized / encapsulated in a single pipeline, and can be arbitrarily and selectively presented to the user using a user interface of a layer connected to the pipeline, thereby improving the automation of the evaluation process.

[0007] According to an embodiment, a method for evaluating an ML model may be provided. The method may include: receiving at least one requirement obtained from a storage layer through a requirement management layer; interpreting at least one requirement through the requirement management layer; and sending a command for performing an ML evaluation process to an execution layer based on the interpreted test parameters through the requirement management layer, wherein the execution layer sends an output signal having a result of the ML evaluation process when the ML evaluation process is completed.

[0008] At least one test parameter may be in the form of a requirements code (RaC: Requirements as Code) file.

[0009] It may be that the storage layer is in communication with a first user interface, the first user interface being configured to enable a user to edit at least one test parameter.

[0010] It may be that the execution layer is configured to send an output signal to a second user interface configured to display a result of the ML evaluation process.

[0011] The first user interface and the second user interface may be displayed simultaneously.

[0012] The execution layer may include a conclusion inference component and a unit test component. When a command for performing an ML evaluation process is received, the conclusion inference component is configured to receive test data and perform a conclusion inference process based on the test data and a command for obtaining output from the ML model. The unit test component is configured to perform an evaluation process based on the output from the ML model and a command for obtaining indicators.

[0013] It may be that the conclusion inference component is configured to receive the test data from the test data storage layer.

[0014] According to an embodiment, a device for evaluating a machine learning (ML) model may be provided. The device may include: at least one memory storing computer executable commands; and at least one processor configured to execute the computer executable commands to perform the following processing: receiving at least one element obtained from the storage layer through the element management layer; interpreting at least one element through the element management layer; and sending a command for performing an ML evaluation process to the execution layer based on the interpreted element through the element management layer, wherein the execution layer sends an output signal having the result of the ML evaluation process when the ML evaluation process is completed.

[0015] The additional aspects will be described in part in the following description and in part will be obvious from the description, or may be achieved through practice of the embodiments presented in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Hereinafter, features, aspects, and advantages of specific preferred embodiments of the present disclosure will be described with reference to the accompanying drawings, in which like reference numerals represent like elements.

[0017] Figure 1 is a diagram of exemplary components of an apparatus of an exemplary embodiment.

[0018] Figure 2 is a system architecture diagram showing the layers and components of one or more exemplary embodiments.

[0019] Figure 3 is a flowchart illustrating a method of evaluating an ML model according to one or more exemplary embodiments.

[0020] Figure 4 is a flowchart of a method for evaluating an ML model using a conclusion inference component and a unit testing component, representing one or more exemplary embodiments.

[0021] Figure 5 A flowchart of a method for evaluating an ML model including adding or updating elements is provided, illustrating one or more exemplary embodiments. DETAILED DESCRIPTION

[0022] The following detailed description of exemplary embodiments refers to the accompanying drawings. The present disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit one or more exemplary embodiments to the exact form disclosed. Modifications and variations may be made in light of the present disclosure, or may be learned from the implementation of one or more exemplary embodiments. Moreover, one or more features or components of an exemplary embodiment may be combined with or may be combined with another exemplary embodiment (or one or more features of another exemplary embodiment). Moreover, it is understood that in the flowcharts and descriptions of the actions provided in this specification, one or more actions may be omitted, one or more actions may be added, one or more actions may be performed (at least partially) simultaneously, and the order of one or more actions may be switched.

[0023] It is apparent that the exemplary embodiments of the systems and / or methods and / or non-transitory computer-readable storage media described in this specification may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the system and / or method is not limited to one or more exemplary embodiments. Therefore, the actions and behaviors of the systems and / or methods and / or non-transitory computer-readable storage media are described in this specification without reference to specific software codes. It is understood that the software and hardware are designed to implement the system and / or method based on the description of this specification.

[0024] A particular combination of features is listed in the claims and / or disclosed in the specification, but this combination is not intended to limit the disclosure of possible exemplary embodiments. In fact, many of the features may be combined in ways not specifically listed in the claims and / or disclosed in the specification. Each dependent claim listed below is directly dependent on only one claim, but the disclosure of possible exemplary embodiments includes each dependent claim in combination with all other claims in the claim set.

[0025] As long as it is not particularly clearly recorded, the elements, behaviors or commands used in this specification should not be interpreted as important or necessary. In addition, the articles "a" and "an" used in this specification are intended to include more than one item and can be used interchangeably with "more than one". In the case of only one item, a term such as "one" or the same term is used. In addition, the terms "has", "have", "having", "include", "including" or similar terms used in this specification are meant to be open-ended terms. Moreover, as long as it is not particularly clearly recorded, phrases such as "based on..." are intended to mean "at least partially based on...". Moreover, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B or both A and B.

[0026] Figure 1 is a diagram of exemplary components of a machine learning (ML) evaluation apparatus 100. Figure 1 As shown, the ML evaluation device 100 may include a bus 110 , a processor 120 , a memory 130 , a storage component 140 , an input component 150 , an output component 160 , and a communication interface 170 .

[0027] The bus 110 includes components that enable communication between components of the ML evaluation device 100. The processor 120 may be implemented by hardware, firmware, or a combination of hardware and software. The processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. In one or more exemplary embodiments, the processor 120 includes one or more processors that can be programmed to perform functions. The memory 130 may be a random access memory (RAM), a read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or commands for use by the processor 120.

[0028] The storage component 140 stores information and / or software associated with the operation and use of the ML evaluation device 100. For example, the storage component 140 may include a corresponding drive and include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk), a compact disc (CD: Compact Disc), a digital versatile disc (DVD: Digital Versatile Disc), a floppy disk, a cartridge, a magnetic tape, and / or other types of non-transitory computer-readable media. The input component 150 may include a component that enables the ML evaluation device 100 to receive information via user input, etc. (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone). In addition or in lieu thereof, the input component 150 may include a sensor that senses information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output component 160 includes a component that provides output information from the ML evaluation device 100 (e.g., a display, a speaker, and / or one or more light emitting diodes (LED: Light Emitting Diode)).

[0029] The communication interface 170 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the ML evaluation device 100 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 170 enables the ML evaluation device 100 to receive information from another device and / or provide information to another device. For example, the communication interface 170 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi (Wireless Fidelity) interface, a cellular network interface, or the like, but is not limited thereto.

[0030] The ML evaluation device 100 may perform one or more exemplary processes described in this specification. According to one or more exemplary embodiments, the ML evaluation device 100 may perform the process in response to the processor 120 executing software commands stored in a non-transitory computer-readable medium such as the memory 130 and / or the storage component 140. The computer-readable medium is defined as a non-transitory storage device in this specification. The storage device includes storage space within a single physical storage device or storage space across multiple physical storage devices.

[0031] The software commands may be read into the memory 130 and / or storage component 140 from another computer-readable medium or from another device via the communication interface 170. When executed, the software commands stored in the memory 130 and / or storage component 140 may cause the processor 120 to perform one or more processes described in this specification.

[0032] Additionally or alternatively, to perform one or more processes described in this specification, hardwired circuits may be used in place of software commands, or in combination with software commands. Therefore, the one or more exemplary embodiments described in this specification are not limited to any specific combination of hardware circuits and software.

[0033] supply Figure 1 The number and configuration of components shown are provided as an example. Figure 1 The ML evaluation device 100 may include additional components, fewer components, different components, or components in a different configuration than shown. Additionally or alternatively, a set of components (e.g., more than one component) of the ML evaluation device 100 may perform one or more functions documented as being performed by another set of components of the ML evaluation device 100.

[0034] Figure 2is a system architecture diagram showing layers and components of one or more exemplary embodiments. The layers (eg, execution layer 200, requirement management layer 210, and storage layer 220) can be combined to be viewed as a machine learning (ML) evaluation pipeline.

[0035] According to an embodiment, an execution layer 200 may be provided. The execution layer 200 may include a conclusion inference component 201 and a unit test component 202. The execution layer 200 may be responsible for the execution of the ML evaluation process for the ML model 240. It should be understood that the execution layer 200 may implement any appropriate means for executing the ML evaluation. The execution layer 200 may be configured to receive commands from the element management layer 210. The execution layer 200 may also interact with the display result UI 260 in order to output the result of the evaluation. In particular, the execution layer 200 may send an output signal including the result of the evaluation when the evaluation is completed, and according to a part of the embodiment, the display result UI 260 may receive the output signal and display the result of the evaluation. The display result UI 260 may be implemented using any appropriate UI. For example, the display result UI 260 may have a table form in order to list the results of the evaluation test, and the display result UI 260 may also emphasize a specific part of the test result based on whether the evaluation test is qualified or unqualified. However, it should be understood that the display result UI 260 may depend on a specific implementation scheme as determined by those skilled in the art.

[0036] The execution layer 200 may also receive test data from the test data storage 222 via the requirement management layer 210 , or directly receive test data from the test data storage 222 .

[0037] According to some embodiments, conclusion inference component 201 may be responsible for receiving data from test data storage 222 and inputting the data to ML model 240 for computing output using ML model 240 (ie, conclusion inference).

[0038] According to some embodiments, the unit test component 202 may be responsible for executing unit tests for the ML model 240. Specifically, the unit test may evaluate the results of the output from the ML model 240 obtained in the conclusion inference process performed by the conclusion inference component 201 for the ML model 240. Metrics related to the performance of the ML model 240 may be obtained using the unit test component 202.

[0039] A requirement management layer 210 may be provided. The requirement management layer 210 may be responsible for the interpretation of files stored in the storage layer (e.g., requirement code (RaC) files 1, 2, ... N (221-1, 221-2, 221-N, ...)) or test data from the test data storage 222. Based on the interpretation of the RaC files, the requirement management layer 210 may send a command to the execution layer 200 to perform ML evaluation.

[0040] The storage layer 220 may include all data used by the execution layer 200 and the element management layer 210. The storage layer 220 may be implemented by any appropriate storage component (e.g., a database, a cloud storage, etc.). The storage layer 220 may include any number of RaC files 221-1, 221-2, 221-N and a test data storage 222. It should be understood that each RaC file and test data storage may be stored in the same storage medium or in different storage media.

[0041] Requirement code (RaC) files 221-1, 221-2, 221-N may be responsible for the storage of the expected behavior of the ML model 240. The expected behavior may have requirements (which may include test parameters). In particular, the RaC file may be, for example, a human-readable form that specifies the performance indicators of the ML model to be tested based on a specific indicator benchmark to be achieved, a standard to be achieved, a type of test, etc. The RaC file may include requirements, qualified benchmarks, test targets, file paths for test data, and test conditions used in the ML evaluation process in the execution layer 200. Therefore, such requirements, qualified benchmarks, test targets, test data paths, and test conditions can be easily interpreted by the requirement management layer 210 to determine how ML testing and evaluation of the ML model 240 should be performed by the execution layer 200.

[0042] According to some embodiments, the RaC files 221-1, 221-2, and 221-N can be edited through the edit prompt UI 250. In particular, the edit prompt UI 250 can be a general command line interface used to edit the source file, or the edit prompt UI 250 can be a graphical user interface (GUI) having elements that enable the user to interact in a manner of dragging and dropping or selecting a specified component. However, it should be understood that any appropriate user interface can be implemented for the edit prompt UI 250.

[0043] The test data storage 222 may include data to be used by the conclusion inference component 201 as input to the ML model 240. According to some embodiments, the test data storage 222 may be stored separately from the RaC file. The specific format of the data file of the test data may be any suitable format.

[0044] It should be understood that the editing prompt UI 250 and the display result UI 260 can be implemented in any appropriate environment, for example, in a web interface, or only on a local user device. It should also be understood that according to some embodiments, the editing prompt UI 250 and the display result UI 260 can be displayed separately or simultaneously.

[0045] Depending on the implementation, a UI (which may include features of the edit prompt UI 250 and the display result UI 260) may also be provided, which may include further features for streamlining the evaluation process. For example, the user interface may communicate with an interface for enabling a user to select which set of elements from the storage layer 220 should be used and their ordering, e.g., whether the elements of the first set and the second set should be evaluated sequentially or in parallel, and communicate with the element management layer 210. The above user interface also enables the user to select a test data set (e.g., from the test data storage 222). It may also be assumed that the user can configure the same set of elements to act on different candidate ML models and automatically show which candidate ML model has the highest metric.

[0046] Figure 3 is a flow chart illustrating a method 300 of evaluating an ML model according to one or more exemplary embodiments.

[0047] Reference Figure 3 In action S310, the requirements including the test parameters are received from the storage layer 220 via the requirement management layer 210. According to some embodiments, the requirements (such as the test parameters) may be one or more of the RaC files 221-1, 221-2, and 221-N.

[0048] After receiving the requirements from the storage layer 220, in action S320, the requirements may be interpreted by the requirement management layer 210 to determine how to instruct the execution layer 200 to perform ML evaluation on the ML model 240. This action S320 may include: interpreting the requirements, qualified benchmarks, test targets, test data paths, model adjustment parameters such as confidence thresholds, and test conditions based on the requirements; and determining appropriate tests and test parameters that should be performed using the execution layer 200. Therefore, the interpreted requirements may be obtained by the requirement management layer 210.

[0049] In action S330, the requirement management layer 210 sends a command to the execution layer 200 based on the test parameters interpreted in action S320 in order to make the execution layer 200 perform ML evaluation on the ML model 240. According to some embodiments, when the execution layer 200 finishes the execution of the ML evaluation, the execution layer 200 sends an output signal including the evaluation result. According to some embodiments, the output signal may be sent to the display result UI 260.

[0050] Figure 4 is a flow chart of a method 400 for evaluating an ML model using a conclusion inference component and a unit testing component, according to one or more exemplary embodiments. Figure 2 The same layers and components as shown can be used to implement method 400. For ease of reading, the same layers and components as shown are omitted. Figure 3 A complete description of the same steps of method 300 is shown.

[0051] Reference Figure 4 In action S410, a command for performing ML evaluation may be sent from the requirement management layer 210 and received by the execution layer 200, so that the execution layer 200 performs ML evaluation on the ML model 240. The command received in action S410 may be the same as that received in Figure 3 The same command is sent in action S330 as described above.

[0052] In action S420, the execution layer 200 may send the test data (e.g., the test data obtained from the test data storage 222) to the conclusion inference component 201, and send a command to the conclusion inference component 201 to perform the conclusion inference action using the ML model 240. The output of this action may include a conclusion inference log, which may include data obtained by executing the conclusion inference action for the ML model 240 using the test data.

[0053] In action S430 , the execution layer 200 may send a conclusion inference log from the conclusion inference component 201 to the unit test component 202 .

[0054] In action S440 , the execution layer 200 may send a command to the unit test component 202 to evaluate the metric (eg, by calculating and comparing the metric with respect to a requirement-based test benchmark).

[0055] In action S450, the execution layer 200 may output the evaluated indicator (which may include the comparison result). According to some implementations, the action S450 may include: sending the indicator and the comparison result to a UI such as the display result UI 260.

[0056] Figure 5 A flowchart of a method for evaluating an ML model including adding or updating elements is provided, showing one or more exemplary embodiments. Figure 2 The same layers and components as shown can be used to implement method 500. For ease of reading, the same layers and components as shown are omitted. Figure 3 The method 300 shown and described above Figure 4 A complete description of the same steps of method 400 is shown.

[0057] Reference Figure 5 In action S510, a requirement including a test parameter is received from storage layer 220 via requirement management layer 210. This can be done with Figure 3 The same as the action S310 mentioned above.

[0058] After receiving the requirements from the storage layer 220, in action S520, the requirements can be interpreted by the requirement management layer 210 to determine how to instruct the execution layer 200 to perform ML evaluation on the ML model 240. This can be done with Figure 3 The same as the action S320 mentioned above.

[0059] In action S530, in order to make the execution layer 200 perform ML evaluation on the ML model 240, the requirement management layer 210 sends a command to the execution layer 200 based on the test parameters interpreted in action S520. Figure 3 The same as the above-mentioned action S330.

[0060] In action S540, a command for performing ML evaluation may be received by the execution layer 200, so that the execution layer 200 performs ML evaluation on the ML model 240. This may be related to Figure 4 The same as the action S410 mentioned above.

[0061] In action S550, the command execution layer 200 may send a command to the unit test component 202 to evaluate the metric (eg, by calculating and comparing the metric with respect to a requirement-based test benchmark). This may be related to Figure 4 Although not explicitly shown, it should be understood that, according to some embodiments, before action S550, actions similar to those of S420 to S430 may also be included.

[0062] In action S560, the execution layer 200 may output the evaluated indicator (which may include the comparison result) and send the indicator and the comparison result to a UI such as the display result UI 260. Figure 4 The same as the action S450 mentioned above.

[0063] In action S570, the execution layer 200 may send a command to the requirement management layer 210 to add a requirement to the storage layer 220 (e.g., by adding a new RaC file to the storage layer 220) or to update an existing requirement (e.g., by editing an existing RaC file in the storage layer 220). The command may be sent to update ML model adjustment parameters, such as confidence thresholds and test parameters for optimizing the overall performance of the system including the ML model 240, but is not limited thereto.

[0064] In action S580, the requirement management layer 210 receives the command sent in action S570, and accordingly adds or updates the requirement in the storage layer 220. Thereafter, the steps from S510 may be repeatedly executed. According to this embodiment, an iterative process of searching for requirements and parameters that may not have been found can be implemented.

[0065] It can be understood that based on the above embodiments, the entire process of constructing and executing the ML evaluation process can be rationalized / encapsulated in a single pipeline, and can be arbitrarily and selectively presented to the user using a user interface of a layer connected to the pipeline, thereby improving the automation of the evaluation process.

[0066] The above disclosure provides examples and illustrations, but is not intended to be exhaustive or to limit one or more exemplary embodiments to the precise forms disclosed. Modifications and variations may be made in light of this disclosure or may be learned from practice of one or more exemplary embodiments.

[0067] One or more exemplary embodiments may involve systems, methods, and / or computer-readable media at any possible level of technical detail of the integration. Moreover, one or more of the above components may be implemented as commands stored in a computer-readable medium and executable by at least one processor (and / or include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium(s)) having computer-readable program commands for causing the processor to perform actions.

[0068] A computer-readable storage medium may be a tangible device that can hold and store commands used by a command execution device. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited to these. A more specific exemplary non-exhaustive list of computer-readable storage media includes the following, namely, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disc read-only memories (CD-ROM), digital versatile discs (DVD), memory sticks, floppy disks, punched cards with commands recorded or mechanically encoded devices such as raised structures in grooves, and any suitable combination of the above. The computer-readable storage medium used in this specification should not be interpreted as a temporary signal itself such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (for example, a light pulse through an optical cable), or an electrical signal sent through a wire.

[0069] The computer-readable program commands recorded in this specification can be downloaded from a computer-readable storage medium to a respective computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can have copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program commands from the network and transmits the computer-readable program commands to be stored in a computer-readable storage medium in the respective computing / processing device.

[0070] The computer readable program code / commands for performing the actions may be source code or object code written in assembler commands, instruction-set-architecture (ISA) commands, machine commands, machine-dependent commands, microcode, firmware commands, state setting data, configuration data for integrated circuits, or any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or similar languages, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program commands may be executed entirely on the user's computer, may be executed partially on the user's computer, may be executed as a stand-alone software package, may be executed partially on the user's computer and partially on a remote computer, or may be executed entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., via the Internet using an Internet service provider). In one or more exemplary embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may personalize the electronic circuit using state information of computer-readable program commands, thereby executing the computer-readable program commands to perform a scheme or action.

[0071] The computer-readable program commands may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device in order to generate a machine, and as a result, the commands executed by the processor of the computer or other programmable data processing device generate components that implement the functions / behaviors specified in the functional blocks or functional blocks (multiple) of the flowchart and / or block diagram. The computer-readable program commands may also be stored in a computer-readable storage medium, wherein the computer-readable storage medium may instruct a computer, a programmable data processing device, and / or other devices to function in a specific manner, and as a result, the computer-readable storage medium having the commands stored therein has an article of manufacture, which includes commands for implementing the scheme of the functions / behaviors specified in the functional blocks or functional blocks (multiple) of the flowchart and / or block diagram.

[0072] Computer-readable program commands may also be loaded onto a computer, other programmable data processing apparatus, or other device, and perform a series of action steps on the computer, other programmable apparatus, or other device to generate a computer-implemented process, as a result of which the commands executed on the computer, other programmable apparatus, or other device implement the functions / behaviors specified in the functional box or functional boxes (multiple) in the flowchart and / or block diagram.

[0073] The flowchart and block diagram in the figure show the architecture, functions and actions of possible exemplary embodiments of the system, method and computer-readable medium of more than one exemplary embodiment. In this regard, each functional block in the flowchart or block diagram may represent a microservice, module, segment or part of a command having more than one executable command to implement the specified logical function. The method, computer system and computer-readable medium may include additional functional blocks, fewer functional blocks, different functional blocks or functional blocks of different configurations compared to those depicted in the accompanying drawings. In more than one alternative exemplary embodiment, the functions recorded in the functional blocks may be generated independently of the order recorded in the figure. For example, two functional blocks shown in succession may be executed in fact or substantially at the same time, or the functional blocks may be executed in reverse order according to the associated functions. It should be noted that each functional block of the block diagram and / or flowchart, and the combination of functional blocks of the block diagram and / or flowchart can be implemented by a dedicated hardware-based system, wherein the system performs a specified function or behavior, or executes a combination of dedicated hardware and computer commands.

[0074] It is apparent that the systems and / or methods described in this specification may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement the system and / or method is not limited to one or more exemplary embodiments. Therefore, it is understood that the actions and behaviors of the system and / or method are described in this specification without reference to specific software codes, and the software and hardware may be designed to implement the system and / or method based on the description of this specification.

Claims

1. A machine learning evaluation method is a method for evaluating a machine learning ML model, wherein: The machine learning evaluation method comprises: Receiving, through the element management layer, at least one element obtained from the storage layer; interpreting the at least one element through the element management layer; and A command for performing an ML evaluation process based on the interpreted requirements is sent to the execution layer through the requirement management layer, wherein the execution layer sends an output signal having a result of the ML evaluation process when the ML evaluation process is completed.

2. The machine learning evaluation method according to claim 1, wherein: The at least one requirement is in the form of a requirement code RaC file.

3. The machine learning evaluation method according to claim 1 or 2, wherein: The storage layer is in communication with a first user interface configured to enable a user to edit the at least one element.

4. The machine learning evaluation method according to claim 3, wherein: The execution layer is configured to send the output signal to a second user interface configured to display the result of the ML evaluation process.

5. The machine learning evaluation method according to claim 4, wherein: The execution layer includes a conclusion inference component and a unit test component. When receiving the command for performing the ML evaluation process, the conclusion inference component is configured to receive test data and perform a conclusion inference process based on the test data and a command for obtaining an output from the ML model. The unit test component is configured to perform an evaluation process based on the output from the ML model and the command for obtaining an indicator.

6. The machine learning evaluation method according to claim 5, wherein: The output from the ML model has a conclusion inference log. When the conclusion inference process is completed, the conclusion inference component is configured to transmit the conclusion inference log to the unit test component. The evaluation process includes evaluating indicators from the conclusion inference log, and the evaluated indicators are displayed on the second user interface.

7. The machine learning evaluation method according to claim 6, wherein: The machine learning evaluation method further includes receiving, through the requirement management layer, a command for adding or updating at least one requirement in the storage layer, wherein when the ML evaluation process is completed, the command is sent from the execution layer based on the evaluated indicator.

8. A machine learning evaluation device is a device for evaluating a machine learning ML model, wherein: The machine learning evaluation device comprises: at least one memory storing computer executable instructions; and at least one processor, The at least one processor is configured to execute the computer executable instructions to perform the following processing: Receiving, through the element management layer, at least one element obtained from the storage layer; Explaining the at least one element through the element management layer; as well as A command for performing an ML evaluation process based on the interpreted requirements is sent to the execution layer through the requirement management layer, wherein the execution layer sends an output signal having a result of the ML evaluation process when the ML evaluation process is completed.

9. The machine learning evaluation device according to claim 8, wherein: The at least one requirement is in the form of a requirement code RaC file.

10. The machine learning evaluation device according to claim 8 or 9, wherein: The storage layer is in communication with a first user interface configured to enable a user to edit the at least one element.

11. The machine learning evaluation device according to claim 10, wherein: The execution layer is configured to send the output signal to a second user interface configured to display the result of the ML evaluation process.

12. The machine learning evaluation device according to claim 11, wherein: The execution layer includes a conclusion inference component and a unit test component. When receiving the command for performing the ML evaluation process, the conclusion inference component is configured to receive test data and perform a conclusion inference process based on the test data and a command for obtaining an output from the ML model. The unit test component is configured to perform an evaluation process based on the output from the ML model and the command for obtaining an indicator.

13. The machine learning evaluation device according to claim 12, wherein: The output from the ML model has a conclusion inference log. When the conclusion inference process is completed, the conclusion inference component is configured to transmit the conclusion inference log to the unit test component. The evaluation process includes evaluating indicators from the conclusion inference log, and the evaluated indicators are displayed on the second user interface.

14. The machine learning evaluation device according to claim 13, wherein: The processor is further configured to execute the computer executable instructions to receive a command for appending or updating at least one requirement in the storage layer through the requirement management layer, wherein the command is sent from the execution layer based on the evaluated indicator when the ML evaluation process is completed.

15. A non-transitory computer-readable recording medium recording instructions executable by at least one processor to cause the processor to perform a machine learning evaluation method, wherein: The machine learning evaluation method comprises: Receiving, through the element management layer, at least one element obtained from the storage layer; interpreting the at least one element through the element management layer; and A command for performing an ML evaluation process based on the interpreted requirements is sent to the execution layer through the requirement management layer, wherein the execution layer sends an output signal having a result of the ML evaluation process when the ML evaluation process is completed.

16. The non-transitory computer-readable recording medium according to claim 15, wherein: The at least one requirement is in the form of a requirement code RaC file.

17. The non-transitory computer-readable recording medium according to claim 15 or 16, wherein: The storage layer is in communication with a first user interface configured to enable a user to edit the at least one element.

18. The non-transitory computer-readable recording medium according to claim 17, wherein: The execution layer is configured to send the output signal to a second user interface configured to display the result of the ML evaluation process.

19. The non-transitory computer-readable recording medium according to claim 18, wherein: The execution layer includes a conclusion inference component and a unit test component. When receiving the command for performing the ML evaluation process, the conclusion inference component is configured to receive test data and perform a conclusion inference process based on the test data and a command for obtaining output from the ML model. The unit test component is configured to perform an evaluation process based on the output from the ML model and the command for obtaining indicators.

20. The non-transitory computer-readable recording medium according to claim 19, wherein The output from the ML model has a conclusion inference log. When the conclusion inference process is completed, the conclusion inference component is configured to transmit the conclusion inference log to the unit test component. The evaluation process includes evaluating indicators from the conclusion inference log, and the evaluated indicators are displayed on the second user interface.

Images