Computer systems and computer programs for machine learning

Abstract

To provide a computer system and a method which train a learning model and thereby create a utility model, and assist with an operation for making practical use thereof.SOLUTION: Provided is a computer system 50 prepares learning models in one or more learning units 102; receiving an input of learning data from a data source 111, and training said one or more learning models using the learning data. One or more utility models are finalized on the basis of said one or more trained learning models, and said one or more utility models are deployed to one or more inference units 114. Each of said one or more inference units 114 receives an input of utility data from the data source 111, provides the utility data to the utility models to execute an inference, and transmits, to a data target 113, inference result data outputted from the utility models.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates generally to machine learning and / or inference using computer systems. [Background technology]

[0002] In recent years, inference using neural networks configured for practical use has been used in image recognition, automatic stock trading, automatic translation, etc. For example, Patent Document 1 discloses that training data, which are images inspected by experts, can be applied to a neural network (hereinafter, referred to as NN) (paragraphs 0056 and 0058). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-517052 Summary of the Invention [Problem to be solved by the invention]

[0004] In general, it is not easy for many users to properly perform machine learning on a neural network according to the intended use, and then deploy the completed machine learning neural network on a computer for practical use.

[0005] Such issues can also occur with computational models that can be machine-learned other than NNs (i.e., so-called AI (Artificial Intelligence) in general). [Means for solving the problem]

[0006] A computer system according to one aspect of the present disclosure includes a physical computer having at least one processor, at least one storage device that stores an instruction set executed by the at least one processor and that can store data that can be processed by the at least one processor, and at least one interface that is connected to the at least one processor and at least one communication network. Prepare one or more learning models, inputting training data from at least one data source through the at least one interface, and storing the input training data in the at least one storage device; generating one or more trained models by performing machine learning on the one or more learning models using the training data; determining at least one working model based on the one or more trained models; inputting operational data from at least one data source through said at least one interface; performing inference using the practical data with the at least one practical model; transmitting inference result data output from the at least one practical model by the inference to at least one data target through the at least one interface; The computer system according to this aspect is configured to assist a user in the tasks of performing machine learning of a learning model, preparing a practical model based on the results of the machine learning, and using the practical model for actual inference.

[0007] According to another aspect of the present disclosure, there is provided a computer system including at least one processor; The system includes at least one storage device that stores an instruction set executed by the at least one processor and that can store data that can be processed by the at least one processor, and at least one interface that is connected to the at least one processor and to at least one communication network. Prepare one or more learning models, inputting at least one unit of training data from at least one data source through the at least one interface, and storing the input training data in the at least one storage device; transmitting the at least one unit of training data to at least one annotator terminal through the at least one interface; receiving at least one annotation result related to each of the at least one unit of training data from the at least one annotator terminal through the at least one interface; storing the received at least one annotation result in the at least one storage device in association with each of the at least one unit of training data; generating one or more trained models by performing the machine learning of the one or more learning models using the at least one unit of training data and the at least one annotation result associated therewith as training data; The computer system according to this other aspect can assist a user in the task of preparing training data and performing machine learning of a learning model using the training data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 shows a physical configuration of a computer system according to an embodiment of the present invention. [Figure 2] 1 shows the logical configuration of a computer system. [Figure 3] It shows the logical configuration of at least one physical computer. [Figure 4]1 shows the configuration details of a virtual unit. [Figure 5] This shows the overall process flow for generating and deploying a practical NN (neural network) that corresponds to the intended use. [Figure 6] 1 shows the configuration of the annotation section of the computer system. [Figure 7] 1 shows the configuration of the annotation management database. [Figure 8] This shows the control flow for starting annotation work. [Figure 9] This shows the control flow for annotation generation in annotation work. [Figure 10] 10 shows the control flow of the assignment process in the flow shown in FIG. 9. [Figure 11] The control flow shown in Figure 9 is expanded by adding a suggestion process. [Figure 12] This shows the control flow for reviewing annotation work and reporting the results. [Figure 13] This shows the control flow for automatic learning that is performed in parallel with annotation work. [Figure 14] The control flow for automatically changing task priorities is shown below. [Figure 15] This shows the control flow for issue management in annotation work. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will be described below.

[0010] In the following description, the term "interface unit" includes one or more interfaces. The one or more interfaces may be one or more interface devices of the same type (for example, one or more NICs (Network Interface Cards)) or two or more interface devices of different types (for example, a NIC and an HBA (Host Bus Adapter)).

[0011] In the following description, "storage unit" includes memory and storage. The storage has one or more storage devices, each of which is typically a non-volatile storage device (e.g., an auxiliary storage device), such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory has one or more storage devices, each of which may be a volatile storage device or a non-volatile storage device.

[0012] In the following description, a "processor unit" includes one or more processors. At least one processor is typically a CPU (Central Processing Unit). The processor may include a hardware circuit that performs some or all of the processing.

[0013] In the following description, a "program" refers to a set of instructions executable by a processor unit. When a program, i.e., an instruction set, is executed by a processor unit, various processes, controls, or functions as described below are performed using a storage unit (e.g., a memory) and / or an interface unit as appropriate. Therefore, in the following description, processes, controls, or functions may be described using a program as the subject, but the subject may also be the processor unit (or a device or system having the processor unit). A program may be installed into a computer system from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs. A program is typically stored in the storage of a computer system.

[0014] Furthermore, a "dataset" is a logical block or collection of electronic data as seen from a program such as an application program, and may be, for example, any of a record, a file, a key-value pair, and a tuple. A dataset or each unit of data contained in a dataset is typically stored in the storage of a computer system.

[0015] FIG. 1 shows the physical configuration of a computer system according to an embodiment of the present invention.

[0016] The computer system 50 is configured from a plurality of (or one) physical computers 201 connected to a network 240.

[0017] The network 240 is one or more communication networks, and may include, for example, at least one of an FC (Fibre Channel) network and an IP (Internet Protocol) network. The network 240 may exist outside the computer system 50.

[0018] Each physical computer 201 is, for example, a general-purpose computer, and has physical computer resources 330. The physical computer resources 330 include an interface unit 251 connected to a network 240, a storage unit 252, and a processor unit 253 connected to these.

[0019] The computer system 50 may be, for example, a cloud computing system that provides XaaS (X as a Service). "XaaS" generally refers to a service that makes available some resources (e.g., hardware, lines, software execution environment, application programs, development environment, etc.) required for system construction or operation via a network such as the Internet. The letter (or word) used for the "X" in XaaS varies depending on the type (service model) of XaaS. Examples of XaaS include PaaS (Platform as a Service), SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), and HaaS (Hardware as a Service).

[0020] FIG. 2 shows the logical configuration of the computer system 50.

[0021] The computer system 50 provides a learning inference support system 100 that supports at least one of learning and inference of one or more computational models capable of machine learning. Here, a "computational model" refers to a computational process performed by a computer, i.e., a computational system implemented on a computer. In this embodiment, the computational model is a neural network (NN). However, NN is merely an example, and the computational model does not necessarily have to be a NN. It may be another type of computational process or computational system. The learning inference support system 100 may be implemented on, for example, a cloud platform provided by the computer system 50. The learning inference support system 100 includes a learning inference system 101 and a management system 161 that manages the learning inference system 101. The management system 161 may also be a system that manages a system other than the learning inference system 101.

[0022] The learning inference system 101 is a system that performs machine learning (e.g., deep learning) using one or more computational models, such as NNs, and inference using one or more computational models, such as NNs, that have completed machine learning. The learning inference system 101 is connected to one or more data sources 111, one or more data targets 113, and one or more user terminals 112 via a network 140. The network 140 may be the above-mentioned network 240 or a network different from the network 240. The network 140 may also be one or more communication networks.

[0023] The data source 111 is the source (read source) of the data set. The data target 113 is the target (output destination) of the data set. Either the data source 111 or the data target 113 may be, for example, a website, network storage, or a user terminal 112.

[0024] The user terminal 112 is a computer (client) used by a user. The user terminal 112 is used, for example, to instruct learning and deployment to the learning inference system 101. The user terminal 112 may be a computer used by a so-called cloud worker.

[0025] At least one of the data source 111, the data target 113, and the user terminal 112 may be a system implemented in the computer system 50 (for example, a virtual computer).

[0026] The learning inference system 101 includes an input API (Application Programming Interface) 106, input meta information 121, a data pool 105, an extraction unit 119, extracted meta information 129, an unstructured storage 104, an annotation unit 103, annotation meta information 130, a learning unit 102, a virtual NN host 116, an inference unit 114, an inference / statistics unit 115, an output API 107, and output meta information 122. At least one of the input API 106 (an example of an input unit), the extraction unit 119, the annotation unit 103, the learning unit 102, the virtual NN host 116, the inference unit 114, the inference / statistics unit 115, and the output API 107 (an example of an output unit) is one or more programs (in other words, a function realized by one or more programs being executed by a processor). In addition, in the learning inference system 101, two or more learning units 102 may be executed in parallel, and two or more inference units 114 may be executed in parallel. There may be two or more of at least one of the elements other than the learning unit 102 and the inference unit 114. Information such as the input meta information 121, the extracted meta information 129, the annotation meta information 130, and the output meta information 122 is stored in one or more storage units 252 in the computer system 50. In addition, storage areas such as the data pool 105 and the unstructured storage 104 are storage areas provided by the one or more storage units 252 in the computer system 50.

[0027] The input API 106 reads one or more data sets from one or more data sources 111 based on the input meta information 121 and stores the read data sets in the data pool 105 .

[0028] The input meta information 121 is information indicating the attributes of a dataset to be read by the input API 106, and includes, for example, information indicating the location where the dataset is stored and the identifier of the dataset.

[0029] The data pool 105 is an example of a logical storage area for storing datasets. Specifically, the data pool 105 stores datasets read by the input API 106 from one or more data sources 111. Furthermore, the data pool 105 may store a dataset as an inference result from at least one of the inference unit 114 and the inference / statistics unit 115. The data pool 105 may contain a mixture of structured datasets and unstructured datasets. Here, structured data refers to data that includes semantics that allow statistical processing by a computer, such as point-of-sale (POS) data indicating information such as "sales amount of product A is B," or weather data indicating information such as "weather in area A is B." On the other hand, unstructured data refers to data that does not include semantics that allow statistical processing by a computer, such as simple image, text, or audio data.

[0030] The extraction unit 119 extracts an unstructured data set from the data pool 105 based on the extracted meta information 129 and stores the unstructured data set in the unstructured storage 104 .

[0031] The extracted meta information 129 is information indicating the attributes of the unstructured dataset that is the target of extraction by the extraction unit 119, and includes, for example, information indicating the identifier of the unstructured dataset. The extracted unstructured dataset is an unstructured dataset for learning or verification, for example, a photo image file containing images of animals such as dogs and cats. The extracted meta information 129 may include information indicating the attributes of the target for extraction for each user's purpose of use.

[0032] The unstructured storage 104 stores unstructured datasets extracted by the extraction unit 119. The extraction unit 119 adds, to the stored unstructured datasets, meta information that may be associated with the unstructured datasets (e.g., meta information extracted from the extracted meta information 129) as probability information. Among the unstructured datasets stored in the unstructured storage 104, a set of data (training data) to be input to the NN through machine learning by the learning unit 102 is referred to as a "training dataset." Each piece of training data included in the training dataset may be associated with an annotation result by the annotation unit 103. Training data and the associated annotation result are referred to as "teaching data," and a set of training data is referred to as a "teaching dataset." Training data may be unstructured data such as simple image data, audio data, or text data, or may be structured data, such as POS data, in which some meaning is previously assigned to the data.

[0033] The annotation unit 103 controls the work of associating annotation results with a learning dataset (hereinafter, this will be referred to as the "annotation work"). Specifically, for example, the annotation unit 103 performs the following based on the annotation meta information 130. A request for input (annotation work) of annotation results (for example, labels indicating the classification names of each piece of learning data) for the learning data set in the unstructured storage 104 is sent to the user terminal 112. The annotation results for the learning data set are received from the user terminal 112 that received the request. · Associate the received annotation results with the training dataset. The training dataset (teacher dataset) associated with the annotation results is stored in the data pool 105.

[0034] More specifically, for example, the annotation unit 103 automatically transmits, periodically or irregularly, to the user terminals 112 of the cloud workers, a request to input annotation results for the training dataset (for example, by sending an email or a message with a link to a web page for inputting the annotation results). Upon receiving the request, the user terminals 112 are provided with a web page displaying training data (for example, one or more photographic images of dogs or cats), a question regarding the training data (for example, "Is the animal in each photographic image a dog or a cat?"), and an input UI (user interface) for inputting annotation results as answers to the questions (for example, a label of "dog" or "cat"). The content of the web page may be determined based on the annotation meta information 130. The annotation unit 103 associates the annotation results input from the user terminals 112 via the web page with the corresponding training data. The annotation unit 103 stores the training data (teacher data) associated with the annotation results in the data pool 105. In this way, annotation results are created and associated with each unit of training data belonging to the training dataset.

[0035] The annotation meta information 130 is information that includes information related to control of the operation of the annotation unit 103. The annotation meta information 130 may include information related to control of the operation of the annotation unit 103 for each purpose of use by the user.

[0036] The learning unit 102 prepares at least one NN (hereinafter referred to as a "learning NN" or, more generally, a "learning model") for machine learning. The learning unit 102 then performs machine learning, i.e., training (hereinafter referred to as a "learning process"), of the learning NN. Specifically, the learning unit 102 reads a training dataset from the data pool 105 based on learning meta-information (not shown) (e.g., information including information indicating the attributes of the training dataset to be read), supplies the read training dataset to the learning NN, and trains the learning NN to be more optimal. In this embodiment, multiple learning units 102 can operate in parallel, and these multiple learning units 102 can train multiple different learning NNs, respectively. These multiple learning NNs may include multiple learning NNs with the same purpose and configuration, multiple learning NNs with the same purpose but different configurations, or multiple learning NNs with different purposes and configurations. The learning meta-information is defined in the learning unit 102, but may alternatively reside outside the learning unit 102. The trained NN after the machine learning in the training unit 102 is completed (hereinafter referred to as "trained NN") is made into a library by the virtual host NN 116 and saved (for example, stored in a predetermined storage area).

[0037] The virtual NN host 116 performs a verification process to calculate the output accuracy of each trained NN. Based on the results of the verification process, the virtual NN host 116 selects one or more trained NNs as NNs to be put to practical use (hereinafter referred to as "practical NNs," or more generally, "practical models") in accordance with predetermined rules. For example, when there are multiple trained NNs for the same purpose, the trained NN with the highest output accuracy is selected as the practical NN. Of the one or more trained NNs stored in the library, the one selected as the practical NN is deployed to the inference unit 114 or the inference / statistics unit 115 depending on its purpose of use and put to practical use.

[0038] The practical NN deployed in the inference unit 114 is a NN model for performing inference for a specific purpose (e.g., identifying the type of animal from a photographic image of the animal) using primarily unstructured data as input. The inference unit 114 executes the practical NN deployed therein to perform inference processing for the specific purpose. For example, the inference unit 114 reads a dataset for the same inference purpose (hereinafter referred to as a "practical dataset," e.g., a dataset of photographic images containing animals) from the data pool 105 based on inference meta-information (not shown) (e.g., an example of APP meta-information containing information indicating the attributes of the dataset to be read), and inputs the practical dataset into the practical NN to obtain output data from the practical NN (e.g., an answer such as "dog"). The set of output data (hereinafter referred to as an "inference result dataset") is stored in the data pool 105. Note that the inference meta-information resides in the inference unit 114, as will be described later. Two or more inference units 114 are executed in parallel.

[0039] The practical NN deployed in the inference / statistics unit 115 is an NN model for performing specific-purpose inference or statistical processing, mainly using inference result data and / or structured data as input. The inference / statistics unit 115 performs at least one of inference processing and statistical processing using the practical NN deployed therein. Specifically, for example, the inference / statistics unit 115 references inference / statistics meta information (not shown). The inference / statistics meta information includes, for example, information indicating the attributes of the datasets to be read (e.g., the inference result dataset and the structured dataset) and information indicating the relationship between the inference result dataset and the structured dataset indicating the values ​​associated with the inference result dataset. The inference / statistics unit 115 reads the inference result dataset and / or the structured dataset based on the inference / statistics meta information, and stores the inference result dataset associated with the values ​​indicated by the structured dataset in the data pool 105 as an inference / statistics result dataset.

[0040] The output API 107 reads a dataset to be output (typically an inference result dataset or an inference / statistical dataset) from the data pool 105 based on the output meta information 122, and outputs the read dataset to the data target 113 that is the output destination specified by the output meta information 122. The output meta information 122 is information that indicates the attributes of the dataset to be read by the output API 122 and the output destination of the dataset, and includes, for example, information that indicates the relationship between the dataset ID and the key, and information that indicates the relationship between the output destination and the key.

[0041] The above is an explanation of the logical configuration of the computer system 50.

[0042] The learning and inference system 101 is provided with a learning unit 102, an inference unit 114, and an inference / statistics unit 115. A practical NN selected from among the trained NNs that have completed machine learning in the learning unit 102 is automatically deployed to an available inference unit 114 or inference / statistics unit 115, and the inference unit 114 or inference / statistics unit 115 performs inference processing using the practical NN. In other words, as long as a training dataset or practical dataset according to the user's intended use is input into the learning and inference system 101, the learning NN or practical NN that matches the intended use will be used. This makes it easy for the user to prepare a practical NN (or computational model) according to the intended use and put it to practical use.

[0043] In the learning and inference system 101, multiple datasets (e.g., including datasets as intermediate results) with different purposes of use and processing methods, such as the training dataset and practical dataset read by the input API 106, the teacher dataset (a training dataset associated with a set of annotation results) obtained by the annotation unit 103, the inference result dataset output from the inference unit 114, and the inference / statistics result dataset obtained from the inference / statistics unit 115, are aggregated in the data pool 105. Since multiple datasets are aggregated in one place in this way, processing such as analysis (e.g., processing performed by the inference / statistics unit 115) is highly convenient. However, it is not necessary for these multiple datasets to be aggregated in a single storage area such as the data pool 105. Multiple datasets may be distributed across multiple different storage areas.

[0044] Also, the inference / statistics unit 115 may be omitted. In that case, an inference result dataset (a dataset not associated with values ​​indicated by a structured dataset) may be output from the output API 107 to the data target 113.

[0045] The deployment of a practical NN in the inference unit 114 (or the inference / statistics unit 115) will be described in detail below.

[0046] Fig. 3 shows the logical configuration of a physical computer 201. Note that the physical computer 201 shown in Fig. 3 is one of all or some of the physical computers 201 in the computer system 50, and each of these physical computers 201 has the logical configuration shown in Fig. 3.

[0047] A host OS (Operating System) 314 runs on a physical computer resource 350 of the physical computer 201. A plurality of virtual units 301 run on the host OS 314. Because the host OS 314 is a common OS for the plurality of virtual units 301, the host OS 314 may also be called a "shared OS 314."

[0048] The virtual unit 301 is a virtual execution environment, and may be, for example, a virtual computer or an execution environment called a container (for example, a container that runs on a Linux OS (Linux is a registered trademark)). In this embodiment, one virtual unit 301 exists for one APP (application program) 302, but two or more APPs 302 may exist in one virtual unit 301.

[0049] A guest OS 304 is executed in a virtual unit 301. There is one guest OS 304 per virtual unit 301, and since it is an OS within the virtual unit 301, it may also be called an "internal OS 304." Middleware 303 is executed on the guest OS 304, and APP 302 is executed on the middleware 303. There may be a virtual unit 301 without middleware 303, and in such a virtual unit 301, APP 302 is executed on the guest OS 304.

[0050] In each virtual unit 301, the APP 302 includes an input module 41, an output module 43, and a control module 45. As described below, a main body module 42 including a practical NN is dynamically deployed (in other words, dynamically incorporated) into the APP 302. In other words, when the APP 302 is initially prepared, the main body module 42 is not yet incorporated therein. An APP 302 without a main body module 42 will be referred to as an "empty APP 302" below, and a virtual unit 301 including an empty APP 302 will be referred to as an "empty virtual unit 301" below. A main body module 42 including a practical NN can be deployed (i.e., incorporated) into an empty APP 302 in an empty virtual unit 301 at any time. An APP 302 with a main body module 42 deployed (embedded) therein will be referred to as an "occupied APP 302" below, and a virtual unit 301 including an empty APP 302 will be referred to as an "occupied virtual unit 301" below.

[0051] The host OS 314 has a host API 315. In each virtual unit 301, the guest OS 304 has a guest API 305. Hereinafter, for each physical computer 201, the address of the host API 315 will be referred to as a "host address," and the address of the guest API 305 in each virtual unit 301 will be referred to as a "guest address." Both the host address and the guest address are, for example, IP (Internet Protocol) addresses. The management system 161 (see FIG. 2) manages the guest address in addition to the host address for each physical computer 201.

[0052] In this embodiment, in at least one physical computer 201, a plurality of free virtual units 301 are initially provided on the host OS 314 running on the physical computer resources 350. The number of virtual units 301 on the host OS 314 may increase or decrease dynamically (for example, the number of virtual units 301 may increase or decrease depending on the load on the physical computer resources 350). However, in this embodiment, the number of virtual units 301 on the physical computer 201 is determined in advance based on the physical computer resources 350 on that physical computer 201. Because a plurality of free virtual units 301 are initially provided on one physical computer 201, even if the number of occupied virtual units 301 changes within the range of the initially provided number (i.e., even if a main body module 42 is deployed on one of the plurality of initially provided free virtual units 301, or even if the deployed main body module 42 is later erased or replaced with another main body module 42), there is no need to restart the physical computer 201. Therefore, even in an environment where multiple virtual units 301 on the same physical computer 201 are used by multiple users (for example, companies), it is possible to avoid stopping the service (the service provided by APP 302) for any of the users.

[0053] Each of the above-described inference units 114 (and inference / statistics units 115) corresponds to one dedicated virtual unit 301 (one dedicated APP 302). The number of inference units 114 may increase dynamically. That is, a new practical NN may be generated and deployed to an available virtual unit 301, thereby creating a new inference unit 114. Similarly, each of the above-described learning units 102 may correspond to one dedicated virtual unit 301, or may instead be a program pre-installed on a certain physical computer 201. Furthermore, at least one program other than the inference units 114 and the learning units 102 (for example, at least one of the input API 106, the output API 107, the virtual NN host 116, and the management system 161) may also correspond to one dedicated virtual unit 301, or may instead be a program pre-installed on a certain physical computer 201. Furthermore, for example, the virtual NN host 116 may be at least a part of the host OS 314 in the physical computer 201.

[0054] FIG. 4 shows the configuration of the virtual unit 301 in more detail.

[0055] As described above, in the virtual unit 301, the middleware 303 runs on the guest OS 304, and the APP 302 runs on the middleware 303.

[0056] APP302 has an input module 41, an output module 43, and a control module 45 in advance (i.e., in the state of an empty APP302). A main body module 42 is dynamically incorporated into or removed from APP302. As described above, an APP302 in which a main body module 42 is not incorporated is called an empty APP302, and an APP302 in which a main body module 42 is incorporated is called an occupied APP302. The main body module 42 is composed of APP meta information 421, a data check module 422, and a practical NN 423.

[0057] The input module 41, the output module 43, and the control module 45 are defined in advance as general-purpose modules whose structures do not depend on the contents of the main module 42 (i.e., the configuration and attributes of the practical NN). That is, the input module 41, the output module 43, and the control module 45 are examples of two or more general-purpose modules. Another general-purpose module may be prepared in place of or in addition to at least one general-purpose module.

[0058] The input module 41 reads a practical dataset from a storage area (for example, a storage area in the data pool 105) based on the APP meta information 421, and supplies the read practical dataset to the main module 42. The dataset to be read, the storage area from which the practical dataset is read, and the supply destination of the read practical dataset are all identified from the APP meta information 421.

[0059] The output module 43 outputs (stores) the inference result dataset output from the main module 42 (practical NN 423) to a storage area (for example, a storage area in the data pool 105) based on the APP meta information 421. The storage area to which the dataset is output is specified by the APP meta information 421.

[0060] The control module 45 communicates with the outside of the APP 302 (for example, the guest OS 304).

[0061] The APP meta information 421 is meta information related to the operation of the APP 302, and includes, for example, at least one of the following: Parameters set in predefined modules such as the input module 41, the output module 43, and the control module 45. That is, parameters set in the input module 41, parameters set in the output module 43, and parameters set in the control module 45. Each of the input module 41, the output module 43, and the control module 45 operates according to parameters set based on the APP meta information 421. The parameter setting may be performed by each of the input module 41, the output module 43, and the control module 45, or may be performed by the control module 45. Information indicating the attributes of the practical dataset to be read by the input module 41 (for example, the dataset ID (e.g., file name), address of the read source area (e.g., LBA (Logical Block Address))). In other words, information regarding access rights to the practical dataset. Information indicating the attributes of the output destination of the inference result dataset output by the output module 43 (for example, the address of the output destination area (for example, LBA)). Information indicating the attributes of the practical NN 423 (for example, the ID of the practical NN 423, the address (for example, LBA) of the read source area of ​​the practical NN 423, and the input dimension and output dimension of the practical NN 423). Information indicating the attributes of the data check module 422 (for example, the ID of the data check module 422 and the address (for example, LBA) of the area from which the data check module 422 reads). Parameters set in the data check module 422 and corresponding to the practical NN 423 (for example, information indicating the attributes of the input data set to be checked (for example, the dimensions of the data set or the error pattern)). APP meta information 421 including the above-described information exists in APP 302 (main module 42). In other words, all (or at least a part) of the information necessary to operate the practical NN 423 is included in APP 302. The input module 41, output module 43, and control module 45, which are configured as general-purpose modules independent of the practical NN 423, can operate the practical NN 423 by using the APP meta information 421.

[0062] The data check module 422 checks, based on the APP meta information 421 (for example, according to parameters set based on the APP meta information 421), whether the practical dataset input from the input module 41 is a dataset according to the information defined when the main module 42 was deployed (i.e., whether the dataset is correct). If the check result is true, the data check module 422 inputs the input practical dataset to the practical NN 423. This makes it possible to prevent an incorrect dataset from being input to the practical NN 423, causing an error. Note that the data check module 422 is not essential.

[0063] The practical NN 423 receives an input of a practical dataset from a storage area outside the virtual unit 301 through the input module 41, and outputs an inference result dataset (or an inference / statistical result dataset). The output inference result dataset is written to a storage area outside the virtual unit 301 through the output module 43.

[0064] As described above, in this embodiment, the APP 302 is executed on the middleware 303 (guest OS 304) in the virtual unit 301. The functions of the APP 302 are configured as modules 41, 422, 423, 43, and 45, respectively, and each of the modules 41, 422, 423, 43, and 45 is a sub-application program. In other words, the APP 302 is a so-called loosely coupled APP. Specifically, the input module 41, output module 43, and control module 45 are prepared in advance as functional modules other than the main module 42 including the practical NN 423. This ensures versatility. In the APP 302, the input module 41, output module 43, control module 45, data check module 422, and practical NN 423 are all sub-application programs called "microservices." Regardless of the configuration of the practical NN 423, as long as it is deployed in the APP 302 as a single sub-application program, the input module 41 and output module 43 handle data input / output between the practical NN 423 and the outside of the virtual unit 301. This makes it easy to make the practical NN 423 executable. Specifically, for example, when the configuration of the practical NN 423 is updated, a new main body module 42 including the updated practical NN 423 may overwrite the main body module 42 including the old practical NN 423, or the practical NN 423 may be deployed in another free virtual unit 301 (free APP 302). This makes it possible to make the updated practical NN 423 executable (and to replace or add the practical NN 423) without restarting the physical computer 201.

[0065] As a first comparative example, a configuration that does not require a virtual unit is conceivable. In this case, the learning unit deploys the practical NN based on environmental information (e.g., information including information about the input source, information indicating the range of the input dataset, and information indicating the format of the input dataset (e.g., the number of dimensions)). However, in the first comparative example, it is necessary to configure the processing associated with the inference result (e.g., if the inference result is A, perform processing B) for each practical NN, resulting in high deployment costs. In addition, the physical computer must be restarted every time a new practical NN is deployed. While the physical computer is restarting, the services provided by the APP running on that physical computer are stopped.

[0066] As a second comparative example, a configuration is possible in which a virtual unit is employed, but a tightly coupled APP (an APP in which a practical NN and other functions (e.g., data input / output, control, etc.) are integrated) is employed, and the above-mentioned virtual NN host does not exist. In this case, the learning unit defines a virtual unit including the APP integrated with the practical NN and configures the virtual unit. However, in the second comparative example, when a practical NN needs to be updated, it is necessary to replace or add the entire APP integrated with the practical NN, which results in the need to replace or add the virtual unit that corresponds one-to-one with the APP. Therefore, as in the first comparative example, the physical computer must be restarted. Furthermore, if a virtual unit goes down, a new virtual unit is added to which the APP in that virtual unit is deployed, which results in the physical computer being restarted.

[0067] As a third comparative example, a configuration including a common instance for two or more inference units is possible. In this case, the instance reads all inference datasets and inputs them to two or more inference units. Therefore, if the instance goes down, none of the inference units can continue the inference process. Furthermore, since the instance reads and inputs the inference datasets regardless of the state of the inference units, there is a possibility that an error will occur in the inference units. If an error occurs in an inference unit, the instance stops reading the inference datasets for error handling. As a result, the inference process of each inference unit will stop.

[0068] According to this embodiment, as described above, the APP 302 is a loosely coupled APP whose functions are modularized, and therefore the practical NN 423 can be replaced or added without restarting the physical computer 201.

[0069] Furthermore, according to this embodiment, the learning unit 102 does not define a virtual unit including an APP integrated with a practical NN, and the APP meta information 421 contained in the APP 302, and the data check module 422 and practical NN 423 based on the APP meta information 421 are deployed to the hosted virtual unit 301.

[0070] Furthermore, according to this embodiment, as described above, the number of virtual units 301 operating on the physical computer 201 is initially a fixed number equal to or greater than two (for example, a maximum number determined in advance based on the physical computer resources 350). Within the range of the initial number of virtual units 301, practical NNs 423 can be replaced or added without restarting the physical computer 201.

[0071] In addition, in the virtual unit 301, each module in the APP 302 is a sub-application program. Therefore, even if an error occurs in another virtual unit 301, the virtual unit 301 can continue to operate.

[0072] However, loosely coupled applications (general-purpose applications) may have lower performance than tightly coupled applications (dedicated applications) because loosely coupled applications involve communication between modules (e.g., API-based communication).

[0073] Therefore, in this embodiment, as shown in FIGS. 3 and 4, a host API 315 (host address) is provided in the host OS 314, and a guest API 304 (guest address) is provided in the guest OS 304 of each virtual unit 301. Both the host address and the guest address are managed by the management system 161. For the management system 161, the guest address is a private address (e.g., an address independently determined by the user). Therefore, access is possible with the guest address as the end point (in other words, routing by the host API is not required), thereby reducing a degradation in the performance of the learning inference support system 100 as a whole. The management system 161 can manage each virtual unit 301 using the guest address, regardless of which physical computer 201 (or host OS 314) the virtual unit 301 runs on.

[0074] In this embodiment, a guest OS 304 (and middleware 303) exists independently for each APP 302. Each APP 302 (virtual unit 301) can operate independently on the same physical computer resource 350. In other words, each APP 302 (and virtual unit 301) does not need to communicate with other APPs 302 (other virtual units 301) for the operation of that APP 302 (and its virtual unit 301). Similarly, each physical computer 201 (host OS 314) does not need to communicate with other physical computers 201 (other host OSs 314) for the operation of that physical computer 201 (its virtual unit 301).

[0075] Furthermore, the management system 161 may be a system including a management unit and a monitoring unit. The management unit manages the learning inference system 101. The monitoring unit monitors the learning inference system 101. The "management" performed by the management system 161 may include "management" by the management unit and "monitoring" by the monitoring unit.

[0076] For example, the management unit refers to or updates virtual unit management information (not shown). The virtual unit management information is stored in the memory unit 252 of at least one physical computer 201. The virtual unit management information includes, for each guest address to be managed (i.e., for each virtual unit 301 to be managed), a physical computer ID, a host address, a main body module attribute (e.g., the ID of the practical NN), and a status (e.g., "standby" meaning that the main body module 42 is present, or "error" meaning that an error has occurred). By referring to the virtual unit management information, the management unit can identify which virtual unit 301 in which physical computer 201 is an available virtual unit 301.

[0077] Furthermore, for example, the management unit is configured to maintain free virtual units 301 at a predetermined free ratio or more for each physical computer 201. This ensures that there is at least one recovery destination even if an error virtual unit 301, which will be described later, occurs. Note that the "free ratio" is the ratio of free virtual units 301 to the total number of virtual units 301 on the physical computer 201.

[0078] For example, if the available ratio of any physical computer 201 becomes equal to or greater than a predetermined available ratio, the management unit adds a new physical computer 201 (for example, instructs a predetermined system to add a physical computer 201) and allocates a predetermined number (for example, the maximum number) of available virtual units 301 to the new physical computer 201. The available virtual units 301 in the new physical computer 201 can be the deployment destination of the practical NN 423.

[0079] The monitoring unit, for example, refers to virtual unit monitoring information (not shown). The virtual unit management information is stored in the memory unit 252 of at least one physical computer 201. The virtual unit monitoring information includes a physical computer ID and a host address for each guest address to be monitored (i.e., for each virtual unit 301 to be monitored). The monitoring unit monitors the virtual units 301 to be monitored based on the virtual unit monitoring information. If the monitoring unit detects that an error has occurred in any of the virtual units 301, it performs recovery. Specifically, for example, the monitoring unit notifies the monitoring unit of the guest address of the error virtual unit 301 (the virtual unit 301 in which the error has occurred). When the monitoring unit receives this notification, it refers to the virtual unit management information, determines the free virtual unit 301 to be the recovery destination, and decides to deploy a main body module 42 that is the same as the main body module 42 (practical NN 423) in the error virtual unit 301 to the determined free virtual unit 301 (for example, it sends a meta setting instruction (an instruction specifying the guest address of the recovery destination virtual unit 301) of the APP meta information 421 in the same main body module 42) to the free virtual unit 301 to be the recovery destination). The free virtual unit 301 to be the recovery destination may be any of the following. If the free virtual unit 301 to be the recovery destination is (A) of the following, it can be expected that the time required for recovery (the time until the same practical NN 423 as the practical NN 423 in the error virtual unit 301 becomes executable) will be short. (A) Any free virtual unit 301 in the physical computer 201 in which the error virtual unit 301 exists. (B) Any free virtual unit 301 in the physical computer 201 with the smallest free rate.

[0080] Also, as described above, the virtual unit 301 actively reads the data set, which means that the virtual unit 301 (eg, the control module 45) can handle the speed handling.

[0081] Below, taking a certain purpose of use as an example, the overall process flow for generating and deploying a practical NN corresponding to that purpose of use will be explained with reference to FIG.

[0082] In S501, the annotation unit 103 stores a teacher dataset (a learning dataset and a dataset of annotation results associated with the learning dataset) corresponding to the intended use in the data pool 105 based on the annotation meta information 130. Specifically, for example, the annotation unit 103 automatically transmits a request to a large number of user terminals 112 of the crowd workers to input annotation results (e.g., answers) for the learning dataset. In response to the request, the annotation unit 103 associates the annotation results input from the user terminals 112 with the learning dataset, and stores the learning dataset (teacher dataset) associated with the annotation results in the data pool 105. In this way, the generation of the teacher dataset (association of the annotation results with the learning dataset) is performed in response to the request automatically transmitted by the annotation unit 103.

[0083] In S502, a user (e.g., an AI engineer) defines the configuration of at least one learning NN (learning model) according to the intended use (e.g., the number of layers of the NN, the format and number of dimensions of each layer, and the format and parameter set of each inter-layer function). The defined configuration of the learning NN is set, for example, in a storage area of ​​the management system 161. Multiple learning NNs with different configurations may be defined for the same intended use. In this case, multiple learning units 102 each having the multiple learning NNs may be provided on one or more physical computers 201.

[0084] In S503, the parameter set of each of the one or more learning NNs is tuned. Specifically, for example, the management system 161 first initializes the parameter set of each learning NN. Then, based on the output error of each learning NN obtained from the learning process in S504 described below, the parameter set of each learning NN is adjusted to be closer to optimal.

[0085] In S504, each of the one or more learning units 102 performs a learning process. That is, each learning unit 102 reads each piece of teacher data from the teacher dataset in the data pool 105, inputs the learning data from that piece of teacher data into each learning NN, and obtains output data from each learning NN. The error between the output data obtained from each learning NN and the annotation result from that teacher data, i.e., the output error of each learning NN, is calculated. Parameter tuning in S504 described above is performed again according to the output error. Thereafter, another piece of teacher data is read from the teacher dataset, and the processes of S504 and S503 are repeated using that teacher data. When a predetermined learning termination condition is met for each learning NN (for example, the number of repetitions of the processes of S504 and S503 reaches a certain allowable number, or the output error or its reduction rate falls below a certain allowable value), machine learning for each learning NN ends. When machine learning is completed

[0086] In S505, the virtual NN host 116 stores one or more trained NNs, i.e., trained NNs, for which machine learning has been completed by one or more learning units 102, in a library and performs a verification process for the trained NNs. That is, the virtual NN host 116 calculates the accuracy of each of the trained NNs. Specifically, for example, the virtual NN host 116 inputs a training dataset of a training dataset for verification to each trained NN, and compares the output dataset of each trained NN with the set of annotation results for that training dataset. From the comparison results, the virtual NN host 116 calculates the accuracy of the trained NN (e.g., the ratio of the number of correct output data points to the total number of data points in the training dataset). The virtual NN host 116 then evaluates the accuracy of each trained NN using predetermined conditions (e.g., determining whether the accuracy of each trained NN is equal to or greater than a predetermined threshold).

[0087] If the result of the above evaluation indicates that all of the trained NNs are insufficient in accuracy (for example, their accuracy is below the threshold), the virtual NN host 116 notifies the management system 161 of the insufficient accuracy. Upon receiving the notification of the insufficient accuracy, the management system 161 causes each learning unit 102 to perform additional learning processing and parameter tuning (S504 and S503) on all or part of the trained NNs with insufficient accuracy.

[0088] If the evaluation results indicate that the accuracy of at least one of the trained NNs is acceptable (e.g., the accuracy is equal to or greater than the threshold), the virtual NN host 116 selects at least one trained NN with the acceptable accuracy as a practical NN in S506. If multiple trained NNs for the same purpose have acceptable accuracy, the trained NN with the highest accuracy may be selected as the practical model. The virtual NN host 116 then sends a determination notification to the management system 161, including information about the practical NN 423 (e.g., the NN's ID). This allows the management system 161 to identify which NN is the practical NN 423. Depending on the configuration of the practical NN 423, the management system 161 or the virtual NN host 116 prepares a data check module 422 associated with the practical NN 423. In this case, additional machine learning may be performed on all or part of the trained NNs not selected as the practical NN 423. Additional machine learning may also be performed on the NN selected as the practical NN 423.

[0089] In S507, when the management system 161 recognizes the practical NN 423, it automatically selects or prepares a free virtual unit 301 to which the practical NN 423 is to be deployed (507). For example, the management system 161 selects one of the existing free virtual units 301 as the deployment destination based on the virtual unit management information, or prepares a new free virtual unit 301. Note that the management system 161 may determine the free virtual unit 301 to which the practical NN 423 is to be deployed in response to an instruction from the user (for example, when it displays information about the practical NN 423 and receives a deployment instruction as approval of the practical NN 423).

[0090] In S508, the management system 161 performs environment setting for the deployment destination free virtual unit 301. Specifically, for example, the following is performed. The management system 161 generates APP meta information 421 corresponding to the practical NN 423 and sets the APP meta information 421 in the APP 302 in the free virtual unit 301 at the deployment destination. Specifically, for example, the management system 161 transmits a meta setting instruction specifying the guest address of the free virtual unit 301 at the deployment destination to the free virtual unit 301 at the deployment destination. The meta setting instruction is an instruction to set the APP meta information 421. The guest OS 304 (guest API 305) having the specified guest address receives the meta setting instruction through the host OS 314 (host API 315) in the physical computer 201 that has the free virtual unit 301. In response to the meta setting instruction, the control module 45 in the APP 302 receives the instruction from the guest OS 304, and the control module 45 sets the APP meta information 421 in the APP 302. The management system 161 transmits an activation instruction specifying the guest address of the free virtual unit 301 to which the virtual unit is to be deployed. The activation instruction is an instruction to set parameters defined in the APP meta information 421 in the input module 41 and the output module 43. The guest OS 304 receives the activation instruction via the same route as the meta setting instruction. In response to the activation instruction, the control module 45 in the APP 302 receives the instruction from the guest OS 304, and the control module 45 sets the parameters in the APP meta information 421 in the input module 41 and the output module 43. The control module 45 also instructs the input module 41 and the output module 43, for which the parameters have been set, to perform a test operation. In response to the instruction, the input module 41 and the output module 43 perform the test operation (for example, issue a dummy command to test whether the dataset can be read based on the APP meta information 421).

[0091] In S509, the practical NN 423 (and the data check module 422 associated therewith) is deployed. Specifically, for example, the management system 161 transmits a deploy instruction specifying the guest address of the free virtual unit 301 to which the practical NN 423 is to be deployed. The deploy instruction is an instruction to deploy the practical NN 423 defined in the APP meta information 421. The guest OS 304 receives the deploy instruction through the same route as the meta setting instruction and the activation instruction. In response to the deploy instruction, the control module 45 in the APP 302 receives an instruction from the guest OS 304, and the control module 45 reads the practical NN 423 based on the APP meta information 421 (i.e., reads the practical NN 423 identified from the APP meta information 421 from the location identified from the APP meta information 421), and sets the read practical NN 423 in the APP 302.

[0092] In S510, the virtual unit 301 of the deployment destination is added as a new monitoring target. Specifically, for example, the management system 161 sets the guest address of the virtual unit 301 of the deployment destination as a guest address to be newly monitored (for example, adds it to the virtual unit monitoring information). The monitoring and recovery performed by the management system 161 (monitoring unit) are as described above.

[0093] After the above steps are taken, the operation of the deployed virtual unit 301 (APP 302 in which the practical NN 423 is set) begins. Specifically, for example, the APP 302 in the virtual unit 301 is executed as the inference unit 114.

[0094] Although one embodiment of the present invention has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms.

[0095] For example, the present invention can be applied to a system including a module that performs learning other than deep learning (e.g., machine learning). Therefore, the learning model, trained model, practical model, inference model, and inference / statistical model may be a computational model having a configuration other than a neural network.

[0096] Also, for example, the management system 161 may deploy the practical NN 423 (and the data check module 422 ) in addition to the APP meta information 421 to the virtual unit 301 .

[0097] Furthermore, for example, one or more learning units 102, one or more inference units 114, and a virtual NN host 116 may exist on the same physical computer 201. In other words, each of at least some of the physical computers 201 (one or more physical computers 201) may have one or more learning units 102, one or more inference units 114, and a virtual NN host 116.

[0098] Furthermore, for example, when the availability ratio of any physical computer 201 becomes equal to or exceeds a predetermined availability ratio, instead of or in addition to adding a new physical computer 201, the management system 161 may send a removal instruction specifying the guest address of the virtual unit 301 to the virtual unit 301 having the APP 302 including the unnecessary main body module 42. The removal instruction is an instruction to remove the main body module 42. The guest OS 304 receives the removal instruction through the same route as the meta setting instruction and the activation instruction. In response to the removal instruction, the control module 45 in the APP 302 receives an instruction from the guest OS 304, and the control module 45 removes the main body module 42 from the APP 302. As a result, the virtual unit 301 becomes an available virtual unit 301. The control module 45 returns a completion response to the management system 161. In response to the completion response, the management system 161 updates the virtual unit management information; specifically, it updates the status corresponding to the guest address of the virtual unit 301 from “active” to “standby.”

[0099] Generally, individual technologies such as those related to learning and inference are known, but no one-stop technology is known that can perform everything from learning to inference.

[0100] The computer system according to the above-described embodiment receives a training dataset from an external data source, adds annotation results to it to prepare a teacher dataset, and then uses the teacher dataset to perform machine learning on a learning model to create a practical model.

[0101] Furthermore, the system can receive a practical dataset from an external data source, input the practical dataset into a practical model to perform inference processing, and output an inference result dataset indicating the results of the inference processing to an external data target.

[0102] Therefore, when users of this system wish to perform inference using a practical model (for example, human face recognition), they do not need to perform the tedious and complicated tasks of machine learning (for example, training a neural network) to prepare a practical model and deploying the practical model in an inference execution environment. This is expected to facilitate the use of models such as neural networks in various industrial fields, promoting the development of a more advanced and convenient information society.

[0103] The system may further include a means for storing the practical model separately from the learning unit and the inference unit. The learning unit may input the separately stored practical model to perform further learning, i.e., training, of the practical model. This allows the practical model to evolve in response to performance improvements, changes in user requirements, etc.

[0104] The annotation unit 103 of this embodiment will be described in more detail below. FIG.

[0105] The annotation unit 103 is provided, for example, by the processor unit 253 executing an instruction set stored in the storage unit 252 of one or more physical computers 201 of the computer system 50 shown in FIG.

[0106] As shown in Fig. 6, the annotation unit 103 can communicate with multiple external computers (e.g., multiple user terminals 112 shown in Fig. 1) 401, 403, and 405 via a communication network 140. The multiple computers 401, 403, and 405 include one or more client terminals 401 used by one or more requesters 400, one or more annotator terminals 403 used by one or more annotators 402, and one or more reviewer terminals 405 used by one or more reviewers 404. Here, the requester 400 is a person or organization that requests annotation work from the system 50. The annotator 402 is a person or organization that performs annotation work (i.e., the work of inputting annotation results for training data). The reviewer 404 is a person or organization whose job is to review the annotation results generated by the annotators (e.g., to determine the validity of the annotation results, or further modify the annotation results as necessary to determine the final annotation results). The annotators 402 (and possibly the reviewers 404) may be so-called crowdworkers, for example, provided through crowdsourcing.

[0107] The annotation unit 103 has an API 407, a storage 408, and a management database 409. The API 407 communicates with each of the requester terminal device 401, the annotator terminal 403, and the reviewer terminal 405, and transmits and receives information necessary for annotation work with these computers. All or part of the API 407 may be included in the input API 106 or the output API 107 shown in Fig. 2, or may be prepared separately from them.

[0108] The storage 408 may be included in whole or in part in the data pool 105 or unstructured storage 104 shown in FIG. 2, or may be provided separately. The storage 408 stores a set of training data to be annotated. The training data may be in any format, such as image data, video data, text data, audio data, or numerical data. The training data may be unstructured data or structured data. The annotation results for the training data are, for example, data that is expected to be output from a neural network when the training data is input to the neural network. Therefore, training data to be used in neural network training can be prepared from the training data and its annotation results.

[0109] The management database 409 may be included in whole or in part in the data pool 105 or unstructured storage 104 shown in Fig. 2, or may be prepared separately. The management database 409 stores various data for managing annotation work, and an example of the data configuration is shown in Fig. 7.

[0110] FIG. 7 shows an example of the configuration of the management database 409.

[0111] The management database 409 contains the following different types of data units:

[0112] 1) Metadata 523 The training dataset 502 stored in the storage 408 includes multiple (usually a large number) units of training data 521. Metadata 523 for each unit is associated one-to-one with the unit of training data 521. A metadata ID for identifying the metadata and a uniform resource identifier (URI) that is an identification code for uniquely identifying the file of the corresponding unit of training data 521 are recorded in the metadata 523 for each unit. The file of the corresponding training data 521 can be accessed by using the URI. Auxiliary information related to the corresponding training data 521 (e.g., information related to the date and time, training data, or data source) can also be recorded in the metadata 523 for each unit.

[0113] 2) Task 525 A task 525 is a data unit corresponding to one unit of work (i.e., a task) for associating annotation results with training data 521. The task 525 records a task ID for identifying the task, the priority of the task, the maximum number of assignments indicating the maximum number of annotators to whom the task can be assigned, a completion status indicating whether the task has been completed, and a lock status indicating whether the task is locked (no new assignments can be made). The priority of the task 525 is useful for controlling the progress of the task. The task 525 is presented to the annotator 402 who receives a request for the work via the annotator terminal 403. One task 525 can be associated with one or more units of metadata 523. That is, one task can include annotation work for one or more units of training data 521. Furthermore, one or more tasks 525 can be associated with the same unit of metadata 523. That is, multiple tasks can be set for the same training data 521 to obtain annotation results by multiple annotators.

[0114] 3) Assignment 527 An assignment 527 is a data unit corresponding to the assignment of one task 525 to one annotator 402. One purpose of the assignment 527 is to control the assignment process so that the same task 525 is not assigned to too many annotators 402. The assignment 527 records an assignment ID that identifies the assignment, the start time and expiration date of the assignment, and the annotator name that identifies the annotator 402 who has been given the assignment (i.e., who has acquired the corresponding task). When a task is assigned to a certain annotator 402, if the expiration date of the assignment 527 expires before the annotator 402 finishes the annotation work corresponding to the task, the assignment is invalidated, and as a result, another annotator 402 can acquire the same task. This prevents task delays due to annotator negligence. One or more assignments 527 can be associated with one task 525. That is, one task can be assigned to multiple annotators 402. The maximum number of assignments described in the task 525 is the upper limit of the number of annotators 402 to whom the task can be assigned at the same time.

[0115] 4) Result 529 A result 529 is a data unit corresponding to the annotation result of one task performed by one annotator 402. The result 529 records a result ID that identifies the result, the annotation result, the worker name that identifies the annotator 402 who input the annotation result, and the work time that indicates the length of time required from assignment to completion of the corresponding task. When one task is assigned to multiple annotators 402, multiple results 529 corresponding to the annotation results of each annotator 402 are associated with that one task 525.

[0116] Here, the "annotation result" of a certain training data 521 is information that identifies the expected output data that is expected to be output from a training model (training NN) when the training data 521 is input to the training model. In many cases, this is a "label" that represents the classification of the expected output data. For example, if an image of an animal is input and the training model is to learn how to identify the type of animal, the annotation result of the image is a label representing the type of animal, such as "dog" or "cat." Alternatively, if an image showing an object in a partial area is input and the training model is to learn how to identify the object's area within the image, the annotation result of the image is, for example, a filled-in shape or outline shape that identifies the object's area, which can also be referred to as a "label." In the following description, such a classification of expected output data identified by the annotation result is referred to as a "label." The type of data a "label" is (e.g., text, figures, numbers, images, audio, etc.) and the meaning it refers to (e.g., identification of an object or person, spatial or temporal point, area or range, judgment or evaluation, quantity or level, etc.) may vary depending on the characteristics of the training data and the purpose of use of the training model.

[0117] 5) 531 reviews A review 531 is a data unit corresponding to the work of one reviewer 404 reviewing one annotation result (determining its validity and correcting it if necessary). The review 531 records a review ID that identifies the review, the review result (e.g., a judgment on whether the annotation result, i.e., the label, is valid or not, or the label corrected by the reviewer 404, etc.), and the worker name that identifies the reviewer 404. One or more reviews 531 can be associated with the same result 529. That is, the same annotation result can be reviewed by multiple reviewers 404. Instead of or in addition to human review, the system 50 can also automatically review annotation results according to a program (e.g., automatically selecting one label as the correct answer by majority vote from multiple different labels for the same training data 521).

[0118] 5) Issue 533 There may be cases where the annotator is unable to determine or is unsure of what label to assign to the training data 521. An issue 533 is a data unit corresponding to such an event (issue) that requires discussion. The issue 533 records an issue ID that identifies the issue, a problem such as a reason why a label cannot be assigned as reported by the annotator 402, a guide such as advice or instructions that the requester 400 has provided to the annotator 402 in response to the problem, an opinion that the reviewer 404 has provided in response to the guide, and a resolution status indicating whether the issue has been resolved. One or more issues 533 can be associated with one or more tasks 525.

[0119] 6) Project 535 A project 535 is a data unit corresponding to one annotation business project. One or more tasks 525 are associated with one project 535. The project 535 records a project ID that identifies the project, a requester name that identifies the requester, the maximum number of assignments for the project, and the time-out conditions for the project (e.g., the maximum allowable processing time for a task). The maximum number of assignments for each task included in the project is automatically determined based on the maximum number of assignments for the project (e.g., the maximum number of assignments for the project is the same as the maximum number of assignments for a task). The expiration date of each assignment for each task included in the project is automatically determined based on the time-out conditions for the project (e.g., the expiration date is the time obtained by adding the above-mentioned maximum allowable processing time to the assignment start time).

[0120] 7) Worker 537 The worker 537 is a data unit corresponding to each worker (i.e., annotator 402 and reviewer 404). The worker 537 records a worker ID that identifies the worker, the unit price paid for the worker's work, and the worker's work ability (e.g., the average work time required to complete one task). The worker 537 can be associated with one or more projects 535. In other words, one worker can participate in multiple projects.

[0121] 8 to 15 show the control flow of the processes that make up the annotation work. First, Fig. 8 shows the control flow for starting the annotation work.

[0122] In step S601, the requester terminal 401 requests the API 407 to create a new project. In response to this request, the API 407 creates a project 535 in S602, and then returns the project ID of the created project 535 to the requester terminal 401 in S603. In S604, the requester terminal 401 sets the maximum number of assignments and the time-out condition for that project 535 in the API 407. In S605, the API 407 saves the settings for the maximum number of assignments and the time-out condition in the project 535, and then notifies the requester terminal 401 of the completion of the settings in S606.

[0123] In S607, the requester terminal 401 uploads one or more units of learning data and associated information to the API 407. In S608, the API 407 stores the uploaded learning data 521 in the storage 408, creates metadata 523 associated with each of the learning data 521, and then in S609 returns the metadata IDs of the metadata 523 to the requester terminal 401.

[0124] In S610, the requester terminal 401 requests the API 407 to register one or more tasks for each learning data 521, and sets the priority of each task (the priority requested by the requester or the default priority). In response to this, in S611, the API 407 creates one or more tasks 525 corresponding to each learning data 521, and in S612, returns a message to the requester terminal 401 indicating that the tasks 525 have been created.

[0125] Thereafter, in S613, the requester terminal 401 can acquire from the API 407 the work status of any one or more tasks 525 in any project 535, display it, and allow the requester 400 to check it (this process will be described later with reference to FIG. 12). Then, if the requester 400 requests a priority change, in S614 the requester terminal 401 requests the API 407 to change the priority of the task 525 selected by the requester 400 to a value desired by the requester 400. In response to this request, in S615 the API 407 updates the priority of the selected task 525.

[0126] FIG. 9 shows the control flow for annotation generation (i.e., labeling) in the annotation business.

[0127] In S621, a certain annotator 402 requests login from the annotator terminal 403 to the API 407. If the login is successful, in S622 the API 407 sends information representing one or more existing projects 535 to the annotator terminal 403. In response to this information, in S623 the annotator terminal 403 transitions its display from the login screen to an annotation screen for performing annotation creation work.

[0128] Thereafter, the control of steps S624 to S634 described below is repeated for one or more tasks.

[0129] In S624, the annotator terminal 403 selects one project from existing projects and requests at least one task 525 associated with the selected project 535 from the API 407. In response to the request, the API 407 performs an assignment process in S625. The assignment process is a process of finding one or more incomplete tasks 525 already assigned to the annotator 402, notifying the annotator 402 of the incomplete tasks, and further assigning one or more new incomplete tasks to the annotator 402, as necessary. Details of this assignment process will be described later with reference to FIG. 10. When assigning a new task in this assignment process, the API 407 selects a task 525 to be assigned from the incomplete tasks 525 based on the priority of each task 525, and creates an assignment 527 associated with the selected task 525. Furthermore, the API 407 determines the expiration date of the assignment 527 based on the time-out condition of the project 535. After the assignment process, the API 407 returns the assigned task 525 and the expiration date of the assignment 527 to the annotator terminal 403 in S627.

[0130] In S627, the annotator terminal 403 uses the URI and access key associated with the assigned task 525 to request the corresponding learning data 521 from the API 407. In response to the request, in S628, the API 407 determines whether the file of the learning data 521 is accessible using the URI and access key. If the determination result is Yes, the API 407 reads the file of the learning data 521 from the storage 408 and returns the file to the annotator terminal 403 in S629. Upon receiving the file, the annotator terminal 403 displays the learning data 521 in S630.

[0131] In S631, the annotator 402 inputs the annotation results for the displayed learning data 521 (e.g., labels specifying the matters determined by the annotator 402 for the learning data 521) to the annotator terminal 403. In S632, the annotator terminal 403 transmits the input annotations and the end time of the task 525 (e.g., the current time) to the API 407. In response to these data, the API 407 creates a result 529 associated with the task 525 in S633 and records the work time (e.g., the length of time from the start time (assignment time) to the end time of the task) in the result 529. Then, in S634, the API 407 returns a response to the annotator terminal 403 indicating that the annotation results have been accepted.

[0132] FIG. 10 shows an example of the control flow of the assignment process (S625) shown in FIG.

[0133] In S641, the existing assignments 527 of the annotator 402 are searched, and in S642, it is checked whether any of the existing assignments 527 have expired. In S643, if any assignments 527 have expired, they are invalidated, that is, canceled. If a task 525 assigned to the annotator 402 is not completed even after the expiration date, the assignment of the task to the annotator 402 can be canceled, and the task can be assigned again to another annotator 402. This makes it possible to prevent delays in the completion of the task 525. Then, in S644, the annotator 404 is notified of any existing assignments 527 that are within their expiration dates.

[0134] In S645, it is checked whether a new task can be assigned to the annotator 402. For example, this check can be performed based on the capabilities of the annotator 402, the number of existing incomplete assignments that the annotator 402 has, and whether there is a request for a new assignment from the annotator 402. If a new assignment is not possible, control of the assignment process ends.

[0135] If a new assignment is possible, in S646, the existing tasks 525 are searched for unlocked, incomplete tasks 525. Then, in S647, at least one task 525 having a higher priority than the other tasks is selected from the unlocked, incomplete tasks 525 found. In S648, an assignment 527 is created to assign the selected task 525 to the annotator 402, and a start time (e.g., the current time) and an expiration date are recorded in the assignment 527. Here, the expiration date is determined, for example, by applying the time-out condition (e.g., the maximum allowable processing time) set for the project in 535 to the start time.

[0136] In S649, the task 525 and expiration date of the new assignment 527 are notified to the annotator 402. Then, if the new assignment 527 causes the current number of assignments for the task 525 to reach the maximum number of assignments, in S650, the task 525 is locked.

[0137] The above-described assignment process increases the likelihood that a task with a higher priority will be processed first. The priority of each task can be changed arbitrarily by the requester 400, as described with reference to Fig. 8, and can also be changed automatically, as will be described later with reference to Fig. 14. Controlling the priority of a task controls the progress of the task.

[0138] In the assignment process described above, an expiration date is set for each assignment, and any assignment that is not completed after the expiration date is automatically invalidated, allowing the same task to be reassigned to another annotator. This prevents delays in task completion due to annotator negligence. Furthermore, a maximum number of assignments is set for each task, and the same task can be assigned to multiple annotators up to the maximum number of assignments. This allows the same task to be completed by multiple annotators, improving the accuracy of the annotation results and avoiding inefficiencies caused by assigning the same task to too many annotators exceeding the maximum number.

[0139] FIG. 11 shows a control flow in which a suggestion process using a learning model is added to the control of the annotation generation work already described with reference to FIG.

[0140] Here, the suggestion process is a process in which the learning unit 102 in this system 50 generates suggestions for annotation results, i.e., labels, to be added and presents them to the annotator 404 in order to help the annotator 402 and facilitate the annotation work on the training data 521.

[0141] In FIG. 11, the control from SS624 to S628 is the same as that shown in FIG. 9. After reading the file of the training data 521 in S628, the API 407 transmits the file to the training unit 102. Upon receiving the file, the training unit 102 inputs the file into its training model in S642 and executes a machine learning algorithm. This results in output data from the training model for the training data 521, i.e., the label of the training data 521 inferred by the training model (hereinafter referred to as the inferred label). The reliability of the inferred label output from the training model changes depending on the progress of the training of the training model. If the training has progressed to a certain extent, the output inferred label should be reliable enough to assist the annotator 402. In S643, the training unit 102 returns the obtained inferred label to the API 407 as a suggestion. In S644, the API 407 returns the file of the training data 521 and the suggestions to the annotator terminal 403. In S645, the annotator terminal 403 displays the training data 521 and the suggestions (the classification results produced by the training model). The annotator 402 can refer to the suggestions when deciding what labels to assign to the training data. The subsequent control from S631 to S634 is the same as that shown in Figure 9.

[0142] Figure 12 shows the control flow for reviewing annotation work and reporting the results.

[0143] In S651, a reviewer 404 requests login from the reviewer terminal 405 to the API 407. If the login is successful, in S652 the API 407 sends information representing one or more existing projects 535 to the reviewer terminal 405. In response to this information, in S653 the reviewer terminal 405 transitions its display from the login screen to a review screen for performing review work.

[0144] Thereafter, the control of steps S654 to S664 described below is repeated for one or more tasks.

[0145] In S654, the reviewer terminal 405 selects one project from the existing projects and requests the API 407 for tasks 525 associated with the selected project 535. In response to the request, the API 407 selects at least one task 525 to which one or more results 529 have already been assigned in S655 and returns the selected task to the reviewer terminal 405 in S656.

[0146] In S657, the reviewer terminal 405 requests the API 403 for the corresponding learning data 521 using the URI and access key associated with the selected task 525. In response to the request, in S658, the API 407 determines whether the file of the learning data 521 can be accessed using the URI and access key. If the determination result is Yes, the API 407 reads the file of the learning data 521 from the storage 408, and in S659 returns the file of the learning data 521 and the annotation result of the result 529 associated with the task 525 to the reviewer terminal 405. Upon receiving these data, in S660, the reviewer terminal 405 displays the annotation result of the learning data 521.

[0147] In S661, the reviewer 404 inputs the review results for the annotation results of the displayed learning data 521 (for example, a judgment on whether the annotation results, i.e., the labels, are appropriate, or the reviewer has corrected the labels, etc.) into the reviewer terminal 405. In S662, the reviewer terminal 405 transmits the input review results to the API 407. In response to these data, the API 407 creates a review 531 associated with the result 529 in S653. Then, in S664, the API 407 returns a response to the review terminal 405 indicating that the review results have been accepted.

[0148] The requester terminal 402 can select at least any project that it has requested at any time in S665 and request information on desired items (or fixed items) related to the selected project from the API 407. In response to this request, the API 407 prepares report data for the requested information in S666 and returns the report data to the requester terminal 401 in S667. The requester terminal 401 displays the received report data in S648.

[0149] Here, information items that can be included in the report data include various data managed in the management database 409 shown in FIG. 7 , one example of which is information about workers (annotators or reviewers). For example, when information about one or more annotators participating in a certain project is requested from the client terminal 401, the API 407 obtains, from the management database 409, for example, the processing times (length of time from start time to end time) of one or more tasks that each annotator has processed to date, and calculates the average value (average processing time). The API 407 also obtains from the management database 409 the review results that each worker has received to date, and aggregates these review results (e.g., calculates the number of times the annotation results were judged to be appropriate, the number of times they were judged to be inappropriate, and the ratio between those numbers). The API 407 then compiles the processing times for each task, the average processing time, the review results, and the aggregated review results into report data and returns it to the client terminal 401. The client can use the report data as a reference to, for example, evaluate the capabilities of each annotator or adjust the amount of compensation paid to each worker.

[0150] Another example of information that a requester can request is information about the status of the project progress or the status of tasks. For example, when information about all or some of the tasks included in a certain project is requested from the requester terminal 401, the API 407 obtains information from the management database 409, such as the priority, maximum number of assignments, current number of assignments, and completion status of each task, compiles that information into report data, and returns it to the requester terminal 401. From the report data, the requester can understand the progress of each task or the progress of the project and change the priority of each task as necessary. This process corresponds to the processes from S613 to S615 shown in FIG. 8.

[0151] Figure 13 shows the control flow for training a learning model, which is automatically performed in parallel with annotation work.

[0152] In S671, the API 407 automatically requests learning from the learning unit 102 (for example, according to a predetermined time schedule or triggered by a predetermined event such as the completion of review). In response to the request, in S672, the learning unit 102 executes a machine learning algorithm, that is, trains a learning model, using one or more annotation results for which review has been completed at the current time and one or more units of learning data 521 corresponding to the annotation results as training data. This updates the learning model of the learning unit 102. In S673, the learning unit 102 returns a response to the API 407.

[0153] The client terminal 401 can request learning from the API 407 at any time in S674. In this case, the API 407 also requests learning from the learning unit 102 in S675. In response to this, in S676, a machine learning algorithm is executed in the same manner as in the above-mentioned control, and the learning model of the learning unit 102 is updated. In S667 and S668, a response is returned from the learning unit 102 to the API 407 and then to the client terminal 401.

[0154] In this way, machine learning, or training, of the learning model is performed in parallel with the annotation work. Therefore, the optimization of the learning model progresses as the annotation work progresses. The more the optimization of the learning model progresses, the more reliable the suggestions described with reference to Figure 11 become.

[0155] FIG. 14 shows the control flow for automatically changing task priorities.

[0156] As described with reference to FIG. 8, the requester 400 can change the priority of any task included in his / her project at any time. As can be seen from the flow of the assignment process shown in FIG. 10, the higher the priority of a task, the earlier the task will be assigned to the annotator 402, and as a result, the more likely the task will be completed quickly. The system 50 can also automatically change the priority of a task. FIG. 14 shows an example of such automatic priority change control. In the system 50, this control is performed, for example, by the API 407 and the learning unit 102 working together.

[0157] In S681, the API 407 counts the number of completed tasks for which review has been completed and annotation results have been finalized as of now, for each label indicated by the finalized annotation results (for example, counting the number of images with the labels "dog," "cat," and "monkey" assigned to the animal images in the training data). This determines the distribution of the number of completed tasks by label. In S682, the API 407 selects one or more labels with a larger number of completed tasks than other labels (hereinafter referred to as "majority labels") and / or one or more labels with a smaller number of completed tasks than other labels (hereinafter referred to as "minority labels") according to a predetermined rule. The predetermined rule may be set arbitrarily, for example, by selecting the label with the largest or smallest number of completed tasks, selecting the label that falls within a predetermined range in the distribution curve of the number of completed tasks described above, or selecting the label whose ratio of the number of completed tasks compared between labels is higher or lower than a predetermined value.

[0158] In S683, the API 407 selects one or more incomplete tasks for which annotation results are not yet finalized, reads the learning data 521 files associated with the selected incomplete tasks from the storage 408, and provides these files to the learning unit 102. In S684, the learning unit 102 inputs the provided learning data 521 for each incomplete task into a learning model and executes a machine learning algorithm. As a result, a label (inferred label) inferred by the learning model for the learning data 521 for each incomplete task is obtained. If the learning of the learning model has progressed to a certain extent, the obtained inferred label should be reliable to a certain extent. The inferred label obtained for each incomplete task is returned from the learning unit 102 to the API 407.

[0159] In S685, the API 407 selects incomplete tasks whose inferred labels are the same as the minority label or the majority label. Incomplete tasks whose inferred labels are the same as the minority label may be processed with priority over other tasks. Such incomplete tasks are hereinafter referred to as "high-priority tasks." Conversely, incomplete tasks whose inferred labels are the same as the majority label may be processed later than other tasks. Such incomplete tasks are hereinafter referred to as "low-priority tasks." For example, suppose there are many completed tasks that label many animal images with their respective animal species names, and most of the completed tasks are labeled "dog" and very few are labeled "cat." In this case, among the incomplete tasks, tasks that should be labeled "cat" may be processed with priority, and tasks that should be labeled "dog" may be processed later. Selection of one or both of these high-priority and low-priority tasks is performed in S685.

[0160] In S686, the API 407 changes the priority of the uncompleted task selected in S685, for example, by increasing the priority of a high-priority task or decreasing the priority of a low-priority task.

[0161] In this way, the priority of incomplete tasks is automatically controlled according to the distribution of the labels (annotation results) of completed tasks. This priority control contributes to balancing the distribution of the number of training data obtained through annotation work between labels. A balanced set of training data is beneficial for proper machine learning.

[0162] FIG. 15 shows the control flow for issue management in annotation work.

[0163] As already explained with reference to FIG. 2, an issue refers to a matter requiring discussion, in which the annotator is unable to determine or is unsure of what label to assign to the training data 521. The data unit corresponding to an issue is issue 533 shown in FIG. 2. FIG. 15 shows the flow of issue management and resolution. In FIG. 15, the control shown in area A relates to the processing performed by the annotator 402 using the annotator terminal 403, that in area B relates to the processing performed by the reviewer 404 using the reviewer terminal 405, and that in area C relates to the processing performed by the requester 400 using the requester terminal 401.

[0164] As shown in Fig. 15, in S701, the annotator 402 searches for unprocessed tasks that have not yet been annotated from among the tasks assigned to him / her using the annotator terminal 403. This corresponds to the control from S624 to S626 shown in Fig. 9 and the control of the assignment process shown in Fig. 10. If no unprocessed tasks are found, this control of issue management ends.

[0165] When an unprocessed task is found, the annotator 402 performs the content of the unprocessed task (displaying the learning data 521, checking the label to be attached, etc.) in S702, and then determines what the correct label is in S703. Once the correct label is known, the annotator 402 inputs the correct label, that is, the annotation result, in S704. This corresponds to the control from S627 to S631 shown in FIG. 9.

[0166] If the correct label is unknown, in S705 the annotator 402 makes a request to the API 407 from the annotation terminal 403 to list the issues 533 related to the same project 535 that currently exist, and refers to the contents of those issues 533. In S706, the annotator 402 searches for an issue that is applicable to the current task from among the listed issues 533. For example, if there is an issue related to the same or a similar task, that issue is considered to be applicable.

[0167] When an applicable issue 533 is found, the annotator 402 checks whether the issue 533 has been resolved in S707. If it has been resolved, the issue 533 includes a guide, which is instructions or advice from the requester 400 on how to perform annotation (i.e., how to determine the correct label). Therefore, the annotator 402 refers to the guide in S708 and returns to S703 to reconsider the correct label.

[0168] If no applicable issue 533 is found in S706, or if no applicable issue has yet been resolved in S707 (i.e., no guide has yet been entered), the annotator 402 performs a predetermined action on the current task (e.g., suspends the task) in S709 and then moves on to the next task.

[0169] In S711, the reviewer 404 accesses the API 407 from the reviewer terminal 405 to list the pending tasks and refer to the contents of the pending tasks. In S712, the reviewer 404 classifies the listed pending tasks into one or more groups consisting of pending tasks that are similar to each other, and creates an issue 533 associated with each group (i.e., pending tasks that are similar to each other).

[0170] In S713, the requester 400 accesses the API 407 from the requester terminal 401, lists the issues 533 related to his / her project, and inputs a guide for each issue 534.

[0171] In S714, the reviewer 404 accesses the API 407 from the reviewer terminal 405, refers to the contents of the issues 533 related to the reviewer 404, and checks the validity of the guide entered in each issue 533 (for example, whether the guide is clear enough for the annotator to determine the correct label). If the reviewer 404 determines that the guide is invalid, in S715 the reviewer 404 enters his / her opinion about the guide in the issue 533 and requests the requester 400 to revise the guide. If the reviewer 404 determines that the guide is invalid, in S716 the reviewer 404 marks the issue 533 as resolved and returns all pending tasks associated with the issue 533 to the respective annotators 402 (i.e., releases the pending tasks and returns them to normal pending tasks). This allows the annotator 402 who put a pending task on hold in S709 to resume annotation work for that pending task from S702 onwards.

[0172] This type of issue management reduces annotation results based on incorrect or ambiguous judgments, improving the quality of training data for machine learning.

[0173] Although several embodiments of the present disclosure have been described above, these are merely examples for understanding the present disclosure and are not intended to limit the scope of the present disclosure to only these embodiments. The present disclosure can be implemented in various structures or methods different from the above-described embodiments without departing from the spirit of the present disclosure. [Explanation of symbols]

[0174] 100: Learning and inference support system

Claims

1. A learning unit that has at least one learning model and generates one or more trained models by performing machine learning on the learning model; a model evaluation unit that evaluates the one or more trained models and determines at least one practical model based on the evaluation results; an inference unit having the at least one practical model, inputting practical data into the practical model, executing inference using the practical model, and outputting inference result data; a data pool that collects and stores in one place a set of learning data used by the learning unit for the machine learning of the learning model, a set of practical data that the inference unit inputs to the practical model, and a set of inference result data that the inference unit outputs; A computer system comprising:

2. The computer system according to claim 1 , wherein the learning unit is capable of performing additional machine learning on the trained model or the practical model.

3. the evaluation unit selects a model to be subjected to the additional machine learning in accordance with the evaluation result.

3. The computer system according to claim 2.

4. The model evaluation unit If there are multiple trained models for the same purpose, the one with the highest output accuracy is selected as the practical model.

4. The computer system according to claim 1.

5. The inference unit a first inference unit that performs inference using unstructured data as input; a second inference unit that performs inference or statistical processing using the structured data or the inference result data as input; 4. The computer system according to claim 1.

6. The model evaluation unit providing one or more free virtual units, each free virtual unit having an input module and an output module; providing at least one occupied virtual unit by incorporating the at least one working model into at least one of the at least one free virtual unit; In the at least one dedicated virtual unit, the practical data is input to the at least one practical model through the input module, and the inference is performed by the at least one practical model; outputting the inference result data output from the at least one practical model by executing the inference from the dedicated virtual unit; transmitting the inference result data output from the dedicated virtual unit to at least one data target through at least one interface; 4. The computer system according to claim 1.

7. The input module and the output module in each of the free virtual units are general-purpose modules having a configuration that does not depend on a practical model and can be incorporated into each of the free virtual units.

7. The computer system according to claim 6.

8. The model evaluation unit establishing meta-information about the at least one dedicated virtual unit; the meta-information includes parameters relating to the operation of at least one of the generic modules of the input module and the output module; setting the parameters in the at least one general-purpose module based on the meta-information; Operate at least one general-purpose module based on the set parameters; It was configured as 8. The computer system of claim 7.

9. The model evaluation unit establishing meta-information about the at least one dedicated virtual unit; the meta information includes information indicating attributes of the at least one practical model, the attributes including identification information and a location of the at least one practical model; Incorporating the at least one practical model into the at least one free virtual unit based on the meta-information.

7. The computer system according to claim 6.

10. The model evaluation unit initially preparing a predetermined number of free virtual units, two or more, in at least one physical computer; This allows the number of the dedicated virtual units to be changed within a range equal to or less than the predetermined number without requiring the physical computer to be restarted. It was configured as 7. The computer system according to claim 6.

11. The model evaluation unit Investigating the accuracy of the one or more trained models; determining the at least one practical model based on the accuracy of the one or more trained models; 4. The computer system according to claim 1.

12. The model evaluation unit Investigating the accuracy of the one or more trained models; selecting at least one trained model from the one or more trained models based on the accuracy of the one or more trained models, and performing additional machine learning on the selected trained model; 4. The computer system according to claim 1.

13. The model evaluation unit performing statistical processing using at least one of the practical data and the inference result data; 4. The computer system according to claim 1.

14. The model evaluation unit transmitting the training data to at least one annotator terminal through at least one interface; receiving annotation results related to the training data from the at least one annotator terminal through the at least one interface; storing the received annotation results in the data pool in association with the training data; The learning data and the associated annotation results are used as training data to perform the machine learning of the one or more learning models.

4. The computer system according to claim 1, wherein the computer system is configured as follows:

15. A computer program that, when executed by a computer, enables the following steps to be performed, said steps comprising: generating one or more trained models by performing machine learning on at least one learning model; Evaluating the one or more trained models and determining at least one practical model based on the evaluation results; Executing inference using the at least one practical model and outputting inference result data; Collecting and storing a set of learning data used in the machine learning of the learning model, a set of practical data input to the practical model, and a set of inference result data in a single data pool; A computer program comprising:

Images