JUDGMENT PROGRAM, JUDGMENT METHOD, AND INFORMATION PROCESSING APPARATUS
The determination program addresses complex data prediction accuracy issues by categorizing and statistically analyzing data sets to ensure compatibility and accuracy in model selection.
Patent Information
- Application Number
- JP2021028087
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-25
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-02-25
AI Technical Summary
When the causal relationship between input data and prediction results is complex, existing models may not provide prediction results with the expected accuracy, and users are unaware of this before executing predictions.
A determination program that converts data into item value sets, classifies items into categories, calculates statistics based on these categories, and determines model suitability by comparing similarity between data sets to ensure accurate predictions.
Enables the determination of whether prediction with expected accuracy is possible, ensuring high accuracy by selecting models compatible with user data.
Smart Images

Figure 0007594183000001 
Figure 0007594183000002 
Figure 0007594183000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a determination program, a determination method, and an information processing device. [Background technology]
[0002] With the recent development of AI (Artificial Intelligence) technology, data analysis and prediction using models trained by machine learning have become a major social need. Therefore, useful models are required in various industries such as companies and medical facilities. For example, medical facilities can use trained models to determine appropriate treatment plans according to the patient's condition.
[0003] When multiple trained models are available, it may be difficult to determine which model is appropriate to use. Therefore, a trained model providing method has been proposed that selects and provides a trained model that is optimal for use on a user device from multiple trained models stored in advance in a database, for example. A training method that allows selection of an evaluation model in a short time has also been proposed. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] International Publication No. 2018 / 142766 [Patent Document 2] Patent No. 6751955 Summary of the Invention [Problem to be solved by the invention]
[0005] When the causal relationship between the input data and the prediction result is complex, even if a fully trained model is used, the prediction result may not have the expected accuracy. However, in the conventional technology, a user cannot know whether a prediction result with the expected accuracy will be obtained when the user's own data is input into an existing model before executing a prediction using the model.
[0006] In one aspect, the present invention aims to make it possible to determine in advance whether or not it is possible to predict an expected accuracy. [Means for solving the problem]
[0007] In one proposal, a determination program is provided that causes a computer to execute the following processes. The computer classifies items of a plurality of first records included in a first dataset used to generate a trained model by machine learning into one of a plurality of categories based on a first item value set in the items of the plurality of first records. Next, the computer calculates a first statistic in a procedure corresponding to the category to which the items belong based on the item values set in the items of the plurality of first records. Furthermore, the computer calculates a second statistic in a procedure corresponding to the category to which the items belong based on second item values set in the items of a plurality of second records included in a second dataset having the same items as the first dataset. Then, the computer determines whether the second dataset is compatible with the model based on the similarity between the first statistic and the second statistic. Effect of the Invention
[0008] According to one aspect, it becomes possible to determine in advance whether or not the expected accuracy can be predicted. [Brief description of the drawings]
[0009] [Figure 1] FIG. 4 illustrates an example of a determination method according to the first embodiment. [Diagram 2] FIG. 1 illustrates an example of a system configuration. [Diagram 3] FIG. 2 illustrates an example of hardware of a DB server. [Figure 4] FIG. 2 is a block diagram showing an example of functions of each server. [Diagram 5] FIG. 13 is a diagram showing an example of an application target dataset held by a DB server. [Figure 6] FIG. 13 is a diagram illustrating an example of a learning dataset held by a cloud server. [Figure 7] FIG. 13 is a diagram showing an example of a procedure for calculating a predicted value using a model to which a target data set is adapted. [Figure 8] 13 is a flowchart illustrating an example of a procedure for a classification process. [Figure 9] FIG. 11 is a diagram showing a first example of calculation of column statistics. [Figure 10] FIG. 11 is a diagram illustrating a second example of calculation of column statistics. [Figure 11] FIG. 13 is a diagram illustrating a third example of column statistics calculation. [Figure 12] FIG. 13 is a diagram showing a fourth example of calculation of column statistics. [Figure 13] FIG. 13 is a diagram illustrating a fifth example of column statistics calculation. [Figure 14] 1 is a flowchart (part 1) illustrating an example of a procedure of a statistical amount calculation process. [Figure 15] 13 is a second flowchart illustrating an example of a procedure for a statistical amount calculation process. [Figure 16] 11 is a third flowchart illustrating an example of a procedure for a statistical amount calculation process. [Figure 17] 11 is a fourth flowchart illustrating an example of a procedure for a statistical amount calculation process. [Figure 18] FIG. 11 is a diagram illustrating an example of a feature vector generation process. [Figure 19] 13 is a flowchart illustrating an example of a procedure for a similarity calculation process. [Figure 20] 13 is a flowchart illustrating an example of a procedure for a conformity determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the present embodiment will be described with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments as long as no contradiction occurs. First Embodiment First, a first embodiment will be described. The first embodiment is a method for determining whether a dataset owned by a user is compatible with a model generated by machine learning. The compatibility of a dataset with a model means that when the dataset is input to the model, a highly accurate prediction result is obtained.
[0011] Fig. 1 is a diagram showing an example of a determination method according to the first embodiment. Fig. 1 shows an example of a determination method for determining compatibility, which is implemented using an information processing device 10. The information processing device 10 can implement the determination method by, for example, executing a determination program.
[0012] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory or a storage device included in the information processing device 10. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10.
[0013] The storage unit 11 stores a model 1 that has been trained by machine learning, a first dataset 2 that has been used to train the model 1, and a second dataset 3 to which an inference process using the model 1 is to be applied.
[0014] The processing unit 12 compares the first data set 2 with the second data set 3 to determine whether the second data set 3 fits the model 1. In other words, if a data set having a data distribution similar to the data distribution of the first data set 2 used to train the model 1 is used, it is considered that a highly accurate prediction result can be obtained using the model 1. Therefore, the processing unit 12 generates a first feature vector 4 and a second feature vector 5 according to the data distributions of the first data set 2 and the second data set 3, respectively, and determines that the second data set 3 fits the model 1 when the similarity between the vectors exceeds a certain threshold.
[0015] To this end, the processing unit 12 first classifies the items of the first data set 2 and calculates a first statistical amount. For example, the processing unit 12 classifies the items of a plurality of first records included in the first data set 2 used to generate the model 1 into one of a plurality of categories based on the first item values set in the items of the plurality of first records. Next, the processing unit 12 calculates the first statistical amount in a procedure according to the category to which the item belongs based on the item values set in the items of the plurality of first records. For example, the processing unit 12 calculates a plurality of first statistical amounts of different types, such as an average or a median, based on the first item values set in the items of the plurality of first records.
[0016] Furthermore, the processing unit 12 classifies items of the second data set 3 having the same items as those of the first data set 2 and calculates second statistics. For example, the processing unit 12 calculates second statistics in a procedure according to the category to which the items belong, based on second item values set in the items of multiple second records included in the second data set 3. For example, the processing unit 12 calculates multiple second statistics of different types, based on the second item values set in the items of multiple second records.
[0017] Then, the processing unit 12 judges whether or not the second data set 3 matches the model 1 based on the similarity between the first statistics and the second statistics. The similarity is, for example, the similarity between a first feature vector 4 having each of the multiple first statistics as an element and a second feature vector 5 having each of the multiple second statistics as an element. The processing unit 12 judges that the data match if the similarity is equal to or greater than a preset threshold value, for example.
[0018] In this way, by classifying items into categories according to the item values set for the items and calculating statistics in a procedure according to the category, it is possible to obtain statistics that well represent the characteristics of the item values set in the first data set 2 and the second data set 3. As a result, it is possible to calculate the similarity with high accuracy, and to correctly determine whether or not they match.
[0019] Item categories include, for example, nominal binary scale, nominal polynomial scale, numerical data, free text, etc. Nominal binary scale can also be categorized separately depending on whether it is numerical data or not. For example, the processing unit 12 classifies the items into different categories depending on whether the first item value set in the item is a numerical value or not.
[0020] When the first item value set for an item is not a numeric value, the processing unit 12 calculates the number of occurrences of each type of the first item value as the first statistical amount. In calculating the second statistical amount, the processing unit 12 calculates the number of occurrences of each type of the second item value as the second statistical amount. This makes it possible to convert the distribution of non-numeric item values into numeric values and calculate appropriate statistical amounts.
[0021] Furthermore, the processing unit 12 may classify the first item values into different categories depending on whether the number of types of the first item values is less than two. In this case, when the number of types of the first item values set in the items is less than two, the processing unit 12 converts the first item value to 0 or 1 according to the type, and calculates the first statistical amount based on the converted value. Furthermore, the processing unit 12 calculates the first statistical amount based on the converted value, converts the second item value to 0 or 1 according to the type, and calculates the second statistical amount based on the converted value. In this way, when the number of types of the item values is less than two, the statistical amount can be calculated on a unified scale, and the accuracy of the determination of the similarity can be improved.
[0022] The processing unit 12 may also classify the items into different categories depending on whether the ratio of the number of types of first item values to the number of first records is equal to or greater than a predetermined value. For example, when the ratio is less than a predetermined value, the processing unit 12 divides the items of the first dataset 2 into first divided items for each type of first item value. The processing unit 12 then sets a value indicating the presence or absence of a corresponding type of first item value in the first divided item of each of the multiple first records, and calculates a first statistic for each first divided item. The processing unit 12 also divides the items of the second dataset 3 into first divided items. The processing unit 12 then sets a value indicating the presence or absence of a corresponding type of second item value in the first divided item of each of the multiple second records, and calculates a second statistic for each first divided item. This makes it possible to appropriately convert item values other than numerical values into numerical values and calculate features.
[0023] Furthermore, when the ratio value is equal to or greater than a predetermined value, the processing unit 12 divides the items of the first data set 2 into second divided items for each word included in the first item value. The processing unit 12 then sets a value indicating the presence or absence of a corresponding word in the second divided item of each of the multiple first records, and calculates a first statistic for each of the second divided items. The processing unit 12 also divides the items of the second data set 3 into second divided items. The processing unit 12 then sets a value indicating the presence or absence of a corresponding word in the second divided item of each of the multiple second records, and calculates a second statistic for each of the second divided items. This makes it possible to appropriately convert free text into a numerical value and calculate a feature amount.
[0024] Second Embodiment Next, a second embodiment will be described. The second embodiment is a system that predicts any information from the user's own data by utilizing a model provided by a cloud computing system (hereinafter, referred to as a cloud).
[0025] 2 is a diagram showing an example of a system configuration. The DB server 100 manages a data set to which inference using a model is applied. The DB server 100 is connected to a cloud server 200 via a network 20. The cloud server 200 is a server computer provided on a cloud. The cloud server 200 has a plurality of trained models and provides the models in response to a request from the DB server 100.
[0026] The cloud server 200 judges the degree of suitability of the data set held by the DB server to the model based on the similarity to the learning data set used to learn the model. That is, the learned model is optimized so as to obtain a correct inference result when the data set used for learning is input. If the model is a neural network, the weight value indicating the connection strength between neurons is adjusted, and a correct prediction result can be obtained by inputting the data set used for learning to the model. Therefore, if a data set is similar to the data set used for learning, there is a high possibility that a correct prediction result will be obtained when input to the same model.
[0027] Therefore, in order to select an appropriate model for performing inference processing using a data set owned by the DB server 100 as input data, the DB server 100 calculates statistics indicating the characteristics of the data set. The DB server 100 transmits the calculated statistics to the cloud server 200. The DB server 100 also calculates statistics indicating the characteristics of the data set used to generate the model and transmits them to the cloud server 200.
[0028] Cloud server 200 calculates the similarity between the data sets by comparing the feature amounts of the data sets used to generate each of the multiple models owned by cloud server 200 with the feature amounts of the data sets owned by DB server 100. Cloud server 200 transmits to DB server 100 the similarity calculated for each data set used to generate each of the multiple models.
[0029] The DB server 100 judges whether or not the data set held by the DB server 100 is compatible with each of the multiple models based on the similarity acquired from the cloud server 200. Then, the DB server 100 acquires from the cloud server 200 a model whose degree of compatibility with the data set is higher than a threshold, and performs inference processing using the model.
[0030] 3 is a diagram showing an example of hardware of a DB server. The DB server 100 is entirely controlled by a processor 101. A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least a part of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).
[0031] The memory 102 is used as a main storage device of the DB server 100. The memory 102 temporarily stores at least a part of the OS (Operating System) program and application programs to be executed by the processor 101. The memory 102 also stores various data used for processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.
[0032] The peripheral devices connected to the bus 109 include a storage device 103, a GPU (Graphics Processing Unit) 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.
[0033] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for a computer. The storage device 103 stores an OS program, application programs, and various data. Note that, for example, a hard disk drive (HDD) or a solid state drive (SSD) can be used as the storage device 103.
[0034] The GPU 104 is an arithmetic device that performs image processing, and is also called a graphics controller. The GPU 104 is connected to a monitor 21. The GPU 104 displays an image on the screen of the monitor 21 in accordance with an instruction from the processor 101. The monitor 21 may be a display device using an organic EL (Electro Luminescence) display device, a liquid crystal display device, or the like.
[0035] The input interface 105 is connected to the keyboard 22 and the mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. Note that the mouse 23 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.
[0036] The optical drive device 106 uses a laser beam or the like to read data recorded on the optical disc 24 or write data to the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), etc.
[0037] The device connection interface 107 is a communication interface for connecting peripheral devices to the DB server 100. For example, a memory device 25 or a memory reader / writer 26 can be connected to the device connection interface 107. The memory device 25 is a recording medium equipped with a communication function with the device connection interface 107. The memory reader / writer 26 is a device that writes data to the memory card 27 or reads data from the memory card 27. The memory card 27 is a card-type recording medium.
[0038] The network interface 108 is connected to the network 20. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 108 is a wired communication interface that is connected by a cable to a wired communication device such as a switch or a router. The network interface 108 may also be a wireless communication interface that is communicatively connected by radio waves to a wireless communication device such as a base station or an access point.
[0039] The DB server 100 can realize the processing function of the second embodiment by the above hardware. The cloud server 200 can also be realized by the same hardware as the DB server 100. Moreover, the information processing device 10 shown in the first embodiment can also be realized by the same hardware as the DB server 100.
[0040] The DB server 100 realizes the processing function of the second embodiment by executing a program recorded in, for example, a computer-readable recording medium. The program describing the processing contents to be executed by the DB server 100 can be recorded in various recording media. For example, the program to be executed by the DB server 100 can be stored in the storage device 103. The processor 101 loads at least a part of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the DB server 100 can also be recorded in a portable recording medium such as the optical disk 24, the memory device 25, or the memory card 27. The program stored in the portable recording medium becomes executable after being installed in the storage device 103 under the control of, for example, the processor 101. The processor 101 can also directly read and execute the program from the portable recording medium.
[0041] Next, the functions of each server for performing inference processing using a model to which the data set held by the DB server 100 is adapted will be described. 4 is a block diagram showing an example of functions of each server. The DB server 100 includes a storage unit 110, a classification unit 120, a statistics calculation unit 130, a conformance determination unit 140, a model acquisition unit 150, and an inference unit 160.
[0042] The storage unit 110 stores a target data set 111 to which the inference process is to be applied and a model 112 acquired from the cloud server 200. The target data set 111 is, for example, data in a table format, and a record is registered in each row. An item is assigned to each column in each record, and an item value of the corresponding item is set in the column. For example, if the DB server 100 is a system of a medical institution, the target data set 111 includes a record for each patient. Attributes of the item value to be registered (such as name, sex, and disease name) are set in the columns. The storage unit 110 is realized, for example, by a part of the memory area of the memory 102 or the storage device 103 of the DB server 100.
[0043] The classification unit 120 classifies each column of the target data set 111 into a category according to the type of item value set. For example, the classification unit 120 classifies a column in which a numerical value is set into a column in which a character string is set into a different category. The classification unit 120 also classifies into a different category depending on whether the number of types of item values set is only two or three or more. The classification unit 120 transmits information indicating the category to which each column belongs to the statistics calculation unit 130.
[0044] The classification unit 120 also acquires a training dataset used to generate each of the multiple models stored in the cloud server 200. The classification unit 120 then classifies the training dataset into categories for each column, similar to the application dataset 111.
[0045] The statistics calculation unit 130 calculates statistics of the dataset used to generate the model and the target dataset 111 to which the inference process is applied. For example, the statistics calculation unit 130 calculates a predetermined statistic for each column of the target dataset 111 based on the item values of the corresponding column of all records, using a method according to the category to which the corresponding column belongs. Similarly, the statistics calculation unit 130 calculates statistics for each column of the learning dataset acquired from the cloud server 200. The statistics calculation unit 130 uploads to the cloud server 200 a statistic set 61, 62,... that summarizes the statistics calculated for each of the target dataset 111 or the learning datasets 41, 42,...
[0046] The compatibility determination unit 140 determines whether the target dataset 111 for the model-based prediction process is compatible with any of the models 51, 52, .... For example, the compatibility determination unit 140 transmits a similarity determination request to the cloud server 200. In response to the similarity determination request, the compatibility determination unit 140 acquires the similarity between each of the training datasets 41, 42, ... and the target dataset 111 from the cloud server 200. The compatibility determination unit 140 transmits to the model acquisition unit 150 the identification number of the model corresponding to the training dataset whose similarity is equal to or greater than a predetermined threshold.
[0047] The model acquisition unit 150 acquires a model that is compatible with the target dataset 111 from the cloud server 200. For example, the model acquisition unit 150 transmits a model acquisition request specifying the identification number acquired from the compatibility determination unit 140 to the cloud server 200. The cloud server 200 returns a trained model that corresponds to the specified identification number. The model acquisition unit 150 stores the acquired model in the storage unit 110.
[0048] The inference unit 160 performs inference processing using the application target dataset 111 as input data to the model 112, and obtains a predicted value for a predetermined item. The inference unit 160 outputs the obtained predicted value as a prediction result.
[0049] The cloud server 200 includes a storage unit 210 , a learning dataset providing unit 220 , a similarity calculation unit 230 , and a model providing unit 240 . The storage unit 210 stores a plurality of training data sets 41, 42,..., a plurality of models 51, 52,..., and a plurality of statistical quantity sets 61, 62,.... Each of the plurality of training data sets 41, 42,... is a data set used for generating any one of the models. Each of the plurality of statistical quantity sets 61, 62,... is a set of statistics calculated from the plurality of training data sets 41, 42,... or the target data set 111. The storage unit 210 is realized using a part of the memory area of the memory or storage device included in the cloud server 200.
[0050] In response to a request from the DB server 100, the learning data set providing unit 220 acquires a plurality of learning data sets 41, 42, . . . from the storage unit 210 and transmits them to the DB server 100.
[0051] The similarity calculation unit 230 calculates the similarity between each of the training data sets 41, 42, ... and the target data set 111 held by the DB server 100 based on the similarity determination request sent from the DB server 100. For example, the similarity calculation unit 230 generates a feature vector of the target data set 111, whose elements are the numerical values of the statistics calculated for each column of the target data set 111. The similarity calculation unit 230 also generates a feature vector of each of the multiple training data sets 41, 42, ..., whose elements are the numerical values of the statistics calculated for each column. Then, the similarity calculation unit 230 calculates the similarity between the feature vector of the target data set 111 and the feature vector of each of the training data sets 41, 42, .... The similarity calculation unit 230 transmits the calculated similarity for each of the training data sets 41, 42, ... to the DB server 100.
[0052] In response to a model acquisition request from the DB server 100 , the model providing unit 240 acquires a model specified in the model acquisition request from the storage unit 210 , and transmits the acquired model to the DB server 100 .
[0053] Note that the lines connecting the elements shown in Fig. 4 indicate part of the communication paths, and communication paths other than those shown in the figure can also be set. The function of each element shown in Fig. 4 can be realized, for example, by having a computer execute a program module corresponding to that element.
[0054] Fig. 5 is a diagram showing an example of a target dataset held by a DB server. For example, a record for each patient is registered in the target dataset 111. The target dataset 111 has columns in which attributes such as a patient ID, name, disease code, and specimen test information are set. Each record includes item values of the attributes set in the columns. In addition to those shown in Fig. 5, the target dataset 111 has columns such as a treatment code, biopsy information, clinical test item classification code, classification code, identification code, material code, measurement method code, result identification code, and pharmaceutical code.
[0055] 6 is a diagram showing an example of a training dataset held by the cloud server. The training dataset 41 held by the cloud server 200 is assigned an identification number of a model 51 generated using the training dataset 41. The training dataset 41 has a record for each patient, similar to the application dataset 111. Each record of the training dataset 41 has the same type of columns as the records in the application dataset 111.
[0056] The model 51 generated using the training data set 41 is, for example, a neural network. Item values set in one or more columns in the training data set 41 are input to the input layer of the model 51, and calculations are performed according to the model 51, so that a highly accurate predicted value can be obtained as the output of the output layer.
[0057] A user having the DB server 100 may want to obtain a predicted value of a predetermined item based on the target dataset 111 by using one of the multiple models 51, 52, ... that the cloud server 200 has. In this case, it is appropriate to use a model that can obtain a correct predicted value among the multiple models 51, 52, .... Each of the multiple models 51, 52, ... has a parameter value adjusted so as to match the learning dataset used during learning. Therefore, if there is a learning dataset similar to the target dataset 111, it is considered that the target dataset 111 matches the model generated using the learning dataset. Therefore, the cloud server 200 determines a model to which the target dataset 111 matches based on the similarity between each of the learning datasets 41, 42, ... and the target dataset 111, for example.
[0058] 7 is a diagram showing an example of a procedure for calculating a predicted value using a model to which a target dataset is adapted. In response to an input instructing model acquisition, the classification unit 120 of the DB server 100 acquires the training datasets 41, 42, ... used in generating the multiple models 51, 52, ... from the cloud server 200. For example, the classification unit 120 transmits a request to acquire the training datasets 41, 42, ... to the cloud server 200 (step S11). The training dataset providing unit 220 of the cloud server 200 transmits the multiple training datasets 41, 42, ... to the DB server 100 (step S12).
[0059] The classification unit 120 of the DB server 100 classifies the columns of the acquired training data sets 41, 42, ... into one of a plurality of categories (step S13). For example, the classification unit 120 classifies each column included in the training data sets 41, 42, ... into one of nominal scale binomial, numerical data, nominal scale multinomial, and free text.
[0060] Specifically, the classification unit 120 classifies the item value set in the column into a category nominal scale binomial if the item value is a numeric value and has only two types (for example, 0 and 1). The classification unit 120 classifies the item value set in the column into a numeric data if the item value is a numeric value and has three or more types. The classification unit 120 classifies the item value set in the column into a category nominal scale binomial if the item value set in the column is not numeric data and has two types. The classification unit 120 classifies the item value into a category nominal scale multinomial if the number of types of item values set in the column is not numeric data and the ratio of the number of types of item values set to the number of records registered in the learning dataset to be processed is less than 10% (the ratio value is less than 0.1). The classification unit 120 classifies the item value set in the column into a category nominal scale multinomial if the item value set in the column is not numeric data and the number of types of item values set to the number of records registered in the learning dataset to be processed is 10% or more. The statistics calculation unit 130 calculates statistics for each column of the multiple learning data sets 41, 42, ... in a procedure according to the category to which the column belongs (step S14). The statistics calculation unit 130 uploads a set of statistics, which is a collection of statistics generated for each learning data set 41, 42, ..., to the cloud server 200 (step S15).
[0061] When the calculation of statistics for the multiple training datasets 41, 42, ... is completed, the classification unit 120 acquires the target dataset 111 to be subjected to the inference process from the storage unit 110 (step S16). Then, the classification unit 120 classifies each column of the target dataset 111 into one of multiple categories, similar to the case of the training datasets 41, 42, ... (step S17). The statistics calculation unit 130 calculates statistics for each column of the target dataset 111 in a procedure according to the category to which the column belongs (step S18). The statistics calculation unit 130 uploads a statistics set that collects statistics generated based on the target dataset 111 to the cloud server 200 (step S19).
[0062] The conformance determination unit 140 transmits a similarity determination request to the cloud server 200 (step S20). The similarity calculation unit 230 of the cloud server 200 generates feature vectors of the target dataset 111 and the training datasets 41, 42, ... based on the previously uploaded statistical quantity sets 61, 62, ... (step S21). Next, the similarity calculation unit 230 calculates the similarity between each of the training datasets 41, 42, ... and the target dataset 111 based on the generated feature vectors (step S22).
[0063] The similarity calculation unit 230 transmits the similarity for each of the training data sets 41, 42, ... to the DB server 100 (step S23). The suitability determination unit 140 of the DB server 100 determines that a model corresponding to a training data set whose similarity is equal to or greater than a predetermined threshold is a model that is suitable for the application target data set 111 (step S24). The suitability determination unit 140 transmits the identification number of the suitable model to the model acquisition unit 150. The model acquisition unit 150 transmits a model acquisition request specifying the identification number of the recommended model to the cloud server 200 (step S25). The model providing unit 240 of the cloud server 200 transmits the model with the identification number specified in the model acquisition request to the DB server 100 (step S26).
[0064] The model acquisition unit 150 of the DB server 100 stores the model 112 acquired from the cloud server 200 in the storage unit 110. After that, the inference unit 160 executes an inference process according to the model 112, using the item values set in the application target dataset 111 as input values to the model 112 (step S27). A predicted value of a predetermined item is obtained by the inference process.
[0065] The above-mentioned procedure makes it possible to calculate a predicted value using a model that fits the target dataset 111. At this time, the DB server 100 calculates statistics of the corresponding columns in accordance with procedures according to the column categories of the learning datasets 41, 42, ... and the target dataset 111.
[0066] Next, the column sorting process will be described in detail. 8 is a flowchart showing an example of a procedure for classification processing. The processing shown in FIG. 8 will be described below in order of step numbers.
[0067] [Step S101] The classification unit 120 selects one column from the learning datasets 41, 42, . . . or the application dataset 111 to be processed. [Step S102] The classification unit 120 determines whether the data set in the selected column is numeric data. For example, the classification unit 120 determines that the data is numeric data if all item values in the column are numeric. If the data is numeric data, the classification unit 120 proceeds to step S103. If the data is not numeric data, the classification unit 120 proceeds to step S106.
[0068] [Step S103] The classification unit 120 determines whether the item value set in the selected column has only two patterns. If there are only two patterns, the classification unit 120 proceeds to step S104. If there are three or more patterns of item values set, the classification unit 120 proceeds to step S105.
[0069] [Step S104] The classification unit 120 classifies the selected column into a nominal scale binary (numeric data). After that, the classification unit 120 proceeds to step S111. [Step S105] The classification unit 120 classifies the selected column into numeric data, and then the processing proceeds to step S111.
[0070] [Step S106] The classification unit 120 determines whether the item value set in the selected column has only two patterns. If there are only two patterns, the classification unit 120 proceeds to step S107. If there are three or more patterns of item values set, the classification unit 120 proceeds to step S108.
[0071] [Step S107] The classification unit 120 classifies the selected column into a nominal scale binary (text data). After that, the classification unit 120 proceeds to step S111. [Step S108] The classification unit 120 determines whether the number of item value types set in the selected column is less than 10% of the number of records included in the target learning datasets 41, 42, ... or the target dataset 111. If the number of item value types is less than 10%, the classification unit 120 proceeds to step S109. If the number of item value types is 10% or more, the classification unit 120 proceeds to step S110.
[0072] [Step S109] The classification unit 120 classifies the selected column into nominal scale polytomous categories. After that, the classification unit 120 proceeds to step S111. [Step S110] The classification unit 120 classifies the selected column into free text.
[0073] [Step S111] The classification unit 120 determines whether or not there is an unselected column among the columns of the training datasets 41, 42, ... to be processed or the application dataset 111. If there is an unselected column, the classification unit 120 proceeds to step S101. If all columns have been selected, the classification unit 120 ends the classification process.
[0074] By appropriately classifying columns into a plurality of categories, it is possible to correctly reflect in the statistics the characteristics of each of the learning datasets 41, 42, ... and the application dataset 111. Hereinafter, a method for calculating the statistics for each category will be described with reference to Figs.
[0075] Fig. 9 is a diagram showing a first example of column statistics calculation. Fig. 9 shows an example of calculation of statistics for a column that is numeric data and has only two patterns of item values set. For example, only one of two numeric values, "10" or "20", is set in column "V1" of each record of learning dataset 41. In this case, classification unit 120 classifies column "V1" into the category "nominal scale binomial (numeric data)".
[0076] The statistics calculation unit 130 converts the item values of the column "V1" classified into the category "nominal scale binomial (numeric data)" into two values, "0" and "1." In the example of FIG. 9, the statistics calculation unit 130 converts "10" to "0" and "20" to "1." Then, using the converted values, the statistics calculation unit 130 calculates the average, standard deviation, unbiased standard deviation, median, 1Q value (first quartile), and 3Q value (third quartile) as statistics of the column "V1."
[0077] Fig. 10 is a diagram showing a second example of column statistics calculation. Fig. 10 shows an example of calculation of statistics of a column that is numerical data and has three or more patterns of item values set. For example, three or more types of integer item values are set in column "V2" of each record of learning dataset 41. In this case, classification unit 120 classifies column "V2" into the category "numerical data".
[0078] The statistics calculation unit 130 uses the raw item values of column “V2” classified into the category “numerical data” to calculate the mean, standard deviation, unbiased standard deviation, median, 1Q value, and 3Q value as statistics for column “V2”.
[0079] Fig. 11 is a diagram showing a third example of column statistics calculation. Fig. 11 shows an example of calculation of statistics of a column that is not numerical data and has only two patterns of item values set. For example, in column "V3" of each record of learning dataset 41, only one of two types, "male" and "female", is set. In this case, classification unit 120 classifies column "V3" into the category "nominal scale binomial (text data)".
[0080] The statistics calculation unit 130 converts the item values of column "V3" classified into the category "nominal scale binomial (text data)" into two values, "0" and "1." In the example of FIG. 11, the statistics calculation unit 130 converts "male" to "0" and "female" to "1." Then, using the converted values, the statistics calculation unit 130 calculates the average, standard deviation, unbiased standard deviation, median, 1Q value, 3Q value, and the number of occurrences of each item value as statistics for column "V3." The statistics calculation unit 130 also generates a list of item values "male, female" set in column "V3."
[0081] FIG. 12 is a diagram showing a fourth example of column statistics calculation. FIG. 12 shows an example of calculation of column statistics in which the data is other than numerical data and the number of types of item values set is less than 10% of the number of records. For example, one of four blood types ("A type", "B type", "O type", "AB type") is set in column "V4" of each record of learning dataset 41. It is assumed that the number of records in learning dataset 41 is 40 or more. In this case, classification unit 120 classifies column "V4" into the category "nominal scale polynomial".
[0082] The statistics calculation unit 130 divides the column "V4" classified into the category "nominal scale polynomial" into columns for each of the set item values. In the example of FIG. 12, the column is divided into columns "V4_A type", "V4_B type", "V4_O type", and "V4_AB type" corresponding to the item values of "A type", "B type", "O type", and "AB type". In each column after division of each record, if an item value corresponding to the divided column is set in the original column "V4", the item value is set to "1", and if not, the item value is set to "0". Then, the statistics calculation unit 130 calculates the average value, standard deviation, unbiased standard deviation, median, 1Q value, 3Q value, and the number of occurrences of each item value as the statistics of each divided column. The statistics calculation unit 130 also assigns the item value corresponding to the column to the set of statistics of each divided column.
[0083] Fig. 13 is a diagram showing a fifth example of column statistics calculation. Fig. 13 shows an example of calculation of column statistics for a column that is other than numerical data and in which the number of types of item values set is 10% or more of the number of records. For example, an arbitrary character string is set in column "V5" of each record of learning dataset 41. In this case, classification unit 120 classifies column "V5" into the category "free text."
[0084] The statistics calculation unit 130 performs a morphological analysis of the character strings set in the column "V5" classified into the category "free text" and extracts words of a predetermined part of speech, such as nouns. The statistics calculation unit 130 divides the column "V5" into columns for each extracted word. In the example of FIG. 13, the column is divided into columns such as "V5_type 1 diabetes", "V5_atopy", and "V5_type 2 diabetes". In each column after division of each record, if an item value including a character corresponding to the divided column is set in the original column "V5", the item value is set to "1", otherwise, the item value is set to "0". Then, the statistics calculation unit 130 calculates the average value, standard deviation, unbiased standard deviation, median, 1Q value, 3Q value, and the number of occurrences of each item value as the statistics of each divided column. The statistics calculation unit 130 also assigns a word corresponding to the column to the set of statistics of each divided column.
[0085] Next, the procedure of the statistical amount calculation process will be described in detail with reference to FIGS. 14 is a flowchart (part 1) showing an example of a procedure for a statistical amount calculation process. The process shown in FIG. 14 will be described below in order of step numbers.
[0086] [Step S201] The statistics calculation unit 130 selects one column classified as nominal scale binomial (numeric data). [Step S202] The statistics calculation unit 130 binarizes the two types of item values set in the selected column into "0" and "1."
[0087] [Step S203] The statistics calculation unit 130 calculates statistics for the selected column using the binarized values, and generates statistical information for the column including the calculated statistics. [Step S204] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns classified as nominal scale binomial (numeric data). If there is an unselected column, the statistics calculation unit 130 proceeds to step S201. If there is no unselected column, the statistics calculation unit 130 proceeds to step S205.
[0088] [Step S205] The statistics calculation unit 130 selects one column classified as numeric data. [Step S206] The statistics calculation unit 130 calculates statistics for item values set in the selected column, and generates statistical information for the column including the calculated statistics.
[0089] [Step S207] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns classified as numerical data. If there is an unselected column, the statistics calculation unit 130 proceeds to step S205. If there is no unselected column, the statistics calculation unit 130 proceeds to step S211 (see FIG. 15).
[0090] 15 is a flowchart (part 2) showing an example of a procedure for a statistical amount calculation process. The process shown in FIG. 15 will be described below in order of step numbers. [Step S211] The statistics calculation unit 130 selects one column classified as nominal scale binomial (text data).
[0091] [Step S212] The statistics calculation unit 130 binarizes the two types of item values set in the selected column into "0" and "1." [Step S213] The statistics calculation unit 130 calculates statistics for the selected column using the binarized values, and generates statistical information for the column including the calculated statistics.
[0092] [Step S214] The statistics calculation unit 130 adds a character string of the item value corresponding to the selected column to the statistical information of that column. [Step S215] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns classified as nominal scale binomial (text data). If there is an unselected column, the statistics calculation unit 130 proceeds to step S211. If there is no unselected column, the statistics calculation unit 130 proceeds to step S221 (see FIG. 16).
[0093] 16 is a flowchart (part 3) showing an example of a procedure for a statistical amount calculation process. The process shown in FIG. 16 will be described below in order of step numbers. [Step S221] The statistics calculation unit 130 selects one column classified as a nominal scale multinomial.
[0094] [Step S222] The statistics calculation unit 130 divides the selected column into columns for each type of item value. After division, the column for each record is set to a value of "1" or "0" indicating whether or not the item value corresponding to the column exists.
[0095] [Step S223] The statistics calculation unit 130 selects one column corresponding to the type of item value. [Step S224] The statistics calculation unit 130 calculates statistics for item values set in the selected column, and generates statistical information for the column including the calculated statistics.
[0096] [Step S225] The statistics calculation unit 130 adds a character string of an item value corresponding to the selected column to the statistical information of that column. [Step S226] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns for each item value type generated by the division. If there is an unselected column, the statistics calculation unit 130 proceeds to step S223. If there is no unselected column, the statistics calculation unit 130 proceeds to step S227.
[0097] [Step S227] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns classified as nominal scale polynomials. If there is an unselected column, the statistics calculation unit 130 proceeds to step S221. If there is no unselected column, the statistics calculation unit 130 proceeds to step S231 (see FIG. 17).
[0098] 17 is a flowchart (part 4) showing an example of a procedure of a statistical amount calculation process. The process shown in FIG. 17 will be described below in order of step numbers. [Step S231] The statistics calculation unit 130 selects one column classified as free text.
[0099] [Step S232] The statistics calculation unit 130 extracts words by morphological analysis of the character string set in the selected column. [Step S233] The statistics calculation unit 130 divides the selected column into columns for each extracted word. After division, the column of each record is set to a value of "1" or "0" indicating the presence or absence of the word corresponding to the column.
[0100] [Step S234] The statistics calculation unit 130 selects one column corresponding to the extracted word. [Step S235] The statistics calculation unit 130 calculates statistics for the item values set in the selected column, and generates statistical information for the column including the calculated statistics.
[0101] [Step S236] The statistics calculation unit 130 adds the character string of the word corresponding to the selected column to the statistical information including the statistics calculated from the item values of the column. [Step S237] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns for each extracted word. If there is an unselected column, the statistics calculation unit 130 proceeds to step S234. If there is no unselected column, the statistics calculation unit 130 proceeds to step S238.
[0102] [Step S238] The statistics calculation unit 130 determines whether or not there is an unselected column among the columns classified as free text. If there is an unselected column, the statistics calculation unit 130 proceeds to step S231. If there is no unselected column, the statistics calculation unit 130 ends the statistics calculation process.
[0103] 14 to 17 are performed for each of the target dataset 111 and the multiple learning datasets 41, 42, .... Statistics obtained from the target dataset 111 and each of the multiple learning datasets 41, 42, ... are transmitted to the cloud server 200 by the conformance determination unit 140. The similarity calculation unit 230 of the cloud server 200 generates feature vectors for each of the target dataset 111 and the multiple learning datasets 41, 42, ... based on the obtained statistics.
[0104] 18 is a diagram showing an example of the feature vector generation process. A statistics set 70 generated based on the learning dataset 41 includes statistics for each column. For example, column "V1" of the learning dataset 41 is classified into a nominal scale binary. In this case, multiple statistics (a1, a2,...) obtained from binarized values of the item values of column "V1" are included in the statistics set 70. Each of the multiple statistics (a1, a2,...) of column "V1" is set as an element of a feature vector indicating the characteristics of the learning dataset 41.
[0105] Another column "V4" in the training dataset 41 is classified as nominal scale polynomial. In this case, the column "V4" is divided into multiple columns for each item value, and multiple statistics (b1, b2, , c1, c2, , ) are obtained for each divided column. Each of the multiple statistics (b1, b2, , c1, c2, , ) of the column "V4" is set as an element of a feature vector indicating the characteristics of the training dataset 41. For example, the statistical information (b1, b2, ) of the divided column corresponding to the item value "Type A" is set as an element corresponding to the item value "Type A" in the feature vector.
[0106] In this manner, feature vectors of the target dataset 111 and each of the multiple training datasets 41, 42, ... are generated. The similarity calculation unit 230 calculates the similarity between the feature vector of the target dataset 111 and each of the multiple training datasets 41, 42, .... This similarity is also the similarity between each of the multiple training datasets 41, 42, ... and the target dataset 111.
[0107] 19 is a flowchart illustrating an example of a procedure for similarity calculation processing. The processing shown in FIG. 19 will be described below in order of step numbers. [Step S301] The similarity calculation unit 230 selects a set of statistics of the target data set 111.
[0108] [Step S302] The similarity calculation unit 230 generates a feature vector based on the selected set of statistics. [Step S303] The similarity calculation unit 230 selects a set of statistics from the training data set.
[0109] [Step S304] The similarity calculation unit 230 generates a feature vector based on the selected set of statistics. [Step S305] The similarity calculation unit 230 calculates the cosine similarity between the feature vector generated in step S302 and the feature vector generated in step S304.
[0110] [Step S306] The similarity calculation unit 230 determines whether or not there is an unselected statistic set. If there is an unselected statistic set, the similarity calculation unit 230 proceeds to step S303. If all statistic sets have been selected, the similarity calculation unit 230 proceeds to step S307.
[0111] [Step S307] The similarity calculation unit 230 transmits the similarity of each of the multiple learning data sets 41, 42, . . . to the DB server 100. In the DB server 100 that has received the similarity, the suitability determining unit 140 determines whether or not the application target data set 111 is suitable for each of the multiple models 51, 52, . . . based on the similarity.
[0112] 20 is a flowchart showing an example of a procedure for a conformity determination process. The process shown in FIG. 20 will be described below in order of step numbers. [Step S401] The conformance determining unit 140 selects one learning data set.
[0113] [Step S402] The conformance determination unit 140 determines whether the similarity between the selected learning dataset and the target dataset 111 is equal to or greater than a threshold (e.g., 0.8). If the similarity is equal to or greater than the threshold, the conformance determination unit 140 proceeds to step S403. If the similarity is less than the threshold, the conformance determination unit 140 proceeds to step S404.
[0114] [Step S403] The compatibility determination unit 140 determines that the model corresponding to the learning dataset selected in step S402 is a compatible model that is compatible with the target dataset 111, and registers the identification information of the model in a list of compatible models.
[0115] [Step S404] The conformance determination unit 140 determines whether or not there is an unselected training data set. If there is an unselected training data set, the conformance determination unit 140 proceeds to step S401. If all training data sets have been selected, the conformance determination unit 140 ends the conformance determination process.
[0116] In this way, the DB server 100 can obtain a list of compatible models that are compatible with the target dataset 111. The model acquisition unit 150 of the DB server 100 acquires the corresponding models 112 from the cloud server 200 based on the list of compatible models, and stores them in the storage unit 110. Then, the inference unit 160 provides an inference processing service in which item values in the target dataset 111 are input to the models 112, using a technology such as WebAPI (Application Programming Interface).
[0117] The learning dataset used to generate the acquired model 112 and the target dataset 111 have similar contents, and if an inference process is performed using the model 112 with the target dataset 111 as input data, a highly accurate predicted value can be obtained. The inference unit 160 can also calculate a predicted value using multiple models. Even when calculating a predicted value using multiple models, a highly accurate predicted value can be obtained from each model (for example, the prediction accuracy of model A is 95%, and the prediction accuracy of model B is 87%).
[0118] The information transmitted from the DB server 100 to the cloud server 200 is a set of statistics such as the target data set 111. Therefore, even if the target data set 111 contains personal information, the organization (e.g., a hospital) that operates the DB server 100 can determine a suitable model while keeping the personal information contained in the target data set 111 confidential.
[0119] A user of the DB server 100 can use the model 112 acquired from the cloud server 200, for example, to verify an inference process using AI. If it is confirmed by the verification that a highly accurate prediction value can be obtained, the user can tune the acquired model 112, for example, to improve the prediction accuracy. This is expected to significantly reduce the amount of work required to generate a model with high prediction accuracy.
[0120] [Other embodiments] In the second embodiment, the DB server 100 and the cloud server 200 cooperate to perform the match determination, but the functions of the DB server 100 and the cloud server 200 may be integrated into one device.
[0121] Although the embodiments have been described above, the configurations of the parts shown in the embodiments can be replaced with other parts having similar functions. Any other components or steps may be added. Furthermore, any two or more configurations (features) of the above-mentioned embodiments may be combined. [Explanation of symbols]
[0122] 1 Model 2. First Data Set 3. Second Data Set 4 First feature vector 5 Second feature vector 10. Information processing device 11 Storage section 12 Processing section
Claims
1. On the computer, classifying items of a plurality of first records included in a first dataset used to generate a trained model by machine learning into one of a plurality of categories based on first item values set in the items of the plurality of first records; calculating a first statistic based on an item value set in the item of the plurality of first records in a procedure corresponding to a category to which the item belongs; calculating a second statistic based on second item values set in the item of a plurality of second records included in a second data set having the same item as the first data set, in a procedure corresponding to a category to which the item belongs; determining whether the second data set fits the model based on a similarity between the first statistic and the second statistic; A judgment program that executes processing.
2. In the calculation of the first statistics, a plurality of first statistics having different types are calculated based on the first item values set in the items of the plurality of first records; In the calculation of the second statistics, a plurality of second statistics having different types are calculated based on the second item values set in the items of the plurality of second records; the similarity between the first statistics and the second statistics is a similarity between a first feature vector having each of the plurality of first statistics as an element and a second feature vector having each of the plurality of second statistics as an element. The determination program according to claim 1.
3. In the classification into the plurality of categories, the first item value set in the item is classified into different categories depending on whether the first item value is a numerical value or not; When the first item value set for the item is not a numeric value, the calculation of the first statistics calculates the number of occurrences of each type of the first item value as the first statistics, and the calculation of the second statistics calculates the number of occurrences of each type of the second item value as the second statistics. The determination program according to claim 1 or 2.
4. In the classification into the plurality of categories, the classification into different categories is performed depending on whether the number of types of the first item value is less than two or not; When the number of types of the first item value set for the item is less than two, in the calculation of the first statistics, the first item value is converted to 0 or 1 according to the type, and the first statistics is calculated based on the converted value, and in the calculation of the second statistics, the second item value is converted to 0 or 1 according to the type, and the second statistics is calculated based on the converted value.
4. The determination program according to claim 1.
5. In the classification into the plurality of categories, the data is classified into different categories depending on whether a ratio of the number of types of the first item value to the number of first records is equal to or greater than a predetermined value; If the value of the ratio is less than the predetermined value, In the calculation of the first statistics, the items of the first data set are divided into first divided items for each type of the first item value, a value indicating the presence or absence of the first item value of a corresponding type is set in the first divided item of each of the plurality of first records, and the first statistics are calculated for each of the first divided items; In the calculation of the second statistics, the items of the second data set are divided into the first divided items, a value indicating the presence or absence of the second item value of a corresponding type is set in the first divided item of each of the plurality of second records, and the second statistics is calculated for each of the first divided items.
5. The determination program according to claim 1.
6. If the value of the ratio is equal to or greater than the predetermined value, In the calculation of the first statistics, the items of the first data set are divided into second division items for each word included in the first item value, a value indicating the presence or absence of a corresponding word is set in the second division item of each of the plurality of first records, and the first statistics are calculated for each of the second division items; In the calculation of the second statistics, the items of the second data set are divided into the second divided items, a value indicating the presence or absence of a corresponding word is set in the second divided item of each of the second records, and the second statistics is calculated for each of the second divided items. The determination program according to claim 5.
7. The computer classifying items of a plurality of first records included in a first dataset used to generate a trained model by machine learning into one of a plurality of categories based on first item values set in the items of the plurality of first records; calculating a first statistic based on an item value set in the item of the plurality of first records in a procedure corresponding to a category to which the item belongs; calculating a second statistic based on second item values set in the item of a plurality of second records included in a second data set having the same item as the first data set, in a procedure corresponding to a category to which the item belongs; determining whether the second data set fits the model based on a similarity between the first statistic and the second statistic; A method for determining whether to execute a process.
8. a processing unit that classifies items of a plurality of first records included in a first dataset used to generate a trained model by machine learning into one of a plurality of categories based on first item values set in the items of the plurality of first records, calculates a first statistical amount based on the item values set in the items of the plurality of first records in a procedure corresponding to the category to which the items belong, calculates a second statistical amount based on second item values set in the items of a plurality of second records included in a second dataset having the same items as the first dataset in a procedure corresponding to the category to which the items belong, and determines whether or not the second dataset is compatible with the model based on a similarity between the first statistical amount and the second statistical amount; An information processing device having the above configuration.
Citation Information
Patent Citations
Pattern recognition device, method, and program
JP2020502658A
Learning method, evaluation device, and evaluation system
JP6751955B1
Learned model provision method and learned model provision device
WO2018142766A1
Label-collecting device, label collection method, and label-collecting program
WO2019167556A1
