Information processing method, information processing device and program

By clustering and evaluating data subsets for learning models, the method addresses biased data division issues, enhancing model accuracy and preventing overfitting.

JP2024160187AInactive Publication Date: 2024-11-13TERUMO KK +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021152480
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-09-17
Publication Date
2024-11-13
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing image processing methods using learning models face issues with biased data division during additional and first learning, leading to potential overfitting and inaccurate model accuracy due to improper data distribution.

Method used

The method involves clustering training data into multiple subsets, dividing each subset into training, validation, and test data, and using learnability and independence evaluation indices to ensure appropriate model learning by adjusting parameters based on these evaluations.

Benefits of technology

This approach ensures unbiased data distribution, preventing overfitting and improving the accuracy of learning models by ensuring appropriate learning through even data distribution and evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160187000001_ABST
    Figure 2024160187000001_ABST
Patent Text Reader

Abstract

To provide an information processing method and the like capable of appropriately performing additional learning.SOLUTION: An information processing method causes a computer to execute processing for: acquiring a learning data set 69 composed of a plurality of sets of learning data 59 each having input data 511 and correct answer data 512 associated with each other for use in training a learning model 41; clustering at least a portion of the learning data set 69 into a plurality of clusters 55; classifying the learning data 59 included in each of the clusters 55 into a set of training data 56, a set of validation data 57, and a set of test data 58; and training the learning model 41 by using, each as a batch, the set of training data 56, the set of validation data 57 and the set of test data 58 in the plurality of clusters 55.SELECTED DRAWING: Figure 11
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] An image processing device that supports image diagnosis using a learning model generated by supervised machine learning has been proposed (Patent Document 1). An information processing method that performs additional learning on a generated learning model using additional learning data has been proposed (Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-175343 A [Patent Document 2] JP 2019-133626 A Summary of the Invention [Problem to be solved by the invention]

[0004] When performing additional learning, the additional learning data is divided into training data, validation data, and test data, and the parameters of the learning model are adjusted using the training data and validation data.The adjusted learning model is then tested using the test data.However, if there is a bias in the way the additional learning data is divided, the additional learning cannot be performed appropriately.

[0005] Similarly, when performing the initial learning, the learning data is divided into training data, validation data, and test data, and the parameters of the learning model are adjusted using the training data and validation data.The adjusted learning model is then tested using the test data.However, if there is a bias in the way the learning data is divided, there is a risk that the accuracy of the learning model will not reach the target due to overfitting, etc.

[0006] In one aspect, an object of the present invention is to provide an information processing method and the like that enables appropriate learning. [Means for solving the problem]

[0007] The information processing method includes a process in which a computer acquires a learning dataset consisting of multiple sets of learning data in which input data and correct answer data are associated, the learning dataset being used for training a learning model, clusters at least a portion of the learning data into multiple clusters, separates the learning data contained in each of the clusters into training data, validation data, and test data, and compiles the training data, validation data, and test data of the multiple clusters and uses them to train the learning model. Effect of the Invention

[0008] In one aspect, it is possible to provide an information processing method etc. that enables appropriate learning. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of an additional learning method. [Diagram 2] FIG. 1 is an explanatory diagram for explaining an overview of a study model generation method. [Diagram 3] FIG. 2 is an explanatory diagram illustrating a configuration of an information processing device. [Figure 4] FIG. 2 is an explanatory diagram illustrating a configuration of an additional learning dataset. [Diagram 5] FIG. 2 is an explanatory diagram illustrating a configuration of an additional learning dataset. [Figure 6] FIG. 11 is an explanatory diagram for explaining a method for calculating an independence evaluation index. [Figure 7] 11 is a flowchart illustrating a process flow of a program. [Figure 8] 13 is a flowchart illustrating the process flow of a subroutine for extracting question data. [Figure 9]13 is a flowchart illustrating the flow of processing of a subroutine for calculating an evaluation index. [Figure 10] 13 is a flowchart illustrating the flow of processing of a subroutine for calculating an index. [Figure 11] FIG. 1 is an explanatory diagram outlining a method for generating a learning model. [Figure 12] 10 is a flowchart illustrating the flow of processing of a program according to the second embodiment. [Figure 13] FIG. 11 is an explanatory diagram illustrating a configuration of an information processing device according to a third embodiment. [Figure 14] FIG. 13 is a functional block diagram of an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] 1 is an explanatory diagram for explaining an outline of the additional learning method. An additional learning dataset 64 is prepared, which is made up of a large number of sets of additional learning data 51 in which input data 511 and correct answer data 512 are associated with each other. When input data 511 included in one set of additional learning data 51 is input to a learning model 41, output data 53 is output from the learning model 41. Note that the additional learning data 51 is an example of learning data, and the additional learning dataset 64 is an example of a learning dataset.

[0011] The learning model 41 is, for example, a classification model that outputs a label related to a subject appearing in image data when image data is input. The learning model 41 may be a segmentation model that outputs a label corresponding to the subject of each pixel constituting the image data when image data is input. The learning model 41 may be an object detection model that outputs the position and label of a subject appearing in the image data when image data is input.

[0012] The input of the learning model 41 is not limited to image data. For example, the learning model 41 may be a model that receives text input and outputs voice, an image, or a label. The learning model 41 may be a model that receives voice and outputs text, an image, or a label.

[0013] An evaluation value indicating the processing accuracy of the additional learning data 51 is calculated based on a comparison between the supervised answer data 512 associated with the input data 511 and the output data 53. In the following explanation, an example will be given in which the evaluation value is defined so that the evaluation value is high when the output data 53 is close to the supervised answer data 512. Details of the evaluation value will be described later.

[0014] Problem data in which the accuracy of output data 53 is lower than a predetermined threshold is extracted from multiple sets of additional learning data 51. The extracted problem data is divided into multiple clusters 55 by a known clustering method such as the k-means method or Ward's method. The extracted problem data may be automatically divided into multiple clusters 55 based on a predetermined rule.

[0015] 1 shows three clusters 55, namely, a first cluster 551, a second cluster 552, and a third cluster 553. The points inside each cluster 55 indicate the additional learning data 51 that have been determined to be problem data.

[0016] Fig. 2 is an explanatory diagram for explaining an outline of the method for generating the study model 43. Fig. 2 illustrates a schematic diagram of processing for an ith cluster 55i, which is the ith cluster 55. Additional learning data 51 included in the ith cluster 55i is divided into an ith training data 56i, an ith validation data 57i, and an ith test data 58i.

[0017] Using the i-th training data 56i and the i-th validation data 57i, the i-th study model 43i is newly generated by machine learning. The above process is performed for all clusters 55, and the same number of study models 43 as the number of clusters 55 are generated.

[0018] Continuing the explanation by returning to Fig. 1, Fig. 1 shows a schematic example in which a first study model 431 is generated using a first cluster 551, a second study model 432 is generated using a second cluster 552, and a third study model 433 is generated using a third cluster 553.

[0019] For each cluster 55, a learnability evaluation index indicating whether the study model 43 has been properly learned and an independence evaluation index indicating the independence between the clusters 55 are evaluated. Details of the learnability evaluation index and the independence evaluation index will be described later.

[0020] If either or both of the learnability evaluation index and the independence evaluation index do not show good values, the process returns to the clustering step and is repeated. For example, by changing the number of clusters 55 to be generated, the calculation conditions such as the initial values ​​used when creating the clusters 55, the method used for clustering, or the similarity scale between data used for clustering, different clusters 55 can be formed even for the same problem data group.

[0021] If all of the learnability evaluation indices and independence evaluation indices show good values, the additional learning data 51 included in the additional learning dataset 64 is distributed to a training dataset 61, a validation dataset 62 and a test dataset 63.

[0022] The training data set 61 includes training data 56 of each cluster 55 and a portion of the additional learning data 51 determined not to be problem data. The validation data set 62 includes validation data 57 of each cluster 55 and a portion of the additional learning data 51 determined not to be problem data. The test data set 63 includes test data 58 of each cluster 55 and a portion of the additional learning data 51 determined not to be problem data.

[0023] The training data set 61 and the validation data set 62 are used to retune the parameters of the learning model 41. The retuned learning model 41 is then tested using a test data set 63.

[0024] The processing described above prevents bias when dividing the multiple sets of additional learning data 51 contained in the additional learning dataset 64 into training data, validation data, and test data, thereby enabling additional learning of the learning model 41 to be performed appropriately.

[0025] 3 is an explanatory diagram for explaining the configuration of the information processing device 20. The information processing device 20 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, and a bus. The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, or the like. The control unit 21 is connected to each hardware unit constituting the information processing device 20 via the bus.

[0026] The main memory device 22 is a storage device such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs being executed by the control unit 21.

[0027] The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores the learning model 41, an additional learning data DB 45 in which the above-mentioned additional learning data set 64 is recorded, a program to be executed by the control unit 21, and various data required for executing the program. The learning model 41 and the additional learning data DB 45 may be stored in an external large-capacity storage device connected to the information processing device 20. The communication unit 24 is an interface that performs communication between the information processing device 20 and a network.

[0028] The display unit 25 is, for example, a liquid crystal display panel or an organic EL (electro-luminescence) panel, etc. The input unit 26 is, for example, a keyboard or a mouse, etc. The display unit 25 and the input unit 26 may be stacked to form a touch panel.

[0029] The information processing device 20 is a general-purpose personal computer, a tablet, a mainframe computer, a virtual machine running on a mainframe computer, or a quantum computer. The information processing device 20 may be configured with hardware such as a plurality of personal computers or a mainframe computer that perform distributed processing. The information processing device 20 may be configured with a cloud computing system or a quantum computer.

[0030] 4 and 5 are explanatory diagrams for explaining the configuration of the additional learning dataset 64. As described above, the additional learning dataset 64 is a dataset consisting of a plurality of sets of additional learning data 51. In Fig. 4 and Fig. 5, the configuration of the additional learning dataset 64 is explained in the form of a table in which one set of additional learning data 51 is written in one row. Input data 511 is shown in the left column, and correct answer data 512 is shown in the right column.

[0031] 4 shows an example of an additional learning dataset 64 used for additional learning of a learning model 41 that accepts input of image data and outputs a label related to a subject appearing in the image data. For example, additional learning data 51 shown in the top row of Fig. 4 includes input data 511 which is image data used for input, and correct answer data 512 indicating that the label corresponding to the subject of the input data 511 is "dog."

[0032] The control unit 21 inputs the input data 511 to the learning model 41 or the consideration model 43, and obtains the output data 53. When the output data 53 is "dog", the correct answer data 512 and the output data 53 match, so the control unit 21 calculates the evaluation value to be "1". On the other hand, when the output data 53 is "cat", the correct answer data 512 and the output data 53 do not match, so the control unit 21 calculates the evaluation value to be "0".

[0033] The evaluation value is not limited to the binary values ​​of "0" and "1". For example, the evaluation value may be set to three or more values, such as "0.5" when a label indicating a four-legged animal other than "dog", such as "cat", "horse" or "cow" is output for the correct answer data 512 of "dog", and "0" when any other label is output. In the process of extracting question data, the control unit 21 extracts additional learning data 51 having an evaluation value less than a threshold value.

[0034] FIG. 5 shows an example of an additional learning dataset 64 used for additional learning for a learning model 41, which is a so-called segmentation model that accepts input of image data and outputs labels corresponding to the subject of each pixel that constitutes the image data.

[0035] For example, additional learning data 51 shown in the top row of FIG. 5 includes input data 511 which is image data used for input, and supervised answer data 512 which indicates labels corresponding to each pixel constituting the input data 511.

[0036] The control unit 21 inputs the input data 511 to the learning model 41 or the consideration model 43, and obtains the output data 53. The control unit 21 compares the correct answer data 512 with the output data 53 for each pixel to calculate an evaluation value. As the evaluation value, any evaluation value that can be used for evaluating the accuracy of a segmentation model, such as Pixel Accuracy, mIoU (Mean Intersection over Union), or F-value, can be used.

[0037] The control unit 21 extracts problem data having an evaluation value lower than a threshold value from the additional learning data 51. The threshold value is, for example, a predetermined constant. The threshold value may be determined based on the calculated evaluation value. For example, the control unit 21 may determine the threshold value so as to extract additional learning data 51 having an evaluation value in the bottom 10 percent.

[0038] In addition, the evaluation value used when extracting problem data from multiple sets of additional learning data 51 and the evaluation value used when calculating the learnability evaluation index and the independence evaluation index may be the same evaluation value or different evaluation values.

[0039] The method for calculating the learnability evaluation index will now be described. The learnability evaluation index is defined by equation (1).

[0040]

number

[0041] The learnability evaluation index L(F i2, the control unit 21 divides the additional learning data 51 included in the ith cluster 55i into the ith training data 56i, the ith validation data 57i, and the ith test data 58i. The control unit 21 generates the ith consideration model 43i using the ith training data 56i and the ith validation data 57i.

[0042] The control unit 21 inputs each of the input data 511 included in the i-th test data 58i to the i-th study model 43i to obtain the output data 53. The control unit 21 calculates an evaluation value for each of the additional learning data 51 based on a comparison between the output data 53 and the correct answer data 512 associated with the input data 511.

[0043] The control unit 21 calculates a representative value of the evaluation values ​​evaluated for each of the additional learning data 51. The representative value is, for example, the arithmetic mean, geometric mean, harmonic mean, median, or mode of the accuracy evaluated for each of the additional learning data 51. The representative value calculated by the control unit 21 is used as the learning possibility evaluation index L(F i The control unit 21 calculates a learning possibility evaluation index for each cluster 55.

[0044] The learnability evaluation index is an index related to the accuracy of the generated study model 43. If the learnability evaluation index is equal to or less than the threshold, the generated study model 43 is not an appropriate model. If the learnability evaluation index exceeds the threshold, the study model 43 with good accuracy has been generated. An appropriate threshold value is, for example, 0.6.

[0045] The calculation method of the independence assessment index is explained below. The independence assessment index is defined by formula (2). Note that explanations of symbols common to formula (1) are omitted.

[0046]

number

[0047] Based on the definition of the learnability evaluation index shown in equation (1), equation (2) can be transformed into equation (3).

[0048]

number

[0049] Independence evaluation index I(F i First, the learning possibility evaluation index L(F i ) is calculated. The control unit 21 inputs each of the input data 511 included in the j-th test data 58j to the i-th study model 43i to obtain the output data 53. The control unit 21 calculates an evaluation value for each of the additional learning data 51 based on a comparison between the supervised answer data 512 associated with the input data 511 and the output data 53.

[0050] The control unit 21 calculates a representative value of the evaluation values ​​evaluated for each additional learning data 51. The representative value calculated by the control unit 21 is the evaluation index m(i,j) when the ith study model 43i and the jth test data 58j are combined. Note that it is preferable that the representative value used when calculating the learnability evaluation index is the same as the representative value used when calculating the independence evaluation index.

[0051] 6 is an explanatory diagram for explaining a method for calculating the independence evaluation index. The vertical direction indicates i in the upper equations of equations (2) and (3), and the horizontal direction indicates j in the upper equations. Each square except the rightmost square indicates the evaluation index m(i,j). The squares on the diagonal line surrounded by a thick frame are learnability evaluation indexes evaluated by combining the study model 43 and the test data 58 of the cluster 55 used to generate the study model 43.

[0052] Returning to equation (3), the explanation will continue. For each j that is not equal to i, the control unit 21 calculates the learning possibility evaluation index L(F i) and the evaluation index m(i,j). The control unit 21 calculates the arithmetic mean of the calculated differences. The arithmetic mean calculated by the control unit 21 is the independence evaluation index I(F i ).

[0053] The independence evaluation index is an index regarding the independence between clusters 55. When the independence evaluation index is equal to or less than the threshold, clustering is not performed properly. When the independence evaluation index exceeds the threshold, clustering is performed properly. For example, 0.6 is appropriate as the threshold. Note that the threshold for the learnability evaluation index and the threshold for the independence evaluation index may be different values.

[0054] Fig. 7 is a flowchart explaining the flow of processing of the program. Prior to execution of the program of Fig. 7, an unlearned model having a structure corresponding to the learning model 41 is prepared. When an additional learning data set 64 including a sufficient number of additional learning data 51 is recorded in the additional learning data DB 45, the control unit 21 executes the program explained using Fig. 7.

[0055] The control unit 21 starts a question data extraction subroutine (step S501). The question data extraction subroutine is a subroutine that extracts question data from the additional learning data 51 included in the additional learning dataset 64. The process flow of the question data extraction subroutine will be described later.

[0056] The control unit 21 clusters the extracted question data (step S502). The control unit 21 selects one cluster 55 (step S503). The control unit 21 distributes the additional learning data 51 included in the selected cluster 55 into training data 56, validation data 57, and test data 58 (step S504).

[0057] The control unit 21 performs machine learning using the training data 56 and the validation data 57 to generate the study model 43 (step S505). The process of generating a model using supervised machine learning has been performed in the past, so detailed explanations will be omitted. The control unit 21 records the generated study model 43 in the main storage device 22 or the auxiliary storage device 23 in association with the cluster 55 selected in step S503.

[0058] The control unit 21 judges whether the processing of all clusters 55 has been completed (step S506). If it is judged that the processing has not been completed (NO in step S506), the control unit 21 returns to step S503. If it is judged that the processing has been completed (YES in step S506), the control unit 21 starts a subroutine for calculating an evaluation index (step S507). The subroutine for calculating an evaluation index is a subroutine for calculating a learnability evaluation index and an independence evaluation index of the study model 43 generated for each cluster 55. The processing flow of the subroutine for calculating an evaluation index will be described later.

[0059] The control unit 21 determines whether the learnability evaluation index and the independence evaluation index of the study model 43 satisfy the condition (step S508). Specifically, when the learnability evaluation index and the independence evaluation index calculated for each study model 43 all exceed a predetermined threshold, the control unit 21 determines that the condition is satisfied.

[0060] If it is determined that the condition is satisfied (YES in step S508), the control unit 21 distributes the additional learning data 51 included in the additional learning dataset 64 to the training dataset 61, the validation dataset 62, and the test dataset 63 (step S509).

[0061] Here, the training data set 61 includes the training data 56 distributed in step S504 for each cluster 55, and a portion of the additional learning data 51 determined not to be problem data. The validation data set 62 includes the validation data 57 distributed in step S504 for each cluster 55, and a portion of the additional learning data 51 determined not to be problem data. The test data set 63 includes the test data 58 distributed in step S504 for each cluster 55, and a portion of the additional learning data 51 determined not to be problem data.

[0062] The control unit 21 performs additional learning of the learning model 41 using the training data set 61 and the validation data set 62, and performs testing after the additional learning using the test data set 63 (step S510). The additional learning process of the model generated by supervised machine learning has been performed conventionally, so the details will not be described. The control unit 21 determines whether the accuracy of the learning model 41 has improved by the additional learning (step S511).

[0063] When it is determined that there has been an improvement (YES in step S511), the control unit 21 updates the additional learning data DB 45 and the learning model 41 (step S512). Specifically, the control unit 21 deletes the additional learning data 51 used for the additional learning from the additional learning dataset 64 recorded in the additional learning data DB 45. The control unit 21 may record a flag indicating the used additional learning data 51 in the additional learning dataset 64.

[0064] The control unit 21 updates the parameters of the learning model 41 recorded in the auxiliary storage device 23. Note that the control unit 21 may record both the learning model 41 before the additional learning and the learning model 41 after the additional learning in the auxiliary storage device 23. Thereafter, the control unit 21 ends the process.

[0065] When it is determined that the learnability and independence of the study model 43 do not satisfy the conditions (NO in step S508), or when it is determined that the accuracy of the learning model 41 has not been improved by additional learning (NO in step S511), the control unit 21 determines whether or not to end the process (step S521). For example, when step S521 is repeated a predetermined number of times, the control unit 21 determines to end the process.

[0066] When it is determined not to end the process (NO in step S521), the control unit 21 returns to step S502. When it is determined to end the process (YES in step S521), the control unit 21 ends the process. The additional learning data 51 is not deleted from the additional learning dataset 64. After a sufficient number of additional learning data 51 are further added to the additional learning dataset 64, the program described using FIG. 7 is executed again.

[0067] 8 is a flowchart illustrating the process flow of the question data extraction subroutine. The question data extraction subroutine is a subroutine for extracting question data from additional learning data 51 recorded in additional learning dataset 64.

[0068] The control unit 21 acquires one set of additional learning data 51 from the additional learning dataset 64 (step S531). The control unit 21 inputs the input data 511 included in the acquired additional learning data 51 to the learning model 41, and acquires the output data 53 (step S532).

[0069] The control unit 21 calculates an evaluation value based on the correct answer data 512 included in the additional learning data 51 acquired in step S531 and the output data 53 acquired in step S532 (step S533). The control unit 21 associates the additional learning data 51 acquired in step S531 with the calculated evaluation value, and records them in the auxiliary storage device 23 or the main storage device 22 (step S534).

[0070] The control unit 21 determines whether or not the processing of the additional learning data 51 recorded in the additional learning dataset 64 has been completed (step S535). If it is determined that the processing has not been completed (NO in step S535), the control unit 21 returns to step S531.

[0071] If it is determined that the process has ended (YES in step S535), the control unit 21 extracts additional learning data 51, which is question data, based on the evaluation value recorded in step S534 (step S536). For example, the control unit 21 extracts additional learning data 51 whose evaluation value is lower than a predetermined threshold. Note that the control unit 21 may calculate the threshold based on the recorded evaluation value. Thereafter, the control unit 21 extracts question data based on the calculated threshold, and ends the process.

[0072] 9 is a flowchart illustrating the process flow of the evaluation index calculation subroutine. The evaluation index calculation subroutine is a subroutine for calculating the learnability evaluation index and the independence evaluation index of the study model 43 generated for each cluster 55.

[0073] The control unit 21 selects a cluster 55 for which an evaluation index is to be calculated (step S541). The cluster 55 selected in step S541 is the i-th cluster F i The cluster 55 selected in step S541 is an example of the cluster used to generate the study model 43 being processed.

[0074] The control unit 21 acquires the study model 43 generated based on the cluster 55 selected in step S541 (step S542). The study model 43 selected in step S542 is the i-th study model M i is equivalent to.

[0075] The control unit 21 acquires the test data 58 included in the cluster 55 selected in step S541 (step S543). The test data 58 acquired in step S543 corresponds to the test data Fite separated from the i-th cluster 55 in the formulas (1) to (3).

[0076] The control unit 21 starts an index calculation subroutine using the study model 43 acquired in step S542 and the test data 58 acquired in step S543 as arguments (step S544). The index calculation subroutine is a subroutine that calculates an evaluation index m(a,b) obtained by evaluating the a-th study model 43a using the b-th test data 58b. In step S544, a=b=i, and the calculated evaluation index m(i,i) is the learnability evaluation index L(F i The flow of the process of the index calculation subroutine will be described later.

[0077] The control unit 21 selects a cluster 55 of the test data 58 (step S545). The cluster 55 selected in step S545 corresponds to the j-th cluster Fj in equations (2) and (3). The cluster 55 selected in step S545 is an example of an unused cluster that has not been used to generate the study model 43 being processed.

[0078] The control unit 21 acquires the test data 58 included in the cluster 55 selected in step S545 (step S546). The test data 58 acquired in step S546 corresponds to the test data Fjte separated from the j-th cluster 55 in equations (2) and (3).

[0079] The control unit 21 starts an index calculation subroutine (step S547) using the study model 43 acquired in step S542 and the test data 58 acquired in step S546 as arguments. The index calculation subroutine is the same as the subroutine started in step S544. In step S544, a=i, b=j, and the evaluation index m(i,j) in equation (3) is calculated from equation (2).

[0080] The control unit 21 determines whether or not the processing of all clusters 55 other than the cluster 55 selected in step S541 has been completed (step S548). If it is determined that the processing has not been completed (NO in step S548), the control unit 21 returns to step S545.

[0081] When it is determined that the process is completed (YES in step S548), the control unit 21 calculates the independence evaluation index I(F i ) is calculated (step S549).

[0082] The control unit 21 determines whether or not the calculation of the learnability evaluation index and the independence evaluation index for all clusters 55 is completed (step S550). If it is determined that the calculation is not completed (NO in step S550), the control unit 21 returns to step S541. If it is determined that the calculation is completed (YES in step S550), the control unit 21 ends the process.

[0083] 10 is a flowchart illustrating the process flow of the index calculation subroutine. The index calculation subroutine is a subroutine for calculating an evaluation index m(a, b) obtained by evaluating the a-th considered model 43a using the b-th test data 58b.

[0084] The control unit 21 acquires one set of additional learning data 51 from the b-th test data 58b specified by the argument (step S551). The control unit 21 inputs the input data 511 included in the acquired additional learning data 51 to the a-th consideration model 43a specified by the argument, and acquires the output data 53 (step S552).

[0085] The control unit 21 calculates an evaluation value based on the correct answer data 512 included in the additional learning data 51 acquired in step S551 and the output data 53 acquired in step S552 (step S553). The control unit 21 associates the additional learning data 51 acquired in step S551 with the calculated evaluation value, and records them in the auxiliary storage device 23 or the main storage device 22 (step S554).

[0086] The control unit 21 determines whether the processing of the additional learning data 51 included in the b-th test data 58b has been completed (step S555). If it is determined that the processing has not been completed (NO in step S555), the control unit 21 returns to step S551.

[0087] If it is determined that the process is completed (YES in step S555), the control unit 21 calculates a representative value of the evaluation values ​​recorded in step S554 (step S556). The evaluation value calculated in step S556 is the evaluation index m(a, b). After that, the control unit 21 ends the process.

[0088] According to this embodiment, the additional learning data 51 included in the additional learning dataset 64 can be distributed evenly to the training dataset 61, the validation dataset 62, and the test dataset 63. Therefore, the additional learning of the learning model 41 can be performed appropriately.

[0089] According to this embodiment, the additional learning data 51 can be appropriately distributed from two perspectives, that is, the learnability evaluation index and the independence evaluation index.

[0090] According to this embodiment, if the additional learning data 51 included in the additional learning dataset 64 does not satisfy the conditions for performing appropriate additional learning, the additional learning is not performed, and the system waits for more additional learning data 51 to be accumulated in the additional learning dataset 64. This makes it possible to prevent the accuracy of the learning model 41 from being reduced due to inappropriate additional learning.

[0091] [Embodiment 2] This embodiment relates to the generation of a new learning model 41. Explanation of parts common to the first embodiment will be omitted. In the following explanation, an example will be given in which the learning model 41 is generated using the information processing device 20 as in the first embodiment. The generation of the learning model 41 and the additional learning of the learning model 41 explained in the first embodiment may be executed by different hardware.

[0092] 11 is an explanatory diagram for explaining an outline of a method for generating the learning model 41. A learning data set 69 is prepared, which is made up of a large number of sets of learning data 59 in which input data 511 and correct answer data 512 are associated with each other. The learning data set 69 is recorded in a database stored in the auxiliary storage device 23 or an external large-capacity storage device connected to the information processing device 20.

[0093] The configurations of the training data 59 and the training data set 69 are similar to those of the additional training data 51 and the additional training data set 64 described using FIG. 4 and FIG. 5, and therefore detailed description thereof will be omitted.

[0094] The learning data 59 is divided into a plurality of clusters 55 by a known clustering method such as the k-means method or the Ward method. Fig. 11 shows three clusters 55, a first cluster 551, a second cluster 552, and a third cluster 553. The points inside each cluster 55 indicate the learning data 59.

[0095] The learning data 59 included in each cluster 55 is divided into training data 56, validation data 57, and test data 58, and then the study models 43 are generated by machine learning. The above process is performed for all clusters 55, and the same number of study models 43 as the number of clusters 55 are generated.

[0096] For each cluster 55, a learning possibility evaluation index indicating whether or not the learning of the study model 43 was performed appropriately and an independence evaluation index indicating the independence between the clusters 55 are evaluated. The calculation method of the learning possibility evaluation index is the same as the calculation method described using formula (1) in embodiment 1. The calculation method of the independence evaluation index is the same as the calculation method described using formulas (2) and (3) in embodiment 2.

[0097] If one or both of the learnability evaluation index and the independence evaluation index do not show good values, the process returns to the clustering step and is repeated. If all the learnability evaluation indexes and the independence evaluation indexes show good values, the learning data 59 included in the learning dataset 69 is distributed to the training dataset 61, the validation dataset 62, and the test dataset 63.

[0098] The training data set 61 includes training data 56 for each cluster 55. The validation data set 62 includes validation data 57 for each cluster 55. The test data set 63 includes test data 58 for each cluster 55.

[0099] A learning model 41 is generated by machine learning using a training dataset 61 and a validation dataset 62. The generated learning model 41 is tested using a test dataset 63.

[0100] The processing described above prevents bias when dividing the multiple sets of learning data 59 contained in the learning data set 69 into training data, validation data, and test data 58, thereby enabling the generation of the learning model 41 to be performed appropriately.

[0101] Fig. 12 is a flowchart illustrating the flow of processing of the program of embodiment 2. Prior to execution of the program of Fig. 12, an unlearned model having a structure corresponding to the learning model 41 is prepared.

[0102] The control unit 21 clusters the learning data 59 recorded in the learning dataset 69 (step S561). The control unit 21 selects one cluster 55 (step S562). The control unit 21 distributes the learning data 59 included in the selected cluster 55 into training data 56, validation data 57, and test data 58 (step S563).

[0103] The control unit 21 performs machine learning using the training data 56 and the validation data 57 to generate the study model 43 (step S564). The process of generating a model using supervised machine learning has been performed in the past, so detailed explanations will be omitted. The control unit 21 records the generated study model 43 in the main storage device 22 or the auxiliary storage device 23 in association with the cluster 55 selected in step S562.

[0104] The control unit 21 judges whether the processing of all clusters 55 has been completed (step S565). If it is judged that the processing has not been completed (NO in step S565), the control unit 21 returns to step S562. If it is judged that the processing has been completed (YES in step S565), the control unit 21 starts a subroutine for calculating an evaluation index (step S566). The subroutine for calculating an evaluation index is the same as the subroutine described using FIG. 9.

[0105] The control unit 21 determines whether the learnability evaluation index and the independence evaluation index of the study model 43 satisfy the condition (step S567). Specifically, when the learnability evaluation index and the independence evaluation index calculated for each study model 43 all exceed a predetermined threshold, the control unit 21 determines that the condition is satisfied.

[0106] If it is determined that the condition is satisfied (YES in step S567), the control unit 21 distributes the learning data 59 included in the learning dataset 69 to the training dataset 61, the validation dataset 62, and the test dataset 63 (step S568).

[0107] Here, the training data set 61 is composed of the training data 56 distributed in step S563 for each cluster 55. The validation data set 62 is composed of the validation data 57 distributed in step S563 for each cluster 55. The test data set 63 is composed of the test data 58 distributed in step S563 for each cluster 55.

[0108] The control unit 21 performs machine learning using the training data set 61 and the validation data set 62 to generate a learning model 41 (step S569). The control unit 21 tests the generated learning model 41 using the test data set 63. The control unit 21 determines whether or not a learning model 41 having a predetermined accuracy has been generated based on the test result (step S570).

[0109] When it is determined that the learning model 41 has been generated (YES in step S570), the control unit 21 records the parameters of the generated learning model 41 in the auxiliary storage device 23 (step S571). After that, the control unit 21 ends the process.

[0110] When it is determined that the learnability and independence of the study model 43 do not satisfy the conditions (NO in step S567), or when it is determined that a learning model 41 having a predetermined accuracy has not been generated (NO in step S570), the control unit 21 determines whether or not to end the process (step S572). For example, when step S572 is repeated a predetermined number of times, the control unit 21 determines that the process is to end.

[0111] If it is determined that the process should not be terminated (NO in step S572), the control unit 21 returns to step S561. If it is determined that the process should be terminated (YES in step S572), the control unit 21 terminates the process. Note that the control unit 21 may notify the user that an appropriate learning model 41 could not be generated before terminating the process.

[0112] According to this embodiment, bias can be prevented when distributing the learning data 59 recorded in the learning dataset 69 to the training dataset 61, the validation dataset 62, and the test dataset 63, so that an appropriate learning model 41 can be generated with a small number of trials.

[0113] [Embodiment 3] 13 is an explanatory diagram for explaining the configuration of an information processing device 20 according to the third embodiment. This embodiment relates to a form in which the information processing device 20 according to this embodiment is realized by combining and operating a general-purpose computer 90 and a program 97. Explanations of parts common to the first embodiment will be omitted.

[0114] The computer 90 includes a reading unit 29 in addition to the control unit 21, main memory device 22, auxiliary memory device 23, communication unit 24, display unit 25, input unit 26 and bus described above.

[0115] The program 97 is recorded in a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read out the program 97 stored in a semiconductor memory 98 such as a flash memory implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown), and store it in the auxiliary storage device 23.

[0116] The program 97 is installed as a control program for the computer 90, and is loaded into and executed by the main storage device 22. In this manner, the information processing device 20 described in the first embodiment is realized.

[0117] [Embodiment 4] 14 is a functional block diagram of the information processing device 20 according to the embodiment 4. The information processing device 20 includes an acquisition unit 81, a clustering unit 82, a distribution unit 83, and a learning unit 84.

[0118] The acquiring unit 81 acquires a plurality of sets of learning data 59 in which input data 511 and supervised answer data 512 are associated with each other, to be used for learning the learning model 41. The clustering unit 82 clusters the learning data 59 into a plurality of clusters 55.

[0119] The distribution unit 83 divides the learning data 59 included in each cluster 55 into training data 56, validation data 57, and test data 58. The learning unit 84 collects the training data 56, validation data 57, and test data 58 of the multiple clusters 55 and uses them for learning the learning model 41.

[0120] The technical features (constitutive elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0121] 20 Information processing device 21 Control section 22 Main storage 23 Auxiliary storage device 24 Communications Department 25 Display section 26 Input section 29 Reading unit 41 Learning Model 43 Model under consideration 431 First model 432 Second Study Model 433 Third Model 43a Model A under consideration 43i ith model under consideration 45 Additional learning data DB 51 Additional learning data 511 Input Data 512 Correct Data 53 Output Data 55 Cluster 551 Cluster 1 552 Cluster 2 553 Cluster 3 55i i-th cluster 56 Training Data 56i Training Data 57 Validation Data 57i Validation Data 58 Test Data 58b Test Data B 58i Test Data 58j jth test data 59 Training Data 61 training datasets 62 Validation Dataset 63 Test Dataset 64 additional training datasets 69 Training Dataset 81 Acquisition Department 82 Clustering Department 83 Distribution section 84 Learning Department 90 Computer 96 Portable recording media 97 Program 98 Semiconductor Memory

Claims

1. A learning dataset is obtained, which is made up of a plurality of sets of learning data in which input data and correct answer data are associated with each other, and which is used for learning a learning model; clustering at least a portion of the training data set into a plurality of clusters; Dividing the learning data included in each of the clusters into training data, validation data, and test data; The training data, validation data, and test data of the plurality of clusters are each collected and used for learning the learning model. An information processing method in which processing is performed by a computer.

2. generating a plurality of models to be considered through machine learning using each of the plurality of clusters; For each of the generated multiple study models, Calculating a learnability evaluation index regarding the learnability of the use cluster using the test data separated from the use cluster used to generate each of the study models among the plurality of clusters; Calculating an independence evaluation index regarding the independence of the used cluster from the unused cluster by using the test data separated from the unused cluster that is not used in generating each of the study models among the plurality of clusters; If the learnability evaluation index and the independence evaluation index do not satisfy a predetermined condition, the clustering is performed again. The information processing method according to claim 1 .

3. The learnability evaluation index is defined by equation (1), and the independence evaluation index is defined by equation (2). The information processing method according to claim 2 . [0010]

4. If all of the learnability evaluation indices and the independence evaluation indices do not exceed 0.6, the clustering is performed again. The information processing method according to claim 3.

5. The learning data set is an additional learning data set used for additional learning of the learning model, inputting the input data into the learning model, and evaluating the accuracy of the output data based on output data output from the learning model and correct answer data associated with the input data; The clustering is performed on the plurality of sets of learning data, among the additional learning data set, for which the accuracy of the output data is equal to or less than a predetermined threshold.

5. The information processing method according to claim 1.

6. The plurality of sets of learning data whose accuracy of the output data exceeds a predetermined threshold are organized into one cluster. The information processing method according to claim 5.

7. A learning dataset is obtained, which is made up of a plurality of sets of learning data in which input data and correct answer data are associated with each other, and which is used for learning a learning model; Clustering the plurality of training data sets into a plurality of clusters; Dividing the learning data set included in each of the clusters into training data, validation data, and test data; creating a training data set including the training data for each of the plurality of clusters, a validation data set including the validation data for each of the plurality of clusters, and a training data set including the test data for each of the plurality of clusters; An information processing method in which processing is performed by a computer.

8. An acquisition unit that acquires a plurality of sets of learning data in which input data and correct answer data are associated with each other and used for learning a learning model; A clustering unit that clusters the learning data into a plurality of clusters; a distribution unit that divides the learning data included in each of the clusters into training data, validation data, and test data; a learning unit that compiles the training data, validation data, and test data of the plurality of clusters and uses them for learning the learning model; An information processing device comprising:

9. Obtaining a plurality of sets of learning data in which input data and correct answer data are associated with each other and used for learning a learning model; clustering the learning data into a plurality of clusters; Dividing the learning data included in each of the clusters into training data, validation data, and test data; The training data, validation data, and test data of the plurality of clusters are each collected and used for learning the learning model. A program that causes a computer to carry out processing.

Citation Information

Patent Citations

  • Medical image processing apparatus, method, and program

    JP2018175343A

  • Information processing method and information processing system

    JP2019133626A