Proposal device, method, system, program, and information recording medium for proposing factor parameters for estimating target labels

Abstract

The present invention suggests factor parameters for estimating a target label from a plurality of parameters that can be acquired from a target. Provided is a suggestion device (101), wherein a calculation unit (102) trains an estimation model which: estimates, among a plurality of candidates that represent a combination of parameters, a label from the parameters pertaining to the combination with respect to a portion of calculation candidates, with reference to a plurality of records, each of which representing the label imparted to the target and a plurality of parameter values that are acquired from the target; calculates the performance; makes the calculation candidates compete with each other on the basis of the calculated performance; and calculates rating values of the calculation candidates. The estimation unit (103) trains a rating model by means of the combination and rating values of the parameters of the calculation candidates, and estimates rating values of estimation candidates other than the calculation candidates. An output unit (104) outputs, as useful candidates representing factor parameters, candidates that are the calculation candidates of which the calculated or estimated rating values are at a high level.

Description

[Technical Field]

[0001] The present invention relates to a proposal device, a proposal method, a proposal system, a program, and an information recording medium that propose factor parameters for estimating a label of an object from among a plurality of parameters that can be obtained for the object. [Background technology]

[0002] The pathological malignancy of prostate cancer is an important evaluation value in determining a treatment plan. Whether or not a cancer is pathologically malignant and the degree of malignancy can be determined by performing a pathological examination of a specimen obtained from a patient using a microscope. The inventor has proposed a technology that obtains feature parameters or feature vectors by dimensionally compressing a photographic image obtained by photographing a specimen obtained from a patient, and then appropriately classifies the patient as pathologically malignant or not based on the obtained feature parameters or feature vectors (Patent Document 1).

[0003] However, pathological testing is not only invasive, but also takes time to provide results and is expensive. Therefore, although it is very useful for post-mortem analysis after tumor removal surgery, it is difficult to apply it to pre-emptive prediction.

[0004] For this reason, there is a demand for an appropriate combination of various non-invasive tests, such as prostate ultrasound imaging and blood tests, to predict in advance the degree of pathological malignancy, and for further combining information obtained from invasive tests, such as information based on photographic images of specimens prepared from the affected area removed during treatment, to improve the accuracy of predicting prognosis and treatment effects.

[0005] With the advancement of medicine, the number of types of tests, including blood tests, is steadily increasing. Performing each type of test requires time, effort, and costs based on drug prices. Therefore, it is necessary to select a cost-effective combination of tests, i.e., a combination of tests that will significantly contribute to accurately estimating a patient's current condition, prognosis, treatment effects, etc.

[0006] This problem can be generalized as follows: What is needed is a technique for identifying, among multiple parameters obtainable about an object, which factor parameters contribute to determining the label of the object, and the combination of these factor parameters.

[0007] When there are n types of parameters that can be obtained from the target, there are a total of 2 n Therefore, in this technology, two types of parameters are used, each of which represents a possible combination of parameters. n From among the candidates, candidates that represent a combination of factor parameters that is suitable for estimating a label are identified. Here, since a suitable combination of factor parameters is useful for estimating a target label, the candidate that represents the suitable combination can be called a useful candidate.

[0008] where 2 n To narrow down the candidates to useful candidates, one possible method is to use rating systems such as Elo Rating and TrueSkill (trademark) disclosed in Patent Document 2, as well as Glicko Rating and Glicko 2 Rating, to compare the performance of candidates against each other to determine superiority, calculate a rating value for each candidate based on that superiority, and then select the candidates with the highest rating value as useful candidates. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 6945253 [Patent Document 2] U.S. Patent No. 8,538,910 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in order to calculate the performance of candidates representing parameter combinations, it is necessary to proceed with training of an estimation model using techniques such as machine learning, neural networks, deep learning, etc., with parameter values ​​of the parameters related to the combination among parameters obtainable from the target as input data and labels attached to the target as output data. Training of an estimation model often requires a large amount of calculation and a long time.

[0011] On the other hand, rating systems require a large number of candidates to compete against each other. The simplest approach would be to create an estimation model for each candidate and calculate their performance, and then compete against each other under the rating system. However, this method requires enormous amounts of calculation and time, resulting in an explosion in the number of matches.

[0012] Therefore, there is a need for a technology that can appropriately narrow down useful candidates with a small amount of calculation and short calculation time while limiting the learning of the estimation model and the number of matches, and then propose the obtained useful candidates.

[0013] The present invention aims to solve the above-mentioned problems and provides a proposal device, a proposal method, a proposal system, a program, and an information recording medium that propose factor parameters for estimating the label of an object from among multiple parameters that can be obtained for the object. [Means for solving the problem]

[0014] The proposing device according to the present invention proposes, based on a plurality of records, one or more candidates from among a plurality of candidates each representing a combination of parameters obtainable for an object as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object. Here, each record of the plurality of records has a plurality of parameter values ​​obtained by obtaining a plurality of parameters for the object associated with the respective record, and a label assigned to the associated object. The proposing device then: Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the learned estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance; Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, estimate a rating value associated with each of the estimated candidates other than the calculated candidate among the plurality of candidates; The calculated candidate associated with the highest rating value among the calculated or estimated rating values ​​is output as the useful candidate. The proposed system according to the present invention is a prediction system that predicts the prognosis of a patient for a medical professional who treats the patient, Displaying the test values ​​already obtained for the patient on the screen; selecting, as a prediction candidate, a useful candidate whose acquired test value satisfies the combination of test items related to the useful candidate from a plurality of useful candidates each representing a combination of test items; providing the acquired test values ​​that satisfy the combinations related to the selected prediction candidates as inputs to an estimation model trained for the selected prediction candidates to estimate the prognosis of the patient; The estimated prognosis is presented on the screen. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a proposal device, a proposal method, a proposal system, a program, and an information recording medium that propose factor parameters for estimating the label of an object from among multiple parameters that can be obtained for the object. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is an explanatory diagram showing a schematic configuration of a proposed device according to an embodiment of the present invention; [Figure 2] FIG. 1 is an explanatory diagram showing the deep learning network configuration used in the rating model. [Figure 3] 10 is a flowchart illustrating a basic control flow of a proposal process according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating a control flow of a modified example of the proposal process according to the embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram showing an example of outputting useful candidates in the form of a scatter plot. [Figure 6] FIG. 10 is an explanatory diagram showing a display example in which useful candidates are output and displayed on the screen within a scatter diagram. [Figure 7] FIG. 10 is an explanatory diagram showing a display example in which useful candidates are output and displayed on the screen within a scatter diagram. [Figure 8] 1 is an explanatory diagram showing a schematic configuration of a proposal system according to an embodiment of the present invention; [Figure 9] FIG. 2 is an explanatory diagram showing how information is sent and received in the proposed system according to the embodiment of the present invention. [Figure 10] FIG. 2 is an explanatory diagram showing how information is sent and received in the proposed system according to the embodiment of the present invention. [Figure 11] FIG. 1 is an explanatory diagram showing a schematic configuration of a prediction system included in a proposal system according to an embodiment of the present invention. [Figure 12]FIG. 1 is an explanatory diagram showing the appearance of an electronic medical record presented in a prediction system according to an embodiment of the present invention. [Figure 13] FIG. 10 is an explanatory diagram showing how the results of prognosis prediction are presented in an electronic medical record presented in a prediction system according to an embodiment of the present invention. [Figure 14] 10 is a flowchart illustrating a control flow of a prediction process executed in the prediction system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of the present embodiments are replaced with equivalents. Furthermore, elements described in each example can be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included in the scope of the present invention.

[0018] (composition) The proposed device according to this embodiment is typically realized by a computer executing a program. The computer is connected to various output devices and input devices, and transmits and receives information to and from these devices.

[0019] A program executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0020] The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, EEPROM, etc., possessed by the computer. The information processing device of this embodiment is then realized by the computer. Generally, the computer's central processing unit (CPU) reads the program from the information recording medium into random access memory (RAM) under the control of the computer's operating system (OS), and then interprets and executes the code contained in the program. However, in an architecture in which the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

[0021] Furthermore, as mentioned above, it is desirable for the computer to be equipped with a GPU (Graphics Processing Unit) for performing various image processing calculations at high speed. By using a GPU and libraries such as TensorFlow, it becomes possible to use learning functions, suggestion functions, dimensionality reduction functions, etc. in various artificial intelligence processes such as neural networks, machine learning, and deep learning under the control of the CPU.

[0022] It should be noted that the information processing device of this embodiment may be configured using a dedicated electronic circuit rather than a general-purpose computer. In this embodiment, the program may be used as a resource for generating wiring diagrams, timing charts, and the like for the electronic circuit. In this embodiment, an electronic circuit that meets the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the information processing device of this embodiment.

[0023] In the following, for ease of understanding, the proposed device will be described assuming that it is realized by a computer executing a program. Fig. 1 is an explanatory diagram showing the general configuration of the proposed device according to an embodiment of the present invention.

[0024] As shown in the figure, a proposal device 101 according to this embodiment includes a calculation unit 102, an estimation unit 103, and an output unit 104. In addition, a removal unit 105 can be included as an optional element.

[0025] Based on multiple records, the proposing device 101 proposes one or more candidates from multiple candidates each representing a combination of parameters that can be obtained for the object as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object.

[0026] As mentioned above, if n types of parameters can be obtained for a target, the total number of parameter combinations is 2 n Since there are n elements, each combination can be expressed by an n-dimensional binary vector. For example, if the value of the i-th element of the binary vector is 1, the i-th parameter is included in the combination, and if it is 0, the i-th parameter is not included in the combination.

[0027] For example, if there are 13 types of blood tests, namely PSA, WBC, Hb, Plt, LDH, ALP, GOT, GPT, Alb, Bun, cre, Ca, and CRP, then n = 13. The combinations can be expressed as a 13-dimensional binary vector in which each test is listed in order, with 1 / 0 indicating whether it is included in the combination or not.

[0028] For example, a combination in which three tests, PSA (tumor marker), WBC (white blood cell count), and Hb (hemoglobin), are used as parameters to be acquired and tests are performed, while other parameters are not acquired and are not tested, would be represented by a 13-dimensional binary vector, [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0].

[0029] As an estimation model for estimating a label to be assigned to an object from parameter values ​​acquired from the object, various machine learning models and classification models can be adopted, such as SVM (Support Vector Machine), Lasso, Ridge, Elasticnet, Decision Tree, RF (Random Forest), CART, Boosting, XGGoost, various neural networks, etc. Furthermore, these various models can also be used in combination.

[0030] Furthermore, these classification models require the setting of various hyperparameters to adjust their behavior, which means that the behavior of the classification model changes depending on the parameter settings.

[0031] Therefore, each candidate may be associated with the type and specifications of an estimation model.

[0032] For example, if we assume the above 10 types as estimation models for the above blood test and assume five types for each estimation model, the total number of candidates is 2 13 × 10 × 5. It is also possible to always include a parameter in the combination. For example, if PSA is always included, the number of candidates is 2 (13-1) ×10×5, which is half of the above.

[0033] When only one type of estimation model is adopted, it is not necessary to indicate the type and specifications of the model for each candidate.

[0034] The plurality of candidates are generated and stored in the candidate storage area 122 such as a RAM by enumerating combinations.

[0035] As described above, the proposal device 101 according to this embodiment refers to a plurality of records recorded in the database 121 or the like.

[0036] For example, in the case of the prostate cancer patient mentioned above, each record corresponds to an electronic medical record containing the results of 13 blood tests performed on the prostate cancer patient.

[0037] This electronic medical record may include various medical images (pathological photographs obtained by invasive tests such as prostate biopsy, echograms obtained by ultrasound examinations, CT, MRI, PET, bone scintigraphy, plain X-rays, mammography, dermoscopy, fundus photographs, corneal photographs, etc.), as well as feature parameters or feature vectors obtained by dimensionally compressing the medical images using the technology disclosed in Patent Document 1, etc.

[0038] It may also include the Gleason score determined by a doctor or other medical professional observing a pathological specimen (or a pathological photograph image) obtained by an invasive prostate biopsy examination, and the determination result of whether or not the tumor is highly malignant based on the Gleason score.

[0039] In the following, for ease of understanding, as an example, the electronic medical record will be described as including a Gleason score determined by a doctor or the like by observing a pathological specimen obtained by an invasive test such as a prostate biopsy, and an ultrasound image (feature vector and feature parameters) as a medical image, but the scope of application of this embodiment is not limited to this.

[0040] Furthermore, electronic medical records typically include the patient's gender, age, medical history, whether or not they drink or smoke, and how often, as well as test results other than blood tests.

[0041] This information generally corresponds to parameter values ​​of the parameters obtained for the subject.

[0042] Furthermore, the electronic medical record also contains information such as the patient's prognosis after the examination and whether or not there has been a recurrence, etc. This generally corresponds to a label attached to the subject.

[0043] That is, each record of the multiple records has multiple parameter values ​​obtained by acquiring multiple parameters for the object associated with each record, and a label attached to the associated object.

[0044] Now, the calculation unit 102 of the proposing device 101 refers to the plurality of records, for example, the database of the electronic medical records, and performs the following processing.

[0045] That is, the calculation unit 102 refers to a plurality of candidates generated in the candidate storage area 122 or the like, and learns an estimation model for each of the calculation candidates that are a part of the candidates.

[0046] As described above, each candidate is associated with a combination of parameters to be obtained from multiple parameters, and generally, each candidate is also associated with the type and specifications of an estimation model that will be trained using that combination.

[0047] The calculation unit 102 learns an estimation model of the type and specifications indicated in some of the candidates. In other words, in this application, a candidate for which an estimation model associated with the candidate is learned is called a "calculated candidate."

[0048] The estimation model for each calculation candidate has as input the parameter values ​​obtained by obtaining parameters for the combination represented by each calculation candidate for the target, and has as output the target.

[0049] For example, when considering the candidates for the combination expressed by the 13-dimensional binary vector above, [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], the inputs to the estimation model are the test results of three blood tests: PSA, WBC, and Hb.

[0050] Note that, as inputs other than the parameters related to the candidate combinations, it is also possible to use feature parameters or feature vectors obtained by dimensionally compressing an image of a subject. Considering this aspect in the above example, a vector consisting of the test results of the three blood tests PSA, WBC, and Hb and the feature parameters or feature vectors obtained by dimensionally compressing the ultrasound image becomes input data for the estimation model.

[0051] Furthermore, information that can be easily obtained from patients without the need for testing, such as gender, age, medical history, and whether or not they drink or smoke, and how often, can also be used as input data.

[0052] The output of the estimation model is the label (probability) to be assigned to the object related to the input data. The estimation model performs classification by comparing the evaluation value calculated based on the input data with a threshold. Then, by training the estimation model, various coefficients used to calculate the evaluation value (and, depending on the type of model, the threshold to be compared) are determined.

[0053] When labels are expressed as binary expressions of negative and positive, the evaluation value calculated by the estimation model can be configured to represent the probability that the object related to the input data will be classified as one of the two (e.g., positive). In this case, the threshold is a constant 0.5.

[0054] A plurality of records are referenced for training an estimation model. The plurality of records are divided into training records and test records. The test records are the records remaining after removing the training records from the plurality of records. This division may be performed randomly or may be performed regularly based on a predetermined rule or procedure.

[0055] Training data for learning the estimation model is generated from the training records. The input data for each training data set includes parameter values ​​extracted from each training record for the parameters associated with the combinations represented by each candidate. These are called combination input parameters. In the above example, these correspond to the blood test values ​​of PSA, WBC, and Hb in the patient's electronic medical record.

[0056] In addition, information other than the combination of parameters can be added as input data. These are called additional input parameters. In the above example, these include gender, age, medical history, whether or not you drink alcohol or smoke, and how often.

[0057] When it is desired to predict the prognosis or treatment effect of a patient after an invasive procedure, the Gleason score determined by a doctor or other professional observing the pathological specimen obtained from the procedure, and the feature parameters and feature vectors obtained by dimensionally compressing the pathological photograph image can be used as input data. These can be applied to both combined input parameters and additional input parameters.

[0058] The output data (ground truth data) is a label that indicates the prognosis and treatment effect of the patient shown in the training record. In the above example, this label indicates whether or not the prostate cancer recurred within a predetermined period (e.g., one year) after the treatment, and is assigned a value such as 1 if recurrence occurred and 0 if no recurrence occurred.

[0059] In this way, training data for each candidate is generated from the training records, and an estimation model is learned.

[0060] Once the learning of the estimation model for each calculation candidate is completed, the calculation unit 102 calculates the performance of the learned estimation model to estimate a label to be assigned to the target from the parameter values ​​obtained from the parameters related to the combinations represented by each calculation candidate for the target.

[0061] To calculate performance, we use test data generated from test records in the same way as training records.

[0062] That is, for each test data, input data is provided to the trained estimation model, and a test is performed to determine whether the label output from the estimation model matches the output data related to the test data. If the two match, the inference by the estimation model is correct. In the simplest terms, the accuracy rate obtained here can be used as the performance of the trained estimation model.

[0063] The output from the estimation model may also include the label of the estimation result and the probability associated with the label. In this case, the calculation unit 102 can plot a receiver operating characteristic (ROC) curve based on the results of each test to calculate an area under the curve (AUC). The calculation unit 102 can then use the calculated AUC value as the performance of the trained estimation model.

[0064] In addition, even for records in which some parameters have not been obtained, such as the electronic medical record of a patient for whom only some blood tests have been performed, if the parameter values ​​related to the combined input data and the information related to the additional input data are all present in the record, the record can be used as a training record or a test record.

[0065] Once the performance of the trained estimation model related to the calculation candidate is calculated in this manner, the calculation unit 102 calculates a rating value to be associated with the calculation candidate based on the calculated performance.

[0066] The rating value according to this embodiment is similar to that used in Elo Ratings and the like, and the probability of winning is determined from the difference in the rating values ​​of the competitors.

[0067] In this embodiment, the calculated performance of the calculated candidates among the multiple candidates is referenced, and the calculated candidates are pitted against each other, and based on the outcome of the match, the rating values ​​of the competing calculated candidates are updated so that the rating value of the winning calculated candidate increases and the rating value of the losing calculated candidate decreases.

[0068] For example, in Elo Rating, the average opponent's rating, i.e., the initial value of the rating before updating, is a constant. For example, 1500 can be used as the initial value.

[0069] Elo Rating estimates the winning probability of each opponent based on their rating before the match. For example, if the rating values ​​of opponents a and b are Ra and Rb, the winning probability of a and b, respectively, is Wab and Wba. Wab = 1 / [10 (Rb - Ra) / 400 + 1) Wba = 1 / [10 (Ra - Rb) / 400 + 1) = 1 - Wab Here, a constant value of 400 is used, but other values ​​can also be used.

[0070] Then, when the opponents play several times, their ratings are updated according to the difference between the estimated winning rate and the actual winning rate. When opponents a and b play several times and the actual winning rate of opponent a is Sab and the actual winning rate of opponent b is Sba = 1 - Sab, the new rating values ​​Ra' and Rb' of opponents a and b are Ra' = Ra + K×(Sab - Wab), Rb' = Rb + K × (Sba - Wba) Here, the value of K is often 16 or 32.

[0071] In this way, the calculation unit 102 calculates the rating value of the calculated candidate by determining the outcome of the match between the calculated candidates based on their performance and updating the rating value associated with the matched calculated candidate.

[0072] Now, in order to ensure a fair match between the calculation candidates, the training data and test data given to both should be the same.

[0073] Therefore, the calculation unit 102 extracts two calculation candidates from the plurality of candidates. The extraction of the two calculation candidates is typically performed randomly, but various other methods will be described later.

[0074] The calculation unit 102 then repeats the following process several times for the two extracted calculation candidates. The method for determining the number of repetitions, that is, the number of times the same calculation candidates will compete against each other, will be described later. (a) Randomly select training records from multiple records, and set the remaining records as test records. (b) Generate training and test data; (c) Learning an estimation model using training data; (d) Calculating the performance of each of the two candidate calculations based on the two learned estimation models; (e) The winner of this competition between the two evaluation candidates will be determined based on their performance.

[0075] Then, by having the two evaluation candidates play against each other at least once, the probability of winning or losing in the match between the two evaluation candidates can be calculated.Then, the player with the higher probability of winning is designated as the winner, and the player with the lower probability of winning is designated as the loser, and the rating values ​​of both players are updated.The number of times the two evaluation candidates will play against each other can most simply be set to a constant greater than or equal to 1, but it can be set as appropriate.

[0076] In process (a), when the same two evaluation candidates compete against each other two or more times, the training data and test data are randomly selected at the beginning of each match, so the training data and test data are different from one match to another. Therefore, which candidate wins may change from match to match.

[0077] In process (c), if the estimation model has already been trained in the previous match, the previous training results may be carried over, but the estimation model may first be cleared and then trained using the current training data.

[0078] Similar processing can be performed when using a rating system other than Elo Rating, such as Glicko Rating or Glicko 2 Rating. Rating systems such as TrueSkill (trademark) support battles between three or more players. In such rating systems, the calculation unit 102 may extract three or more candidates, have them battle each other, and update the rating value.

[0079] The calculation unit 102 repeats the process of "selecting calculation candidates and having them compete against each other several times, determining the winning probabilities of both players based on the results of the matches, and then updating the rating values ​​of both players." The number of repetitions, i.e., the number of times calculation candidates are extracted, can be set to a constant, for example, but other methods will be described later.

[0080] As described above, the number of candidates becomes enormous depending on the combination of parameters and the type and specifications of the estimation model, so it is difficult to match all combinations of candidates against each other.

[0081] Therefore, once the match has progressed to a certain extent, the calculation unit 102 learns the rating model by referring to the rating values ​​calculated for the calculation candidates. During learning, corrections may be made to change part of the rating values ​​to avoid falling into a local solution, and the corrected rating values ​​may be referenced. The calculation candidates whose rating values ​​are to be changed can be selected, for example, randomly from all the calculation candidates. The amount of change in the rating value can be determined by adding, subtracting, multiplying, dividing, or the like a constant value or a random value.

[0082] As with the estimation model, various types and specifications of the rating model can be adopted. Figure 2 is an explanatory diagram showing the deep learning network configuration used for the rating model. The network configuration shown in this figure uses a simple fully connected deep learning model.

[0083] In the training data for a rating model, the input data is the combination of parameters (and the type and specifications of the estimation model) shown as candidates, and the output data (correct answer data) is the rating value of the candidate.

[0084] In the above example, the input data is a 13-dimensional binary vector (and the type and specifications of the estimation model) related to the calculation candidate.

[0085] In this embodiment, a candidate that has not been extracted as a calculation candidate, and therefore has not played a single match and has not had its rating value calculated (updated from the initial value), is called an estimated candidate. When a combination of parameters represented by an estimated candidate (and the type and specifications of the estimation model) is given as input to a trained rating model, the rating model estimates the rating value of the estimated candidate and outputs the estimated rating value.

[0086] Candidates with similar parameter combinations often have similar rating values, and the rating model can be thought of as learning the aspects of "similarity between candidates" and "similarity between rating values."

[0087] Then, the calculation unit 102 estimates a rating value associated with each estimated candidate based on the learned rating model.

[0088] In this way, candidates whose estimation model has been learned through matches and whose rating values ​​have been calculated will be classified as calculated candidates, while candidates who have not yet played matches and whose estimation model has not been learned will have their rating values ​​estimated by the rating model and will be classified as estimated candidates.

[0089] Then, the output unit 104 outputs the calculated candidate associated with the highest rating value among the calculated or estimated rating values ​​as a useful candidate.

[0090] For example, the output unit 104 selects a predetermined number of candidates (hereinafter referred to as "top candidates") that have the highest associated rating values ​​from among the multiple candidates. If all of the top candidates are evaluation candidates, the output unit 104 outputs the top candidates as useful candidates.

[0091] At this time, the output unit 104 can further output an inference model trained for the useful candidate. For example, when a doctor selects a test to be performed on a patient, he or she compares and considers the combinations of tests suggested by the output useful candidates, selects one of the combinations, and performs the test. After that, the doctor can predict the prognosis of the patient by providing the test value to the inference model associated with that combination. An embodiment for comparing and contrasting the output useful candidates will be described later.

[0092] Furthermore, the output unit 104 can display a plurality of useful candidates in a scatter diagram to make it easier for a person to determine which useful candidate should be adopted.

[0093] Here, the first axis of the scatter plot (e.g., the vertical axis) can indicate the number of parameters for the combination represented by the useful candidate, or the total cost for obtaining the parameters for the combination (e.g., the drug price when a test related to the parameters is performed), and the second axis of the scatter plot (e.g., the horizontal axis) can indicate the performance (e.g., the AUC value) or rating value calculated for the useful candidate.

[0094] When the above horizontal-vertical axis setting is adopted, useful candidates located on the lower side of the vertical axis (small number of parameters or total cost) and on the right side of the horizontal axis (high performance or rating value) in the scatter plot are desirable.

[0095] If some or all of the top candidates are inferred candidates, the above process must be repeated because an inferred model has not been learned for the inferred candidates.

[0096] In the simplest case, the top candidates are extracted as estimated candidates, and the calculation unit 102 performs calculation and the estimation unit 103 performs estimation, and then the output unit 104 outputs the results.

[0097] In the simplest case, in a mode in which candidates to be matched up are randomly selected each time, if the entire process is repeated, eventually all of the top candidates will become calculated candidates. However, in order to reduce matches that are not profitable, at an appropriate time, for example, when it is determined that some or all of the top candidates are estimated candidates and therefore it is necessary to repeat the process again, the removal unit 105 may remove estimated candidates with lower associated rating values ​​from the candidate storage area 122, etc., to narrow down the candidates.

[0098] Furthermore, in step S205, the calculation unit 102 may randomly select, from among the multiple candidates, candidates for which an estimation model has not been trained or candidates for which the number of times of training has been small, and use the selected candidates as the calculation candidates for subsequent matches. That is, the probability of a candidate being extracted may be changed depending on whether or not an estimation model has been trained and the number of times. Alternatively, a candidate for which an estimation model has been trained or a candidate for which the number of times of training has been large may not be extracted thereafter.

[0099] (Basic flow) The basic control flow according to this embodiment will be described below: Fig. 3 is a flowchart showing the basic control flow of the proposal process according to this embodiment of the present invention.

[0100] When the processing starts, the proposal device 101 according to this embodiment performs various initializations, such as acquiring multiple records from the database 121, setting multiple parameters that can be acquired from the target, and setting the type and specifications of the estimation model that can be adopted (step S201).

[0101] Next, the proposal device 101 generates multiple candidates by listing combinations of the multiple set parameters and the type and specifications of the estimation model, and stores them in the candidate storage area 122 in association with the rating value for which a default value has been set (step S202).

[0102] After this, if the higher-level condition is not satisfied (step S203; No), the proposing device 101 repeats the following process. As the higher-level condition, for example, it is possible to adopt that all of the candidates with the highest rating values ​​are calculation candidates. When step S203 is reached for the first time, there are no calculation candidates and the rating values ​​of all candidates should be the same initial value, so the higher-level condition is not satisfied and control proceeds to step S204.

[0103] Next, if the calculation completion condition is not satisfied (step S204; No), the proposing device 101 repeats the following processing. The calculation completion condition can be that the learning of the estimation model, the calculation of the performance, and the calculation of the rating value have been completed for a certain number of calculation candidates, or, most simply, that a predetermined number of repetitions have been performed. When step S203 is reached for the first time, there are no calculation candidates, so the calculation completion condition is not satisfied and control proceeds to step S205.

[0104] First, in the proposing device 101, the calculation unit 102 extracts calculation candidates from the multiple candidates stored in the candidate storage area (step S205). The number of calculation candidates extracted here is the number of competitors who can participate in one match in the adopted rating system, for example, 2 in Elo Rating.

[0105] Then, the calculation unit 102 repeats the following process a predetermined number of times (step S206). The number of times of competition may be a fixed number of times, such as one or more, or may be varied as appropriate.

[0106] First, the calculation unit 102 randomly divides a plurality of records to generate training records and test records (step S207). The number of training records is generally several to several tens of times the number of test records.

[0107] Then, the calculation unit 102 repeats the following process for each of the calculation candidates extracted in the immediately preceding step S206 (step S208).

[0108] First, the calculation unit 102 generates training data and test data from training records and test records based on the combinations represented by the calculation candidates (step S209).

[0109] Next, the calculation unit 102 applies the generated training record to an estimation model related to the type and specifications represented by the calculation candidate, and trains the estimation model (step S210). Note that if the estimation model has already been trained in a previous iteration, the estimation model may be cleared, or additional training may be performed without clearing.

[0110] After the estimation model is trained, the calculation unit 102 provides the trained estimation model with the generated test data and calculates the performance of the estimation model (step S211).

[0111] After the performance of the estimation model is calculated for each of the estimation candidates extracted by the repetition (step S212), the calculation unit 102 determines the outcome of the match between the extracted estimation candidates based on the calculated performance (step S213).

[0112] After a predetermined number of matches have been played by repeating the process (step S214), the calculation unit 102 updates the rating values ​​stored in the candidate storage area in association with each of the extracted calculation candidates based on the number of wins and losses in the predetermined number of matches (step S215), and returns control to step S204. The updated rating values ​​become the most recently calculated rating values ​​for the calculation candidates.

[0113] In this manner, in this embodiment, the candidates extracted in step S206 become the candidates for evaluation, and through the processing of steps S207-S215, the estimation model is trained, its performance is calculated, and its rating value is calculated.

[0114] Now, when the calculation completion condition is satisfied (step S204; Yes), the estimation unit 103 learns a rating model using the combination of parameters represented by the calculation candidate (and the estimation model type and specifications) and the rating value calculated for the calculation candidate as training data (step S221). If the entire process is repeated, additional learning may be performed using the rating model learned up to the previous time as is, or the rating model may be cleared and then learned.

[0115] Then, the following process is repeated for each of the remaining candidates that are not calculated candidates, that is, each of the estimated candidates, among the plurality of candidates (step S222).

[0116] First, the parameter combination (and the estimation model type and specifications) represented by the estimation candidate is given as input to the trained rating model, and a rating value is estimated (step S223).

[0117] Next, the rating value stored in the candidate storage area in association with the inferred candidate is updated with the inferred rating value (step S224).

[0118] Whether a candidate is a calculated candidate or an estimated candidate, i.e., whether the rating value stored in the candidate storage area is a calculated value or an estimated value, can be identified by whether the estimation model for the candidate has been trained.

[0119] When the estimation of rating values ​​for all estimated candidates is completed (step S225), the proposing device 101 acquires the top candidates with the highest rating values ​​from the candidates stored in the candidate storage area (step S226), and returns control to step S203. Here, the simplest way is to select a predetermined number of candidates from the top when the candidates are sorted in descending order of rating value as the top candidates. In addition, the definition of "having a top rating value" may take into account the number of parameters included in each combination of candidates, a score obtained by weighting and adding the parameters included in each combination of candidates (for example, the drug price when a test related to the combination is performed), etc.

[0120] If the upper-level condition is satisfied (step S203; Yes), the output unit 104 outputs useful candidates based on the acquired upper-level candidates (step S231), and ends this process.

[0121] In the simplest case, the output unit 104 outputs the top candidates as useful candidates as they are.

[0122] In addition, instead of designating all of the top candidates as useful candidates, the top candidates may compete against each other in a round-robin format to further update their rating values, and then some of the top candidates with high rating values ​​after the round-robin format may be output as useful candidates.

[0123] Up to this point, the basic control flow of this embodiment has been explained, but below, an example in which this has been modified will be explained.

[0124] (Transformation Flow) Below, we will explain a modified example in which an upper limit k is set on the number of parameters represented by useful candidates, and the battle is held in order of the number of parameters. In this modified example, the timing of determining the higher-level conditions and calculation completion conditions in the basic flow can be thought of as being divided into multiple parts. Figure 4 is a flowchart showing the control flow of a modified example of the proposal process according to an embodiment of the present invention.

[0125] In this process, as in the basic flow, initialization (step S201) and candidate generation (step S202) are performed, and then the process is repeated in order for stages t = 1, 2, ..., k (step S301).

[0126] The proposed device 101 repeats the following process a number of times according to the stage t (step S302). The number of repetitions can be a constant regardless of the value of t. It is also possible to configure the number of repetitions to decrease as the value of t increases.

[0127] That is, the calculation unit 102 performs processing similar to steps S204 to S225 on the extracted evaluation candidates, limited to those having a combination of t parameters, to calculate or estimate the rating value of the candidate having t parameters (step S303).

[0128] Then, the removal unit 105 removes candidates with lower rating values ​​from the candidate storage area 122 among the candidates with the parameter number t (step S304), thereby narrowing down the candidates. The candidates to be removed may be limited to estimated candidates with lower rating values, or may be both calculated candidates and estimated candidates with lower rating values.

[0129] For example, when the remaining candidates with parameter number t are sorted in order of rating value, the candidates to be deleted may be those ranked up to a predetermined percentage (e.g., half, one-third, one-quarter, etc.) from the lowest rating. The percentage may also be adjusted according to the number of iterations for stage t.

[0130] After the repetition (step S305) is completed, a predetermined number (e.g., 3 to 5) of candidates with the highest rating values ​​are selected as top candidates from among the candidates with the parameter number t (step S306). In this embodiment, a round-robin competition of the top candidates is conducted later, so at this stage, even estimated candidates can be selected as top candidates.

[0131] After the repetition of each stage (step S307) is completed, a round-robin competition is conducted for the top candidates, and the rating values ​​of the top candidates are calculated (step S308). Through this process, all the top candidates become calculated candidates, and the estimation model is trained.

[0132] Next, among the top candidates, a predetermined number (for example, 10) of candidates with the highest rating values ​​are output as useful candidates (step S309), and this process ends.

[0133] In addition, if the estimation and updating of rating values ​​in step S303 is performed not only for the estimated candidates with parameter number t but also for the estimated candidates with parameter number (t+1), it becomes possible to appropriately adjust the rating values ​​of the candidates with parameter number (t+1) before the match takes place in stage (t+1).

[0134] Also, in step S304, the candidates to be removed may be those with the lowest rating value among the candidates with parameter number t or (t+1). In this case, it is possible to narrow down the candidates with parameter number (t+1) before the match is held in stage (t+1).

[0135] (Experimental Example 1) In the following, the label y is calculated based on the 13 parameters X1, X2, ..., X9, XA, XB, ..., XD. y = 1, if 0.5>X1*sin(X2) / X3 and X1*sin(X2) / X3>-0.1; = 0, otherwise Consider an object that is nonlinearly defined as follows. In this object, the label y depends on X1, X2, and X3, so the factor parameters are X1, X2, and X3.

[0136] Therefore, 300 records were prepared in which 13 parameters were set to random values ​​and the label y was determined by the above-mentioned judgment formula based on the random values, and typical hyperparameters were adopted for the estimation models of RF, Ridge, and Lasso types, with the upper limit of the number of parameters set to k=4, and applied to the above-mentioned embodiment. As a result, the following useful candidates were obtained from the records having the above-mentioned hidden associations.

[0137] The list below shows useful candidates such as "Estimation model type (factor parameters) ... rating value". RF(X1,X2,X3) ...... 1898.964893 RF(X1,X2) ..........1836.415016 Ridge(X1,X3,X9) ...... 1712.232568 Ridge(X1,X3)...... 1684.863008 Ridge(X1,X6,XC,XD) ... 1441.144343 Ridge(X1)..........1415.236034 Ridge(X1,X6,X8,XD) ... 1404.945883 Ridge(X1,X5,X9,XB) ... 1382.644057 Ridge(X1,X5,X7) ...... 1376.231578 RF(X2,X4) ..........1352.316014 RF(X2)..........1289.258478 Ridge(X3)..........1205.748127

[0138] The percentage of useful candidates that each parameter combination appears in is shown below. X1 ......... 75% X2 ......... 33.3% X3 ......... 33.3% X1,X3 ...... 25% X1,X2 ...... 16.7% X1,X5 ...... 16.7% X1,X6 ...... 16.7% X1,X9 ...... 16.7% X1,XD ...... 16.7% X1, XD, X6 ... 16.7% X5 ......... 16.7% X6 ......... 16.7% X9 ......... 16.7% XD ......... 16.7% XD,X6 ...... 16.7%

[0139] As described above, the parameter combination for the most highly rated useful candidate is X1, X2, and X3, which occupy the top positions in terms of parameter appearance rate. Therefore, the factor parameters are correctly proposed along with the appropriate type of estimation model, and the accuracy of this embodiment was confirmed.

[0140] (Experimental Example 2) Below, we will discuss the following for patients suspected of having prostate cancer: Dimensionally compressed 3D feature vector of prostate echo image, As a result of the above 13 blood tests, Whether or not a physician examines a biopsy specimen taken from a patient and determines whether the Gleason score is 8 or higher (whether or not the tumor is highly malignant) We prepared 87 records.

[0141] When it is desired to predict whether a patient has pathologically high-grade prostate cancer, the above embodiment suggests a blood test to be combined with a prostate ultrasound image.

[0142] The estimation model used representative hyperparameters, RF, Ridge, and Lasso, as in Experimental Example 1. The upper limit of the number of parameters was set to k=5, and the model was asked to suggest which blood tests should be performed.

[0143] The results are as follows: Each useful candidate is shown with an underlined list of the estimation model type ("RF"), the name of the blood test ("echo"), and a rating value. RF_echo_PSA_WBC_Hb........... 1635.710833 RF_echo_PSA_WBC .......... 1578.544444 RF_echo_PSA_Hb..........1575.473352 RF_echo_PSA_WBC_Hb_Alb_Ca.... 1532.266206 RF_echo_PSA_WBC_Hb_CRP ....... 1520.272607 RF_echo_PSA_WBC_Hb_Alb ....... 1516.125097 RF_echo_PSA_Hb_cre ..... 1509.518065 RF_echo_PSA_WBC_Hb_Plt ....... 1507.569392 RF_echo_PSA_Hb_GOT .......... 1502.329218 RF_echo_PSA_WBC_Hb_GPT_cre ... 1497.454263 PSA .......... 84% Hb ........... 72% PSA,Hb ....... 64% PSA, WBC ...... 48% PSA, WBC, Hb... 44%

[0144] Furthermore, to determine whether these test values ​​could predict pathologically high-grade cancer, the AUC values ​​were calculated 100 times using RF and the average values ​​were calculated. The following results were obtained: RF: Echo image (PSA, WBC, Hb) ... 0.828 RF: Echo image (PSA, Hb) ....... 0.818 RF: Echo image (PSA) ........... 0.795 RF: Echo image (Hb) ........... 0.687 RF: Echo image (WBC) ........... 0.622

[0145] In Experimental Example 2, the results were output in a scatter diagram. Figure 5 is an explanatory diagram showing an example of useful candidates being output in a scatter diagram. In the scatter diagram shown in this figure, the vertical axis represents cost, and the horizontal axis represents rating. Therefore, the cost and rating of a useful candidate are indicated by a point in the scatter diagram. The further to the bottom right a point is located in the scatter diagram, the better the cost performance of that useful candidate.

[0146] Thus, this Experimental Example 2 provided new insight that combining WBC (white blood cell count), Hb (hemoglobin), and PSA (tumor marker) is useful for prostate ultrasound images, i.e., that these are prognostic factors.

[0147] In Experimental Example 2, the training data used ultrasound images and pathological images (derived from Gleason scores) based on invasive examinations. On the other hand, predictions are made before invasive procedures. Therefore, ultrasound images are used for prediction, but pathological images are not required.

[0148] That is, in Experimental Example 2, a prostate ultrasound image is required and any blood test is to be combined, but other medical images including pathological photograph images are not to be combined.

[0149] In Experimental Example 2, the range of combinations is more limited than when proposing a combination of any medical image and any blood test, or when proposing a combination of the presence or absence of an ultrasound image and any blood test, and the amount and time of calculations are reduced. Generally, limiting the range of combinations is useful in that it reduces the amount and time of calculations required to obtain a proposed result.

[0150] For example, if only "combinations including certain fixed parameters" (certain fixed tests are always performed) are considered as initial candidates, the number of candidates can be reduced, and the calculation time and amount can be reduced.

[0151] As can be seen from Experimental Examples 1 and 2, in this embodiment, by adopting a rating model, it is possible to propose factor parameters with high accuracy while suppressing the number of matches.

[0152] Furthermore, for images that utilize feature parameters and feature vectors along with combinations of parameters such as blood test type, for example, the number of initial candidates when the combination includes only the presence or absence of an "echo image" is fewer than the number of initial candidates when the combination includes the presence or absence of an "echo image" and the presence or absence of a "pathological image," and if the "echo image" is limited to the presence, the number of initial candidates is further reduced. In this way, by always using a certain type of image in a combination, it is possible to reduce the calculation time and amount.

[0153] In addition, it is generally known that blood test values ​​vary between hospitals and between testing equipment, but by using predicted ratings estimated by deep learning, it is believed that the effects of overfitting in the estimation model are suppressed.

[0154] Furthermore, in this embodiment, various estimation models can be adopted, and appropriate hyperparameters can also be proposed.

[0155] Furthermore, according to this embodiment, highly accurate prediction is possible by combining feature parameters and feature vectors obtained from pathological photograph images and ultrasound photograph images and then applying them to an estimation model related to useful candidates.

[0156] Furthermore, according to this embodiment, data such as various medical images and various test results, which have often been handled separately until now, can be analyzed in a cross-sectional and integrated manner, making it possible to predict prognosis that is more suited to individual patients than ever before.

[0157] To the inventor's knowledge, this is the first technology to utilize artificial intelligence technology (machine learning, deep learning, etc.) to propose prognostic factor parameters for a patient's (subject's) disease based on the patient's (subject's) ultrasound image and various test results, and to make highly accurate disease prognosis predictions based on predicted values ​​(rating scores) whose usefulness has been verified.

[0158] (output to screen) As described above, the output unit 104 can output useful candidates on the screen in the form of a scatter diagram. An example of display on the screen will be described below.

[0159] The scatter plot may be output by the output unit 104 in a number of ways, such as being displayed on an individual's electronic medical record, on the analysis screen of an electronic medical record system, or on a database system that is not limited to an electronic medical record or an electronic medical record system.

[0160] FIG. 6 is an explanatory diagram showing an example of a display in which useful candidates are output within a scatter plot and displayed on the screen. This diagram shows an example in which electronic medical record information and various data sets are displayed on the screen, with the scatter plot displayed alongside other information. Additionally, oval marks entered by the doctor are superimposed on the scatter plot, indicating that the doctor has focused on the combination. Alternatively, a predetermined rating score and cost may be set in the proposed device in advance, and useful candidates within the set range may be automatically surrounded by oval marks on the electronic medical record screen, or useful candidates may be displayed in a distinguishable manner using a different color or mark from other candidates.

[0161] Figure 7 is an explanatory diagram showing an example of a display in which useful candidates are output and displayed on the screen within a scatter plot. This figure is an example of a display that allows a doctor to check more specific information about the information displayed in the scatter plot. When the doctor performs a predetermined operation, for example, by selecting an item from the MENU on the left side of the screen, the rating score of the combination displayed in the scatter plot or the proportion (percentage) included in the top combinations is displayed in a subwindow, allowing the doctor to check specific information.

[0162] By clicking on an area of ​​the scatter plot with the mouse that has high cost performance, the AUC (1-year prognosis prediction) of the combination of areas with high cost performance will be displayed in a subwindow, allowing you to check the prediction accuracy.

[0163] In addition, by performing a predetermined operation, it is possible to display the medical photograph images of the combination that the user wishes to check in a sub-window (not shown).

[0164] The various new interfaces (the scatter plots in Figures 6 and 7 and their information provision functions) provided by the highly accurate disease prognosis prediction technology, the usefulness of which has been verified, will be useful for doctors in making diagnoses to provide cost-effective and appropriate medication and treatment for specific patients (subjects).

[0165] (Examples to which this embodiment can be applied) In the above-mentioned Experimental Example 2, an example was explained in which medical images consisting of test results of a prostate cancer patient and ultrasound images were combined, but this embodiment can be applied to various aspects such as the following.

[0166] First, in this embodiment, the target patients may be those with general diseases other than cancer, including non-tumor diseases such as inflammatory diseases and degenerative diseases. For example, the present invention may be applied to autoimmune diseases, infectious diseases, wound (healing), neurodegenerative diseases, ulcerative colitis (a potentially applicable disease), new infectious diseases causing pandemics, and post-wound keloids.

[0167] Furthermore, the target patients include those with almost all cancers (including all malignant tumors, such as epithelial tumors, "cancer," as well as leukemia) including breast cancer, lung cancer, liver cancer, pancreatic cancer, bile duct cancer, esophageal cancer, stomach cancer, colon cancer, bladder cancer, kidney cancer, skin cancer, cervical cancer, uterine cancer, ovarian cancer, testicular cancer, sarcoma, blood tumors, brain tumors, and prostate cancer.

[0168] In the embodiment where various test results and feature parameters or feature vectors obtained by dimensionally compressing images are used, images can be electron microscope images, 3D tissue images using transparentized tissue, 3D cell images, etc. Also usable are cytology images, immunohistochemistry images, fluorescent staining images, and images obtained by a technique for detecting the distribution and amount of DNA or mRNA using in situ hybridization. In addition to various photographs taken before a procedure, various pathological images used in the pathology department after a procedure can also be used.

[0169] Furthermore, the image may be applied to images obtained by CT, MRI, PET, bone scintigraphy, plain X-ray, mammography, dermoscopy, fundus photography, corneal photography, etc., in addition to echographic images.

[0170] (Proposed system) An embodiment of a proposal system using the above-described proposal device will be described below. Fig. 8 is an explanatory diagram showing a schematic configuration of the proposal system according to an embodiment of the present invention.

[0171] As shown in this figure, in the proposed system 501, a big data server 502 and an external device 503 are connected via a computer communication network 504 such as the Internet.

[0172] The big data server 502 functions as the proposed device 101 described above, and also executes various data processing and data management.

[0173] The external device 503 is realized by a computer used in medical treatment / examination institutions such as hospitals, research / analysis institutions such as universities, etc. Each external device 503 may be configured by a single computer, or may be configured by a combination of a tissue server that acts as an intermediary between the big data server and the external device 503, and a terminal that serves as an interface between the tissue server and doctors, researchers, etc.

[0174] Test results and medical images of patients are stored in external devices 503 used in community hospitals and university hospitals. The records stored in these external devices 503 are anonymized and then transmitted to a big data server 502, where they are stored and managed in a database within the big data server 502.

[0175] The big data server 502 functions as the above-mentioned proposal device 101. Here, the proposal device 101 realized by the big data server 502 can function as follows: Fig. 9 is an explanatory diagram showing how information is sent and received in the proposal system according to an embodiment of the present invention.

[0176] That is, the proposal device 101 can receive (601) from the external device 503 the specifications of "parameters that can be acquired for the target" and "type of label."

[0177] For example, if there are restrictions on the tests that can be performed at a hospital or the like, the external device 503 can specify the tests that can be performed as "parameters that can be obtained for the subject" to the big data server 502.

[0178] The "type of label" varies depending on the purpose of the request by the hospital or the like, i.e., the purpose for which the system 501 is to be used. For example, if the purpose of the request is to predict the prognosis of cancer recurrence in one year, the label to be attached to the target would be "whether or not there will be one-year recurrence or the predicted recurrence value (a percentage representing the probability of recurrence)." The "type of label" related to the purpose of the request can also be specified from the external device 503 to the big data server 502. Note that if the purpose of the request is to predict the prognosis of cancer recurrence in three years, the label to be attached to the target would be "whether or not there will be three-year recurrence or the predicted recurrence value (percentage)," and various types of labels may be set based on the purpose of the request.

[0179] Upon receiving the above-mentioned designation, the big data server 502 functions as the above-mentioned proposing device 101 based on the designation, performs the proposing process (602), and outputs useful candidates and the like.

[0180] The big data server 502 may be equipped with models (useful candidates) that have been prepared in advance according to typical purposes by learning from various medical images and test results of a large number of patients (subjects). In this case, when the purpose of prognosis prediction is input, a pre-prepared model (useful candidate) that matches the purpose can be immediately output by reusing the results of past proposal processing.

[0181] In addition, several typical purposes, such as one-year prognosis prediction for prostate cancer recurrence and three-year prognosis prediction for prostate cancer, may be prepared, and prognosis prediction models corresponding to these typical purposes may be set as applications in the big data server 502. Users (doctors, etc.) may access the big data server 502 from the screen of the electronic medical record system and use these applications according to their purposes.

[0182] The useful candidates are transmitted from the big data server 502 to the external device 503 (603), and the external device 503 presents the performance and rating value of the useful candidates, as well as the number of parameters and total cost (604), which doctors and others can use as reference when deciding on future treatment and examination policies.

[0183] There may be cases where test values ​​(parameter values) and various medical photographs (images) have already been examined for a specific patient (subject). In such cases, when a doctor reviews the patient's electronic medical record, the following processing can be performed. Figure 10 is an explanatory diagram showing the transmission and reception of information in the proposed system according to an embodiment of the present invention.

[0184] That is, the external device 503 transmits the parameter values ​​and images already acquired for the target to the big data server 502 along with the type of test that can be performed (acquirable parameters) and the request content (type of label) (701).

[0185] In the big data server 502, useful candidates are output by the proposal process (702).

[0186] Also, calculation candidates to which the transmitted combination of acquired parameter values ​​and images can be applied, that is, calculation candidates representing partial combinations that are part or all of the combination of acquired parameters, are extracted (703).

[0187] In addition, in the case of outputting useful candidates prepared in advance, the calculated candidates created in the advance preparation are also cached, and extraction is performed from this cache, so that calculated candidates representing partial combinations that are part or all of the combinations of parameters that have already been acquired can be extracted in a short time.

[0188] Next, from the extracted estimation candidates, the estimation candidate with the best performance or rating value (hereinafter referred to as the "acquired candidate") is selected (704). The performance or rating value of the acquired candidate obtained here is transmitted to the external device 503 together with information on the useful candidates (705). The external device 503 presents the estimated performance or rating value based on the acquired parameter values ​​for the target to the requester (706).

[0189] Furthermore, following the above process, the big data server 502 can also provide the transmitted acquired parameter values ​​and images as input to the estimation model for the acquired candidate (707) to estimate the label.

[0190] The estimated label is transmitted to the external device 503 (708). The external device 503 presents the transmitted label to the requester as a label estimated based on the parameter values ​​already acquired for the target (709). The accuracy of the presented label can be known from the performance and rating values ​​of the already acquired candidates also presented to the requester (706).

[0191] Furthermore, the probability of being classified into that label can also be used as the certainty of the label. As described above, the estimation model calculates the probability of being positive (or negative) and compares that probability with a threshold to output a positive or negative label. Therefore, the probability of the label can be communicated and presented to the client along with the label. "Whether or not there will be a one-year recurrence" predicts which of the two labels the patient will fall into, and is obtained by comparing the probability output by the estimation model with a threshold. Furthermore, the probability output by the estimation model can be used directly as the "recurrence prediction value (a percentage representing the probability of recurrence)."

[0192] If the requester considers the accuracy (performance, rating value, etc.) of the acquired candidates to be sufficient, the labels (and their probabilities) presented here can be used as a reference when predicting the patient's prognosis.

[0193] If the requester considers that the performance of the acquired candidates is insufficient, the requester refers to the useful candidates output from the big data server 502 to the external device 503 and determines the additional tests to be performed.

[0194] To aid in this decision, the external device 503 can present to the doctor or other person the difference between the performance and rating value of the useful candidate and the performance and rating value of the acquired candidate, along with the additional costs of additional tests to be performed to apply the useful candidate, additional medical image capture, etc. The presentation of these additional test items, additional medical images, and their rating scores and costs may be displayed on the screen of an electronic medical record, electronic medical record system, database system, etc. as one of the information provision functions of the scatter diagram mentioned above. Details of this will be described later.

[0195] In addition, in the scatter diagram shown in Figure 7, Acquired candidates and Useful candidates with better performance and rating values ​​than the acquired candidates, but with higher parameter counts and costs, You can highlight the two to make it easier to compare them.

[0196] Then, doctors and other medical professionals can decide which additional tests to perform, taking into account the additional cost and the improvement in accuracy.

[0197] When new parameters for the target are acquired, the above process can be repeated. In other words, if the acquired parameter values ​​and images for the target are transmitted again from the external device 503 to the big data server 502, a label estimated with higher accuracy can be obtained from the big data server 502.

[0198] For ease of understanding, an example in which Experimental Example 2 is applied to this system 501 will be considered.

[0199] For a patient, if a prostate ultrasound image, PSA, and Hb tests have been performed, but a WBC test has not been performed, the candidates for the partial combination are as follows: RF_echo_PSA RF_echo_PSA_Hb RF_echo_Hb This becomes:

[0200] Of these, RF_echo_PSA_Hb and RF_echo_Hb are the top 10 useful candidates, but RF_echo_PSA_Hb has a higher rating. Also, RF_echo_PSA is not a useful candidate. RF_echo_PSA_Hb has the best performance.

[0201] Therefore, the acquired candidate is RF_echo_PSA_Hb, whose rating value is 1575.473352 and AUC value is 0.818.

[0202] A useful candidate with a better rating value and performance than the acquired candidate is RF_echo_PSA_WBC_Hb (rating value 1635.710833, AUC value 0.828). Therefore, doctors and other medical professionals can refer to the performance and rating values ​​of RF_echo_PSA_Hb, which is an acquired candidate, and RF_echo_PSA_WBC_Hb, which is a useful candidate.

[0203] In this example, by performing an additional WBC test, the number of tests increases by 1 and the cost increases by the drug price of the additional WBC test, but the rating value of the resulting prognosis prediction label increases by 1635.710833-1575.473352 = 60.237481 and the AUC value increases by 0.828-0.818 = 0.010.

[0204] Therefore, let's say that a doctor or other medical professional decides to perform an additional WBC test, taking into account the increased cost and the increased likelihood of the predicted results. In this case, the doctor can also consider the number of additional tests (1 in this example) and the additional cost (the drug price for the WBC test).

[0205] If additional tests are performed on the patient and WBC is obtained as an additional test value, this will be sent together with the previously tested test values ​​as a "newly acquired parameter value" from the external device 503 to the big data server 502. The acquired candidate will be RF_echo_PSA_WBC_Hb, and a label estimated for the patient with an AUC value of 0.828, i.e., material for more accurate prognosis prediction, will be obtained.

[0206] Note that data containing test results or pathological images that are unavailable at the time of prognosis prediction may be excluded from the list of useful candidates. For example, a data containing a total resection pathological image that is preoperative but only available after surgery is excluded from the list of useful candidates by the big data server 502's judgment based on the purpose of the request. In addition, if MRI is not available in a hospital on a remote island, data containing MRI images is excluded from the list of useful candidates by the big data server 502's judgment based on the hospital information of the requesting hospital for prognosis prediction.

[0207] In this way, in addition to proposing disease prognostic factor parameters based on the provided patient (subject) medical images and various test results, the function of suggesting candidate additional test items and additional medical images that have not yet been provided and that may have a higher predictive value (rating score) can further improve the accuracy of prognosis predictions for patients (subjects).In addition, by presenting the proposed additional test items and the additional costs associated with additional medical diagnoses, doctors can be provided with decision-making information that takes into account the patient's (subject's) situation, etc., when deciding on the necessity of additional tests and image acquisition, and the additional test items and additional medical images.

[0208] (Order of matches) In the above embodiment, at stage t (t = 1, 2, ..., k), candidates with the number of parameters related to the combination equal to t are extracted and competed against each other (see steps S302-S306). First, we match candidates with a parameter number of 1, Next, we match candidates with two parameters, Next, candidates with three parameters are matched against each other. Then, we matched candidates with four parameters, ... Next, we match candidates with (k-3) parameters, Next, candidates with the number of parameters (k-2) are matched against each other. Then, candidates with the number of parameters (k-1) are matched against each other, Finally, we match candidates with k parameters. The following order was adopted. However, this order can be changed arbitrarily.

[0209] In general, a permutation of integers from 1 to k, a1, a2, ..., a k At stage t, the number of parameters related to the combination is a t In the above example, the permutation can be a t = t. In this permutation, the elements are arranged in ascending order, starting from 1 and increasing by 1.

[0210] Alternatively, odd-numbered elements of the permutation may be arranged in ascending order, increasing by one from 1, and even-numbered elements of the permutation may be arranged in descending order, decreasing by one from k. That is, First, we match candidates with a parameter number of 1, Next, we match candidates with k parameters, Next, candidates with two parameters are matched against each other. Then, candidates with the number of parameters (k-1) are matched against each other, Next, we matched candidates with three parameters, Next, candidates with the number of parameters (k-2) are matched against each other. ... In this case, if k is even, for integers m = 1, 2, ... (where 2×m-1≦k and 2×m≦k), the permutations are as follows: a (2×m - 1) = m; a (2×m)= k - m + 1; It is defined as follows:

[0211] Alternatively, the elements of the permutation may be integers from 1 to k arranged randomly.

[0212] Depending on the application and training data, changing the order of matches as described above may encourage convergence to the same optimal solution rather than a local solution.

[0213] For example, by sandwiching the number of parameters from above and below rather than increasing the number of parameters by one in order, the convergence rate to the optimal solution improved from 75 percent to 90 percent.

[0214] In general, it is expected that convergence to the optimal solution will differ depending on the application and training data. Therefore, it is thought that processing with multiple different permutations will enable the search for the optimal solution without falling into a local optimum.

[0215] (Use multiple combinations of useful candidates and estimation models) In the above embodiment, the calculation candidates with the highest rating values ​​are output as useful candidates, but the term "highest" here does not mean just one, but can also mean a high-ranking group of candidates.

[0216] For example, the proposing device 101 may select a plurality of combinations of useful candidates and their estimation models from those with the highest rating values.

[0217] In the proposed system 501 , the external device 503 transmits the acquired parameter values ​​to the proposed device 101 .

[0218] In the proposing device 101, some or all of the transmitted parameter values ​​are given as input to each estimation model of the top combination, and each estimation model is made to output a label.

[0219] The obtained multiple labels are then transmitted to the external device 503. The user of the external device 503 can get a sense of how reliable the labeling in the proposed device 101 is by checking whether the labels output by the different estimation models all match, or if they differ, what the label distribution is.

[0220] For example, if it is rare for the multiple labels output for previous test values ​​to not match, and the label output for the current test value does not match, it can be assumed that the current test value contains an abnormal value.

[0221] Furthermore, if there are many cases where the labels output for test values ​​previously performed do not match, it can be assumed that the training data contains abnormal values.

[0222] Generally, since multimodal analysis involves many combinations, even if no major abnormal values ​​(testing errors or input errors) are detected in individual tests, a synergistic effect of anomalies may occur, in which a combination of minor abnormal values ​​may result in an abnormal prediction by chance. In this embodiment, risk management for the output of an estimation model can be performed by performing pattern analysis of the output of a top-performing estimation model.

[0223] At first glance, this method appears similar to conventional ensemble analysis in that it uses multiple estimation models. However, while conventional ensemble analysis aims to improve accuracy, this method differs in that it manages risk by analyzing different models from multiple perspectives.

[0224] For example, typically, within each estimation model, a value within a certain interval (for example, from 0 to 1) is output as an intermediate parameter, and a label is determined by comparing this with a threshold value in the estimation model.

[0225] Therefore, it is thought that it will be possible to analyze the pattern of the output by treating the intermediate parameters in multiple estimation models as a vector and detecting abnormal values ​​(outliers) of that vector.A variety of techniques can be applied to detect outliers, including statistics-based outlier detection such as the Generalized ESD method, distance-based outlier detection such as the Mahalanobis distance, and outlier detection based on density ratio estimation such as One Class SVM.

[0226] As a preliminary experiment, we selected several estimation models with the highest performance based on the training data, (1) Laboratory values ​​without abnormal values; (2) Test values ​​containing one abnormal value; (3) Test values ​​containing two abnormal values When outlier detection was performed on Regarding (1), 10% were mistakenly detected as outliers, Regarding (2), 40% of test values, including abnormal values, can be detected. Regarding (3), all test values, including abnormal values, were detected.

[0227] Therefore, this embodiment, which utilizes multiple estimation models, can be considered useful for warning of the possibility that test values ​​contain abnormal values.

[0228] (Prediction System) The following describes an embodiment in which part or all of the proposed system 501 functions as a prediction system that predicts the prognosis of a patient for medical professionals who treat the patient. Fig. 11 is an explanatory diagram showing the general configuration of a prediction system included in the proposed system according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0229] As described above, the prediction system 801 performs some or all of the functions of the proposal system 501 for medical professionals who treat patients, and is realized by sharing the following roles among devices such as personal computers used by medical professionals, local servers that work with the devices to locally manage electronic medical records and the like at medical institutions, and a big data server 502 that collects and analyzes data from medical institutions and provides various medical information, particularly the above-mentioned useful candidates and estimation models, or by one device performing all of the following roles. The terminals used by medical professionals and the local server that manages electronic medical records correspond to the above-mentioned external device 503. Furthermore, in this diagram, the prediction system 801 forms part of the proposal system 501, and the proposal device 101 itself is not included in the prediction system 801.

[0230] The prediction system 801 includes a test value presenting unit 802, a prediction candidate selecting unit 803, a prognosis estimating unit 804, and a prognosis presenting unit 805. Furthermore, the system may further include a recommendation presenting unit 806 and a similarity presenting unit 807 as optional elements.

[0231] Here, the test value presenting section 802 presents the test values ​​already obtained for the patient on a screen 911 of a terminal used by a medical professional attending to the patient.

[0232] Here, the acquired test value for a certain test item of a certain patient means the acquired parameter value for a certain parameter of a certain target in the above embodiment.

[0233] Furthermore, the test value presenting section 802 may further present photographic images such as medical photographic images and ultrasound photographic images taken of the patient.

[0234] 12 is an explanatory diagram showing the state of an electronic medical record presented in the prediction system according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0235] As shown in this figure, an electronic medical record 901 displayed on a terminal screen 911 displays the patient's name, patient number, and the like in an identification information field 902. The patient's test items and their test values ​​(or whether they have not yet been tested) are displayed in rows in a parameter field 904 and a parameter value field 905 in a parameter table 903. For test items that have been tested, the numerical values ​​of the test values ​​obtained are displayed, and for test items that have not yet been tested, a line is displayed indicating that the test has not yet been tested. Additionally, in the example shown in this figure, a photo field 906 can display ultrasound images, etc., taken of the patient through non-invasive procedures. After an invasive or surgical procedure is performed on the patient, a photo of the affected area or a photo of a specimen of the excised area can also be displayed in the photo field 906.

[0236] The various information displayed in the electronic medical record 901 is generally managed on a terminal used by a medical professional or a local server, but it can also be stored using a cloud service or entrusted to a big data server 502, etc.

[0237] For ease of understanding, the following describes an embodiment in which the subject is a patient suspected of having prostate cancer, various blood tests that can be administered to the patient at the medical institution are used as test items, and ultrasound photographs that are less physically demanding for the patient are used as photographic images, and the prognosis of the patient is estimated using an estimation model related to useful candidates and used as material for consideration by medical professionals. However, it goes without saying that the present invention can also be applied when it is desired to make various prognostic predictions or obtain information for judgment based on test values ​​of other diseases or other test items.

[0238] In the electronic medical record 901 of this example, a list box 908 for selecting a desired prediction and a button 909 for executing the selected prediction are provided. Note that the user interface for executing the prediction is not limited to the list box 908 and the button 909, as long as it performs the same functions as these.

[0239] In the example shown in this figure, the list box 908 provides various selectable candidate items for prognosis prediction, including "recurrence of prostate cancer after 1 year," "recurrence of prostate cancer after 3 years," and "recurrence of prostate cancer after 5 years."

[0240] Here, it is assumed that the medical professional selects "prostate cancer recurrence after one year" in the list box 908 and selects the button 909. Then, the prediction system 801 functions as follows.

[0241] That is, the predicted candidate selection unit 803 selects, from a plurality of useful candidates each representing a combination of test items, a useful candidate whose acquired test value satisfies the combination of test items related to the useful candidate, as a predicted candidate.

[0242] Meanwhile, the prognosis estimation unit 804 provides the acquired test values ​​that satisfy the combination related to the selected prediction candidate as input to the estimation model trained for the selected prediction candidate, thereby estimating the prognosis of the patient.

[0243] Furthermore, the prognosis presenting unit 805 presents the estimated prognosis on a screen 911 .

[0244] Hereinafter, a specific example will be described.

[0245] As described above, the proposing device 101 outputs a plurality of useful candidates, and the prediction candidate selecting unit 803 of the prediction system 801 selects a prediction candidate from these. Here, one useful candidate represents a combination of parameters, in this example, a combination of blood tests to be applied to a patient.

[0246] When all test values ​​for the blood tests related to the combination have been obtained, i.e., when the acquired test values ​​satisfy the combination of test items, the test values ​​that satisfy the combination of test items are input to the estimation model trained by the proposing device 101 for the valid candidate, and a label for the patient is output. This label estimates the prognosis of the patient.

[0247] In other words, a prediction candidate is a useful candidate that enables prognosis prediction using test values ​​obtained for test items that have already been administered to a patient. In other words, a combination of useful candidates corresponds to a subcombination of a combination of test items that have already been administered to a patient.

[0248] Consider the case of predicting the prognosis of prostate cancer recurrence one year from now. Here, recurrence includes not only the recurrence of symptoms after a complete cure, but also the persistence of the disease.

[0249] As described above, in the training data, if there is a recurrence after one year, the label "1" (Positive) can be assigned, and if there is no recurrence after one year, the label "0" (Negative) can be assigned.

[0250] In this case, when the patient's test values ​​are input as parameter values ​​to the estimation model, the neural network of the estimation model outputs a numerical value between 0 and 1. This output value is compared with a threshold value that has been trained together with the neural network.

[0251] If the output value is closer to "0", the label "0" will indicate that there will be no recurrence after one year, and if the output value is closer to "1", the label "1" will indicate that there will be a recurrence after one year. These are the predicted prognoses for the patient.

[0252] The output value compared with the threshold value can be considered to represent the likelihood of recurrence one year later, i.e., the output value can be considered to be the probability of recurrence one year later, or a numerical value that has a positive correlation with that probability.

[0253] As described above, the inputs to the estimation model can include parameter values ​​consisting of test values ​​for test items, as well as the patient's specifications (which may include, for example, gender, age, height, weight, whether or not the patient is a smoker, whether or not the patient is a drinker, and medical history), the patient's photographic image itself, or feature parameters or feature vectors obtained by dimensionally compressing the photographic image.

[0254] It also represents the tests that will be applied to the patient currently being treated by the medical professional.

[0255] As mentioned above, useful candidates represent a combination of test items that are useful for predicting the prognosis of a certain disease. In the above example, useful candidates are explained for patients suspected of having prostate cancer, using gender, age, ultrasound images of the affected area, and blood test values. In other words, in the above example, the following five points are listed as useful candidates and the performance of their estimation models. RF: Echo image (PSA, WBC, Hb) ... 0.828 RF: Echo image (PSA, Hb) ....... 0.818 RF: Echo image (PSA) ........... 0.795 RF: Echo image (Hb) ........... 0.687 RF: Echo image (WBC) ........... 0.622

[0256] For example, if only the blood test Hb has been applied to the patient, the one that can be used as a prediction candidate, that is, the combination of test items that satisfies only the Hb value, is one of the following. RF: Echo image (Hb) ........... 0.687

[0257] When only the PSA blood test is applied to a patient, the combination of test items that can be used as a prediction candidate, i.e., the combination of test items that satisfies the PSA value alone, is one of the following. RF: Echo image (PSA) ........... 0.795

[0258] 12, the patient has been tested for Hb and PSA blood tests, but not for WBC blood tests. In this case, the following three items can be used as prediction candidates, i.e., the combinations of Hb and PSA values ​​satisfy the test item combinations: RF: Echo image (PSA, Hb) ....... 0.818 RF: Echo image (PSA) ........... 0.795 RF: Echo image (Hb) ........... 0.687

[0259] In this example, the prediction candidate selection unit 803 performs the following depending on the embodiment: All combinations that satisfy the acquired test values ​​(in the above example, RF: echo image (PSA, Hb), RF: echo image (PSA), RF: echo image (Hb)) may be selected as prediction candidates, or Among the combinations that satisfy the acquired test values, the one with the highest performance (in the above example, one of RF: echo image (PSA, Hb)) may be selected as a prediction candidate. Of the combinations that satisfy the acquired test values, a predetermined number or a predetermined percentage of high-performance combinations (for example, if the predetermined number is the top two, then RF: echo image (PSA, Hb) and RF: echo image (PSA)) may be selected as prediction candidates.

[0260] For each of the selected prediction candidates, the prognosis estimation unit 804 provides the estimation model associated with the prediction candidate with the test values ​​(and the patient's specifications and photographic images) associated with the combination of the prediction candidates to estimate the prognosis.

[0261] When the estimation model runs on the big data server 502, the patient's test values ​​and the like are passed from an external device 503 such as a terminal or a local server to the big data server 502. Therefore, it is possible to configure the big data server 502 to take on the roles of the prediction candidate selection unit 803 and the prognosis estimation unit 804.

[0262] On the other hand, while training an estimation model requires the use of a high-performance computer such as the big data server 502, once a trained estimation model is obtained, it is also possible to operate the trained estimation model on an external device 503, such as a terminal or a local server, by obtaining the configuration settings of the estimation model from the big data server 502. In this case, the roles of the prediction candidate selection unit 803 and the prognosis estimation unit 804 can be entrusted to the external device 503, such as a terminal or a local server. Alternatively, the selection of prediction candidates by the prediction candidate selection unit 803 may be performed on the external device 503 side, but the estimation model itself may be prepared in the big data server 502, and the prognosis estimation unit 804 may be controlled by the external device 503 side to estimate a prognosis. The estimation model is surrounded by a dotted line in FIG. 11 to indicate that the estimation model may be included in the prediction system 801 or may be prepared outside the prediction system 801.

[0263] The prognosis estimated by the estimation model is presented on a screen 911. Fig. 13 is an explanatory diagram showing how the results of prognosis prediction are presented in an electronic medical record presented in a prediction system according to an embodiment of the present invention. The following description will be given with reference to this diagram.

[0264] In the example shown in this figure, the predicted prognosis result is displayed in a predicted result field 932 in a pop-up window 931 superimposed on the electronic medical record 901 shown in Fig. 12. Note that specific wording and the like are omitted from the illustration in this figure. Specific examples of information displayed in the predicted result field 932 will be described below.

[0265] The simplest prognostic prediction result is to simply answer "yes" or "no" for the selected "recurrence of prostate cancer after one year." Alternatively, prognostic information can be explained by text including the name and performance of the estimation model used, such as "According to the estimation model [RF echo image (PSA, Hb) / AUC 0.818], it is estimated that the patient is unlikely to have a recurrence of prostate cancer after one year."

[0266] When prognosis is estimated using multiple estimation models, all estimation models and estimated prognoses may be listed and explained.

[0267] In this case, the similarity display unit 807 displays the following on the screen 911: The reliability or risk of multiple prognoses estimated for the selected multiple prediction candidates, and Reliability or abnormality of previously obtained test results As related information relating to the above, the similarity or variability of a plurality of prognoses estimated for a plurality of selected prediction candidates may be presented.

[0268] If the multiple estimated prognoses all match (or match among a predetermined number of the top-performing estimation models among the prediction candidates), that is, if the multiple prognoses have little variation and high similarity, the top-performing estimation model and its performance can be displayed, and medical professionals can be informed of this by providing related information such as text such as "This estimation result matches the other two estimation models."

[0269] In this way, if the estimated prognosis does not vary, it can be predicted that the reliability of the estimated prognosis is high, the risk is low, and the test values ​​obtained for the patient are also highly reliable and do not contain any abnormal values.

[0270] On the other hand, if the estimated multiple prognoses have a large variance and low similarity, it is acceptable to inform medical professionals of this by providing related information such as text such as, "This estimated result does not match the other two estimation models, so retesting or additional testing is recommended." At this stage, it is desirable to also present the estimated multiple prognoses and the names and performance of the estimation models.

[0271] (Proposing new test items) In the above example, five useful candidates for predicting the prognosis of prostate cancer were obtained, but only two test values ​​were obtained: Hb and PSA. Therefore, of the useful candidates for the combination satisfied by these two test values, the one with the highest performance was "RF: echo image (PSA, Hb) ... AUC 0.818."

[0272] However, among the five useful candidates, there is one that outperforms "RF: echo image (PSA, Hb) ... AUC 0.818", namely "RF: echo image (PSA, WBC, Hb) ... AUC 0.828".

[0273] In the useful candidate "RF: echo image (PSA, WBC, Hb) ... AUC 0.828", the test item that is not satisfied by the already acquired test value is WBC.

[0274] Therefore, in this embodiment, A predicted candidate selection unit 803 selects, from the plurality of useful candidates, a useful candidate whose performance is superior to that of the selected predicted candidate, as a recommended candidate; The recommendation presentation unit 806 displays the following on the screen 911: Test items that are not covered by the test values ​​already obtained among the combinations related to the selected recommended candidates; Comparison information comparing the performance of the selected recommendation candidate with the performance of the selected prediction candidate; It may also be possible to present the following.

[0275] For example, in the above example, the information displayed in the prediction result field 932 is Information on test items that are not covered by the test values ​​already obtained among the combinations related to the selected recommendation candidates, such as "We propose WBC as an additional test." "By performing this additional test, the prognostic prediction performance will increase by 0.010, from 0.818 to 0.828. The drug cost of this test is XXXX." By adding this information, medical professionals can be advised to conduct additional testing.

[0276] (Suggestion on whether or not to have the treatment) The following describes an aspect of giving advice on whether or not to perform an invasive or surgical procedure such as surgery on the patient.

[0277] In the above example of prostate cancer, when a medical professional must consider whether or not to perform a surgical or invasive procedure on a patient suspected of having prostate cancer, the medical professional can receive advice based on the estimation model of the present invention.

[0278] In this embodiment, The estimation model estimates two prognoses: the prognosis when a patient undergoes a surgical or invasive procedure, and the prognosis when the patient does not undergo the procedure. The prognosis presentation unit 805 displays the following on the screen 911: Two estimated prognoses, and Whether there is a difference between the two estimated prognoses Present at least one of the following: Configure it as follows.

[0279] For this reason, in the above-mentioned proposed device 101, proposed system 501, and prediction system 801, whether or not the patient has undergone a specified treatment (particularly, an invasive or surgical procedure) is added as an input to the estimation model in addition to the aspects of the above-mentioned embodiments.

[0280] Then, in the learning stage of the estimation model, for the training cases that received the treatment, Parameter values ​​of the patient related to the training case (which may include specifications such as the patient's gender and an echo image of the patient), A parameter indicating that the patient has undergone treatment; is the input data, Labels for classifying 1-year recurrence prognosis Prepare training data with the above as output data (correct answer data), and for training cases that have not received the treatment, the same parameter values ​​as above for the patient in the training set; A parameter indicating that the patient did not receive treatment; is the input data, Labels for classifying 1-year recurrence prognosis Let the training data be the output data (correct answer data).

[0281] Then, as in the above embodiment, the estimation model is trained, candidate verbs related to the estimation model are compared, and useful candidates related to the prediction are selected.

[0282] In the prediction stage, when medical professionals respond to patients suspected of having prostate cancer, the estimated model is Enter the patient's parameter values ​​and the fact that they will undergo treatment to estimate their prognosis (prognosis with treatment). Along with Enter the parameter values ​​of the patient and the fact that they will not undergo treatment to estimate the prognosis (prognosis without treatment). It shall be so decided.

[0283] Then, when the prognosis with treatment and the prognosis without treatment match, medical personnel can obtain information to determine that treatment is not necessary.

[0284] Furthermore, if the prognosis with and without treatment do not match, and the prognosis with treatment is better, medical professionals will have information to help them decide that it would be better to perform the treatment.

[0285] Furthermore, if the prognosis with and without treatment do not match, and the prognosis without treatment is better, medical professionals can obtain information to determine that treatment should not be performed.

[0286] Therefore, based on whether the two estimated prognoses with and without treatment match, the prognosis presenting unit 805 notifies the medical staff as information to be displayed in the prediction result field 932 on the screen 911, "The one-year prognosis is the same regardless of whether invasive or surgical treatment is performed, so there is little need for such treatment." "The prognosis after one year is better with invasive or surgical treatment than without it, so it is considered a good idea to do so." "The prognosis after one year is better than without invasive or surgical treatment, so this treatment should not be performed." Advice can be given through texts such as:

[0287] It should be noted that the above advice may not be presented in the pop-up window 931 but may be displayed in a form other than the pop-up window 931, or may be cited and inserted into the doctor's findings column in the electronic medical record 901 so that the doctor can refer to it as needed.

[0288] (Prediction processing) The following describes the prediction processing executed by the prediction system 801. Fig. 14 is a flowchart showing the control flow of the prediction processing executed by the prediction system according to an embodiment of the present invention. The following description will be made with reference to this figure.

[0289] When this process starts, the prediction system 801 first displays the electronic medical record 901, including the test values ​​and the like already acquired for the patient, on the screen 911 and provides it to the medical staff (step S951).

[0290] Then, the prediction system 801 receives a selection of a desired prediction from the medical worker (step S952).

[0291] Next, the prediction system 801 acquires a plurality of useful candidates related to the selected prediction from the proposing device 101 or from a set of useful candidates prepared in advance in the prediction system 801 (step S953).

[0292] Furthermore, the prediction system 801 selects a predicted candidate from the acquired useful candidates based on the test values ​​already acquired for the patient (step S954).

[0293] Then, for each of the selected prediction candidates (step S955), the prediction system 801 inputs the patient's acquired test values ​​as input to the estimation model associated with the prediction candidate, and estimates the prognosis (step S956). This process is repeated (step S957). Note that in step S956, prognosis with and without treatment may be estimated separately.

[0294] Next, the prediction system 801 generates advice to be provided to a medical professional based on the estimated prognosis (step S958).

[0295] The advice generated here includes, as mentioned above, Estimation results using the best-performing estimation model The estimation results of each of the multiple estimation models, Variability and similarity of multiple estimation models, as well as related information on the reliability, risk, and presence of outliers of predictions and test values Does the prognosis change depending on whether or not the treatment is performed? The information can be selected and included depending on the purpose.

[0296] Furthermore, the prediction system 801 checks whether there is a useful candidate that has better performance than the predicted candidate with the highest performance, that is, whether there is a recommended candidate (step S959). If there is no recommended candidate (step S959; No), the process proceeds to step S962.

[0297] On the other hand, if there are recommended candidates (step S959; Yes), the test items to be added are obtained by excluding the test items of the predicted candidates with the highest performance from the recommended candidate test items (step S960). Additional inspection items, How much will the performance of the estimation model improve by adding that test item? Costs incurred by carrying out the test items The above information is selected and recommended depending on the purpose and added to the advice (step S961).

[0298] Finally, the prediction system 801 presents the advice on the screen 911 (step S962), and ends this process.

[0299] (summary) As described above, the present invention is a novel method that has never been conceived before, which was arrived at by the inventors through their ingenuity and ingenuity, and through the use of artificial intelligence technology, in order to solve the above-mentioned problems. When this novel method is used for predicting the prognosis of diseases such as cancer, it has the great effect of providing knowledge for appropriate medication and treatment with high cost performance.

[0300] As described above, the proposal device according to this embodiment is a proposal device that proposes, based on a plurality of records, one or more candidates from among a plurality of candidates each representing a combination of parameters that can be obtained for an object, as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records includes a plurality of parameter values ​​respectively obtained by obtaining a plurality of parameters for the object associated with the respective record, and a label assigned to the associated object; Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the trained estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation Department, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. Estimation department, an output unit that outputs, as the useful candidate, a calculated candidate associated with a rating value that is higher among the calculated or estimated rating values; The device is configured to include the following.

[0301] In addition, in the proposed device according to this embodiment, Each of the plurality of candidates further indicates a type and specifications of an estimation model; The calculation unit learns an estimation model related to the type and specifications represented by each of the calculation candidates, The output unit further outputs an estimation model trained for the useful candidate. It can be configured as follows.

[0302] In addition, in the proposed device according to this embodiment, The estimated candidate associated with the highest rating value among the calculated or estimated rating values ​​is set as the calculated candidate, and then the calculation by the calculation unit and the estimation by the estimation unit are repeated, and then the output is performed by the output unit. It can be configured as follows.

[0303] In addition, in the proposed device according to this embodiment, a removal unit that removes at least the estimated candidate having the lowest associated rating value from the plurality of candidates; Furthermore, In the repeated processing, the removal unit further removes the It can be configured as follows.

[0304] In addition, in the proposed device according to this embodiment, training data for learning the estimation model is generated from training records randomly selected from the plurality of records; test data for calculating the performance of the learned estimation model is generated from test records that are the remainder after removing the training records from the plurality of records; The training data and the test data are parameter values ​​of the parameters related to the combinations represented by the respective calculation candidates among the parameter values ​​in the training records and the test records are used as input data; The labels in the training record and the test record are used as output data. It can be configured as follows.

[0305] In addition, in the proposed device according to this embodiment, Each of the records further includes a feature parameter or a feature vector obtained by dimensionally compressing an image of an object associated with each of the records, The feature parameters or feature vectors are included in the input data for the estimation model. It can be configured as follows.

[0306] In addition, in the proposed device according to this embodiment, The calculation unit, based on a rating system, determining performance for two or more candidates randomly selected from the plurality of candidates by sharing the training record and the test record; determining a match result between the two or more candidates based on the determined performance; updating the rating values ​​of the plurality of candidates based on the determined match results; Repeat the match process It can be configured as follows.

[0307] In addition, in the proposed device according to this embodiment, The calculation unit preferentially extracts, from the plurality of candidates, a candidate for which an estimation model has not been learned or a candidate for which learning has been performed a small number of times, and sets the extracted candidate as the calculation candidate. It can be configured as follows.

[0308] In addition, in the proposed device according to this embodiment, The rating system is one of Elo Rating, Glicko Rating, Glicko 2 Rating, and TrueSkill™. It can be configured as follows.

[0309] In addition, in the proposed device according to this embodiment, where n is the number of the plurality of parameters, the combination is represented by an n-dimensional binary vector; The rating model is When a binary vector representing the combination represented by the estimated candidates is input, Outputting a rating value for the putative candidate It can be configured as follows.

[0310] In addition, in the proposed device according to this embodiment, In the suggestion device, the number of parameters related to the combinations represented by the useful candidates is k or less, and a permutation a1, a2, ..., a k For stages t = 1, 2, ..., k, In the battle processing, candidates having a number of parameters related to a combination of t are extracted as the two or more candidates from the plurality of candidates, and the candidates are subjected to a battle; After repeating the battle process, the estimation unit and the elimination unit narrow down the plurality of candidates. It can be configured as follows.

[0311] In addition, in the proposed device according to this embodiment, For each candidate of the plurality of candidates, the rating value estimated at stage t is set as the initial rating value at stage (t+1). It can be configured as follows.

[0312] In addition, in the proposed device according to this embodiment, In the step t, the removal unit removes candidates having a number of parameters related to a combination of t or (t+1) and having a low estimated rating value; The output unit At stage t = 1, 2, ..., k, a candidate having a combination of t parameters and having a high updated or estimated rating value is selected; The calculation unit causes a round-robin battle between the extracted candidates. It can be configured as follows.

[0313] In addition, in the proposed device according to this embodiment, The elements of the permutation are arranged in ascending order, starting from 1 and increasing by 1. It can be configured as follows.

[0314] In addition, in the proposed device according to this embodiment, The odd-numbered elements of the permutation are arranged in ascending order, starting from 1 and increasing by 1, and the even-numbered elements of the permutation are arranged in descending order, starting from k and decreasing by 1. It can be configured as follows.

[0315] In addition, in the proposed device according to this embodiment, The elements of the permutation are the integers from 1 to k arranged randomly. It can be configured as follows.

[0316] In addition, in the proposed device according to this embodiment, the output unit outputs the useful candidates by plotting them on a scatter plot; a first axis of the scatter plot indicates the number of parameters related to the combination represented by the useful candidates or the total cost for obtaining the parameters related to the combination; The second axis of the scatter plot indicates the performance or rating value calculated for the useful candidate. It can be configured as follows.

[0317] In addition, in the proposed device according to this embodiment, the subject is a patient; the plurality of parameters are a plurality of test items applicable to the patient; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the patient, the label identifies a prognosis for the patient; The factor parameters are prognostic factors that contribute to predicting the prognosis of the patient. It can be configured as follows.

[0318] In addition, in the proposed device according to this embodiment, the subject is a cancer patient, the plurality of parameters are a plurality of test items applicable to the cancer patient, including a plurality of blood tests applicable to the cancer patient and a score or classification determination based on medical photograph images; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the cancer patient, the factor parameter is a prognostic factor that contributes to predicting the prognosis of the cancer patient; the image is a medical photograph image taken of the cancer patient; The label indicates whether or not the cancer patient has recurred. It can be configured as follows.

[0319] In addition, in the proposed device according to this embodiment, the subject is a patient suspected of having cancer; the plurality of parameters being a plurality of blood tests applicable to the patient; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the patient, The factor parameter is a factor that contributes to predicting whether the patient is suffering from a high-grade malignant tumor; the image is an echographic image taken of the patient; The labels in the plurality of records are scores or classification decisions based on medical specimens. It can be configured as follows.

[0320] The proposal system according to this embodiment includes an external device and the proposal device described above. The external device is Specifying the obtainable parameters to the proposed device; The proposed device is outputting the useful candidates to the external device; The external device is Accepts a combination of parameters already obtained for the target, Regarding the output useful candidates, Among the parameters related to the combination represented by the useful candidates, parameters that are not the accepted acquired parameters (hereinafter referred to as "unacquired parameters") Present Configure it as follows.

[0321] In addition, in the proposed system of this embodiment, The external device is Regarding the output useful candidates, The number of unobtained parameters; and An additional cost for acquiring the unacquired parameters; The performance or rating value calculated for the useful candidate; and Furthermore, present at least one of the following: It can be configured as follows.

[0322] In addition, in the proposed system of this embodiment, The external device is further specifying the combination of the received acquired parameters to the proposing device; The proposed device is Further outputting to the external device a candidate (hereinafter referred to as an "acquired candidate") with the best calculated performance or rating value among the calculated candidates representing subcombinations included in the specified combination, and the performance or rating value of the acquired candidate; The external device is Further presenting the output performance or rating values ​​for the acquired candidates. It can be configured as follows.

[0323] In addition, in the proposed system of this embodiment, The proposed device is If the acquired candidate is not the calculated candidate, an estimation model for the acquired candidate is learned to calculate performance; Further outputting the calculated performance of the acquired candidate to the external device; The external device is Further presenting the output performance for the acquired candidates. It can be configured as follows.

[0324] In addition, in the proposed system of this embodiment, The external device is The performance or rating value calculated for the useful candidate is presented as a difference from the performance or rating value output for the acquired candidate. It can be configured as follows.

[0325] In addition, in the proposed system of this embodiment, When the external device specifies the obtainable parameters, or after the unobtained parameters are presented and new parameters are obtained for the target, transmitting parameter values ​​of the parameters already acquired for the target to the suggestion device; The proposed device is a part or all of the transmitted parameter values ​​are provided as input data to an estimation model for the useful candidate, and a label to be assigned to the object is estimated; communicating the estimated label to the external device; The external device is Present the transmitted label It can be configured as follows.

[0326] In addition, in the proposed system according to this embodiment, The proposed device is outputting a plurality of combinations of the useful candidates and the estimation models trained for the useful candidates; estimating a plurality of labels to be assigned to the object by providing a part or all of the transmitted parameter values ​​as input data to each estimation model of the plurality of combinations; communicating the estimated labels to the external device; The external device is Presenting the transmitted plurality of labels It can be configured as follows.

[0327] In addition, in the proposed system according to this embodiment, the external device is a terminal for a medical professional attending to a patient, the parameters are test items applicable to the patient; the acquired parameters are test items applied to the patient; the unobtained parameters are test items that have not yet been applied to the patient; the parameter value is a test value to be obtained by administering the test item to the patient; the prognosis of the patient is classified according to the label; The terminal displays on a screen the test values ​​already acquired for the patient; the proposing device or the terminal selects, from the plurality of useful candidates, a useful candidate whose acquired test value satisfies a combination of test items related to the useful candidate, as a predicted candidate; the prognosis prediction device provides the acquired test values ​​that satisfy the combination related to the selected prediction candidates as inputs to an estimation model trained for the selected prediction candidates, thereby estimating the prognosis of the patient; The terminal displays the estimated prognosis on the screen. It can be configured as follows.

[0328] The proposed system according to this embodiment is a prediction system for medical professionals who deal with patients, a test value display unit that displays test values ​​already acquired for the patient on a screen; a prediction candidate selection unit that selects, from a plurality of useful candidates each representing a combination of test items, a useful candidate whose acquired test value satisfies a combination of test items related to the useful candidate, as a prediction candidate; a prognosis estimation unit that estimates the prognosis of the patient by providing the acquired test values ​​that satisfy the combination related to the selected prediction candidate as input to an estimation model trained for the selected prediction candidate; a prognosis display unit that displays the estimated prognosis on the screen; The device is configured to include the following.

[0329] In addition, in the proposal system (prediction system) according to this embodiment, Each useful candidate of the plurality of valid candidates further represents a performance of the estimation model trained for the useful candidate; the predicted candidate selection unit selects, from the plurality of useful candidates, a useful candidate whose performance is superior to that of the selected predicted candidate, as a recommended candidate; On the screen, Test items among the combinations related to the selected recommendation candidates that are not covered by the acquired test values; comparison information comparing the performance of the selected recommendation candidate with the performance of the selected prediction candidate; Recommendation section that presents The device may be configured to further include:

[0330] In addition, in the proposal system (prediction system) according to this embodiment, the selected prediction candidates are plural, On the screen, A plurality of prognostic reliabilities or risks estimated for the selected plurality of prediction candidates; and The reliability or abnormality of the test results obtained a similarity display unit that displays similarities or variations of the plurality of prognoses estimated for the plurality of selected prediction candidates as related information related to the The device may be configured to further include:

[0331] In addition, in the proposal system (prediction system) according to this embodiment, the test value presenting unit further presents a photographic image taken of the patient; The estimation model is further provided with feature parameters or a feature vector obtained by dimensionally compressing the photographic image as the input. It can be configured as follows.

[0332] In addition, in the proposal system (prediction system) according to this embodiment, the patient is a patient suspected of having cancer, the plurality of test items are a plurality of blood tests applicable to the patient; The photographic image is a medical photographic image or an echographic image. It can be configured as follows. 22. The proposal system according to claim 21.

[0333] In addition, in the proposal system (prediction system) according to this embodiment, The estimation model estimates two prognoses: a prognosis when a surgical or invasive procedure is performed on the patient, and a prognosis when the procedure is not performed on the patient; The prognosis presenting unit The two estimated prognoses, and Is there a difference between the two estimated prognoses? Present at least one of the following: It can be configured as follows.

[0334] The proposed method according to this embodiment is as follows: A proposal method in which a proposal device proposes, based on a plurality of records, one or more candidates from among a plurality of candidates each representing a combination of parameters obtainable for an object as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records has a plurality of parameter values ​​respectively obtained by obtaining a plurality of parameters for an object associated with the respective record, and a label assigned to the associated object; Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the learned estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation process, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. estimation process, an output step of outputting, as the useful candidate, a calculated candidate associated with a higher rating value among the calculated or estimated rating values; The device is configured to include the following.

[0335] A program according to this embodiment is a program that causes a computer to propose, based on a plurality of records, one or more of a plurality of candidates each representing a combination of parameters that can be obtained for an object, as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records has a plurality of parameter values ​​obtained by obtaining a plurality of parameters for the object associated with each record, and a label assigned to the associated object, and the program causes the computer to: Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the learned estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation Department, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. Estimation department, an output unit that outputs, as the useful candidate, a calculated candidate associated with a rating value that is higher among the calculated or estimated rating values; It can be configured to function as

[0336] The program according to the present embodiment can be distributed or sold by recording it on a non-transitory computer-readable information recording medium, or via a temporary transmission medium such as a computer communication network.

[0337] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of the present invention. This application claims priority based on patent application No. 2022-155487, filed in Japan on Wednesday, September 28, 2022 (Reiwa 4), and patent application No. 2023-86586, filed in Japan on Friday, May 26, 2023 (Reiwa 5), ​​and the contents of said basic applications are incorporated into this application to the extent permitted by the laws and regulations of the designated countries. [Industrial Applicability]

[0338] According to the present invention, it is possible to provide a proposal device, a proposal method, a proposal system, a program, and an information recording medium that propose factor parameters for estimating the label of an object from among multiple parameters that can be obtained for the object. [Explanation of symbols]

[0339] 101 Proposed device 102 Calculation Department 103 Estimation part 104 Output section 105 Removal section 121 databases 122 Candidate storage 501 Proposal System 502 Big Data Server 503 External device 504 Computer Communication Network 801 Prediction System 802 Test value presentation unit 803 Prediction Candidate Selection Unit 804 Prognosis Estimation Department 805 Prognosis Presentation Department 806 Recommendation section 807 Similarity presentation section 901 Electronic Medical Records 902 Identification information field 903 Parameter Table 904 Parameter column 905 Parameter value field 906 Photo column 908 List Box 909 Button 911 Screen 931 Pop-up Windows 932 Prediction result column

Claims

1. A suggestion device that proposes, based on a plurality of records, one or more candidates from a plurality of candidates each representing a combination of parameters obtainable for an object as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records includes a plurality of parameter values ​​respectively obtained by obtaining a plurality of parameters for an object associated with the respective record, and a label assigned to the associated object; Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the learned estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation Department, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. Estimation department, an output unit that outputs, as the useful candidate, a calculated candidate associated with a rating value that is higher among the calculated or estimated rating values; A proposed device comprising:

2. Each of the plurality of candidates further indicates a type and specifications of an estimation model; The calculation unit learns an estimation model related to the type and specifications represented by each of the calculation candidates, The output unit further outputs an estimation model trained for the useful candidate.

2. The proposal device according to claim 1,

3. The estimated candidate associated with the highest rating value among the calculated or estimated rating values ​​is set as the calculated candidate, and then the calculation by the calculation unit and the estimation by the estimation unit are repeated, and then the output is performed by the output unit.

2. The proposal device according to claim 1,

4. a removal unit that removes at least the estimated candidate having the lowest associated rating value from the plurality of candidates; Furthermore, In the repeated processing, the removal unit further removes the 4. The proposal device according to claim 3.

5. training data for learning the estimation model is generated from training records randomly selected from the plurality of records; test data for calculating the performance of the learned estimation model is generated from test records that are the remainder after removing the training records from the plurality of records; The training data and the test data are parameter values ​​of the parameters related to the combinations represented by the respective calculation candidates among the parameter values ​​in the training records and the test records are used as input data; The labels in the training record and the test record are used as output data.

5. The proposal device according to claim 4.

6. Each of the records further includes a feature parameter or a feature vector obtained by dimensionally compressing an image of an object associated with each of the records, The feature parameters or feature vectors are included in the input data for the estimation model.

6. The proposal device according to claim 5.

7. The calculation unit, based on a rating system, determining performance for two or more candidates randomly selected from the plurality of candidates by sharing the training record and the test record; determining a match result between the two or more candidates based on the determined performance; updating the rating values ​​of the plurality of candidates based on the determined match results; Repeat the match process 6. The proposal device according to claim 5.

8. The calculation unit preferentially extracts, from the plurality of candidates, a candidate for which an estimation model has not been learned or a candidate for which learning has been performed a small number of times, and sets the extracted candidate as the calculation candidate.

8. The proposal device according to claim 7.

9. In the proposed device, where n is the number of the plurality of parameters, the combination is represented by an n-dimensional binary vector; The rating model is When a binary vector representing the combination represented by the estimated candidates is input, outputting a rating value for the putative candidate; The number of parameters related to the combinations represented by the useful candidates is k or less, and a permutation a of integers from 1 to k is 1 , a 2 , ..., a k For stages t = 1, 2, ..., k, In the battle processing, candidates having a number of parameters related to a combination of t are extracted as the two or more candidates from the plurality of candidates, and the candidates are subjected to a battle; After repeating the battle process, the estimation unit and the elimination unit narrow down the plurality of candidates.

8. The proposal device according to claim 7.

10. the output unit outputs the useful candidates by plotting them on a scatter plot; a first axis of the scatter plot indicates the number of parameters related to the combination represented by the useful candidates or the total cost for obtaining the parameters related to the combination; The second axis of the scatter plot indicates the performance or rating value calculated for the useful candidate.

2. The proposal device according to claim 1,

11. the subject is a patient; the plurality of parameters are a plurality of test items applicable to the patient; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the patient, the label identifies a prognosis for the patient; The factor parameters are prognostic factors that contribute to predicting the prognosis of the patient.

2. The proposal device according to claim 1,

12. the subject is a cancer patient, the plurality of parameters are a plurality of test items applicable to the cancer patient, including a plurality of blood tests applicable to the cancer patient and a score or classification determination based on medical photograph images; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the cancer patient, the factor parameter is a prognostic factor that contributes to predicting the prognosis of the cancer patient; the image is a medical photograph image taken of the cancer patient; The label indicates whether or not the cancer patient has recurred.

7. The proposal device according to claim 6.

13. the subject is a patient suspected of having cancer; the plurality of parameters are a plurality of test items including a blood test applicable to the patient; the plurality of parameter values ​​are test result values ​​obtained by administering the plurality of test items to the patient, The factor parameter is a factor that contributes to predicting whether the patient is suffering from a high-grade malignant tumor; the image is an echographic image taken of the patient; The labels in the plurality of records are scores or classification decisions based on medical specimens.

7. The proposal device according to claim 6.

14. A proposal system comprising an external device and the proposal device according to claim 2, The external device is Specifying the obtainable parameters to the proposed device; The proposed device is outputting the useful candidates to the external device; The external device is Accepts a combination of parameters already obtained for the target, Regarding the output useful candidates, Among the parameters related to the combination represented by the useful candidates, parameters that are not the accepted acquired parameters (hereinafter referred to as "unacquired parameters") Present The proposed system is characterized by:

15. The external device is Regarding the output useful candidates, The number of unobtained parameters; and An additional cost for acquiring the unacquired parameters; The performance or rating value calculated for the useful candidate; and Furthermore, present at least one of the following:

15. The proposal system according to claim 14.

16. The external device is further specifying the combination of the received acquired parameters to the proposing device; The proposed device is Further outputting to the external device a candidate (hereinafter referred to as an "acquired candidate") with the best calculated performance or rating value among the calculated candidates representing subcombinations included in the specified combination, and the performance or rating value of the acquired candidate; The external device is Further presenting the output performance or rating values ​​for the acquired candidates.

16. The proposal system according to claim 15,

17. The external device transmits parameter values ​​of the parameters already acquired for the target to the suggestion device; The proposed device is outputting a plurality of combinations of the useful candidates and the estimation models trained for the useful candidates; estimating a plurality of labels to be assigned to the object by providing a part or all of the transmitted parameter values ​​as input data to each estimation model of the plurality of combinations; communicating the estimated labels to the external device; The external device is Presenting the transmitted plurality of labels 17. The proposal system according to claim 16,

18. 18. The proposal system according to claim 17, the external device is a terminal for a medical professional attending to a patient, the parameters are test items applicable to the patient; the acquired parameters are test items applied to the patient; the unobtained parameters are test items that have not yet been applied to the patient; the parameter value is a test value to be obtained by administering the test item to the patient; the prognosis of the patient is classified according to the label; The terminal displays on a screen the test values ​​already acquired for the patient; the proposing device or the terminal selects, from the useful candidates, a useful candidate whose acquired test value satisfies a combination of test items related to the useful candidate, as a predicted candidate; the prognosis prediction device provides the acquired test values ​​that satisfy the combination related to the selected prediction candidates as inputs to an estimation model trained for the selected prediction candidates, thereby estimating the prognosis of the patient; The terminal displays the estimated prognosis on the screen. The proposed system is characterized by:

19. 1. A recommendation system for a medical professional serving a patient, comprising: a test value display unit that displays test values ​​already acquired for the patient on a screen; a prediction candidate selection unit that selects, from a plurality of useful candidates each representing a combination of test items, a useful candidate whose acquired test value satisfies a combination of test items related to the useful candidate, as a prediction candidate; a prognosis estimation unit that estimates the prognosis of the patient by providing the acquired test values ​​that satisfy the combination related to the selected prediction candidate as input to an estimation model trained for the selected prediction candidate; a prognosis display unit that displays the estimated prognosis on the screen; A proposal system comprising:

20. Each useful candidate of the plurality of useful candidates further represents a performance of the estimation model trained for the each useful candidate; the predicted candidate selection unit selects, from the plurality of useful candidates, a useful candidate whose performance is superior to that of the selected predicted candidate, as a recommended candidate; On the screen, Test items among the combinations related to the selected recommendation candidates that are not covered by the acquired test values; comparison information comparing the performance of the selected recommendation candidate with the performance of the selected prediction candidate; Recommendation section that presents 20. The proposal system of claim 19, further comprising:

21. the selected prediction candidates are plural, On the screen, A plurality of prognostic reliabilities or risks estimated for the selected plurality of prediction candidates; and The reliability or abnormality of the test results obtained a similarity display unit that displays similarities or variations of the plurality of prognoses estimated for the plurality of selected prediction candidates as related information related to the 20. The proposal system of claim 19, further comprising:

22. the test value presenting unit further presents a photographic image taken of the patient; The estimation model is further provided with feature parameters or a feature vector obtained by dimensionally compressing the photographic image as the input.

20. The proposal system according to claim 19,

23. the patient is a patient suspected of having cancer, the plurality of test items are a plurality of blood tests applicable to the patient; The photographic image is a medical photographic image or an echographic image.

23. The proposal system of claim 22.

24. The estimation model estimates two prognoses: a prognosis when a surgical or invasive procedure is performed on the patient, and a prognosis when the procedure is not performed on the patient; The prognosis presenting unit The two estimated prognoses, and Is there a difference between the two estimated prognoses? Present at least one of the following:

24. The proposal system according to claim 23.

25. A proposal method in which a proposal device proposes, based on a plurality of records, one or more candidates from among a plurality of candidates each representing a combination of parameters obtainable for an object as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records has a plurality of parameter values ​​respectively obtained by obtaining a plurality of parameters for an object associated with the respective record, and a label assigned to the associated object; Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the trained estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation process, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. estimation process, an output step of outputting, as the useful candidate, a calculated candidate associated with a higher rating value among the calculated or estimated rating values; The proposed method is characterized by comprising:

26. A program that causes a computer to propose, based on a plurality of records, one or more of a plurality of candidates each representing a combination of parameters that can be obtained for an object, as useful candidates representing a combination of factor parameters that contribute to determining a label to be assigned to the object, wherein each record of the plurality of records has a plurality of parameter values ​​obtained by obtaining a plurality of parameters for an object associated with the respective record, and a label assigned to the associated object, and the program causes the computer to: Referring to the plurality of records, learning an estimation model for each of the calculation candidates that are part of the plurality of candidates; calculating a performance of the learned estimation model to estimate a label to be assigned to the target from parameter values ​​obtained by using parameters related to combinations represented by each of the estimation candidates for the target; Calculating a rating value associated with each of the evaluation candidates based on the calculated performance. Calculation Department, Referring to the rating value calculated for the evaluation candidate, Learn the rating model, Based on the learned rating model, a rating value associated with each of the estimated candidates other than the calculated candidate is estimated from among the plurality of candidates. Estimation department, an output unit that outputs, as the useful candidate, a calculated candidate associated with a rating value that is higher among the calculated or estimated rating values; A program characterized by functioning as

27. A non-transitory computer-readable information recording medium on which the program according to claim 26 is recorded.

Images