Machine learning model setting method, and program

Abstract

To set a cut-off value of a machine learning model so that sensibility and specificity should have a desired balance.SOLUTION: A machine learning model setting method includes the steps of: detecting an input operation for designating at least one of sensitivity and specificity of a machine learning model for medical diagnosis; based on at least one of the sensitivity and the specificity designated by the detected input operation, determining a pair of combination of values of the sensitivity and the specificity; and setting a cut-off value of the machine learning model corresponding to the pair of determined combination.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The disclosure of this specification relates to a method for setting a machine learning model and a program.

Background Art

[0002] In the medical field, diagnostic support using machine learning models has been conventionally performed. For example, in addition to those that output disease names described in Patent Document 1, those that discriminate between positive and negative for a predetermined test are known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a machine learning model for discriminating between positive and negative, the cut-off value serving as the discrimination criterion is set in advance to a fixed value so that the sensitivity and specificity have a predetermined balance. However, the optimal balance between sensitivity and specificity can vary depending on various factors such as the usage scenario.

[0005] Based on the above circumstances, an object according to one aspect of the present invention is to provide a technique capable of setting the cut-off value of a machine learning model so that the sensitivity and specificity have a desired balance.

Means for Solving the Problems

[0006] A method for setting a machine learning model according to an aspect of the present invention includes a step of detecting an input operation for specifying at least one of sensitivity or specificity of a machine learning model for medical diagnosis, and based on at least one of the sensitivity or the specificity specified by the detected input operation, determining a set of combinations of values of the sensitivity and the specificity, and setting a cut-off value of the machine learning model corresponding to the determined set of combinations.

[0007] A program according to an aspect of the present invention causes a computer to execute a process of detecting an input operation for specifying at least one of sensitivity or specificity of a machine learning model for medical diagnosis, a process of determining a set of combinations of values of the sensitivity and the specificity based on at least one of the sensitivity or the specificity specified by the detected input operation, and a process of setting a cut-off value of the machine learning model corresponding to the determined set of combinations.

Advantages of the Invention

[0008] According to the above aspect, the cut-off value of the machine learning model can be set so that the sensitivity and the specificity have a desired balance.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0010] FIG. 1 is a diagram illustrating the configuration of a system according to an embodiment. The system shown in FIG. 1 is a diagnostic support device using a machine learning model for medical diagnosis, and is configured to determine positive / negative for a predetermined disease based on the input image and output the determination result. Further, the diagnostic support device shown in FIG. 1 is configured such that a user can set a cutoff value for identifying positive / negative. Hereinafter, the configuration of the diagnostic support device will be described with reference to FIG. 1.

[0011] The diagnostic support device shown in FIG. 1 includes an information processing device 1, an input device 2, a display device 3, and a photographing device 4. The information processing device 1 is a device including a processor 10 and a storage device 20, and may be, for example, a general-purpose personal computer. The input device 2 is, for example, a mouse, a keyboard, a touch device, or the like. The display device 3 is, for example, a liquid crystal display or the like. The photographing device 4 is a device for acquiring medical images, and is, for example, a dermoscopy camera or the like.

[0012] The storage device 20 of the information processing device 1 stores a program 21, a machine learning model 22, model information 23, a setting file 24, and the like. The machine learning model 22 is a trained machine learning model for medical diagnosis that outputs a determination result (positive or negative) for a predetermined disease in response to the input of a medical image.

[0013] The model information 23 includes information indicating the performance of the machine learning model 22 for each cut-off value, which is a criterion for distinguishing between positive and negative. Specifically, the model information 23 includes a combination of sensitivity and specificity, which are accuracy indices of the machine learning model 22 calculated for each cut-off value. That is, the model information 23 includes a combination of sensitivity, specificity, and the cut-off value.

[0014] The setting file 24 is a file in which the settings of the diagnostic support device are recorded. The setting file 24 records, for example, the cut-off value set in the machine learning model 22. In addition to the cut-off value, the setting file 24 may store display settings of a setting screen described later for setting the cut-off value of the machine learning model 22.

[0015] In the information processing apparatus 1 configured as described above, when the processor 10 executes the program 21, determination processing using the machine learning model 22 is performed on the medical image acquired by the imaging apparatus 4, and a determination result indicating positive or negative is output. Further, by executing the program 21, the information processing apparatus 1 changes the setting of the machine learning model 22 based on the operation by the user using the input device 2. Specifically, the cut-off value of the machine learning model 22 is changed to change the balance between the sensitivity and specificity of the machine learning model 22.

[0016] FIG. 2 is a flowchart showing the procedure of cut-off value setting processing performed in the system according to an embodiment. Hereinafter, the processing for changing the cut-off value according to the operation of the user performed by the information processing apparatus 1 will be described with reference to FIG. 2. The processing shown in FIG. 2 is started when the processor 10 executes the program 21.

[0017] When the processing shown in FIG. 2 is started, first, the processor 10 causes the display device 3 to display a setting screen for setting the cut-off value of the machine learning model 22 (step S1). The setting screen is a screen on which at least one of the sensitivity or specificity of the machine learning model 22 can be specified instead of directly specifying the cut-off value of the machine learning model 22.

[0018] Note that the sensitivity of the machine learning model 22 is the ratio of being positive in the target disease group, that is, the ratio (true positive rate) at which the machine learning model 22 correctly determines positive as positive. Further, the specificity of the machine learning model 22 is the ratio of being negative in the non-disease group, that is, the ratio (true negative rate) at which the machine learning model 22 correctly determines negative as negative.

[0019] After that, when the user designates at least one of the sensitivity and specificity of the machine learning model 22 on the setting screen, the processor 10 detects the input operation (step S2). The sensitivity and specificity of the learned machine learning model 22 are in a relationship where if one is determined, the other is determined. Utilizing this, the processor 10 determines a set of combinations of the values of sensitivity and specificity based on at least one of the sensitivity or specificity designated by the detected input operation (step S3).

[0020] When the combination of sensitivity and specificity is determined, the processor 10 sets the cut-off value corresponding to the combination in the machine learning model 22 (step S4) and ends the cut-off value setting process shown in FIG. 2. In step S4, the processor 10, for example, reads out the cut-off value corresponding to the combination of sensitivity and specificity determined in step S3 from the model information 23 and sets the read cut-off value in the machine learning model 22.

[0021] In order to change the operation of the machine learning model 22 so that the sensitivity and specificity have a desired balance, it is necessary to change the cut-off value. However, the cut-off value is an operation parameter of the machine learning model 22, and it is difficult for the user to intuitively grasp what value to set to achieve an operation that exhibits the desired performance (in this example, the balance between sensitivity and specificity) in the machine learning model 22. Therefore, it is difficult for the user to directly specify the cut-off value so that the sensitivity and specificity have a desired balance.

[0022] In contrast, in the information processing apparatus 1, the user only needs to specify at least one of the sensitivity and specificity, which is a combination having a desired balance, instead of the cut-off value itself. The information processing apparatus 1 determines the cut-off value to be set in order to achieve the desired balance between sensitivity and specificity based on the designation operation of the user and sets it in the machine learning model 22. That is, the user only needs to directly specify the desired performance (the balance between sensitivity and specificity), and the information processing apparatus 1 can set the necessary cut-off value in the machine learning model 22 based on the specification.

[0023] As an application example of such a cut-off value setting method, it is conceivable to change the cut-off value according to the usage scenario. For example, in the case of a screening test, it is desirable to increase the sensitivity with priority given to avoiding overlooking lesions. On the other hand, in a case where the resection site during surgery is being considered, it is necessary to avoid accidentally resected non-lesion parts, and in some cases, it may be desirable to increase the specificity compared to the above-described screening test. Although the optimal balance between sensitivity and specificity may change according to the usage scenario, according to the information processing apparatus 1 according to the present embodiment, since the user can directly specify the sensitivity and specificity, it is possible to easily realize the setting of the optimal cut-off value according to the usage scenario. Therefore, according to the information processing apparatus 1, it is possible to perform diagnostic support with an optimal setting according to the usage scenario.

[0024] Hereinafter, a specific example of a method in which the information processing apparatus 1 sets a cut-off value that exhibits the performance by specifying the performance of the machine learning model 22 will be described.

[0025] <First Embodiment> FIG. 3 is a flowchart showing the procedure of cut-off value setting processing performed in the system according to the present embodiment. FIG. 4 is an example of a screen displayed in the system according to the present embodiment. The present embodiment is an example in which an ROC curve (Receiver Operating Characteristic) showing the relationship between sensitivity and specificity is displayed on the setting screen. Note that the configuration of the system according to the present embodiment is the same as the system (diagnostic support apparatus) shown in FIG. 1. In the system according to the present embodiment, the cut-off value setting processing shown in FIG. 3 is performed.

[0026] The process shown in Fig. 3 is started by the processor 10 executing the program 21. When the process shown in Fig. 3 is started, the processor 10 first causes the display device 3 to display a setting screen 100 shown in Fig. 4 for setting a cutoff value of the machine learning model 22, and causes the display device 3 to display a graph 101 indicating the relationship between the sensitivity and specificity of the machine learning model 22 within the setting screen 100 (step S11).

[0027] 4 is an ROC curve with sensitivity and specificity on two axes, more precisely, the vertical axis indicates sensitivity and the horizontal axis indicates 1-specificity. The graph 101 is displayed based on information on the combination of sensitivity and specificity for each cutoff value, which is calculated in advance for the machine learning model 22 and is stored in the storage device 20 as model information 23.

[0028] Thereafter, when the user specifies the sensitivity and specificity of the machine learning model 22 on the setting screen 100, the processor 10 detects the input operation (step S12). As shown in FIG. 4, the setting screen 100 is provided with a slider 103 and a slider 104 that slide on each axis indicating the sensitivity and specificity of a graph 101. The user can specify the sensitivity and specificity by roughly moving the point 102 on the graph 101 by operating the slider 103 or the slider 104 provided on the setting screen 100. The sliders 103 and 104 are configured so that when one of them is moved, the other also moves in conjunction with it.

[0029] The setting screen 100 also has a spin button 105 for finely adjusting the sensitivity and a spin button 106 for finely adjusting the specificity. In step S12, the user can operate the spin button 105 or the spin button 106 provided on the setting screen 100 to slightly move the point 102 on the graph 101 and specify the sensitivity and specificity. Note that the spin button 105 and the spin button 106 are configured so that when one is changed, the other also changes in conjunction with it.

[0030] That is, by operating the slider or spin button on the setting screen 100, the user can move a point 102 indicating the sensitivity and specificity to be set in the machine learning model 22 among a plurality of points on a graph 101 corresponding to a plurality of combinations of sensitivity and specificity stored in the model information 23. These operations are an example of operations for specifying the sensitivity and specificity with respect to the area where the graph 101 is displayed.

[0031] Based on the input operation detected in step S12, the processor 10 determines the sensitivity and specificity corresponding to the point 102 as a set of combinations of the sensitivity and specificity desired by the user (step S13). At this time, the determined sensitivity and specificity are displayed on the spin button. That is, in the cut-off value setting process shown in FIG. 3, a process of causing the display device 3 to display the values of the sensitivity and specificity determined in step S13 is performed.

[0032] When the combination of sensitivity and specificity is determined, the processor 10 sets the cut-off value corresponding to the combination in the machine learning model 22 (step S14), and ends the cut-off value setting process shown in FIG. 3. The process of step S14 is the same as the process of step S4 shown in FIG. 2.

[0033] As described above, in the cut-off value setting method shown in FIG. 3, based on the operations of the user for specifying the sensitivity and specificity, the cut-off value of the machine learning model 22 can be set so that the sensitivity and specificity have a desired balance. In particular, in the present embodiment, since the system displays the ROC curve on the setting screen 100, the user can grasp the relationship between the sensitivity and specificity peculiar to the machine learning model 22, so that the user can easily make settings according to the usage scenario. As a result, according to the present embodiment, it is possible to operate the machine learning model 22 with an optimal setting. Further, in the present embodiment, since the values of the sensitivity and specificity specified on the graph are displayed on the display device 3, the user can accurately grasp the specified values of the sensitivity and specificity.

[0034] Figure 5 shows another example of the screen displayed by the system according to this embodiment. In FIG. 4, an example was shown in which the point 102 was roughly moved by the slider provided on the setting screen 100 to specify approximate sensitivity and specificity, but the method of specifying sensitivity and specificity is not limited to this method. As shown in FIG. 5, sensitivity and specificity may be specified by directly specifying a point on the graph 101 with the cursor 201. Note that the fact that the sensitivity and specificity specified with the cursor 201 can be adjusted with the spin buttons (spin button 105, spin button 106) is the same as the example shown in FIG. 4.

[0035] In the cutoff value setting method shown in FIG. 3, even when the setting screen 200 shown in FIG. 5 is displayed, the same effect as when the setting screen 100 shown in FIG. 4 is displayed can be obtained.

[0036] <Second Embodiment> FIG. 6 is a flowchart showing the procedure of the cutoff value setting process performed by the system according to this embodiment. FIG. 7 is an example of the screen displayed by the system according to this embodiment. This embodiment is an example in which a graph showing the change in sensitivity and a graph showing the change in specificity are displayed on the setting screen. Note that the configuration of the system according to this embodiment is the same as the system (diagnostic support device) shown in FIG. 1. In the system according to this embodiment, the cutoff value setting process shown in FIG. 6 is performed.

[0037] The process shown in FIG. 6 is started when the processor 10 executes the program 21. When the process shown in FIG. 6 is started, first, the processor 10 causes the display device 3 to display the setting screen 300 shown in FIG. 7 for setting the cutoff value of the machine learning model 22, and within the setting screen 300, a graph 301 (sensitivity graph) showing the change in sensitivity with respect to the setting of the machine learning model 22 and a graph 302 (specificity graph) showing the change in specificity with respect to the setting of the machine learning model 22 are displayed on the display device 3 (step S21).

[0038] The graph 301 shown in FIG. 7 is a graph with the horizontal axis representing the setting and the vertical axis representing the sensitivity. The graph 302 shown in FIG. 7 is a graph with the horizontal axis representing the setting and the vertical axis representing the specificity. The settings shown by these graphs correspond one-to-one with the cut-off values. By sharing the axis representing the setting, the graph 301 and the graph 302 display the sensitivity and specificity corresponding to the same setting side by side on a straight line in the vertical axis direction. Note that the graph 301 and the graph 302 are displayed based on the information of the combination of sensitivity and specificity for each cut-off value, which is calculated in advance for the machine learning model 22 and stored in the storage device 20 as the model information 23.

[0039] After that, when the user specifies the sensitivity and specificity of the machine learning model 22 on the setting screen 300, the processor 10 detects the input operation (step S22). As shown in FIG. 7, the setting screen 300 is provided with a slider 308 that slides on the axis representing the setting. The movement range of the slider 308 corresponds to the area 307 between the setting (line 305) corresponding to the sensitivity of 100% indicated by the peak of the graph 301 and the setting (line 306) corresponding to the specificity of 100% indicated by the peak of the graph 302. The user can roughly move the point 303 on the graph 301 indicating the sensitivity and the point 304 on the graph 302 indicating the specificity by operating the slider 308 provided on the setting screen 300, thereby specifying the sensitivity and the specificity.

[0040] In addition, the setting screen 300 is also provided with a spin button 309 for finely adjusting the sensitivity and a spin button 310 for finely adjusting the specificity. In step S22, the user can finely move the point 303 and the point 304 on the graph 301 and the graph 302 by operating the spin button 309 or the spin button 310 provided on the setting screen 300, thereby specifying the sensitivity and the specificity.

[0041] Based on the input operation detected in step S22, the processor 10 determines, as a set of combinations of the sensitivity corresponding to point 303 and the specificity corresponding to point 304, the combinations of the sensitivity and specificity desired by the user (step S23). At this time, the determined sensitivity and specificity are displayed on the spin button. That is, in the cut-off value setting process shown in FIG. 6, the process of displaying the values of the sensitivity and specificity determined in step S23 on the display device 3 is performed.

[0042] When the combination of sensitivity and specificity is determined, the processor 10 sets the cut-off value corresponding to the combination in the machine learning model 22 (step S24), and ends the cut-off value setting process shown in FIG. 6. The process of step S24 is the same as the process of step S4 shown in FIG. 2.

[0043] As described above, also in the cut-off value setting method shown in FIG. 6, similar to the cut-off value setting method shown in FIG. 3, based on the operation of specifying the sensitivity and specificity by the user, the cut-off value of the machine learning model can be set so that the sensitivity and specificity have a desired balance. Further, in the present embodiment, instead of the ROC curve, two graphs regarding the sensitivity and specificity are displayed, but these two graphs are also graphs showing the relationship between the sensitivity and specificity of the machine learning model 22. That is, also in the present embodiment, a graph showing the relationship between the sensitivity and specificity of the machine learning model 22 is displayed on the display device 3, and the sensitivity and specificity corresponding to the cut-off value to be set are determined by the operation of specifying the sensitivity and specificity for the area where the graph is displayed. This is the same as in the first embodiment. Therefore, it is possible to make the user grasp the relationship between the sensitivity and specificity peculiar to the machine learning model 22 by the graph, and the user can also easily make settings according to the usage scene, which is the same as in the first embodiment. Therefore, also according to the present embodiment, the machine learning model 22 can be operated with an optimal setting.

[0044] FIG. 8 shows another example of a screen displayed by the system according to the present embodiment. In FIG. 7, an example was shown in which points 303 and 304 were roughly moved by a slider provided on the setting screen 300 to specify approximate sensitivity and specificity, but the method of specifying sensitivity and specificity is not limited to this method. As shown in FIG. 8, sensitivity and specificity may be specified by directly designating a point on graph 301 or graph 302 displayed on the setting screen 400 with cursor 401. Note that the point that the sensitivity and specificity designated with cursor 401 can be adjusted with spin buttons (spin button 309, spin button 310) is the same as the example shown in FIG. 7.

[0045] <Third Embodiment> FIG. 9 is a flowchart showing the procedure of cutoff value setting processing performed by the system according to the present embodiment. FIG. 10 is an example of a screen displayed by the system according to the present embodiment. This embodiment is an example in which one of sensitivity and specificity is specified from among the options displayed in pull-down list 501. Note that the configuration of the system according to the present embodiment is the same as the system (diagnosis support device) shown in FIG. 1. In the system according to the present embodiment, the cutoff value setting processing shown in FIG. 9 is performed.

[0046] The processing shown in FIG. 9 is started when the processor 10 executes the program 21. When the processing shown in FIG. 9 is started, first, the processor 10 causes the display device 3 to display the setting screen 500 shown in FIG. 10 for setting the cutoff value of the machine learning model 22, and displays the pull-down list 501 and the pull-down list 502 in the setting screen 500 (step S31).

[0047] Note that the pull-down list 501 is a pull-down list for selecting sensitivity from a plurality of options. The plurality of options included in the pull-down list 501 are a plurality of sensitivities that constitute combinations of sensitivity and specificity for each cutoff value stored in the storage device 20 as model information 23. Also, the pull-down list 502 is a pull-down list for selecting specificity from a plurality of options, and these plurality of options are a plurality of specificities that constitute combinations of sensitivity and specificity for each cutoff value stored in the storage device 20 as model information 23.

[0048] After that, when the user specifies the sensitivity and specificity of the machine learning model 22 on the setting screen 500, the processor 10 detects the input operation (step S32) and determines the sensitivity and specificity (step S33).

[0049] For example, when the user operates the pull-down list 501 to select sensitivity, the specificity corresponding to the sensitivity selected on the pull-down list 501 is automatically selected on the pull-down list 502, and thereby a set of sensitivity and specificity is determined. Also, when the user operates the pull-down list 502 to select specificity, the sensitivity corresponding to the specificity selected on the pull-down list 502 is automatically selected on the pull-down list 501, and thereby a set of sensitivity and specificity is determined. In this way, the determined sensitivity and specificity are displayed in the pull-down list. That is, in the cutoff value setting process shown in FIG. 9, the process of causing the display device 3 to display the sensitivity and specificity determined in step S33 is performed.

[0050] When the combination of sensitivity and specificity is determined, the processor 10 sets the cutoff value corresponding to the combination in the machine learning model 22 (step S34) and ends the cutoff value setting process shown in FIG. 9. The process of step S34 is the same as the process of step S4 shown in FIG. 2.

[0051] As described above, in the cutoff value setting method shown in FIG. 9 as well, similar to the cutoff value setting method shown in FIG. 3, based on the operation of specifying sensitivity and specificity by the user, the cutoff value of the machine learning model can be set so that the sensitivity and specificity have a desired balance.

[0052] <Fourth Embodiment> FIG. 11 is a flowchart showing the procedure of the cutoff value setting process performed by the system according to this embodiment. FIG. 12 is an example of a screen displayed by the system according to this embodiment. This embodiment is an example of specifying a combination of sensitivity and specificity from among the options for the combination of sensitivity and specificity displayed in a table format. Note that the configuration of the system according to this embodiment is the same as the system (diagnostic support device) shown in FIG. 1. In the system according to this embodiment, the cutoff value setting process shown in FIG. 11 is performed.

[0053] The process shown in FIG. 11 is started when the processor 10 executes the program 21. When the process shown in FIG. 11 is started, first, the processor 10 causes the display device 3 to display the setting screen 600 shown in FIG. 12 for setting the cutoff value of the machine learning model 22, and displays a table 601 showing a plurality of combinations of sensitivity and specificity and radio buttons 602 for selecting one combination from the plurality of combinations shown by the table 601 in the setting screen 600 (step S41). Note that the plurality of combinations included in the table 601 are combinations of sensitivity and specificity for each cutoff value stored in the storage device 20 as the model information 23.

[0054] Thereafter, when the user specifies the sensitivity and specificity of the machine learning model 22 on the setting screen 600, the processor 10 detects the input operation (step S42) and determines the sensitivity and specificity (step S43). Specifically, the sensitivity and specificity are determined when the user operates the radio button 602 to specify one of the plurality of combinations included in the table 601.

[0055] When the combination of sensitivity and specificity is determined, the processor 10 sets the cut-off value corresponding to the combination in the machine learning model 22 (step S44), and ends the cut-off value setting process shown in FIG. 11. The process of step S44 is the same as the process of step S4 shown in FIG. 2.

[0056] As described above, also in the cut-off value setting method shown in FIG. 11, similar to the cut-off value setting method shown in FIG. 3, based on the operation of specifying sensitivity and specificity by the user, the cut-off value of the machine learning model can be set so that the sensitivity and specificity have a desired balance. Further, in the present embodiment, since a plurality of combinations of sensitivity and specificity are displayed in a list in the table 601, it is possible for the user to grasp the relationship between the sensitivity and specificity peculiar to the machine learning model 22, and the user can easily make settings according to the usage scene. Therefore, also according to the present embodiment, the machine learning model 22 can be operated with an optimal setting.

[0057] <Fifth Embodiment> FIG. 13 is a flowchart showing the procedure of the cut-off value setting process performed in the system according to the present embodiment. The present embodiment is an example in which the user can freely set the display setting of the setting screen, and the display setting of the setting screen is saved together with the set cut-off value. The configuration of the system according to the present embodiment is the same as the system (diagnostic support device) shown in FIG. 1. In the system according to the present embodiment, the cut-off value setting process shown in FIG. 13 is performed.

[0058] The process shown in FIG. 13 is started when the processor 10 executes the program 21. When the process shown in FIG. 13 is started, first, the processor 10 causes the display device 3 to display a setting screen for setting the cut-off value of the machine learning model 22 (step S51). The setting screen displayed in step S51 is, for example, any one of the setting screens 100 to 600 described above.

[0059] After that, the processor 10 determines whether an update operation has occurred for the display settings of the setting screen (step S52). The display settings of the setting screen include display settings related to the method of specifying sensitivity and specificity, settings related to the numerical display of sensitivity and specificity, settings related to the display of the cut-off value, and the like.

[0060] The display settings related to the method of specifying sensitivity and specificity include, for example, settings for displaying the ROC curve shown in FIG. 4 or FIG. 5, settings for displaying the sensitivity graph and specificity graph shown in FIG. 7 or FIG. 8, settings for displaying the pull-down list shown in FIG. 10, settings for displaying the table and radio buttons shown in FIG. 12, and the like. The user may select any one of these, or may select and set a plurality of these.

[0061] The settings related to the numerical display of sensitivity and specificity are the settings for the display form of sensitivity and specificity displayed in the spin buttons shown in FIG. 4, FIG. 5, FIG. 7 or FIG. 8, the pull-down list shown in FIG. 10, and the table shown in FIG. 11, and include, for example, percentage / percent display settings, significant digit display settings, and the like.

[0062] The settings related to the display of the cut-off value include settings for the cut-off value corresponding to the specified sensitivity and specificity, that is, whether to display the cut-off value to be set. Further, it includes settings for the display form of the cut-off value when displaying, for example, significant digit display settings and the like.

[0063] In step S52, when the processor 10 determines that there has been an update operation for the display settings, it updates the display settings to the settings after the change (step S53). After that, when the user specifies the sensitivity and specificity of the machine learning model 22 on the setting screen, the processor 10 detects the input operation (step S54), determines the sensitivity and specificity (step S55), and sets the cut-off value (step S56). The processing from step S54 to step S56 is the same as the corresponding processing in the cut-off value setting processing according to the above-described embodiment.

[0064] Finally, the processor 10 associates the display settings of the setting screen updated in step S53 with the cut-off value set in step S56, saves them in the setting file 24 (step S57), and ends the cut-off value setting process shown in FIG. 13.

[0065] As described above, in the cut-off value setting method shown in FIG. 13 as well, similar to the cut-off value setting method shown in FIG. 3, based on the operation of specifying sensitivity and specificity by the user, the cut-off value of the machine learning model can be set so that the sensitivity and specificity have a desired balance. Also, in the present embodiment, by making the display settings of the setting screen configurable, the user can freely select the method of setting the cut-off value. Furthermore, by saving the appropriately changed display settings together with the cut-off value in the setting file, when the display screen is displayed next time, the setting screen can be automatically displayed with the preferred display settings without readjusting the display settings.

[0066] The above-described embodiments are presented with specific examples for ease of understanding of the invention, and the present invention is not limited to the above-described embodiments, but should be understood to include various modifications and alternative forms of the above-described embodiments. For example, it will be understood that the above-described embodiments can be embodied by modifying the components without departing from the gist thereof. Also, it will be understood that various embodiments can be implemented by appropriately combining the plurality of components disclosed in the above-described embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments or adding some components to the components shown in the embodiments. That is, the above-described image processing apparatus, image processing method, and program can be variously modified and changed without departing from the scope of the claims.

[0067] In the above-described embodiment, an example in which a spin button is displayed together with a graph in the setting screen has been shown, but the display of the spin button may be omitted. Further, instead of the spin button, a pull-down list as shown in FIG. 10, or a combination of a table and radio buttons as shown in FIG. 12 may be displayed.

[0068] In the above-described embodiment, the identification target of the machine learning model 22 is not specifically limited, but the identification target is, for example, a tumor. The machine learning model 22 may be a model that outputs whether it is a tumor or not as positive or negative.

[0069] In the above-described embodiment, an example in which the diagnostic support device includes the imaging device 4 has been shown, but the imaging device 4 may not be included. The diagnostic support device may be an image processing device that analyzes medical images acquired by another device.

Explanation of Signs

[0070] 1: Information processing device 2: Input device 3: Display device 4: Imaging device 10: Processor 20: Storage device 21: Program 22: Machine learning model 23: Model information 24: Setting file 100, 200, 300, 400, 500, 600: Setting screen 101, 301, 302: Graph 102, 303, 304: Point 103, 104, 308: Slider 105, 106: Spin button 201, 401: Cursor 305, 306: Line 307: Region 309, 310: Spin button 501, 502: Pull-down list 601: Table 602: Radio button

Claims

1. detecting an input operation for specifying at least one of sensitivity or specificity of a machine learning model for medical diagnosis; determining a set of combinations of values of the sensitivity and the specificity based on at least one of the sensitivity or the specificity specified by the detected input operation; setting a cut-off value of the machine learning model corresponding to the determined set of combinations, characterized by a method for setting a machine learning model.

2. In the method for setting a machine learning model according to Claim 1, further comprising: causing a display device to display a graph showing the relationship between the sensitivity and the specificity of the machine learning model, wherein the input operation is an operation for specifying the sensitivity and the specificity with respect to the area where the graph is displayed characterized by a method for setting a machine learning model.

3. In the method for setting a machine learning model according to Claim 2, the graph is an ROC curve with the sensitivity and the specificity on two axes characterized by a method for setting a machine learning model.

4. In the method for setting a machine learning model according to Claim 2, the graph includes: a sensitivity graph showing a change in the sensitivity with respect to the setting of the machine learning model, and a specificity graph showing a change in the specificity with respect to the setting of the machine learning model, wherein the sensitivity graph and the specificity graph share an axis indicating the setting of the machine learning model characterized by a method for setting a machine learning model.

5. In the method for setting a machine learning model according to any one of Claims 1 to 4, further comprising: causing a display device to display the values of the sensitivity and the specificity of the determined set of combinations characterized by a method for setting a machine learning model.

6. In the method for setting a machine learning model according to any one of Claims 1 to 4, further comprising: associating and storing a display setting of a setting screen for setting the cut-off value of the machine learning model and the set cut-off value characterized by a method for setting a machine learning model.

7. On a computer, a process of detecting an input operation for specifying at least one of sensitivity or specificity of a machine learning model for medical diagnosis; a process of determining a set of combinations of values of the sensitivity and the specificity based on at least one of the sensitivity or the specificity specified by the detected input operation; Execute a process of setting a cut-off value of the machine learning model corresponding to the determined set of combinations. A program characterized by the above.

Images