Model selection method and image processing method

Abstract

To provide a model selection method for selecting a machine learning model from a large number of machine learning models with high accuracy and efficiency without using Ground Truth, and an image processing method using the selected machine learning model.SOLUTION: A model selection method comprises a first model selection step S33 and a second model selection step S35. In the first model selection step S33, a plurality of machine learning models are selected from all machine learning models to form a first model group using a first image group including at least one confirmation image. In the second model selection step S35, at least one machine learning model is selected from machine learning models included in the first model group using a second image group. The number of confirmation images included in the second image group is greater than that in the first image group. By gradually increasing the number of confirmation images and narrowing down the number of models to be selected, it is possible to select an appropriate machine learning model with high accuracy without using Ground Truth.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a model selection method for selecting an appropriate model from a plurality of machine learning models for performing image processing, and an image processing method using the selected machine learning model. [Background technology]

[0002] Conventionally, methods for staining and observing pathological tissue specimens, cultured cells, etc. are known. In this case, it is important to extract cellular regions from images of the pathological tissue specimen, etc. In particular, in order to perform single-cell analysis of the expression levels of biological substances such as proteins on digital images of stained pathological tissue specimens, it is necessary to more accurately identify individual cellular regions. A conventional image processing method for extracting cellular regions is described, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2021-506022 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in such specimen observation, differences in observation method, cell type, specimen collection site, and staining material type may occur. Even if these factors are the same, the tissue condition and staining conditions may vary. Therefore, it is difficult to identify cell regions using rule-based image processing. Furthermore, even when machine learning models are used, a model that can accurately extract cell regions for one specimen may not be able to accurately extract cell regions for other specimens.

[0005] When identifying cell regions using deep learning, one example is to apply a segmentation task to train a model on images with annotated cell regions, and then segment and output cell regions for unknown images.

[0006] When ground truth (Ground Truth) is available, numerical accuracy information such as F-score and recall is often used to select superior models from the many machine learning models that have been created. However, creating accurate Ground Truth is a time-consuming manual process, making it inefficient to create Ground Truth for model evaluation for each analysis target. Furthermore, when complete Ground Truth exists, there is no need to identify cell regions using a machine learning model, so Ground Truth is usually not available when the analysis is performed.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide a technology for selecting a machine learning model from a large number of machine learning models with high accuracy and efficiency without using ground truth. [Means for solving the problem]

[0008] In order to solve the above problem, the first invention of the present application is a model selection method for selecting at least one machine learning model from a plurality of machine learning models that perform image processing using a plurality of confirmation images, comprising: a) a first model selection step of using a first image group including at least one of the confirmation images to select a plurality of the machine learning models from the machine learning models included in an entire model group including all machine learning models to form a first model group; and b) after step a), a second model selection step of using a second image group including a plurality of the confirmation images to select at least one of the machine learning models from the machine learning models included in the first model group to form a second model group, wherein the number of the confirmation images included in the second image group is greater than the number of the confirmation images included in the first image group.

[0009] The second invention of the present application is a model selection method of the first invention, wherein the step a) includes: a1) a step of inputting all of the confirmation images included in the first image group into all of the machine learning models included in the total model group to obtain output results; a2) a step of displaying the output results obtained in the step a1); and a3) a step of inputting from outside a selection result that selects a portion of the machine learning models included in the total model group.

[0010] The third invention of the present application is a model selection method according to the first or second invention, wherein step b) includes: b1) obtaining output results by inputting all of the confirmation images included in the second image group into all of the machine learning models included in the first model group; b2) displaying the output results obtained in step b1); and b3) inputting selection results from outside that select some of the machine learning models included in the first model group.

[0011] A fourth invention of the present application is a model selection method of any of the first to third inventions, wherein the second model group includes a plurality of the machine learning models, and c) after step b), the method further includes a third model selection step of selecting one of the machine learning models included in the second model group using a third image group including a plurality of the confirmation images, and the number of the confirmation images included in the third image group is greater than the number of the confirmation images included in the second image group.

[0012] The fifth invention of the present application is a model selection method of any of the first to fourth inventions, further comprising: d) a first image selection step before step a), of selecting one or more images from all of the confirmation images to form the first image group; and e) a second image selection step after step a) and before step b), of selecting multiple images from all of the confirmation images to form the second image group.

[0013] The sixth invention of the present application is a model selection method of the fifth invention, wherein the step d) includes the steps of: d1) obtaining output results by inputting all of the confirmation images into all of the machine learning models included in the entire model group; d2) calculating the variability of the output results obtained in the step d1) for each of the confirmation images; and d3) selecting the confirmation images with the large variability calculated in the step d2) from all of the confirmation images to form the first image group.

[0014] The seventh invention of the present application is a model selection method according to any one of the first to sixth inventions, wherein the machine learning model performs image processing to recognize and segment cellular regions in an image containing cells.

[0015] The eighth invention of the present application is an image processing method for performing image processing on a plurality of analysis images using a machine learning model, comprising: s) a step of selecting one of the machine learning models using the model selection method of any of the first to seventh inventions; and t) a step of performing image processing on the plurality of analysis images using the machine learning model selected in step s), wherein the confirmation image in step s) is the analysis image that is the target of processing in step t). [Effects of the Invention]

[0016] According to the first to eighth aspects of the present invention, a large number of machine learning models are roughly selected using a small number of confirmation images in a first model selection step, and then a larger number of confirmation images are used in a second model selection step to further select machine learning models. By gradually increasing the number of confirmation images and narrowing down the number of machine learning models to be selected, it is possible to accurately select an appropriate machine learning model while minimizing the number of model output results to be confirmed during model selection. Therefore, a machine learning model can be selected from a large number of machine learning models with high accuracy and efficiency without using ground truth.

[0017] In particular, according to the second aspect of the present invention, the first model selection step can be performed at the discretion of a user who has confirmed the output results of the machine learning model, thereby making it possible to appropriately select the first model group.

[0018] In particular, according to the third aspect of the present invention, the second model selection step can be performed at the discretion of a user who has confirmed the output results of the machine learning model, thereby enabling an appropriate selection of the second model group.

[0019] In particular, according to the fourth aspect of the present invention, a machine learning model is selected in three stages: a first model selection step, a second model selection step, and a third model selection step. This allows a more accurate and appropriate machine learning model to be selected when there are a large number of machine learning models to select from, compared to a two-stage machine learning model selection.

[0020] In particular, according to the fifth aspect of the present invention, the confirmation image used in the first model selection step and the confirmation image used in the second model selection step are selected from a large number of confirmation images, thereby enabling more accurate selection of a machine learning model.

[0021] In particular, according to the sixth invention of the present application, by using confirmation images that are likely to cause variation in the output results of the machine learning model to be selected in the first model selection process, it is possible to select a machine learning model with high accuracy even when selection is made using a small number of confirmation images.

[0022] In particular, according to the seventh aspect of the present invention, the input image of the machine learning model varies greatly depending on the type of cell or reagent, and the observation environment, making it important to select an appropriate machine learning model.

[0023] In particular, according to the seventh aspect of the present invention, a machine learning model is selected using an analysis image that is the analysis target, thereby making it possible to select a machine learning model that is optimal for the analysis image. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing a functional configuration of an image analysis device according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a functional configuration of a machine learning model selection device according to a first embodiment. [Figure 4] 10 is a flowchart showing the flow of an image analysis process. [Figure 5] 1 is a flowchart showing the flow of a machine learning model selection process. [Figure 6] The image below shows the images used at each stage of the machine learning model selection process and the models to be selected. [Figure 7] FIG. 1 is a diagram schematically illustrating output results obtained by inputting one confirmation image into multiple machine learning models. [Figure 8] 10 is a flowchart showing a flow when the first image selection step and the second image selection step are performed manually. [Figure 9] 10 is a flowchart showing a flow when the first image selection step is performed automatically. [Figure 10] 10 is a flowchart showing the flow of a first model selection step, a second model selection step, and a third model selection step. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0026] 1. First Embodiment <1-1. Image analysis device configuration> FIG. 1 is a diagram showing the hardware configuration of an information processing device 100 that realizes an image analyzing device 20 including a machine learning model selection device 1 according to the first embodiment.

[0027] 1, the information processing device 100 has a configuration as a computer. Specifically, the information processing device 100 has a processor 101, a RAM 102, a device I / F 103, a communication I / F 104, a storage unit 105, an input unit 106, and a display unit 107. The processor 101, the RAM 102, the device I / F 103, the communication I / F 104, the storage unit 105, the input unit 106, and the display unit 107 are electrically connected to one another via a bus 108.

[0028] Specifically, the processor 101 includes a CPU or a GPU. The RAM 102 is a storage medium from which information can be read and written, and is specifically an SDRAM.

[0029] The device I / F 103 is an interface for electrically connecting external devices such as the imaging device 9 (described later) to the information processing device 100. The communication I / F 104 is an interface for connecting the information processing device 100 to a network such as the Internet. The external devices such as the imaging device 9 may be connected to the information processing device 100 via the communication I / F 104.

[0030] The storage unit 105 is a recording medium from which information can be read and written, and specifically includes an HDD (hard disk drive) or an SSD (solid state drive). The storage unit 105 may also include a portable optical disk, magnetic disk, semiconductor memory, or the like. The storage unit 105 stores a program P. The processor 101 uses the RAM 102 as a working area to execute the program P, thereby realizing various functions of the image analysis device 20 and the machine learning model selection device 1. The program P may be provided or distributed to the information processing device 100 via a network.

[0031] The input unit 106 is an input device that accepts user operation input, and is specifically a mouse or a keyboard. The display unit 107 is a display device that displays images representing various information, and is specifically a liquid crystal display. Note that the input unit 106 and the display unit 107 may be a single device, such as a touch panel display.

[0032] FIG. 2 is a diagram showing the functional configuration of an image analysis device 20 implemented by the information processing device 100 of the first embodiment. The image analysis device 20 is a device that analyzes an image input from an external device. The image analysis device 20 of this embodiment is a device for analyzing a cell image input from an imaging device 9 provided in a microscope. The image analysis device 20 has an image cropping unit 21, a machine learning model selection unit 22, and an image analysis unit 23. The image cropping unit 21, the machine learning model selection unit 22, and the image analysis unit 23 are functions implemented by the processor 101 operating in accordance with the program P.

[0033] The image cropping unit 21 obtains multiple images for analysis from an input image Ge input from an imaging device. In this embodiment, the imaging device 9 is a camera provided in a microscope, and the input image Ge is an image of a fluorescently stained tissue specimen captured by the imaging device 9 provided in the microscope. Note that the analysis target in this embodiment is a fluorescently stained tissue specimen, but the present invention is not limited to this. The analysis target is not limited to a tissue specimen, and may be other biological samples such as cultured cells or blood cells, as long as it is an image of a sample that requires segmentation. Furthermore, the analysis target is not limited to a fluorescently stained sample, and may be an unstained sample or a sample stained with a non-fluorescent staining agent.

[0034] The image cutout unit 21 cuts out a plurality of narrow-field-of-view images from the input image Ge captured with a wide field of view to obtain a plurality of analysis images, and stores the plurality of cut-out analysis images as a whole image group G0 in the storage unit 105.

[0035] The image cropping unit 21 of this embodiment displays a wide-field input image Ge on the display unit 107. Then, the user operates the input unit 106 to specify multiple regions from the input image Ge. The image cropping unit 21 crops the specified regions to create a narrow-field analysis image. Note that the image cropping unit 21 may automatically crop the analysis image without going through the above procedure. In that case, for example, the input image Ge may be automatically cropped into tiles of a predetermined size.

[0036] The machine learning model selection unit 22 uses multiple confirmation images to select one machine learning model from multiple machine learning models that perform image processing. Specifically, it reads out a group of all images G0 stored in the storage unit 105 and a group of all models M0 including multiple machine learning models, and uses the group of all images G0 to select and output one machine learning model from the group of all models M0 as a selected model Mf. In this embodiment, when an observation image of fluorescently stained cells is input, each machine learning model identifies the cellular region of each cell in the observation image and outputs the region.

[0037] In this embodiment, the analysis images that will actually be the subject of image analysis are used as confirmation images for selecting a machine learning model. That is, the multiple analysis images included in the entire image group G0 acquired by the image cropping unit 21 are used as confirmation images. The detailed configuration of the machine learning model selection unit 22 will be described later.

[0038] The image analysis unit 23 analyzes the analysis images using image processing. In this embodiment, the image analysis unit 23 identifies cellular regions for each of the multiple analysis images extracted by the image extraction unit 21 using the selected model Mf selected by the machine learning model selection unit 22. Then, the area and distribution of stained regions in the identified cellular regions are calculated for each type of staining.

[0039] <1-2. Configuration of machine learning model selection device> Next, the machine learning model selection device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the machine learning model selection device 1 implemented in the information processing device 100 of the first embodiment. This machine learning model selection device 1 uses a plurality of confirmation images to select one machine learning model from a plurality of machine learning models that perform image processing.

[0040] As described above, this machine learning model selection device 1 selects a machine learning model that performs image processing to recognize and segment cellular regions in images containing cells. Images containing cells look different depending on the shooting conditions, whether they are stained, and the staining agent. Therefore, the optimal machine learning model for identifying cellular regions differs for each image. For this reason, it is preferable to use this machine learning model selection device 1 to select the optimal machine learning model each time the analysis target changes.

[0041] For ease of understanding, the following describes a specific example in which 80 confirmation images are used to select one machine learning model from 100 machine learning models by repeating the image selection process and model selection process three times, while referring to the specific numbers of confirmation images and machine learning models. However, the number of confirmation images and the number of machine learning models to be selected are not limited to this, and any number is possible as long as there is more than one.

[0042] The machine learning model selection device 1 includes the above-described machine learning model selection unit 22. As shown in FIG. 3 , the machine learning model selection unit 22 includes an image processing unit 31, a first image selection unit 32, a first model selection unit 33, a second image selection unit 34, a second model selection unit 35, a third image selection unit 36, and a third model selection unit 37. The image processing unit 31, the first image selection unit 32, the first model selection unit 33, the second image selection unit 34, the second model selection unit 35, the third image selection unit 36, and the third model selection unit 37 are functions realized by the processor 101 operating in accordance with the program P. The machine learning model selection device 1 also includes the above-described storage unit 105, input unit 106, and display unit 107.

[0043] The storage unit 105 stores a total image group G0 and a total model group M0. The total image group G0 includes a plurality of confirmation images g1 to g80. In this embodiment, the confirmation images g1 to g80 included in the total image group G0 are analysis images that are actually used for analysis. The total model group M0 includes a plurality of machine learning models. These machine learning models may include models created using the same algorithm but with different parameters, or may include machine learning models created using various types of algorithms.

[0044] The image processing unit 31 inputs all of the confirmation images included in the total image group G0 to all of the machine learning models included in the total model group M0, and obtains a total output result group D0. The image processing unit 31 passes the total image group G0 and the total output result group D0 to the first image selection unit 32, the second image selection unit 34, and the third image selection unit, respectively. For example, the total output result group D0 includes a total of 8,000 output results obtained by inputting all 80 confirmation images to 100 machine learning models.

[0045] The first image selection unit 32 selects one or more images from all of the confirmation images included in the total image group G0 to create a first image group G1. The first image selection unit 32 then transfers the first image group G1 and a first output result group D1, which are output results obtained by inputting the first image group G1 into each machine learning model included in the total model group M0, to the first model selection unit 33. The first output result group D1 includes all output results obtained by inputting the confirmation images included in the first image group G1 from the total output result group D0 into all machine learning models included in the total model group M0. For example, if the first image group G1 includes two confirmation images and the total model group M0 includes 100 machine learning models, the first output result group D1 will include 200 output results.

[0046] The first model selection unit 33 uses the first image group G1 and the first output result group D1, which is the output result of the first image group G1 / total model group M0, to select multiple machine learning models from the machine learning models included in the total model group M0 to create a first model group M1. The number of machine learning models included in the first model group M1 is smaller than the number of machine learning models included in the total model group M0. The first model selection unit 33 passes the selected first model group M1 to the second image selection unit 34.

[0047] The second image selection unit 34 selects a plurality of images from all of the confirmation images included in the total image group G0 to form the second image group G2. In this embodiment, the second image selection unit 34 selects confirmation images using the second output result group D2, which is the output result of the total image group G0 / first model group M1, and forms the selected plurality of confirmation images into the second image group G2. For example, if the total image group G0 includes 80 confirmation images and the first model group M1 includes 30 machine learning models, the second output result group D2 will include 2,400 output results.

[0048] The second image selection unit 34 then transfers the second image group G2, the first model group M1, and the third output result group D3, which is the output result of the second image group G2 and the first model group M1, to the second model selection unit 35. The number of confirmation images included in the second image group G2 is greater than the number of confirmation images included in the first image group G1. The second image group G2 may include all, some, or none of the confirmation images included in the first image group G1. For example, if the second image group G2 includes five confirmation images and the first model group M1 includes 30 machine learning models, the third output result group D3 will include 150 output results.

[0049] The second model selection unit 35 uses the second image group G2 and the third output result group D3, which is the output result of the second image group G2 / first model group M1, to select multiple machine learning models from the machine learning models included in the first model group M1 to create a second model group M2. The number of machine learning models included in the second model group M2 is smaller than the number of machine learning models included in the first model group M1. The second model selection unit 35 passes the selected second model group M2 to the third image selection unit 36.

[0050] The third image selection unit 36 ​​selects multiple images from all of the confirmation images included in the total image group G0 to form the third image group G3. In this embodiment, the third image selection unit 36 ​​selects confirmation images using the fourth output result group D4, which is the output result of the total image group G0 / second model group M2, and forms the selected multiple confirmation images into the third image group G3. For example, if the total image group G0 includes 80 confirmation images and the second model group M2 includes five machine learning models, the fourth output result group D4 will include 400 output results.

[0051] The third image selection unit 36 ​​then transfers the third image group G3, the second model group M2, and a fifth output result group D5, which is the output result of the third image group G3 and the second model group M2, to the third model selection unit 37. The number of confirmation images included in the third image group G3 is greater than the number of confirmation images included in the second image group G2. The third image group G3 may include all, some, or none of the confirmation images included in the first image group G1 and the second image group G2. For example, if the third image group G3 includes 40 confirmation images and the second model group M2 includes five machine learning models, the fifth output result group D5 will include 200 output results.

[0052] The third image selection unit 36 ​​may also select all confirmation images included in the entire image group G0 as the third image group G3. For example, if the number of all confirmation images included in the entire image group G0 is equal to or less than a predetermined threshold, all confirmation images included in the entire image group G0 may also be selected as the third image group G3. The third image selection unit 36 ​​may also randomly select confirmation images from among the confirmation images included in the entire image group G0 to select them as the third image group G3.

[0053] The third model selection unit 37 uses the third image group G3 and the fifth output result group D5, which is the output result of the third image group G3 / second model group M2, to select one from the multiple machine learning models included in the second model group M2 as the selected model Mf.

[0054] <1-3. Image processing flow> Next, the flow of the image analysis process including the model selection process will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the image analysis process.

[0055] First, the image analyzing device 20 acquires a machine learning model to be selected (step S1: machine learning model acquisition step). In this embodiment, 100 machine learning models are acquired. Specifically, the 100 candidate machine learning models are stored in the storage unit 105. As long as these machine learning models perform image processing for the same purpose, they may include multiple models with different parameters created using the same type of algorithm, or may include machine learning models created using multiple types of algorithms.

[0056] Next, a plurality of images to be used as confirmation images are acquired (step S2: image acquisition process). In this embodiment, as described above, the image cropping unit 21 crops the input image Ge input from the imaging device 9, thereby acquiring a plurality of analysis images as confirmation images. That is, as confirmation images, observation images of the tissue specimen that will be the subject of image analysis in the image analysis process S4, which will be described later, are used. In this embodiment, 80 confirmation images are acquired. The acquired plurality of confirmation images are stored in the memory unit 105.

[0057] Next, the machine learning model selection unit 22 selects one machine learning model from all 100 machine learning models using the confirmation image acquired in the image acquisition process of step S2 (step S3: machine learning model selection process). In this embodiment, the image for image analysis is used as the confirmation image, so the machine learning model that is optimal for the image to be analyzed in the subsequent image analysis process S4 is selected. Details of the machine learning model selection process performed in step S3 will be described later.

[0058] Finally, the image analysis unit 23 performs image analysis on the observation image for image analysis (step S4: image analysis step). Specifically, first, the machine learning model selected in the machine learning model selection step of step S3 is used to identify cellular regions in each image for image analysis. Then, within the identified cellular regions, the brightness and area of ​​each stained color are calculated. This makes it possible to estimate the expression level and distribution area of ​​the biological substance visualized by staining.

[0059] <1-4. Machine learning model selection process> Next, the machine learning model selection process in step S3 will be described with reference to FIGS.

[0060] Figure 5 is a flowchart showing the flow of the machine learning model selection step S3. As shown in Figure 5, in the machine learning model selection step of this embodiment, a machine learning model is selected in stages from among a large number of machine learning models. In this embodiment, one machine learning model is selected through three stages: a first model selection step S33, a second model selection step S35, and a third model selection step S37, which will be described later. Note that the present invention is not limited to this, and as long as there are multiple model selection steps, a machine learning model may be selected through a two-stage selection step, or a machine learning model may be selected through four or more stages of selection steps.

[0061] 6 is an image diagram of the image groups used at each stage of the machine learning model selection step S3 in this embodiment, and the number of model groups to be selected. In Fig. 6, the numbers in parentheses after the names of the image groups and model groups indicate the number of images included in each image group and the number of machine learning models included in each model group.

[0062] FIG. 7 is a diagram showing the output results obtained by inputting one confirmation image g1 to all machine learning models m1 to m100 included in the entire image group G0. As shown in FIG. 7, confirmation image g1 includes an image of a cell. Each of the machine learning models m1 to m100 then identifies a cellular region in confirmation image g1. In FIG. 7, the identified region is shown in black.

[0063] In a machine learning model suitable for the verification image g1, most cells are recognized and non-cellular regions are less likely to be mistakenly recognized. On the other hand, in a machine learning model that is not suitable for the type of verification image g1, some cells are not recognized or non-cellular regions are mistakenly recognized as cells. In the example of Figure 7, model m3 recognizes all cells, but models m1, m2, and m99 fail to recognize some. On the other hand, model m100 recognizes non-cellular regions as cells.

[0064] To select the machine learning model that performs the most appropriate image processing for all confirmation images g1 to g80 from these output results, for example, a user may check and analyze the output results of all machine learning models m1 to m100 for all confirmation images g1 to g80. However, checking such output results for all confirmation images g1 to g80 and all machine learning models m1 to m100 and using the optimal identification result takes time and effort.

[0065] Therefore, in this machine learning model selection step S3, as shown in Figure 6, when there are many machine learning models to select from, a small number of confirmation images are used to select a model, and the number of machine learning models to select is gradually reduced, while the number of confirmation images is gradually increased to confirm the output of the machine learning model. This makes it possible to select a more appropriate machine learning model while reducing the selection effort.

[0066] In the machine learning model selection step S3, first, the image processing unit 31 inputs all of the confirmation images included in the total image group G0 to all of the machine learning models included in the total model group M0, and obtains a total output result group D0 (total image processing step: step S31). In this embodiment, the total output result group D0 includes a total of 8,000 output results obtained by inputting all of the 80 confirmation images to the 100 machine learning models.

[0067] Next, the first image selection unit 32 selects one or more images from all of the confirmation images included in the entire image group G0 to form the first image group G1 (first image selection step: step S32). This first image selection step may be performed manually, automatically, or a combination of manually and automatically.

[0068] 8 is a flowchart showing the flow when the first image selection step S32 and the second image selection step S34 are performed manually. As shown in Fig. 8, when the first image selection step S32 is performed manually, the first image selection unit 32 first acquires the entire image group G0 and the entire output result group D0 from the image processing unit 31 (step S51).

[0069] Next, first image selection unit 32 displays each of the confirmation images included in all image group G0 and the output results of all model group M0 for that confirmation image so that they can be confirmed on display unit 107 (step S52). That is, 100 output results are displayed for each of the 80 confirmation images in all image group G0.

[0070] Then, the user visually checks the display in step S52 (step S53). After that, the user selects a confirmation image that he or she wants to use in the first model selection step S33, and inputs it to the first image selection unit 32 via the input unit 106 (step S54).

[0071] The first image selection unit 32 recognizes the input confirmation images as the first image group G1, and transfers the first image group G1 and the first output result group D1, which is the output result of the first image group G1 / all model group M0, to the first model selection unit 33 (step S55). In this embodiment, the first image group G1, which includes two confirmation images, and the first output result group D1, which includes 200 output results of the first image group G1 / all model group M0, are transferred to the first model selection unit 33.

[0072] Fig. 9 is a flowchart showing the flow when the first image selection step S32 is performed automatically. As shown in Fig. 9, when the first image selection step S32 is performed automatically, the first image selection unit 32 first acquires the entire image group G0 and the entire output result group D0 from the image processing unit 31 (step S61).

[0073] Next, for each confirmation image included in the entire image group G0, the first image selection unit 32 detects a segment in each output result for that confirmation image, that is, detects a region identified as a cellular region in the output result of the entire model group M0 for each confirmation image (step S62).

[0074] Next, the first image selection unit 32 calculates the variation of segments in all output results for each confirmation image (step S63). In this embodiment, the variance of the number of segments in the output results is used as the value indicating the variation. Note that the value indicating the variation is not limited to this, and an index indicating the variation, such as the variance or standard deviation, may be used for various parameters such as the number of segments, average area, total area, and center of gravity position. Furthermore, the value indicating the variation is not necessarily limited to one type, and two or more types of values ​​may be used.

[0075] Next, the first image selection unit 32 selects confirmation images with large variations between models from all 80 confirmation images included in the total image group G0, and designates them as the first image group G1 (step S64).Then, the first image group G1 and the first output result group D1, which is the output result of the first image group G1 / total model group M0, are passed to the first model selection unit 33 (step S64).

[0076] After the first image selection step of step S32 is completed, the first model selection unit 33 then selects a machine learning model from the entire model group M0 (step S33: first model selection step). The first model selection unit 33 uses the first image group G1 and a first output result group D1, which is the output result of the first image group G1 / the entire model group M0, to select multiple machine learning models from the machine learning models included in the entire model group M0, and sets the selected models as the first model group M1. The first model selection unit 33 then passes the first model group M1 to the second image selection unit 34.

[0077] FIG. 10 is a flowchart showing the flow of the first model selection step S33, the second model selection step S35, and the third model selection step S37.

[0078] When performing the first model selection process S33, as shown in FIG. 10, the first model selection unit 33 first acquires the first image group G1 to be used for selection and the first output result group D1, which is the output result of the first image group G1 / total model group M0 (step S71).

[0079] Next, the first model selection unit 33 displays, in a confirmable manner, each of the confirmation images included in the first image group G1 and the output results of the entire model group M0 for that confirmation image on the display unit 107 (step S72). That is, 100 output results are displayed for each of the two confirmation images in the first image group G1.

[0080] The user then visually confirms the display in step S72 (step S73). After that, the user selects a machine learning model that the user determines to be appropriate and inputs it to the first model selection unit 33 via the input unit 106 (step S74). In this embodiment, 30 machine learning models are selected at this time. Based on the input, the first model selection unit 33 selects the machine learning models selected from the entire model group M0 as a first model group M1 and passes them to the second image selection unit 34 (step S75).

[0081] The first model selection step S33 may be performed automatically by the processor 101 at the discretion of the user. For example, automatic selection may be performed when the number of machine learning models included in the total model group M0 is equal to or greater than a certain number. Also, for example, the first model selection unit 33 may display a histogram of the number of segments in the output result on the display unit 107, and if the user desires automatic selection based on that display, automatic selection may be performed.

[0082] When the first model selection step S33 is performed automatically, when the user inputs an automatic selection command into the input unit 106, the first model selection unit 33 selects, for example, a predetermined number (30 in this embodiment) of machine learning models whose number of segments is close to the average or most frequent value of the entire model group M0, and sets these as the first model group M1.

[0083] Following the first model selection step S33, the second image selection unit 34 selects a second image group G2 to be used in the second model selection step S35 (second image selection step: step S34). The number of confirmation images included in the second image group G2 is greater than the number of confirmation images included in the first image group G1.

[0084] In this second image selection step S34, a plurality of images are selected from all of the confirmation images included in the total image group G0 using the second output result group D2, which is the output result of the total image group G0 / first model group M1, to form the second image group G2. This second image selection step may be performed manually, automatically, or a combination of manual and automatic selection.

[0085] When the second image selection step S34 is performed manually, as shown in FIG. 8, the second image selection unit 34 first acquires the entire image group G0 and the entire output result group D0 from the image processing unit 31 (step S51).

[0086] Next, the second image selection unit 34 displays, on the display unit 107, the confirmation images included in the entire image group G0 and the second output result group D2, which is the output result of the entire image group G0 / first model group M1, so that they can be confirmed (step S52). That is, 30 output results are displayed for each of the 80 confirmation images.

[0087] Then, the user visually checks the display in step S52 (step S53). After that, the user selects a confirmation image that he or she wants to use in the second model selection step S35, and inputs it to the second image selection unit 34 via the input unit 106 (step S54).

[0088] The second image selection unit 34 recognizes the input confirmation images as the second image group G2, and transfers the second image group G2 and the third output result group D3, which is the output result of the second image group G2 / first model group M1, to the second model selection unit 35 (step S55). In this embodiment, the second image group G2, which includes five confirmation images, and the third output result group D3, which includes 150 output results of the second image group G2 / first model group M1, are transferred to the second model selection unit 35.

[0089] After the second image selection process of step S34 is completed, the second model selection unit 35 then selects a machine learning model from the first model group M1 (step S35: second model selection process). The second model selection unit 35 uses the second image group G2 and a third output result group D3, which is the output result of the second image group G2 / first model group M1, to select multiple machine learning models from the machine learning models included in the first model group M1, and sets the selected models as the second model group M2. The second model selection unit 35 then passes the second model group M2 to the third image selection unit 36.

[0090] When performing the second model selection process S35, as shown in FIG. 10, the second model selection unit 35 first acquires the second image group G2 to be used for selection and the third output result group D3, which is the output result of the second image group G2 / first model group M1 (step S71).

[0091] Next, the second model selection unit 35 displays each of the confirmation images included in the second image group G2 and the output results of the first model group M1 for that confirmation image so that they can be confirmed on the display unit 107 (step S72). That is, 30 output results are displayed for each of the five confirmation images in the second image group G2.

[0092] The user then visually confirms the display in step S72 (step S73). After that, the user selects a machine learning model that the user determines to be appropriate and inputs it to the second model selection unit 35 via the input unit 106 (step S74). In this embodiment, five machine learning models are selected at this time. Based on the input, the second model selection unit 35 selects the machine learning models from the first model group M1 as the second model group M2 and passes them to the third image selection unit 36 ​​(step S75).

[0093] As mentioned above, the first model selection step S33 can be performed automatically or manually, but repeated automatic selection does not improve accuracy significantly. To ensure a certain level of quality, it is preferable to perform the model selection steps from the second model selection step S35 onwards manually.

[0094] Following the second model selection step S35, the third image selection unit 36 ​​selects a third image group G3 to be used in the third model selection step S37 (third image selection step: step S36). Then, the third image selection unit 36 ​​passes the third image group G3 to the third model selection unit 37. The number of confirmation images included in the third image group G3 is greater than the number of confirmation images included in the second image group G2.

[0095] In the third image selection process S36, the third image selection unit 36 ​​may, similar to the first image selection process S32 and the second image selection process S34, select multiple images to form the third image group G3 using all confirmation images included in the total image group G0 and the fourth output result group D4, which is the output result of the total image group G0 / second model group M2.

[0096] The third image selection unit 36 ​​may perform the third image selection step S36 using a method different from that used in the first image selection step S32 and the second image selection step S34. For example, the third image selection unit 36 ​​may select all of the confirmation images included in the entire image group G0 as the third image group G3. The third image selection unit 36 ​​may also randomly select images from the confirmation images included in the entire image group G0. In this case, the confirmation images may include those included in the first image group G1 and those included in the second image group G2.

[0097] The third image selection unit 36 ​​transfers the third image group G3, the second model group M2, and a fifth output result group D5 which is the output result of the third image group G3 / second model group M2 to the third model selection unit 37. In this embodiment, the third image group G3 including 40 confirmation images and the fifth output result group D5 including 200 output results of the third image group G3 / second model group M2 are transferred to the third model selection unit 37.

[0098] After the third image selection process of step S36 is completed, the third model selection unit 37 then selects a machine learning model from the second model group M2 (step S37: third model selection process). The third model selection unit 37 uses the third image group G3 and the fifth output result group D5, which is the output result of the third image group G3 / second model group M2, to select one machine learning model from the machine learning models included in the second model group M2 as the selected model Mf. The third model selection unit 37 then outputs the selected model Mf.

[0099] When performing the third model selection process S37, as shown in FIG. 10, the third model selection unit 37 first acquires the third image group G3 to be used for selection and the fifth output result group D5, which is the output result of the third image group G3 / second model group M2 (step S71).

[0100] Next, for all of the confirmation images included in the third image group G3, the third model selection unit 37 displays one of the confirmation images and all of the output results of the second model group M2 for that confirmation image so that they can be compared.The user then visually confirms the display (step S72).This is performed for all of the confirmation images included in the third image group G3.

[0101] The user compares the confirmation image with the output result, selects a machine learning model that is deemed to provide an appropriate output result, and inputs the selected model to the input unit 106 (step S73). In this embodiment, one machine learning model is selected at this time. The third model selection unit 37 selects a machine learning model from the second model group M2 based on the input content as a selected model Mf and outputs the selected model (step S74).

[0102] According to the above procedure, in the machine learning model selection step S3, machine learning models are selected in three stages: a first model selection step S33, a second model selection step S35, and a third model selection step S37. In each selection step, the number of confirmation images to be checked is gradually increased from a small number, and the number of models to be selected is gradually reduced.

[0103] That is, after roughly selecting a large number of machine learning models using a small number of confirmation images, a larger number of confirmation images is used in a later model selection process to further select machine learning models. By gradually increasing the number of confirmation images and narrowing down the number of machine learning models to be selected in this way, it is possible to accurately select an appropriate machine learning model while minimizing the number of model output results to be confirmed during model selection. Therefore, it is possible to accurately and efficiently select a machine learning model from a large number of machine learning models without using ground truth.

[0104] In particular, by going through a model selection process of three or more stages, when there are a large number of machine learning models to select from, it is possible to select an appropriate machine learning model with greater accuracy than when selecting a machine learning model in two stages.

[0105] In this embodiment, the first model selection step S33 and the second model selection step S35 can be performed at the discretion of the user who has confirmed the output results of the machine learning model, thereby making it possible to appropriately select the first model group M1 and the second model group M2.

[0106] Furthermore, in this embodiment, the confirmation images used in each model selection step are selected from a large number of confirmation images, which allows for more accurate selection of machine learning models.

[0107] Furthermore, in this embodiment, when the first image selection step S32 is performed automatically, confirmation images that are likely to cause variability in the output results of the machine learning model to be selected are selected, which allows for accurate selection of a machine learning model even when selection is performed using a small number of confirmation images.

[0108] In this embodiment, the machine learning model to be selected performs image processing to recognize and segment cellular regions in an image containing cells. In this type of machine learning model, the input image varies significantly depending on the type of cells and reagents and the observation environment. Therefore, selecting an appropriate machine learning model is important.

[0109] In this embodiment, the machine learning model is selected using the analysis image that is the analysis target, thereby making it possible to select the machine learning model that is optimal for the analysis image.

[0110] <2. Modifications> Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment.

[0111] In the above embodiment, a three-stage model selection process, including a first model selection process, a second model selection process, and a third model selection process, was performed to select a machine learning model, but the present invention is not limited to this. Two-stage, or four or more-stage model selection processes may be performed to select a machine learning model. The number of stages in the model selection process can be freely set depending on the number of confirmation images and the number of machine learning models to be selected.

[0112] In the above embodiment, the second image selection step is performed after the first model selection step using the first model group selected in the first model selection step. The third image selection step is performed after the second model selection step using the second model group selected in the second model selection step. However, the present invention is not limited to this. The second image selection step and the third image selection step may be performed together with the first image selection step before the first model selection step. In this case, the second image selection step and the third image selection step are performed using the entire model group.

[0113] In the above embodiment, the confirmation images used in the first model selection step, the second model selection step, and the third model selection step were selected in the first image selection step, the second image selection step, and the third image selection step, respectively. However, these image selection steps may not be performed and the confirmation images may be selected randomly. Alternatively, only the first image selection step, or only the first image selection step and the second image selection step may be performed.

[0114] Furthermore, in the above-described embodiment, the machine learning model to be selected performs image processing to identify and segment cellular regions in an image containing cells. However, in the present invention, the machine learning model to be selected is not limited to this. The machine learning model to be selected in the present invention may be any machine learning model that performs image processing. For example, it may be a machine learning model that segments multiple cells and performs multi-label classification, a machine learning model that identifies objects in an image, or a machine learning model that performs image processing in a manner that makes the image easier to observe.

[0115] Furthermore, the elements appearing in the above-described embodiments and modifications may be combined as appropriate within the scope of not causing any contradiction. [Explanation of symbols]

[0116] 1. Machine learning model selection device 22 Machine Learning Model Selection Section 32 First image selection section 33 First Model Selection Section 34 Second image selection section 35 Second Model Selection Section 36 Third image selection section 37 Third Model Selection Section G0 All images G1 First image group G2 Second image group G3 Third image group All M0 models M1 First model group M2 2nd model group Mf selection model S3 Machine learning model selection process S32 First image selection step S33 First model selection process S34 Second image selection step S35 Second model selection process S36 Third image selection process S37 3rd model selection process

Claims

1. A model selection method for selecting at least one machine learning model from a plurality of machine learning models that perform image processing using a plurality of confirmation images, comprising: a) a first model selection step of selecting, using a first image group including at least one of the confirmation images, a plurality of the machine learning models from the machine learning models included in an entire model group including all machine learning models, to form a first model group; b) after the step a), a second model selection step of selecting at least one machine learning model from the machine learning models included in the first model group using a second image group including a plurality of the confirmation images to form a second model group; and A model selection method, wherein the number of the confirmation images included in the second image group is greater than the number of the confirmation images included in the first image group.

2. 2. The model selection method according to claim 1, The step a) a1) obtaining output results by inputting all of the confirmation images included in the first image group into all of the machine learning models included in the total model group; a2) displaying the output result obtained in step a1); a3) a step of externally inputting a selection result for selecting a portion of the machine learning models included in the entire model group; A model selection method, including:

3. 3. The model selection method according to claim 1 or 2, The step b) b1) obtaining output results by inputting all of the confirmation images included in the second image group into all of the machine learning models included in the first model group; b2) displaying the output result obtained in step b1); b3) a step of externally inputting a selection result for selecting a portion of the machine learning models included in the first model group; A model selection method, including:

4. 4. The model selection method according to claim 1, further comprising: the second model group includes a plurality of the machine learning models; c) after the step b), a third model selection step of selecting one of the machine learning models included in the second model group using a third image group including a plurality of the confirmation images. and A model selection method, wherein the number of the confirmation images included in the third image group is greater than the number of the confirmation images included in the second image group.

5. 5. The model selection method according to claim 1, further comprising: d) a first image selection step, prior to step a), of selecting one or more images from all of the confirmation images to form the first image group; e) a second image selection step, after the step a) and before the step b), of selecting a plurality of images from all of the confirmation images to form the second image group; The model selection method further comprises:

6. 6. The model selection method according to claim 5, The step d) d1) obtaining output results by inputting all of the confirmation images into all of the machine learning models included in the entire model group; d2) calculating the variability of the output results obtained in step d1) for each of the confirmation images; d3) selecting the confirmation images having the large variation calculated in step d2) from all the confirmation images to set the first image group; A model selection method, including:

7. 7. A model selection method according to claim 1, further comprising: A model selection method in which the machine learning model performs image processing to recognize and segment cellular regions in images containing cells.

8. An image processing method for performing image processing on a plurality of analysis images using a machine learning model, s) selecting one of the machine learning models by the model selection method according to any one of claims 1 to 7; t) performing image processing on the plurality of analysis images using the machine learning model selected in step s); and An image processing method, wherein the confirmation image in step s) is the analysis image to be processed in step t).

Images