Image analysis device

Abstract

The image analysis device (1) includes: an image holding unit (8) that holds an image; a learned model registration unit (10) configured to register a learned model created by machine learning; a learned model holding unit (12) that holds a learned model registered by the learned model registration unit (10); an algorithm holding unit (14) that holds a plurality of analysis algorithms for performing analysis processing of an image; a process creation unit (18) configured to create, for an analysis target image arbitrarily selected from the image held by the image holding unit (8), an analysis process for performing analysis of the analysis target image by combining a learned model arbitrarily selected from the learned models held by the learned model holding unit (10) and an analysis algorithm arbitrarily selected from the analysis algorithms held by the algorithm holding unit (14); and an analysis execution unit (20) configured to perform analysis of the analysis target image based on the analysis process created by the process creation unit (18).

Description

Technical Field

[0001] This invention relates to an image analysis device. Background Technology

[0002] In image processing analysis, estimation processing is performed to estimate the region and location of the target objects such as cells and cell nuclei captured in the image. However, this estimation processing requires parameters to distinguish between the target objects and non-target objects. Setting such parameters is not easy and is a time-consuming task.

[0003] In recent years, image analysis schemes using machine learning have been proposed and implemented (see Patent Document 1). In machine learning-based image analysis, a computer compares the image of the object to be analyzed with a label image (an image representing the boundaries, etc., of the object being analyzed captured in the image). The computer automatically acquires parameters, such as those needed to determine the region and specific location of the object being analyzed within the image. Then, the computer memorizes the acquired results as a learned model and applies the learned model to other images of the object being analyzed. This allows the computer to automatically estimate the region and specific location of the object being analyzed within the image.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-148950 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] The type of analysis performed on the target image varies from user to user. However, existing image analysis devices have the following problems: it is not easy to create a new learned model suitable for the target image, or to freely combine the desired analysis processing with the newly created learned model, resulting in a heavy workload for users.

[0009] Therefore, the purpose of this invention is to reduce the workload of users in image parsing using machine learning.

[0010] Solution for solving the problem

[0011] A first embodiment of the image parsing apparatus according to the present invention includes: an image holding unit that holds an image; a learning completed model registration unit configured to register a learning completed model created through machine learning; a learning completed model holding unit that holds the learning completed models registered by the learning completed model registration unit; an algorithm holding unit that holds a plurality of parsing algorithms for performing image parsing processing; a process manufacturing unit configured to: for a parsing target image arbitrarily selected from the images held by the image holding unit, combine a learning completed model selected from the learning completed models held by the learning completed model holding unit and a parsing algorithm arbitrarily selected from the parsing algorithms held by the algorithm holding unit, thereby creating a parsing process for parsing the parsing target image; and a parsing execution unit configured to execute the parsing of the parsing target image based on the parsing process created by the process manufacturing unit.

[0012] Here, "parse process" refers to a set of multiple algorithms required to perform the desired parsing on an image of the parsing object.

[0013] A second embodiment of the image analysis apparatus according to the present invention includes: an image holding unit that holds multiple images obtained by taking pictures of multiple cell culture wells disposed in a cell culture plate in states associated with the culture conditions of the cells in the cell culture wells; a process fabrication unit that is configured to: select multiple learning images from the images held by the image holding unit that correspond to multiple analysis target images arbitrarily selected from the images held by the image holding unit, combine the corresponding analysis target images and the learning images based on the culture conditions associated with each image to create multiple learning datasets, thereby creating an analysis process that uses the learning datasets to perform machine learning to create a learned model for image analysis; and an analysis execution unit that is configured to create the learned model based on the analysis process created by the process fabrication unit.

[0014] The effects of the invention

[0015] According to a first embodiment of the image parsing apparatus of the present invention, the learned model holding unit is able to hold not only existing learned models but also new learned models. Furthermore, it can combine any parsing algorithm held by the learned model holding unit and the parsing algorithm held by the algorithm holding unit for the parsing target image to create a parsing process. Therefore, it is easy to create the parsing process required to perform the parsing processing desired by the user. This reduces the workload for the user in image parsing using machine learning.

[0016] According to a second embodiment of the image analysis apparatus of the present invention, a learning image corresponding to each of the multiple analysis object images is automatically selected based on the training conditions associated with the image, thereby creating a learning dataset for machine learning, thus reducing the user's workload related to the creation of the learned model. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating one embodiment of an image resolution apparatus.

[0018] Figure 2 This is a flowchart that schematically illustrates an example of actions prior to image parsing in the same embodiment.

[0019] Figure 3 This is a flowchart illustrating an example of an action involving image parsing using a learned model.

[0020] Figure 4 This is a flowchart illustrating an example of a series of actions related to creating a learned model through machine learning.

[0021] Figure 5 This is a flowchart showing an example of a list of training datasets used to create a learned model.

[0022] Figure 6 This is a concept diagram illustrating an example of the process of creating a label image.

[0023] Figure 7 This is an example of learning to display a list of datasets. Detailed Implementation

[0024] Hereinafter, an embodiment of the image analysis apparatus according to the present invention will be described with reference to the accompanying drawings.

[0025] Figure 1 The outline structure of the image analysis device 1 is shown in the figure.

[0026] The image analysis device 1 includes an information processing device 2 and an information display device 4. The information processing device 2 has the function of acquiring and analyzing image data obtained by the image acquisition unit 6. The information processing device 2 is a computer device (e.g., a personal computer) equipped with electronic circuitry, including an information storage medium such as a hard disk drive and a CPU (central processing unit). The information processing device 2 is loaded with computer programs to implement the functions described later. The information display device 4 is a display (e.g., a liquid crystal display) that is communicatively connected to the information processing device 2. As the image acquisition unit 6, a microscope (e.g., a phase-contrast microscope, a fluorescence microscope) for taking pictures of the wells of a cell culture plate can be used.

[0027] The information processing device 2 includes an image holding unit 8, a learned model registration unit 10, a learned model holding unit 12, an algorithm holding unit 14, an algorithm registration unit 16, a process manufacturing unit 18, and a parsing execution unit 20. The learned model registration unit 10, the algorithm registration unit 16, the process manufacturing unit 18, and the parsing execution unit 20 are functionally implemented by the CPU executing a predetermined computer program imported into the information processing device 2. The image holding unit 8, the learned model holding unit 12, and the algorithm holding unit 14 are functionally implemented by a portion of the storage area of ​​the information storage medium within the information processing device 2.

[0028] In the image holding unit 8, images obtained by taking pictures of multiple cell culture wells placed in a cell culture plate using a phase-contrast microscope, images obtained by fluorescently staining cells or nuclei in the same cell culture well and taking pictures using a fluorescence microscope, and labeled images obtained by processing fluorescence microscope images are maintained in a state associated with the culture conditions of the cells in each cell culture well. The association between images and culture conditions can be achieved, for example, by appending culture conditions to the filename of the image data according to a prescribed rule.

[0029] The learned model registration unit 10 is configured to register new learned models required for image parsing. A learned model is information such as parameters required for image parsing, obtained through machine learning performed using a specific image and its label image. By applying the learned model to other images to be parsed, it is possible to estimate the cell regions captured in the image, and to perform parsing operations such as calculating the area of ​​cell regions and counting the number of cell nuclei using the estimation results. The learned models registered by the learned model registration unit 10 can be created using the image parsing device 1 or other image parsing devices. The learned models registered by the learned model registration unit 10 are stored in the learned model storage unit 12.

[0030] The algorithm holding unit 14 holds various parsing algorithms required for performing image parsing. Among the parsing algorithms held by the algorithm holding unit 14 are algorithms for performing parsing processes such as estimating cell regions in the parsing target image using a learned model, calculating the area of ​​cell regions in the image, and counting the number of cells. They also include algorithms for performing learning processes to create a learned model using the parsing target image and the label image. In other words, this image parsing device 1 has the function of creating a new learned model using the parsing target image and the label image even if no existing learned model exists, and performing the desired parsing processes using that learned model.

[0031] The algorithm registration unit 16 is configured to register new parsing algorithms. The parsing algorithms registered by the algorithm registration unit 16 are stored in the algorithm storage unit 14.

[0032] The process manufacturing unit 18 is configured to create the parsing process required to perform the parsing processing desired by the user on the parsing object image. When creating the parsing process, the user can arbitrarily select the learned model to be applied to the parsing object image and the parsing processing algorithm to be performed. The process manufacturing unit 18 combines the learned model and parsing algorithm selected by the user to create the parsing process for parsing the parsing object image.

[0033] The parsing execution unit 20 is configured to perform parsing processing on the parsing target image based on the parsing process produced by the process manufacturing unit 18.

[0034] use Figure 2 The flowchart illustrates the series of processes that precede the parsing of the object image.

[0035] The manufacturing department 18 displays an input screen on the information display device 4 for selecting the target image to be analyzed, the learned model, and the analysis algorithm. On this screen, the user selects the target image to be analyzed (step 101), the learned model to be analyzed (step 102), and the analysis algorithm to be analyzed (step 103). Furthermore, when selecting a learned model, if no existing learned model exists, if no learned model is available for the target image to be analyzed, or if image analysis is desired without using a learned model for estimation processing, the user can choose not to select any learned model. Additionally, when selecting an analysis algorithm, the user can select multiple analysis algorithms.

[0036] The process manufacturing unit 18 determines whether a new fully learned model needs to be created based on the user input information from steps 101 to 103 (step 104). If a new fully learned model does not need to be created, it creates a parsing process that includes estimation processing using the selected fully learned model, or a parsing process that does not use the fully learned model (if no fully learned model is selected) (step 105). The parsing execution unit 20 executes the parsing processing using the selected parsing algorithm according to the parsing process created by the parsing process manufacturing unit 18 (step 106).

[0037] Furthermore, if the process manufacturing unit 18 determines that a new learned model needs to be created ("Yes" in step 104), after creating the learning dataset required for creating the learned model, it creates a parsing process that includes machine learning (step 107). The parsing execution unit 20 creates the learned model according to the parsing process created by the parsing process manufacturing unit 18 (step 108), and performs parsing processing using the created learned model and the selected parsing algorithm (step 109).

[0038] Figure 3 This is an example of a process related to the analysis of a learned model.

[0039] In the analytical processing using the learned model, the learned model is applied to the image to be analyzed (step 201), and the boundary positions of cell regions and the positions of cell nuclei in the image to be analyzed are estimated using the parameter information of the learned model (step 202). In subsequent analytical processing, the estimated boundary positions of cell regions and the positions of cell nuclei are used to calculate the total area of ​​cell regions and count cell nuclei (step 203).

[0040] Simultaneously refer to Figures 5-7 and Figure 4 The flowchart illustrates a series of actions related to the creation of the model after learning.

[0041] like Figure 5 As shown, to create a fully learned model, it is necessary to parse the object images and the corresponding label images for each parsed object image. Figure 6 As shown, the label image can be obtained in the following way: after fluorescent staining the cells or nuclei in the same cell culture well as the image to be analyzed, the fluorescent staining image is obtained by taking a picture with a fluorescence microscope. Each image in the fluorescent staining image is processed by binarization, the boundary part of the object to be analyzed in each image is numerated, and the numerated processed images are synthesized.

[0042] Here, the label image used for machine learning along with each parsing object image, or the image that forms the basis of the label image (e.g., a fluorescently stained image), is defined as the "learning image". Then, the set of parsing object images and the corresponding label images or the images that form the basis of the label images is defined as the "learning dataset".

[0043] When according to Figure 4When explaining the flowchart, the process manufacturing unit 18 selects learning images from the images held by the image holding unit 8, which are used together with the analysis object images for machine learning, during the production of the analysis process including machine learning (step 301). Since each image held by the image holding unit 8 is associated with information related to the culture conditions of the cells in which the image was taken, the learning image corresponding to each analysis object image can be determined by referring to the culture conditions associated with each image. The process manufacturing unit 18 creates a learning dataset by combining analysis object images with the same culture conditions (well positions, etc.) with the learning images (step 302), and displays a list on the information display device 4 in a way that makes it easy to visually identify the analysis object images and learning images constituting each learning dataset (step 303). The user can confirm the list display on the information display device 4 and make edits as needed, such as changing the images constituting each learning dataset (step 304). When the learning image is the base image for a label image, the user can also select processing methods to make that image a label image.

[0044] exist Figure 7 In the example list, the learning datasets are arranged vertically, with information related to the culture conditions of each dataset displayed in the leftmost column. The parsed object images and learning images that make up each dataset are arranged horizontally, with information such as the associated culture conditions displayed below each image.

[0045] Furthermore, the list displays items for defining the purpose of each learning dataset, allowing you to assign each dataset to any of the "Learning," "Evaluation," or "Testing" categories. The learning dataset assigned to "Learning" is used to create a learned model through machine learning. The learning dataset assigned to "Evaluation" (also known as the evaluation dataset) is used to evaluate the created learned model. When multiple learning datasets are assigned to "Learning," multiple learned models are created, but the evaluation dataset is used to evaluate each model, and only the model with the highest evaluation is ultimately used. The learning dataset assigned to "Testing" (also known as the test dataset) is used to test the ultimately adopted learned model.

[0046] The allocation of uses for each learning dataset can be arbitrarily performed by the user, or it can be automatically performed by the process manufacturing department 18 when the user expects it. When it is desired that the use of each learning dataset be automatically allocated, the process manufacturing department 18 divides each learning dataset into multiple categories according to different cultivation conditions, and allocates the use of each learning dataset in such a way that the learning datasets for "learning", "evaluation" and "testing" exist in each category in a way that is roughly equal.

[0047] As described above, when the editing of the learning dataset is completed (the manufacturing process is completed), the parsing execution unit 20 performs preprocessing on each image as needed (e.g., binarization of fluorescence detection images), and then performs machine learning using each learning dataset allocated to "learning" (step 306) to create a learned model (step 307). Afterwards, the created learned model is evaluated using the evaluation dataset (step 308). The learned model registration unit 10 registers the learned model with the highest evaluation, and the learned model holding unit 12 holds that learned model (step 309).

[0048] The embodiments described above are merely examples of implementations of the image analysis apparatus of the present invention, and the implementations of the image analysis apparatus of the present invention are as follows.

[0049] In a first embodiment of the image parsing apparatus according to the present invention, the apparatus comprises: an image holding unit that holds an image; a learning completed model registration unit configured to register a learning completed model created through machine learning; a learning completed model holding unit that holds the learning completed models registered by the learning completed model registration unit; an algorithm holding unit that holds a plurality of parsing algorithms for performing image parsing processing; a process manufacturing unit configured to combine a learning completed model selected from the learning completed models held by the learning completed model holding unit and a parsing algorithm selected arbitrarily from the parsing algorithms held by the algorithm holding unit for a parsing target image arbitrarily selected from the images held by the image holding unit, thereby creating a parsing process for parsing the parsing target image; and a parsing execution unit configured to execute the parsing of the parsing target image based on the parsing process created by the process manufacturing unit.

[0050] In the first aspect of the first embodiment described above, the algorithm holding unit holds an analytical algorithm for performing machine learning to create a learned model. The process manufacturing unit is configured to: when it is determined based on information input by the user that a learned model needs to be created, create an analytical process for the analytical object image that includes machine learning for creating the learned model; and when creating the analytical process that includes machine learning, select a learning image for the machine learning from the images held by the image holding unit, and create a learning dataset formed by the learning image and the analytical object image. The learned model registration unit is configured to register the learned model created by using the machine learning through the learning dataset. According to this method, the user only needs to prepare a learning image corresponding to the analytical object image to automatically create the dataset required for machine learning and automatically perform machine learning based on that dataset to obtain a new learned model.

[0051] As a first embodiment of the aforementioned first method, the following example can be cited: the image is an image obtained by taking pictures of multiple cell culture wells placed in a cell culture plate, and each image is held in the image holding unit in a state associated with the culture conditions of the cells in the cell culture wells. The process manufacturing unit is configured to: when manufacturing the analysis process including the machine learning, if there are multiple analysis target images and multiple learning images, combine the corresponding analysis target images and learning images based on the culture conditions associated with each image to create multiple learning datasets. When there are a large number of analysis target images and learning images, the task of associating these analysis target images with learning images to create machine learning datasets is time-consuming, and there is a possibility of incorrect combination of analysis target images and learning images. However, in this first embodiment, the analysis target images and learning images are automatically combined according to the culture conditions associated with each image to create learning datasets, thus significantly reducing the user's workload when performing machine learning.

[0052] In the first embodiment described above, the process manufacturing unit may also be configured to present to the user, when producing multiple learning datasets, a list of the parsed object images constituting each learning dataset and the list of learning images, along with the cultivation conditions associated with each learning dataset. In this manner, the user can easily visually confirm the details of each automatically produced learning dataset.

[0053] Alternatively, in the first embodiment described above, the process fabrication unit may be configured to: based on information input by the user, set a portion of the plurality of learning datasets as an evaluation dataset for evaluating the fabricated learned model. In this case, the parsing execution unit may be configured to: fabricate a learned model by performing machine learning using the learning dataset, evaluate the fabricated learned model using the evaluation dataset, and register the learned model with the highest evaluation result. In this manner, a learned model with high resolution for the parsed object image can be obtained.

[0054] In the above scenario, the manufacturing process can also be configured such that, when a user expects to automatically select the learning dataset as the evaluation dataset, the multiple learning datasets are divided into multiple categories according to the training conditions, and at least one learning dataset belonging to each category is designated as the evaluation dataset. In this manner, a fully learned model applicable to a wide range of training conditions can be obtained.

[0055] In the second embodiment of the first method described above, an algorithm registration unit is further included. This algorithm registration unit is configured to register new parsing algorithms not held by the algorithm holding unit, and the algorithm holding unit is configured to hold the parsing algorithms registered by the algorithm registration unit. According to this method, it is possible to increase the number of parsing algorithms that can be executed on the parsing target image.

[0056] In a second embodiment of the image analysis apparatus according to the present invention, the apparatus comprises: an image holding unit that holds multiple images obtained by taking pictures of multiple cell culture wells disposed in a cell culture plate in states associated with the culture conditions of the cells in the cell culture wells; a process fabrication unit configured to: select multiple learning images from the images held by the image holding unit that correspond to multiple analysis target images arbitrarily selected from the images held by the image holding unit, and combine the corresponding analysis target images and the learning images based on the culture conditions associated with each image to create multiple learning datasets, thereby creating an analysis process that uses the learning datasets to perform machine learning to create a learned model for image analysis; and an analysis execution unit configured to create the learned model based on the analysis process created by the process fabrication unit.

[0057] In the first aspect of the second embodiment described above, the process manufacturing unit is configured to present to the user, when producing multiple learning datasets, a list of the parsed object images constituting each learning dataset and the list of learning images, along with the cultivation conditions associated with each learning dataset. In this manner, the user can easily visually confirm the details of each automatically produced learning dataset.

[0058] Furthermore, in the second aspect of the second embodiment described above, the process manufacturing unit is configured to: based on information input by the user, set a portion of the plurality of learning datasets as an evaluation dataset for evaluating the manufactured learned model; and the parsing execution unit is configured to: manufacture the learned model by performing machine learning using the learning dataset, and perform evaluation on the manufactured learned model using the evaluation dataset. According to this method, the evaluation of the learned model can be performed automatically.

[0059] In the second approach described above, the manufacturing process can also be configured such that, when a user expects to automatically select the learning dataset as the evaluation dataset, the multiple learning datasets are divided into multiple categories according to the training conditions, and at least one learning dataset belonging to each category is designated as the evaluation dataset. In this manner, it is possible to evaluate whether the manufactured, trained model can be applied to a wide range of training conditions.

[0060] Explanation of reference numerals in the attached figures

[0061] 1: Image analysis device; 2: Information processing device; 4: Information display device; 6: Image acquisition unit; 8: Image storage unit; 10: Model registration unit after learning; 12: Model storage unit after learning; 14: Algorithm storage unit; 16: Algorithm registration unit; 18: Process manufacturing unit; 20: Analysis and execution unit.

Claims

1. An image analysis device, comprising: The image holding unit holds the image; The Model Registration Department is configured to register models that have been learned through machine learning. The learning-completed model retention unit retains the learning-completed models registered by the learning-completed model registration unit. The algorithm holding unit holds multiple parsing algorithms used to perform parsing processing of the image; The process manufacturing unit is configured to: for a parsing object image arbitrarily selected from the images held by the image holding unit, combine a learned model selected from the learned models held by the learned model holding unit and a parsing algorithm arbitrarily selected from the parsing algorithms held by the algorithm holding unit, thereby manufacturing a parsing process for parsing the parsing object image. as well as The parsing execution unit is configured to perform parsing of the parsing object image based on the parsing process produced by the process manufacturing unit. The algorithm holding section holds an analytical algorithm for performing machine learning to create a learned model. The process manufacturing unit is configured to: when it is determined, based on information input by the user, that a fully learned model needs to be manufactured, manufacture a parsing process for the parsing object image that includes machine learning for manufacturing the fully learned model; and is further configured to: when manufacturing the parsing process that includes machine learning, select a learning image for the machine learning from the images held by the image holding unit, and manufacture a learning dataset formed by the learning image and the parsing object image. The learning-completed model registration unit is configured to register learning-completed models created using the machine learning method with the learning dataset. The images are obtained by taking pictures of multiple cell culture wells placed in a cell culture plate, and each image is held in the image holding unit in a state associated with the culture conditions of the cells in the cell culture well in which it was taken. The process manufacturing unit is configured such that, when manufacturing the analysis process that includes the machine learning, if there are multiple analysis object images and multiple learning images, the analysis object images and the learning images corresponding to each image are combined based on the training conditions associated with each image to create multiple learning datasets.

2. The image analysis apparatus according to claim 1, wherein, The manufacturing process is configured such that, when producing multiple learning datasets, it presents to the user the parsed object images constituting each learning dataset, a list of the learning images, and the cultivation conditions associated with each learning dataset.

3. The image analysis apparatus according to claim 1, wherein, The manufacturing process is configured to: based on information input by the user, designate a portion of the multiple learning datasets as an evaluation dataset for evaluating the manufactured, fully-learned model. The parsing execution unit is configured to: create a learned model by performing machine learning using the learning dataset; and evaluate the created learned model using the evaluation dataset. The learning completion model registration unit is configured to register the learning completion model with the highest evaluation result.

4. The image analysis apparatus according to claim 3, wherein, The process manufacturing department is configured such that, when a user expects to automatically select the learning dataset as the evaluation dataset, it divides the multiple learning datasets into multiple categories according to the training conditions, and sets at least one learning dataset belonging to each category as the evaluation dataset.

5. The image analysis apparatus according to claim 1, wherein, It also includes an algorithm registration unit, which is configured to register new parsing algorithms that are not held in the algorithm holding unit. The algorithm holding unit is configured to hold the parsing algorithm registered by the algorithm registration unit.

6. An image analysis device, comprising: The image holding unit holds multiple images obtained by taking pictures of multiple cell culture wells set in a cell culture plate in a state that is associated with the culture conditions of the cells in the cell culture wells, respectively. The process manufacturing unit is configured to: select multiple learning images from the images held by the image holding unit that correspond to multiple parsing object images arbitrarily selected from the images held by the image holding unit; combine the corresponding parsing object images and the learning images based on the training conditions associated with each image to create multiple learning datasets; and thereby create a parsing process that uses the learning datasets to perform machine learning to create a learned model for image parsing; and The parsing execution unit is configured to produce the learned model based on the parsing process produced by the process manufacturing unit.

7. The image analysis apparatus according to claim 6, wherein, The manufacturing process is configured such that, when producing multiple learning datasets, it presents to the user the parsed object images constituting each learning dataset, a list of the learning images, and the cultivation conditions associated with each learning dataset.

8. The image analysis apparatus according to claim 6, wherein, The manufacturing process is configured to: based on information input by the user, designate a portion of the multiple learning datasets as an evaluation dataset for evaluating the manufactured, fully-learned model. The parsing execution unit is configured to: create a learned model by performing machine learning using the learning dataset, and evaluate the created learned model using the evaluation dataset.

9. The image analysis apparatus according to claim 8, wherein, The process manufacturing department is configured such that, when a user expects to automatically select the learning dataset as the evaluation dataset, it divides the multiple learning datasets into multiple categories according to the training conditions, and sets at least one learning dataset belonging to each category as the evaluation dataset.

Images