Image analysis equipment

Abstract

To provide an image analyzer that lightens workload of a user in image analysis using machine learning.SOLUTION: An image analyzer comprises: an image holding unit which holds an image; a learned model registration unit which is constituted to register a learned model created by machine learning; a learned model holding unit which holds a registered learned model; an algorithm holding unit which holds a plurality of analysis algorithms for executing analysis processing of an image; a recipe creation unit which is constituted to combine a selected learned model and an arbitrarily selected analysis algorithm to an image to be analyzed which has been arbitrarily selected from images held in the image holding unit, and create an analysis recipe for analyzing the image to be analyzed; and an analysis execution unit which is constituted to execute analysis of the image to be analyzed on the basis of the analysis recipe created by the recipe creation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image analysis device. [Background technology]

[0002] In analysis using image processing, an estimation process is performed to estimate the area and position of the object to be analyzed, such as cells and their nuclei, that are visible in the image to be analyzed. However, this estimation process requires parameters to distinguish between the object to be analyzed and other parts, and setting such parameters is not easy and is a time-consuming task.

[0003] In recent years, image analysis using machine learning has been proposed and implemented (see Patent Document 1). In image analysis using machine learning, a computer compares an image to be analyzed with a label image (an image showing the boundary of the object to be analyzed that is shown in the image to be analyzed), and the computer automatically acquires parameters and the like necessary to identify the area of ​​the object to be analyzed and the position of a specific part in the image to be analyzed. The acquired results are then memorized by the computer as a trained model, and the trained model can be applied to other images to be analyzed, allowing the computer to automatically estimate the area of ​​the object to be analyzed and the specific position in the image to be analyzed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-148950 Summary of the Invention [Problem to be solved by the invention]

[0005] The type of analysis to be performed on an image to be analyzed varies from user to user. However, with existing image analysis devices, it is not easy to create a new trained model suited to the image to be analyzed or to freely combine a desired analysis process with the newly created trained model, which creates a problem of a heavy workload for the user.

[0006] Therefore, an object of the present invention is to reduce the workload on users in image analysis using machine learning. [Means for solving the problem]

[0007] A first embodiment of an image analysis device according to the present invention comprises an image storage unit that stores images, a trained model registration unit configured to register trained models created by machine learning, a trained model storage unit that stores trained models registered by the trained model registration unit, an algorithm storage unit that stores a plurality of analysis algorithms for performing image analysis processing, a recipe creation unit that is configured to combine, for an analysis target image arbitrarily selected from the images stored in the image storage unit, a trained model selected from the trained models stored in the trained model storage unit and an analysis algorithm arbitrarily selected from the analysis algorithms stored in the algorithm storage unit, to create an analysis recipe for analyzing the analysis target image, and an analysis execution unit that is configured to perform analysis of the analysis target image based on the analysis recipe created by the recipe creation unit.

[0008] Here, the term "analysis recipe" refers to a set of multiple algorithms required to perform a desired analysis on an image to be analyzed.

[0009] A second embodiment of the image analysis device of the present invention includes an image storage unit that stores multiple images obtained by capturing images of each of multiple cell culture wells in a cell culture plate, each image being associated with the culture conditions of the cells in the cell culture wells being captured; a recipe creation unit that is configured to select multiple training images corresponding to multiple analysis target images arbitrarily selected from the images stored in the image storage unit, combine the analysis target images and the training images that correspond to each other based on the culture conditions associated with each image, create multiple training datasets, and perform machine learning using the training datasets to create a trained model for image analysis; and an analysis execution unit that is configured to create the trained model based on the analysis recipe created by the recipe creation unit. [Effects of the Invention]

[0010] According to the first embodiment of the image analysis device of the present invention, not only existing trained models but also new trained models can be stored in the trained model storage unit, and an analysis recipe can be created for an image to be analyzed by combining a trained model stored in the trained model storage unit with any of the analysis algorithms stored in the algorithm storage unit. This makes it possible to easily create an analysis recipe required to execute the analysis process desired by the user. This reduces the workload on the user in image analysis using machine learning.

[0011] According to the second embodiment of the image analysis device of the present invention, a learning image corresponding to each of a plurality of images to be analyzed is automatically selected based on the culture conditions associated with the image, and a learning dataset for machine learning is created, thereby reducing the workload on the user in creating a trained model. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an embodiment of an image analysis device; [Figure 2] 10 is a flowchart showing an example of an operation up to image analysis in the embodiment. [Figure 3] 10 is a flowchart illustrating an example of an image analysis operation using a trained model. [Figure 4] 1 is a flowchart illustrating an example of a series of operations related to creating a trained model through machine learning. [Figure 5] 10 is a flowchart illustrating an example of a list display of training datasets used to create a trained model. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a procedure for creating a label image. [Figure 7] 10 is an example of a list display of training data sets. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of an image analysis device according to the present invention will be described below with reference to the drawings.

[0014] FIG. 1 shows a schematic configuration of an image analysis device 1.

[0015] The image analysis device 1 includes an information processing device 2 and an information display device 4. The information processing device 2 has a function of importing image data acquired by an image acquisition unit 6 and performing analysis processing. The information processing device 2 is a computer device (e.g., a personal computer) equipped with an information storage medium such as a hard disk drive and an electronic circuit equipped with a CPU (central processing unit), and is installed with a computer program for realizing each function described below. The information display device 4 is a display (e.g., a liquid crystal display) connected to the information processing device 2 so as to be able to communicate with it. The image acquisition unit 6 can be a microscope (e.g., a phase-contrast microscope, a fluorescence microscope) for capturing images of the inside of each well of a cell culture plate.

[0016] The information processing device 2 includes an image storage unit 8, a trained model registration unit 10, a trained model storage unit 12, an algorithm storage unit 14, an algorithm registration unit 16, a recipe creation unit 18, and an analysis execution unit 20. The trained model registration unit 10, the algorithm registration unit 16, the recipe creation unit 18, and the analysis execution unit 20 are functions realized by a CPU executing a predetermined computer program installed in the information processing device 2. The image storage unit 8, the trained model storage unit 12, and the algorithm storage unit 14 are functions realized by a partial storage area of ​​an information storage medium within the information processing device 2.

[0017] The image storage unit 8 stores images of the inside of each of a plurality of cell culture wells provided in the cell culture plate taken with a phase-contrast microscope, images of cells or nuclei in the same cell culture wells fluorescently stained and taken with a fluorescence microscope, labeled images obtained by processing the fluorescence microscope images, and the like, all associated with the culture conditions of the cells in each cell culture well. The association of images with the culture conditions can be realized, for example, by adding the culture conditions to the file names of the image data according to a predetermined rule.

[0018] The trained model registration unit 10 is configured to register a new trained model required for image analysis. A trained model is information such as parameters required for image analysis, acquired by machine learning performed using an image and a labeled image of that image. By applying the trained model to another image to be analyzed, it is possible to estimate the cellular region appearing in the image to be analyzed, and the estimation result can be used to perform analyses such as determining the area of ​​the cellular region in the image or counting the number of cell nuclei. The trained model registered by the trained model registration unit 10 may be created by this image analysis device 1 or by another image analysis device. The trained model registered by the trained model 10 is stored in the trained model storage unit 12.

[0019] The algorithm storage unit 14 stores multiple types of analysis algorithms required to perform image analysis. The analysis algorithms stored in the algorithm storage unit 14 include algorithms for performing analysis processes such as estimating a cellular region in an image to be analyzed using a trained model, calculating the area of ​​a cellular region in the image, and counting the number of cells, as well as algorithms for performing a learning process to create a trained model using the image to be analyzed and a labeled image. In other words, even if an existing trained model does not exist, this image analysis device 1 has the function of creating a new trained model using the image to be analyzed and a labeled image, and performing a desired analysis process using the trained model.

[0020] The algorithm registration unit 16 is configured to register a new analysis algorithm. The analysis algorithm registered by the algorithm registration unit 16 is held in the algorithm holding unit 14.

[0021] The recipe creation unit 18 is configured to create an analysis recipe required to execute the analysis processing desired by the user on the image to be analyzed. When creating the analysis recipe, the user can arbitrarily select a trained model to be applied to the image to be analyzed and an algorithm for the analysis processing to be executed, and the recipe creation unit 18 combines the trained model and analysis algorithm selected by the user to create an analysis recipe for the analysis processing on the image to be analyzed.

[0022] The analysis execution unit 20 is configured to execute analysis processing on the image to be analyzed based on the analysis recipe created by the recipe creation unit 18.

[0023] The flow of the process up to the execution of the analysis processing on the image to be analyzed will be explained using the flowchart in FIG.

[0024] The recipe creation unit 18 displays an input screen on the information display device 4 for selecting an image to be analyzed, a trained model, and an analysis algorithm, and on that screen the user selects an image to be analyzed (step 101), selects a trained model (step 102), and selects an analysis algorithm (103). When selecting a trained model, the user can choose not to select any trained model if there is no existing trained model, if there is no trained model to be applied to the image to be analyzed, or if image analysis that does not require estimation processing by applying a trained model is desired. Furthermore, when selecting an analysis algorithm, the user can select multiple analysis algorithms.

[0025] The recipe creation unit 18 determines whether or not it is necessary to create a new trained model based on the information input by the user in steps 101 to 103 (step 104), and if it is not necessary to create a new trained model, creates an analysis recipe including an estimation process using the selected trained model, or an analysis recipe that does not use a trained model (if no trained model is selected) (step 105).The analysis execution unit 20 executes an analysis process using the selected analysis algorithm in accordance with the analysis recipe created by the analysis recipe creation unit 18 (step 106).

[0026] Furthermore, if the recipe creation unit 18 determines that a new trained model needs to be created (step 104: Yes), it creates a training dataset necessary for creating the trained model, and then creates an analysis recipe including machine learning (step 107). The analysis execution unit 20 creates a trained model in accordance with the analysis recipe created by the analysis recipe creation unit 18 (step 108), and executes analysis processing using the created trained model and the selected analysis algorithm (step 109).

[0027] Figure 3 shows an example of the flow for analysis using a trained model.

[0028] In the analysis process using the trained model, the trained model is applied to the image to be analyzed (step 201), and the boundary positions of the cellular regions, the positions of the cell nuclei, etc. in the image to be analyzed are estimated using the parameter information of the trained model (step 202).In the subsequent analysis process, the boundary positions of the cellular regions, the positions of the cell nuclei, etc. estimated by the estimation process are used to calculate the total area of ​​the cellular regions, count the number of cell nuclei, etc. (step 203).

[0029] A series of operations related to the creation of a trained model will be described with reference to FIGS. 5 to 7 along with the flowchart of FIG.

[0030] To create a trained model, images to be analyzed and corresponding labeled images for each image to be analyzed are required, as shown in Figure 5. As shown in Figure 6, labeled images can be obtained by subjecting cells and nuclei in the same cell culture well as the image to be analyzed to fluorescent staining and capturing them under a fluorescent microscope to processing such as binarization to digitize the boundary areas of the area to be analyzed in each image, and then combining the digitized processed images.

[0031] Here, the labeled images or the images that are the basis for the labeled images (e.g., fluorescently stained images) used together with each analysis target image for machine learning are defined as "learning images."The set of the analysis target image and its corresponding labeled images or the images that are the basis for the labeled images is defined as a "learning dataset."

[0032] Referring to the flowchart of FIG. 4, when creating an analysis recipe including machine learning, the recipe creation unit 18 selects training images to be used for machine learning along with each analysis target image from among the images stored in the image storage unit 8 (step 301). Since each image stored in the image storage unit 8 is associated with information about the culture conditions of the cells depicted in that image, the training image corresponding to each analysis target image can be identified by referencing the culture conditions associated with each image. The recipe creation unit 18 creates a training dataset by combining the analysis target image and training images with the same culture conditions (e.g., well position) (step 302), and displays a list of the analysis target images and training images constituting each training dataset on the information display device 4 so that they can be easily recognized visually (step 303). While checking the list display on the information display device 4, the user can edit the images constituting each training dataset, such as by changing them, as necessary (step 304). If the training images are images that serve as the basis for label images, the user can also select processing to turn the images into label images.

[0033] In the example of the list display in Figure 7, each training dataset is displayed vertically, with information about the culture conditions for each dataset displayed in the leftmost column. The analysis target images and training images that make up each dataset are displayed horizontally, with information about the culture conditions and other information associated with each image displayed below each image.

[0034] Furthermore, this list display displays an item for setting the purpose of each training dataset, allowing each training dataset to be assigned to either "training," "evaluation," or "test." Training datasets assigned to "training" are used to create trained models through machine learning, while training datasets (also called evaluation datasets) assigned to "evaluation" are used to evaluate the created trained models. If multiple training datasets are assigned to "training," multiple trained models will be created, but each trained model will be evaluated using the evaluation dataset, and only the trained model with the highest evaluation will be ultimately adopted. Training datasets (also called test datasets) assigned to "test" are used to test the trained model that is finally adopted.

[0035] The allocation of a purpose to each training dataset can be arbitrarily performed by the user, or, if desired by the user, can be performed automatically by the recipe creation unit 18. When automatic allocation of a purpose to each training dataset is desired, the recipe creation unit 18 classifies each training dataset into a plurality of categories based on differences in culture conditions, and assigns a purpose to each training dataset so that the training datasets assigned to "training," "evaluation," and "test" are present in each category approximately equally.

[0036] Once the editing of the training datasets is completed (the creation of the recipe is completed) in this manner, the analysis execution unit 20 performs preprocessing of each image as necessary (for example, binarization of the fluorescence detection image, etc.), then executes machine learning using each training dataset assigned to "learning" (step 306), and creates a trained model (step 307). The created trained model is then evaluated using an evaluation dataset (step 308). The trained model registration unit 10 registers the trained model with the highest evaluation and stores it in the trained model storage unit 12 (step 309).

[0037] The above-described embodiment is merely an example of an embodiment of the image analysis device according to the present invention, and other embodiments of the image analysis device according to the present invention are as follows.

[0038] A first embodiment of the image analysis device according to the present invention includes an image storage unit that stores images, a trained model registration unit that is configured to register trained models created by machine learning, a trained model storage unit that stores trained models registered by the trained model registration unit, an algorithm storage unit that stores a plurality of analysis algorithms for performing image analysis processing, a recipe creation unit that is configured to combine, for an analysis target image arbitrarily selected from the images stored in the image storage unit, a trained model selected from the trained models stored in the trained model storage unit and an analysis algorithm arbitrarily selected from the analysis algorithms stored in the algorithm storage unit, to create an analysis recipe for analyzing the analysis target image, and an analysis execution unit that is configured to execute analysis of the analysis target image based on the analysis recipe created by the recipe creation unit.

[0039] In a first aspect of the first embodiment, the algorithm holding unit holds an analysis algorithm for performing machine learning to create a trained model, and the recipe creation unit is configured to create an analysis recipe including machine learning for creating the trained model of the image to be analyzed when it determines that a trained model needs to be created based on information input by a user, and is configured to select training images to be used in the machine learning from images held in the image holding unit and create a training dataset consisting of the training images and the image to be analyzed, and the trained model registration unit is configured to register the trained model created by the machine learning using the training dataset. With this aspect, a user only needs to prepare training images corresponding to the image to be analyzed, and the dataset required for machine learning is automatically created, and machine learning based on the dataset is automatically performed to obtain a new trained model.

[0040] As a first example of the first aspect, the images are images obtained by capturing images of the interiors of multiple cell culture wells provided in a cell culture plate, and each image is stored in the image storage unit in association with the culture conditions of the cells in the captured cell culture well. The recipe creation unit is configured to, when creating the analysis recipe including the machine learning, combine corresponding analysis target images and training images based on the culture conditions associated with each image, to create multiple training datasets, if there are multiple analysis target images and multiple training images. When there are multiple analysis target images and training images, linking the analysis target images and training images to create a machine learning dataset takes time, and there is a risk of incorrect combinations of the analysis target images and training images. However, in this first example, the analysis target images and training images are automatically combined based on the culture conditions associated with each image to create a training dataset, significantly reducing the user's workload when performing machine learning.

[0041] In the first embodiment, the recipe creation unit may be configured to, when creating a plurality of training data sets, present to a user a list of the analysis target images and the training images constituting each training data set, together with the culture conditions associated with each training data set. This allows the user to easily visually check the details of each automatically created training data set.

[0042] In the first embodiment, the recipe creation unit may be configured to use some of the training datasets as an evaluation dataset for evaluating the created trained model based on information input by a user, in which case the analysis execution unit may be configured to create a trained model by performing the machine learning using the training dataset and to evaluate the created trained model using the evaluation dataset, and the trained model registration unit may be configured to register the trained model with the highest evaluation result. This aspect makes it possible to obtain a trained model that can provide high analysis accuracy for an image to be analyzed.

[0043] In the above case, when a user requests automatic selection of the training dataset to be used as the evaluation dataset, the recipe creation unit may be configured to classify the multiple training datasets into multiple categories based on the culture conditions, and to use at least one training dataset belonging to each category as the evaluation dataset. With this configuration, it is possible to obtain a trained model that is applicable to a wide range of culture conditions.

[0044] In a second aspect of the first embodiment, the apparatus further includes an algorithm registration unit configured to register a new analysis algorithm not stored in the algorithm storage unit, and the algorithm storage unit is configured to store the analysis algorithms registered by the algorithm registration unit. This aspect makes it possible to increase the number of analysis algorithms that can be executed on the image to be analyzed.

[0045] A second embodiment of the image analysis device of the present invention includes an image storage unit that stores multiple images obtained by capturing images of the insides of multiple cell culture wells in a cell culture plate, each image being associated with the culture conditions of the cells in the cell culture wells being captured; a recipe creation unit that is configured to create an analysis recipe that selects multiple training images corresponding to multiple analysis target images arbitrarily selected from the images stored in the image storage unit, combines the analysis target images and training images that correspond to each other based on the culture conditions associated with each image, and performs machine learning using the training datasets to create a trained model for image analysis; and an analysis execution unit that is configured to create the trained model based on the analysis recipe created by the recipe creation unit.

[0046] In a first aspect of the second embodiment, the recipe creation unit is configured, when creating a plurality of the training data sets, to present to a user a list of the analysis target images and the training images constituting each training data set, together with culture conditions associated with each training data set. This aspect allows a user to easily visually check the details of each automatically created training data set.

[0047] In a second aspect of the second embodiment, the recipe creation unit is configured to use some of the training datasets as an evaluation dataset for evaluating the created trained model based on information input by a user, and the analysis execution unit is configured to create a trained model by performing the machine learning using the training dataset and to evaluate the created trained model using the evaluation dataset. This aspect makes it possible to automatically evaluate the trained model.

[0048] In the second aspect, when a user requests automatic selection of the training dataset to be used as the evaluation dataset, the recipe creation unit may be configured to classify the training datasets into a plurality of categories based on the culture conditions, and to use at least one training dataset belonging to each category as the evaluation dataset. This aspect makes it possible to evaluate whether the created trained model is applicable to a wide range of culture conditions. [Explanation of symbols]

[0049] 1. Image analysis device 2. Information processing equipment 4 Information display device 6 Image acquisition unit 8 Image storage unit 10 Trained model registration section 12 Trained model storage unit 14 Algorithm storage unit 16 Algorithm Registration Section 18 Recipe Creation Department 20 Analysis Execution Department

Claims

1. An algorithm storage unit that stores an algorithm for creating a trained model for image analysis by machine learning using a training dataset including an analysis target image and training images; an image storage unit that stores a plurality of images obtained by capturing images of the interiors of a plurality of cell culture wells provided in the cell culture plate, each image being associated with the culture conditions of the cells in the captured cell culture wells; a recipe creation unit configured to select a plurality of learning images corresponding to a plurality of analysis target images arbitrarily selected from the images stored in the image storage unit, combine the analysis target images and the learning images corresponding to each other based on the culture conditions associated with each image to create a plurality of learning datasets, and perform machine learning using the learning datasets by the algorithm stored in the algorithm storage unit to create an analysis recipe for creating a trained model for image analysis; an analysis execution unit configured to create the trained model using the algorithm based on the analysis recipe created by the recipe creation unit; a trained model registration unit configured to register the trained model created by the analysis execution unit as a trained model for the image analysis, the recipe creation unit is configured to prompt a user to assign an evaluation purpose to any training data set among the plurality of training data sets, and to assign training purposes to at least a portion of the remaining training data sets; The analysis execution unit is configured to create one or more trained models by performing the machine learning using one or more training datasets assigned to a training purpose, and to evaluate the created one or more trained models using the training datasets assigned to an evaluation purpose; The image analysis device, wherein the trained model registration unit is configured to register only the trained model having the highest evaluation result among the one or more trained models created by the analysis execution unit as the trained model for the image analysis.

2. 2. The image analysis device according to claim 1, wherein, when the plurality of training data sets are created, the recipe creation unit is configured to present to a user a list of the analysis target images and the training images constituting each training data set, together with culture conditions associated with each training data set.

3. 2. The image analysis device according to claim 1, wherein the recipe creation unit is configured to classify the plurality of training datasets into a plurality of categories based on the culture conditions and assign the evaluation purpose to at least one of the training datasets belonging to each category when a user desires automatic selection of the training dataset to which the evaluation purpose is to be assigned.

Images