Object classification apparatus, object classification system, and object classification method

Abstract

A representative feature amount representing a kind or a state of the object is extracted, the kind or the state of the object is discriminated based on the representative feature amount, the object is classified, and a result of the classification of the object and the representative feature amount are associated with the image and output.

Description

Technical Field

[0001] This invention relates to an object classification device, an object classification system, and an object classification method. Background Technology

[0002] When observing cells and materials, optical microscopes, electron microscopes, etc., are generally used. However, observing microscope images with the naked eye is very time-consuming and often requires specialized knowledge. Therefore, to assist in the evaluation of cells or materials using microscope images, techniques have been developed to automate some of the processing using image processing.

[0003] For example, Patent Document 1 discloses a method for determining the state of cells by using the temporal changes in the morphological characteristics of two different types of cells. However, the surfaces of cells or materials are mostly amorphous, and sometimes it is not possible to determine the state with high precision using only manually designed feature quantities.

[0004] On the other hand, in recent years, there have been reports of improvements in the accuracy of microscope image resolution compared to previous methods achieved through the use of machine learning, primarily deep learning. In the case of deep learning, features are learned automatically, sometimes yielding features that are difficult to design manually.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2011-229409 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] By using deep learning to generate classifiers for objects, it is possible to automatically learn feature quantities that are effective in improving classification accuracy compared to manually designing feature quantities.

[0010] However, when using a typical deep learning classifier, the user only receives the classification result. For example, when the human eye observes a microscope image, even if features considered valid for classification exist in a specific part of the object, it is uncertain whether the feature quantity has been correctly learned and extracted from that part. Therefore, it is difficult to classify objects within an image with high accuracy.

[0011] The purpose of this invention is to classify objects within an image with high accuracy in an object classification device.

[0012] Methods for solving problems

[0013] An object classification apparatus according to one aspect of the present invention is an object classification apparatus that classifies objects within an image to determine the type or state of the objects. The object classification apparatus includes: an object region calculation unit that calculates the object region of the object within the image; a feature selection unit that uses the object region to select feature values ​​of the object and outputs a feature selection signal; a feature extraction unit that extracts multiple feature values ​​from the image based on the feature selection signal; a feature classification unit that classifies the multiple feature values ​​to extract representative feature values ​​that determine the type or state of the object; an object classification unit that classifies the object based on the representative feature values ​​to determine the type or state of the object and outputs a classification result of the object; and an output unit that associates the classification result of the object and the representative feature values ​​with the image and outputs the result.

[0014] Invention Effects

[0015] According to one aspect of the present invention, objects within an image can be classified with high precision in an object classification device. Attached Figure Description

[0016] Figure 1 This illustrates an example of the hardware structure of the object classification system in Embodiment 1.

[0017] Figure 2 This is an example of a functional block diagram showing the object classification device of Embodiment 1.

[0018] Figure 3 This is an example of a microscope image obtained by photographing ES cells.

[0019] Figure 4 This is an example of the result of a region segmentation application.

[0020] Figure 5 This represents an example of separation of contacting objects inferred from a distance image.

[0021] Figure 6 An example of extracting characteristic quantities from the cell nucleus.

[0022] Figure 7 An example representing the cell membrane region and the cytoplasm region.

[0023] Figure 8 This represents an example of calculating the representative characteristic of an object.

[0024] Figure 9 An example of a table representing the correspondence between characteristic quantities and object classes.

[0025] Figure 10 This is an example of displaying the object classification results of Example 1.

[0026] Figure 11 This illustrates an example of the hardware structure of the object retrieval system in Embodiment 2.

[0027] Figure 12A This represents an example of the GUI in Example 2.

[0028] Figure 12B This represents an example of the GUI in Example 2.

[0029] Figure 13 This illustrates an example of the hardware structure of the object statistics information calculation system in Embodiment 3.

[0030] Figure 14 This represents an example of the GUI in Example 3.

[0031] Figure 15 This is an example of a processing flowchart for Example 1.

[0032] Figure 16 This is an example of a microscope image obtained by photographing the surface of a raw material.

[0033] Figure 17 An example of a table representing the correspondence between characteristic quantities and object classes.

[0034] Figure 18 This is an example of displaying the object classification results of Example 1.

[0035] Figure 19 This is an example of the statistical information display results for Example 3. Detailed Implementation

[0036] Hereinafter, embodiments of the object classification apparatus, method, and system of the present invention will be described with reference to the accompanying drawings. Furthermore, in the following description and drawings, the same reference numerals are used for constituent elements having the same functional structure, and repeated descriptions are omitted.

[0037] Example 1

[0038] use Figure 1 An example of an in-image object classification system from Example 1 will be described.

[0039] The object classification system 100 includes an object classification device 101, an interface unit 110, a processing unit 111, a memory 112, and a bus 113. The interface unit 110, the processing unit 111, and the memory 112 transmit and receive information via the bus 113. In addition, the object classification device 101 is connected to the imaging device 120 and the display device 121 via the interface unit 110.

[0040] Each part of the object classification device 101 will be described.

[0041] The interface unit 110 is a communication device for transmitting and receiving signals with devices located outside the object sorting device 101. As a device for communicating with the interface unit 110, it includes an imaging device 120 and a display device 121. Details of the imaging device 120 and the display device 121 will be described later.

[0042] The arithmetic unit 111 is a device that performs various processes within the object classification device 101, such as a CPU (Central Processing Unit) or an FPGA (Field-Programmable Gate Array). Regarding the functions performed by the arithmetic unit 111, [the following is a description of its functions]. Figure 2 To be described later.

[0043] The memory 112 is a device for storing programs, parameters, coefficients, processing results, etc. executed by the arithmetic unit 111, and can be HDD, SSD, RAM, ROM, flash memory, etc.

[0044] The imaging device 120 is a device for capturing images of an object, such as a camera or microscope. The imaging device 120 sends the captured images to the object classification device 101.

[0045] The display device 121 is a device for displaying object classification information output by the object classification device 101, such as a monitor or printer.

[0046] The object classification device 101 will be described in detail below.

[0047] Figure 2 This is an example of a functional block diagram of Embodiment 1 of the object classification device 101. Furthermore, these functions can be implemented by dedicated hardware or by software operating on the arithmetic unit 111.

[0048] The object classification device 101 includes an image input unit 201, an object region calculation unit 202, a feature selection unit 203, a feature extraction unit 204, a feature classification unit 205, an object classification unit 206, and an output unit 207. Each unit will be described below.

[0049] The image input unit 201 receives an image of the subject being photographed from the interface unit 110. The input image is then input to the object region calculation unit 202, the feature extraction unit 204, and the output unit 207 via the memory 112.

[0050] Description of object area calculation unit 202.

[0051] The object region calculation unit 202 uses the input image received by the image input unit 201 to extract the object region that will be the classification target. As an example of an object region extraction method, the case using a region segmenter will be described. Region segmentation is an image processing method that divides an image into sets of meanings, pixel by pixel. For example, the pixels containing the object that will be the classification target are treated as foreground, and the pixels outside are treated as background for binary classification. The combined regions of the regions classified as foreground are detected, thereby detecting regions for each object.

[0052] Figure 3 Examples of microscope images representing ES cells.

[0053] Assume that an ES cell population exists on culture dish 301. The imaging device 120 captures an image of the cell population from above the culture dish, resulting in a microscope image 302. Cells 303 and 304 are captured in microscope image 302. ES cells possess the pluripotency to differentiate into various tissues. However, once differentiated, this pluripotency is impaired, therefore it is necessary to confirm whether ES cells have differentiated. Furthermore, differentiated ES cells tend to have increased cytoplasm and thickened cell membranes; therefore, a method for determining ES cell differentiation using these characteristics will be described.

[0054] Figure 4 An example of the result of applying region segmentation to microscope image 302 is shown.

[0055] In the region segmentation result 401, the area of ​​cells is represented by black as the foreground region (object region), and all other areas are represented by white as the background region. The object regions corresponding to cells 303 and 304 in the microscope image 302 are object region 402 and object region 403, respectively.

[0056] In this way, the object region calculation unit 202 calculates and outputs the region information for classifying or determining the target object. Figure 3 In the example, in order to determine cell differentiation, the cell's regional information is calculated as the object's regional information.

[0057] Next, in Figure 16 An example of judging the quality of raw materials is shown. An example of judging the quality of raw material 1601 by observing the surface condition of raw material 1601 is explained.

[0058] exist Figure 16In the example, when the surface texture of raw material 1601 is composed of large, circular elements, it is judged to be of good quality. Microscopic image 1602 is an image obtained by photographing raw material 1601 using imaging device 120. Microscopic image 1602 includes the background 1604 of the surface of raw material 1601 and the elements 1603 constituting the texture. For simplicity, not all elements 1603 within microscopic image 1602 are labeled, but the area outside the background 1604 within microscopic image 1602 is considered element 1603. Region segmentation result 1605 is an example of the region segmentation result of microscopic image 1602. Figure 16 In the example, the target object is element 1603, the area of ​​element 1603 is represented by black, and the area of ​​background 1604 is represented by white.

[0059] The above example illustrates binary classification of foreground and background, but object region calculation methods are not limited to this. For instance, in the ES cell example above, where cells are in contact within the microscope image, object regions may also be in contact with each other in the aforementioned region segmentation method. Contacting object regions are subsequently classified as a single cell in the functional section, becoming a major obstacle to accurate feature extraction and classification.

[0060] Therefore, to separate and extract individual object regions, region segmentation can proceed not through binary classification of foreground and background, but by estimating a distance image, which represents the shortest distance from each foreground pixel to a background pixel as a brightness value. In the distance image, the brightness is high near the center of the object, and decreases as the region approaches the background. Therefore, by applying thresholding to the estimated distance image, the separation accuracy of contacting objects is improved.

[0061] Figure 5 This represents an example of separation of contacting objects inferred from a distance image.

[0062] Cells 502 and 503 in contact are captured in microscope image 501. As described above, in the case of conventional binary region segmentation, the object regions of cells 502 and 503 are highly likely to be combined. Distance image 504 is an example of a result obtained by applying distance image estimation to microscope image 501. It is known that the closer to the center of the cell, the higher the brightness.

[0063] Image 505 after thresholding is obtained by thresholding the distance image 504, replacing areas with brightness values ​​less than the threshold with white, and areas with brightness values ​​above the threshold with black. It can be seen that the object regions of cells 502 and 503 are separated. However, in image 502 after thresholding, the object regions of cells 502 and 503 are estimated to be slightly smaller than the actual cells. Therefore, methods such as Watershed or Graph Cut can also be used to interpolate the object regions.

[0064] The interpolated image 506 is the result of interpolation based on the Watershed method, performed on the thresholded image 505 and the distance image 504. As shown in the interpolated image 506, the overall shape of the object region can be obtained while separating the object region.

[0065] As described above, the object region calculation unit 202 extracts the object region that is the classification target from the input image received by the image input unit 201.

[0066] Next, the feature selection unit 203 will be explained.

[0067] The feature selection unit 203 generates a feature selection signal based on the object region information output from the object region calculation unit 202 and the input image received from the image input unit, and outputs it to the feature extraction unit 204. The feature selection signal includes the type and size of the feature to be extracted, as well as the region information for feature extraction.

[0068] The following example of ES cell differentiation determination will be used for illustration. As mentioned in the description of the object region calculation unit 202, the amount of cytoplasm and the thickness of the cell membrane are important characteristic quantities in ES cell differentiation determination. As a method for determining the amount of cytoplasm, a method for evaluating the size of the cell nucleus relative to the cell region will be described. Figure 6 This illustrates an example of using the size of the cell nucleus as a feature. Microscopic images 601 and 602 are images obtained by photographing the same target.

[0069] However, it is assumed that the magnification of microscope image 602 is higher than that of microscope image 601. The targets are cells 603 and 604, with cell 603 having less cytoplasm and cell 604 having more cytoplasm. Filters 610, 611, 612, 613, 614, 615, and 616 are filters used to extract characteristic quantities of the cell nucleus. The black circle located in the center of each filter represents the cell nucleus, with filter 610 representing the smallest nucleus, and the size of the nucleus increasing as it becomes filter 616.

[0070] The highest score is obtained when a filter closest in size to the cell nucleus is applied to the cells within the microscope image. Figure 6 In the example, the filter most suitable for the nuclei of cells 603 and 604 in microscope image 601 is filter 613, and the filter most suitable for the nuclei of cells 603 and 604 in microscope image 602 is filter 616.

[0071] Here, for example, cell 604 in microscope image 601 and cell 603 in microscope image 602 are roughly the same size. However, regarding the proportion of the cell nucleus to the cell, cell 603 in microscope image 602 is larger, so it can be said that cell 603 has less cytoplasm and cell 604 has more cytoplasm.

[0072] To automatically determine this situation, it is necessary to select a feature quantity extraction filter for the cell nucleus based on the size of the entire cell. For example, in the case of cell 604 in microscope image 601, filters 613 to 616 are selected, and the feature quantity selection unit 203 automatically selects filters in such a way that filter 613 is determined to have a small cell nucleus (i.e., more cytoplasm) and as it moves toward filter 616, it is determined to have a large cell nucleus (i.e., less cytoplasm).

[0073] Similarly, for example, if the size of the cell region is the same as that of cell 603 in microscope image 601, filters 610 to 613 are selected, and feature quantity selection unit 203 automatically selects filters in such a way that filter 610 is determined to have a small cell nucleus (i.e., a lot of cytoplasm), and as it moves toward filter 613, it is determined to have a large cell nucleus (i.e., a little cytoplasm).

[0074] As an automatic selection method for filters, there are methods that pre-set multiple thresholds for the area of ​​the cell region. Figure 6 In the example shown, first to fourth thresholds are preset. The first threshold is set to be the lowest, and the values ​​increase as they move towards the fourth threshold.

[0075] First, the area of ​​each cell is calculated based on the cell region output by the object region calculation unit 202. If the cell area is less than a first threshold, filters 610 to 613 are selected; if the cell area is greater than or equal to the first threshold but less than a second threshold, filters 611 to 614 are selected; if the cell area is greater than or equal to the second threshold but less than a third threshold, filters 612 to 615 are selected; and if the cell area is greater than or equal to the third threshold, filters 613 to 616 are selected. Thus, filters are automatically selected.

[0076] In addition, it is not limited to the above method. Any method that selects a portion of features from multiple features based on information obtained from the object region output by the object region calculation unit 202 is acceptable.

[0077] Thus, the feature selection unit 203 automatically selects the size of the feature based on the size of the object region. The size information of the feature is output as part of the feature selection signal.

[0078] In addition, the feature selection unit 203 selects which feature to extract from which region. For example, when using a filter to extract features related to the thickness of the cell membrane, if the filter is applied to regions outside the cell membrane inside the cell, the cell membrane features may not be obtained correctly.

[0079] Therefore, the feature selection unit 203 extracts the cell membrane periphery region based on the object region information calculated by the object region calculation unit 202, and outputs a feature selection signal so that the filter for extracting cell membrane features is applied only to the cell membrane periphery region.

[0080] As an example of a method for extracting the periphery of the cell membrane, the method using edge extraction will be described. Figure 7 This represents an example of the result of extracting the cell membrane region and cytoplasm region from the object region.

[0081] Cell membrane region 702 is through the... Figure 4 The region segmentation result 401 is obtained by extracting edges to calculate the cell membrane region. Additionally, the cytoplasmic region 703 is the cytoplasmic region obtained by subtracting the cell membrane region 702 from the entire object region. Regions of interest 704 and 705 are examples of regions from which cytoplasmic feature quantities are extracted. Examples of edge extraction methods include Sobel filters or Laplace filters. As a region for extracting feature quantities related to the cell membrane, the feature quantity selection unit 203 outputs the cell membrane region 702 as a feature quantity selection signal.

[0082] Furthermore, the feature selection unit 203 can select the size and type of filter based on the location of the target region and local information. For example, when using a filter to extract cytoplasmic features from the cytoplasmic region 703, it is preferable to use a region within the filter that does not contain the cell membrane in order to accurately extract the cytoplasmic features. Therefore, a large filter can be selected in areas far from the cell membrane, such as the region of interest 704, and a small filter can be selected in areas close to the cell membrane, such as the region of interest 705.

[0083] Next, an example of the operation of the feature quantity selection unit 203 in the above example of raw material quality determination will be explained.

[0084] In the quality assessment of raw materials, the shape and size of each element constituting the texture are selected as feature quantities. The methods for extracting shape and size feature quantities will be described later. Additionally, the output is the feature quantity extraction area. Figure 16 The region segmentation result shown is 1605.

[0085] As described above, the feature selection unit 203 outputs a feature selection signal containing the type and size of the feature to be extracted, as well as region information for extracting the feature. These feature selection signals are output according to the number of categories of the feature to be extracted. For example, in the example of ES cell differentiation determination described above, the feature categories are cell membrane and cytoplasm.

[0086] Therefore, the output signal selects features related to the cell membrane and cytoplasm. In the example of raw material quality assessment, the feature categories are the roundness and size of the elements constituting the surface texture of the raw material. Therefore, the output signal selects features related to roundness and size.

[0087] The calculation methods for the aforementioned feature extraction regions can also be combined with image correction or transformation processes. For example, in the cell membrane region example above, dilation or smoothing can be used to expand the edge region, or thresholding can be combined for binarization. Alternatively, the aforementioned distance image can be used to specify boundary portions, the center of an object, or its middle section.

[0088] Furthermore, feature extraction regions can also be selected based on the brightness or RGB values ​​of the input image within the target region. For example, in the case of stained tissues or cells, the color is determined by the location (cell nucleus or cell membrane, etc.) or property (presence or absence of antigens, etc.). Therefore, feature extraction regions can be extracted, for example, by setting a certain range for RGB values ​​or hues.

[0089] The feature extraction unit 204 will be explained.

[0090] The feature extraction unit 204 extracts features for object classification from the selected region within the input image or object region information using the selected feature quantity based on the feature selection signal output by the feature selection unit 203.

[0091] For example, in the ES cell differentiation determination example described above, the feature extraction unit 204 uses feature filters (e.g., filters 613, 614, 615, and 616) to evaluate the cytoplasm selected by the feature selection unit 203, and extracts features from the region specified by the feature selection unit 203. Furthermore, the feature extraction region for the cytoplasm is the object region calculated by the object region calculation unit 202. The feature extraction filters can be generated, for example, using machine learning such as deep learning, or designed manually.

[0092] Furthermore, the feature extraction unit 204 can extract feature quantities from the object region information output by the object region calculation unit 202 based on the selection made by the feature selection unit 203. For example, in the example of determining the quality of raw materials described above, roundness and size are extracted from the object region information. As a method for calculating roundness, for example, there are roundness indices as shown in (Equation 1).

[0093] [Formula 1]

[0094] Ic=4πS / L 2

[0095] L represents the circumference of the object region, and S represents the area. The closer the object region is to a circle, the closer the index Ic is to 1. In addition, details will be discussed later, but besides roundness, the rectangularity index shown in (Equation 2) can also be used in conjunction with the roundness index to determine the shape of the object.

[0096] [Formula 2]

[0097] Is = S / (WH)

[0098] W and H are the width and height of the smallest rectangle encompassing the entire object region, and S, like in equation (1), is the area of ​​the object region. Furthermore, the size is determined using the area of ​​the object region. Additionally, the length, width, aspect ratio, and other features of the object region can be extracted from the object region information.

[0099] The feature quantity classification section 205 will be explained.

[0100] The feature classification unit 205 calculates the representative features of each object based on the object region information calculated by the object region calculation unit 202, the feature selection signal calculated by the feature selection unit 203, and the feature extraction result output by the feature extraction unit 204.

[0101] As an example, using Figure 8This example illustrates the state determination of ES cells described above. Microscopic image 801 is a microscopic image obtained by photographing ES cells 804. Assume that the membrane of ES cells 804 is mostly composed of a thick membrane 805, but partly composed of a thin membrane 806. Feature extraction region 802 is the feature extraction region output by feature selection unit 203, and feature extraction result 803 is a graph visualizing the feature extraction results. Feature 807 indicates that the thick membrane 805 was detected by feature extraction unit 204, and feature 808 indicates that the thin membrane 806 was detected.

[0102] like Figure 8 As shown, in actual microscope images, the same feature quantities are not always detected in a single object; sometimes, partially different feature quantities are obtained. However, for object classification as discussed later, it is necessary to obtain a comprehensive set of feature quantities specific to the cell membrane. This comprehensive set of feature quantities is called the representative feature quantity.

[0103] One method for determining representative features is as follows: In the feature extraction region 802, the feature with the highest detected pixel count or score is accumulated based on the proportion or score of each feature, and the feature with the highest score is output as the representative feature. For example, in... Figure 8 In the example, feature 807 (thick film) with the largest proportion was selected as the representative feature quantity.

[0104] For each feature category selected by the feature selection unit 203, one or more representative features are selected. While one representative feature can be selected as described above, a threshold can also be preset for the score or proportion, and all features exceeding the threshold can be used as representative features.

[0105] Additionally, representative features can also be appended with continuous values ​​indicating the strength or likelihood of the feature. For example, if it is Figure 8 Examples include the proportion of feature 807 in the feature extraction region or the sum of feature scores.

[0106] Alternatively, representative features can be determined based on multiple indicators derived from the entire feature extraction region. For example, in the above example of raw material quality determination, an example of a method for the feature classification unit 205 to extract representative features related to the shape of the object will be explained.

[0107] In the example of raw material quality determination, the feature extraction unit 204 extracts roundness and rectangularity as features from the entire object area. The feature classification unit 205 classifies the object as round when the roundness is higher than the rectangularity, and otherwise classifies it as rectangular, and outputs it as a representative feature. Alternatively, classification can be based on the values ​​of roundness and rectangularity, such as round (high roundness) or round (low roundness). Furthermore, regarding the size of the object, for example, thresholds 1 and 2 for area are predetermined. If the size of the object area is less than threshold 1, it is classified as small; if it is greater than or equal to threshold 1 but less than threshold 2, it is classified as medium; and if it is greater than or equal to threshold 2, it is classified as large, and output as a representative feature.

[0108] Explanation of object classification section 206.

[0109] The object classification unit 206 classifies objects based on representative features calculated by the feature classification unit 205. Regarding the object classification method, for example, an object classification table recording one or more combinations of the aforementioned representative features and object classes is pre-stored in the memory 112, and the object class is determined by comparing the obtained feature values ​​with the aforementioned object classification table.

[0110] Figure 9 This is an example of the object classification table in the differentiation determination of ES cells mentioned above.

[0111] As representative characteristics, the characteristic classification unit 205 outputs judgment results related to membranes and cytoplasm. Figure 9 The table indicates that if the membrane is thin and the cytoplasm is sparse, it is considered undifferentiated; if the membrane is thick and the cytoplasm is abundant, it is considered differentiated. In this case, for example, if an object presents a combination of characteristics such as a thin membrane and abundant cytoplasm that does not match any of the representative features in the object classification table, it is classified as an unknown object and output.

[0112] Figure 17 This is an example of the object classification table in the quality judgment of the aforementioned raw materials.

[0113] As representative features, the feature classification unit 205 outputs a determination result related to the size and shape of the feature region. Figure 17 In the table, a large size and a round shape are considered high quality, while a small size and a rectangular shape are considered low quality.

[0114] Furthermore, to allow for a small number of misclassifications of features, a permissible range can be determined for the number of consistent features. For example, if the permissible range is set to 1, even if one representative feature in the object classification table is inconsistent, if the remaining representative features are consistent, the corresponding object class will be output as the result. In this case, depending on the combination, it may sometimes be consistent with multiple objects.

[0115] For example, in Figure 9 In the example, the combination of representative characteristics such as thin membrane and abundant cytoplasm means that, relative to either undifferentiated or differentiated, one representative characteristic is consistent and the other is inconsistent.

[0116] In such cases, for example, when there are differences in the number of consensus values ​​representing the features, the decision can be made based on the number of consensus values, or by weighting the representative features based on their importance. Alternatively, if the feature classification unit 205 outputs the likelihood of the features, the likelihood of each feature can be weighted and summed to make the decision. Alternatively, all object classes within the allowed range can be presented to the user as candidate suggestions.

[0117] As described above, the object classification unit 206 outputs the classification result of the object class and the extraction result of the representative feature quantity used for classification. In addition, when an allowable range for the aforementioned inconsistent feature quantity is set, a flag indicating whether each representative feature quantity is a feature quantity that is consistent with the conditions of the determined object class can also be output as additional information.

[0118] Furthermore, when a user wants to add a new object class to confirm the classification results and classification criteria displayed via output unit 207, or wants to edit the conditions of an existing object class, the object classification table inside the memory can be edited. The object classification table can be overwritten in the memory 112 after being edited on an external device such as a PC, although this is not specifically described in the documentation. Figure 1 However, the object classification table in memory 112 can be directly edited using input devices such as a keyboard.

[0119] Furthermore, the object classification unit 206 can group unknown objects based on combinations of representative feature quantities. For example, as a result of comparing with the object classification table, if an object is classified as an unknown object, the combination of representative feature quantities is added to the object classification table as unknown object A.

[0120] Therefore, when an unknown object with the same characteristic values ​​appears again, it is classified as unknown object A. On the other hand, if an object with characteristic values ​​that are inconsistent with both known objects and unknown object A appears, a combination of characteristic values ​​is added to the object classification table again, and it is designated as unknown object B. By repeating this operation, unknown objects can be grouped.

[0121] The output unit 207 will be explained. The output unit 207 displays the classification results of the objects attached to the input image and information about the representative feature quantities used as the basis for classification. As an example of the display, in... Figure 10Examples of the differentiation determination of ES cells described above are shown. Microscopic image 1001 is an example of an image obtained by taking a picture of ES cells with a microscope; cell 1002 is an undifferentiated cell; and cells 1003 and 1004 are differentiated cells.

[0122] In each cell, the object classification result and the representative feature quantity used as the basis for the classification result are displayed in the dialog bubble 1005. For example, cell 1002 is judged to have a thin membrane and little cytoplasm, and is therefore classified as undifferentiated. On the other hand, cell 1004 is a differentiated cell, but because it does not meet the criteria for a differentiated cell, it is misclassified as unknown. By checking the feature quantity in the dialog bubble 1005, the user can easily understand why it does not meet the criteria for a differentiated cell—it was judged to have little cytoplasm.

[0123] Furthermore, based on the cell 1004's classification result, the user added "medium" to the cytoplasm classification result and edited the object classification table to classify the cell as differentiated when the membrane is thick and the cytoplasm is medium. This makes it relatively easy to add classification rules even for unknown combinations of characteristics.

[0124] Traditional machine learning approaches address this by appending images similar to cell 1004 to the learning data and then relearning. However, because it's unclear why the recognizer misidentified cell 1004 or what features should be appended, time is sometimes spent improving the recognizer's accuracy. Furthermore, since all features are relearned, it's possible to shift to features relevant to the unaffected membrane. According to the method of the present invention, since the features requiring relearning are clearly defined, it's easy to collect effective learning data for relearning, and relearning can be performed without affecting other feature extractors.

[0125] Figure 18 This illustrates an example of the aforementioned raw material quality assessment. Microscopic image 1801 is an example of an image obtained by photographing the surface of the raw material using a microscope. Element 1802, constituting the surface texture of the raw material, appears when the raw material quality is high; element 1803 appears when the raw material quality is both high and low; and element 1804 appears when the raw material quality is low. For each element, the quality assessment result and the representative characteristic quantity used as the basis for the assessment are displayed in the speech bubble 1805.

[0126] However, in Figure 18 The text only displays whether each element appears under high quality conditions, without indicating the overall quality evaluation result of the raw materials. The overall quality evaluation of the raw materials is explained in Example 3.

[0127] Furthermore, by assigning benchmark consistency information to representative feature quantities, text colors and other elements can be changed based on whether they are consistent or inconsistent with the benchmark, displaying the text in a way that indicates consistency or inconsistency. This makes it easier to identify feature quantities that do not meet the conditions.

[0128] Furthermore, in the example above, the judgment results and feature extraction results were displayed as text. However, they could also be displayed by assigning colors, textures, symbols, etc., or by combining them with text. Additionally, to improve visibility, classification and feature information could be displayed only when the mouse hovers over the object. Moreover, the output method is not limited to images; for example, object region information could be represented by the coordinates of the start and end points, output as text or binary data.

[0129] The details of Embodiment 1 have been described above for each functional block. However, the implementation of the present invention is not necessarily limited to... Figure 2 The function blocks are configured such that they can handle the actions performed by each function block. Figure 15 This is an example of a processing flowchart for Example 1. Each step is related to... Figure 2 The elements of the functional block diagram shown correspond to each other.

[0130] In image input step 1501, an image obtained by photographing the subject is received.

[0131] In object region calculation step 1502, the input image received in image input step 1501 is used to extract the object region that is the classification target. The method for extracting the object region is as described in the object region calculation unit 202 above.

[0132] In feature selection step 1503, a feature selection signal is generated based on the object region information calculated in object region calculation step 1502 and the input image received in image input step 1501. The method for generating the feature selection signal is as described in the feature selection unit 203 above.

[0133] In feature extraction step 1504, based on the feature selection signal generated in feature selection step 1503, the selected feature quantities are used to extract feature quantities for object classification from the selected region within the input image or object region information. The feature extraction method is as described in the feature extraction unit 204 above.

[0134] In the feature classification step 1505, representative features of each object are calculated based on the object region information calculated in the object region calculation step 1502, the feature selection signal calculated in the feature selection step 1503, and the feature extraction result calculated in the feature extraction step 1504. The method for calculating representative features is as described in the feature classification section 205 above.

[0135] In object classification step 1506, objects are classified based on representative feature quantities calculated in feature quantity classification step 1505. The object classification method is as described in object classification section 206 above.

[0136] In output step 1507, the object classification result calculated in object classification step 1506 and the representative feature quantity calculated in feature quantity classification step 1505 are associated with the input image and presented to the user. The method of providing the prompt to the user is as described in the output unit 207 above.

[0137] As described above, feature quantities are automatically selected based on the size of the object, enabling more robust feature extraction and determination. Furthermore, by correlating the object classification results and representative feature quantities with the input image, users can confirm not only the object classification results but also the extracted representative feature quantities.

[0138] Example 2

[0139] Example 2 uses the object classification results and feature information output by the object classification device described in Example 1 to highlight or display the object classification system for a specific object.

[0140] Figure 11 This diagram illustrates the hardware structure of Embodiment 2. The only difference between the hardware structure of Embodiment 2 and Embodiment 1 is the addition of an input device 1101 and a GUI generation device 1102. Therefore, only the input device 1101 and the GUI generation device 1102 will be described below. The object classification device 101, the imaging device 120, and the display device 121 are the same hardware components as in Embodiment 1, and therefore their descriptions are omitted.

[0141] The input device 1101 is a device that accepts user operations, such as a keyboard or mouse. The user operation information received by the input device 1101 is input into the GUI generation device 1102, which will be described later.

[0142] The GUI generation device 1102 generates classification results and feature values ​​for user-specified objects, and only highlights or displays GUIs for objects whose classification results and feature values ​​are consistent with the above-specified values, and displays them on the display device 121.

[0143] Figure 12A This illustrates an example of a GUI generated by the GUI generation device 1102. The GUI window 1201 consists of a display unit 1202 and a selection unit 1203. The display unit 1202 displays the input image by default. In the selection unit 1203, the classification results and feature extraction results included in the output of the object classification device 101 are listed as selection options.

[0144] Each option begins with a checkbox. The GUI generation device 1102 generates object search criteria based on the state of the checkbox group within the selection unit 1203, and extracts objects that meet the search criteria. Then, the area of ​​consistent objects within the display unit 1202 is highlighted. One method for generating object search criteria based on the state of the checkbox group is as follows: First, the items checked for each category are combined using OR, and the OR combination result is combined using AND, thereby obtaining the search criteria.

[0145] For example, if it is Figure 12A If the checkbox is set to a certain state, then "(feature value = thickness) AND (determination result = unknown)" becomes the search condition. If targeting... Figure 12A If the checkbox status is further adjusted so that "thin" is also selected for "cell membrane", then "(feature value = thick OR thin) AND (determination result = unknown)" becomes the search condition.

[0146] exist Figure 12A In the example, the area of ​​the object that matches the condition within the display section 1202 is highlighted with a dashed line. Figure 12A One example of the display method is that the display can be emphasized using colors, speech bubbles, etc., or the area of ​​the object that meets the conditions can be clipped and displayed as a list in the display unit 1202.

[0147] in addition, Figure 12B This represents another example of a GUI generated by the GUI generation device 1102. Its components and... Figure 12A The process is largely the same, but a sensitivity adjustment bar 1204 is provided at the end of each feature quantity in the selection unit 1203. The sensitivity adjustment bar 1204 is a user interface for adjusting the extraction sensitivity of each feature quantity. Based on the value adjusted via the sensitivity adjustment bar 1204, for example, in the feature extraction unit 203 within the object classification device 101, a bias is applied to the output of the corresponding feature quantity filter to adjust the extraction sensitivity. This allows observation of the likelihood at which the feature quantities of each object are calculated.

[0148] In addition, although not specifically in Figure 12A , Figure 12BThe diagram shows the results, but information such as the number of objects that meet the criteria and their proportion relative to the total number of objects can also be displayed in the GUI. Furthermore, multiple selection units 1203 can be set, allowing multiple criteria to be set simultaneously. In this case, the results can be visually distinguished by changing the color, etc., for each criterion.

[0149] Figure 12A as well as Figure 12B The GUI shown is an example; the arrangement and display of elements are not limited to this. Any GUI that can achieve the same user operation and display is acceptable.

[0150] As a result, users can easily discover objects with specific features within an image.

[0151] Example 3

[0152] Example 3 is an object classification system that uses the classification results and feature information of objects in multiple input images output by the object classification device described in Example 1 to calculate the number or proportion of objects with specific classification results and feature values.

[0153] Figure 13 A hardware structure diagram of Embodiment 3 is shown. The only difference between the hardware structure of Embodiment 3 and that of Embodiment 2 is the addition of a statistical information calculation device 1302. Therefore, only the statistical information calculation device 1302 will be described below. The object classification device 101, the imaging device 120, the display device 121, and the input device 1101 are the same as the hardware components of Embodiment 2, and therefore their descriptions are omitted.

[0154] The statistical information calculation device 1302 accumulates and stores the object classification and feature extraction results output by the object classification device 101, calculates statistical information based on the user's selection received via the input device 1101, and displays it on the display device 121. In this embodiment, one or more input images are input from the imaging device 120 to the object classification device 101.

[0155] Whenever the object classification device 101 outputs a result for the input image, the statistical information calculation device 1302 stores it in its memory. Statistical information is calculated using the sum of multiple object classification results and feature extraction results stored in the memory.

[0156] Figure 19This is an example of displaying statistical information results. The statistical information display window 1901 is generated by the statistical information calculation device 1302 and is used to display statistical information to the user; it is displayed on the display device 121. The statistical information display window 1901 displays statistical information on judgment results and feature quantities calculated based on multiple objects present in one or more images.

[0157] in addition, Figure 19 This represents one example of statistical information displayed in the quality assessment of the aforementioned raw materials. Additionally, although in Figure 19 While not specifically stated, a comprehensive judgment can be made based on the judgment results or statistical information of characteristic quantities, and displayed in the statistical information display window 1901. For example, in the case of raw material quality judgment, thresholds for the proportion of "high" and the proportion of "medium" can be predetermined based on the quality judgment results of the elements constituting the surface texture of the raw material. If the proportions of "high" and "medium" in the statistical information of the element quality judgment results exceed the thresholds, the comprehensive quality of the raw material is judged as "high," and displayed in the statistical information display window 1901.

[0158] Alternatively, the statistical information to be calculated can be determined based on the user's selection. The user's selection and the display of the results are performed via a GUI generated by the statistical information calculation device 1302. Figure 14 An example of a GUI for calculating statistical information based on user selection is shown. The GUI window 1401 consists of three parts: a denominator condition setting section 1402, a numerator condition setting section 1403, and a statistical information display section 1404. The denominator condition setting section 1402 and the numerator condition setting section 1403 display the same content, listing the classification results and feature quantities of the object.

[0159] In addition, checkboxes are provided at the beginning of each item. The statistical information calculation device 1302 counts the number of objects with classification results and feature values ​​for which checkboxes are selected. The number of objects that meet the conditions specified by the denominator condition setting unit 1402 or the numerator condition setting unit 1403 is displayed in the denominator or numerator column of the statistical information display unit 1404. In addition, the result obtained by dividing the number of numerators by the number of denominators is displayed in the proportion item. Thus, it is possible to calculate the proportion of objects with specific classification results and feature values ​​present in multiple input images.

[0160] According to Example 3, it is possible to calculate the number and proportion of objects with specific classification results and feature values ​​for objects existing in more than one input image. This assists in a more detailed evaluation of the classification target object, tendency observation, and research on effective feature values ​​for classification.

[0161] <Variation Example>

[0162] In the object region calculation unit 202, a region segmentation method is used as an example of an object detection method. However, methods such as YOLO or SSD, which are used to estimate the rectangles surrounding each object, can also be used to calculate object region information. Alternatively, after estimating the rectangles surrounding the objects, region segmentation can be applied, combining the above methods. Furthermore, it is also possible to calculate the distance image based on the results of binary classification of the foreground and background, without estimating the distance image.

[0163] In the feature selection unit 203, it is explained that the scale of the feature to be selected is determined based on the size of the object region, but the input image can also be enlarged or reduced based on the size of the object region. In addition, rotation and deformation can also be performed based on the shape of the object region.

[0164] In the object classification section 206, a method for setting an allowable range for the number of consistent feature quantities during object classification is described. However, an allowable range can also be set for the number of inconsistent feature quantities, or an allowable range can be set for the consistency rate or inconsistency rate as the denominator of the number of feature quantities that will be conditions.

[0165] In the GUI generation apparatus 1102 of Embodiment 2, as an example of a method for adjusting sensitivity, a method of adding a bias value to the output of the feature filter is described. However, not only addition operations can be used, but also multiplication operations or exponential operations, or combinations thereof.

[0166] In the above embodiments, the region information of each object within the image is calculated. Based on the region information, the type, size, and application area of ​​the features to be extracted are automatically selected. The features required for object classification are extracted based on the selection results. Then, the extracted features are classified, and representative features used to determine the state of the object (hereinafter referred to as representative features) are calculated. Object classification is performed based on the combination of representative features. Therefore, feature extraction corresponding to parts within the object is possible, enabling more accurate classification of objects within the image.

[0167] Furthermore, by associating the object's classification results and representative features with the input image, the system can inform the user of the object's classification results and the representative features that form the basis for classification.

[0168] According to the above embodiments, feature quantities corresponding to parts within an object can be extracted, enabling more accurate classification of objects within an image. Furthermore, the classification results and the feature quantities used as the basis for classification can be presented to the user.

[0169] Explanation of reference numerals in the attached figures

[0170] 100: Object Classification System

[0171] 101: Object classification device

[0172] 110: Interface Department

[0173] 111: Arithmetic Department

[0174] 112: Memory

[0175] 113: Bus

[0176] 120: Filming equipment

[0177] 121: Display device

[0178] 201: Image Input Section

[0179] 202: Object Region Calculation Unit

[0180] 203: Feature Selection Section

[0181] 204: Feature Extraction Section

[0182] 205: Feature Classification Department

[0183] 206: Object Classification Department

[0184] 207: Output Section

[0185] 301: Petri dish

[0186] 1101: Input device

[0187] 1102: GUI generation device

[0188] 1201: GUI window

[0189] 1202: Display Unit

[0190] 1203: Selection Department

[0191] 1204: Sensitivity Adjustment Bar

[0192] 1302: Statistical Information Calculation Device

[0193] 1401: GUI Window

[0194] 1402: Denominator Condition Setting Section

[0195] 1403: Molecular Condition Setting Section

[0196] 1404: Statistical Information Display Department

[0197] 1901: Statistical information display window.

Claims

1. An object classification device, which classifies objects within an image to determine the type or state of the objects, characterized in that, The object classification device has: An object region calculation unit calculates the object region of the object within the image; The feature selection unit uses the object region to select the feature values ​​of the object and outputs a feature selection signal; A feature extraction unit extracts multiple features from the image based on the feature selection signal; The feature classification unit classifies multiple features to extract representative features that determine the type or state of the object. The object classification unit classifies the object by determining the type or state of the object based on the representative feature quantity, and outputs the classification result of the object; as well as The output unit associates the classification result and the representative feature quantity of the object with the image and outputs the result. The feature selection unit uses the object region to select a portion of the feature quantities from a plurality of feature quantities, and outputs a feature selection signal containing the type and size of the feature quantity and the region information for extracting the feature quantity.

2. The object classification device according to claim 1, characterized in that, The feature extraction unit extracts the feature quantities corresponding to the parts of the object. The object classification department classifies the objects based on the combination of the representative feature quantities. The output unit outputs the classification result of the object and the representative feature for each representative feature.

3. The object classification device according to claim 1, characterized in that, The feature classification unit extracts the representative feature of the object based on the object region calculated by the object region calculation unit, the feature selection signal calculated by the feature selection unit, and the feature quantity extracted by the feature extraction unit.

4. The object classification device according to claim 1, characterized in that, The feature selection unit selects the type of feature applicable to the image based on the size of the object region.

5. The object classification device according to claim 1, characterized in that, The object region calculation unit separates the objects that are in contact with each other by estimating the distance image.

6. The object classification device according to claim 1, characterized in that, The object classification department classifies the objects by referring to an object classification table that stores the representative feature quantities in relation to the types or states of the objects.

7. An object classification system, characterized in that, have: The object classification device according to claim 1; The imaging device captures the image input to the object classification device; and A display device that associates the classification result of the object with the representative feature quantity and displays it in the image.

8. An object classification system, characterized in that, have: The object classification device according to claim 1; A camera device that captures the image input to the object classification device; An input device that receives user input information; A GUI generation apparatus that generates a GUI for highlighting a predetermined object based on at least one of the classification result of the object selected by the user from the input device and the representative feature quantity. as well as A display device that displays the GUI generated by the GUI generation device.

9. The object classification system according to claim 8, characterized in that, The GUI generation device generates the GUI with a sensitivity adjustment bar, which can adjust the extraction sensitivity of the representative feature quantity.

10. An object classification system, characterized in that, have: The object classification device according to claim 1; A camera device that captures the image input to the object classification device; A statistical information calculation device calculates statistical information containing the quantity or distribution of the objects based on the classification result of the objects output from the object classification device and the representative feature quantity; as well as A display device that displays the statistical information output by the statistical information calculation device.

11. An object classification method, which classifies objects within an image to determine the type or state of the objects, characterized in that, The object classification method has the following steps: The object region calculation step calculates the object region of the object within the image. The feature selection step involves using the object region to select the feature values ​​of the object and outputting a feature selection signal. The feature extraction step involves selecting a signal based on the feature quantities and extracting multiple feature quantities from the image. The feature classification step involves classifying multiple features to extract representative features that determine the type or state of the object. The object classification step involves classifying the object based on the representative feature quantity to determine its type or state, and then outputting the classification result. as well as The output step involves associating the classification result and representative feature of the object with the image and outputting the result. In the feature selection step, the object region is used to select a portion of the features from a plurality of features, and a feature selection signal containing the type and size of the features and the region information for extracting the features is output.

12. The object classification method according to claim 11, characterized in that, In the feature extraction step, the feature quantities corresponding to the parts of the object are extracted. In the object classification step, the objects are classified according to the combination of the representative feature quantities. In the output step, for each of the representative features in the combination of representative features, the classification result of the object is associated with the representative feature and output.

13. The object classification method according to claim 11, characterized in that, In the feature classification step, the representative feature of the object is extracted based on the object region calculated in the object region calculation step, the feature selection signal calculated in the feature selection step, and the feature quantity extracted in the feature extraction step.

14. The object classification method according to claim 11, characterized in that, In the feature selection step, the type of feature applicable to the image is selected based on the size of the object region.

15. The object classification method according to claim 11, characterized in that, In the object region calculation step, the objects that are in contact with each other are separated by estimating the distance image.

Images