Image analysis program, image analysis method, and image analysis device

Abstract

The present invention provides an image analysis program, an image analysis method, and an image analysis apparatus that can improve the accuracy of analyzing objects captured in an image. [Solution] The image analysis program causes the processor to perform the following actions: estimate at least one of the object's attributes, such as the number or size, from a first image of the object; evaluate the estimation results of the object's attributes based on prior information about the object; and record or aggregate the estimation results that are deemed reasonable.

Description

Technical Field

[0001] The present disclosure relates to an image analysis program, an image analysis method, and an image analysis apparatus.

Background Art

[0002] As described in Patent Document 1, an apparatus that determines the number of cells present in a microscope image using a trained model is known.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] When discriminating an object shown in an image using a trained model, an incorrect discrimination result may be obtained. Errors in the discrimination result reduce the analysis accuracy of characteristics such as the number or size of the objects shown in the image. There is a demand for improving the analysis accuracy of objects shown in images.

[0005] The present disclosure has been made in view of the above points, and an object thereof is to provide an image analysis program, an image analysis method, and an image analysis apparatus that can improve the analysis accuracy of an object shown in an image.

Means for Solving the Problem

[0006] (1) An image analysis program according to several embodiments causes a processor to perform the following actions: estimate at least one of the number or size of an object as an attribute of the object from a first image of the object; evaluate the estimation results of the object's attributes based on prior information about the object; and record or aggregate the estimation results that are deemed valid. In this way, even if the estimation model incorrectly detects an object from the image or incorrectly estimates the attributes of an object in the image, the incorrect estimation results are excluded from recording or aggregation. By excluding incorrect estimation results, the accuracy of the analysis of objects in the image is improved.

[0007] (2) If the image analysis program described in (1) above is evaluated as not being valid, the processor may be instructed to re-estimate the attributes of the object from the first image and to record or aggregate the re-estimate results.

[0008] (3) If the image analysis program described in (1) or (2) above is evaluated as not valid, the processor may be instructed to estimate the attributes of the object from a backup second image that can replace the first image.

[0009] In this way, even if the estimated attributes of an object in some of the images taken for a sample are deemed inaccurate, the lack of images necessary for recording or summarizing the estimated attributes of objects in that sample is compensated for. As a result, the accuracy of recording or summarizing the estimated attributes of objects in the sample is improved.

[0010] (4) In the image analysis program described in any one of (1) to (3) above, the prior information of the object may include information about the shape or size of the object. Including information about the size of the object in the prior information improves the accuracy of evaluating the validity of the estimation result of the size of the object. Including information about the shape of the object in the prior information improves the accuracy of detecting the object from the image. As a result, the accuracy of analyzing the object in the image is improved.

[0011] (5) In the image analysis program described in any one of (1) to (4) above, the prior information of the object may include, if the object is an aggregate containing aggregate elements, the ratio of the size of the aggregate to the number of aggregate elements, or the ratio of the size of the first image to the number of aggregate elements. Including information regarding the number of aggregate elements in the prior information improves the accuracy of estimating the number of objects in the image. As a result, the accuracy of analyzing the objects in the image is improved.

[0012] (6) The image analysis program described in any one of (1) to (5) above may cause the processor to further estimate at least one of the color or texture of the object as an attribute of the object from the first image. This improves the detection accuracy of individual aggregate elements in the aggregate.

[0013] (7) In the image analysis program described in any one of (1) to (6) above, the prior information of the object may include information about the color or texture of the object. Including information about the color or texture of the object in the prior information improves the accuracy of object detection from the image. As a result, the accuracy of analysis of objects in the image is improved.

[0014] (8) Image analysis methods according to several embodiments include a processor estimating at least one of the number or size of an object as an attribute of the object from a first image of the object, evaluating the estimation results of the object's attributes based on prior information about the object, and recording or aggregating the estimation results that are deemed reasonable.

[0015] (9) Image analysis devices according to several embodiments include an estimation unit that estimates at least one of the number or size of an object as an attribute of the object from a first image of the object, and an evaluation unit that evaluates the estimation results of the attributes of the object based on prior information of the object, and records or compiles the estimation results that are evaluated as valid.

[0016] (10) The image analysis device described in (9) above may further include a re-estimation unit that, if the estimated result of the attributes of the object shown in the first image is evaluated as not being appropriate, re-estimates the attributes of the object from the first image and records or compiles the results of the re-estimate.

[0017] (11) In the image analysis apparatus described in (9) or (10) above, if the estimation unit evaluates that the estimation result of the attributes of the object shown in the first image is not valid, it may estimate the attributes of the object from a spare second image that can replace the first image. [Effects of the Invention]

[0018] The image analysis program, image analysis method, and image analysis device described herein improve the accuracy of analyzing objects captured in an image. [Brief explanation of the drawing]

[0019] [Figure 1] This is a block diagram showing an example configuration of an image analysis device related to a comparative example. [Figure 2] This block diagram shows an example configuration of the image analysis system related to this disclosure. [Figure 3] This flowchart shows an example of the procedure for the image analysis method related to this disclosure. [Figure 4] It is a diagram illustrating an example of an image capturing an object. [Figure 5] It is a block diagram illustrating a configuration example of an image analysis device according to another embodiment. [Figure 6] It is a flowchart illustrating a procedure example of an image analysis method according to another embodiment. MODE FOR CARRYING OUT THE INVENTION

[0020] (Comparative Example) As shown in Figure 1, an image analysis device 90 according to the comparative example includes an input unit 91, an estimation unit 92, and a recording unit 94. The input unit 91 receives input of an image capturing an object to be subjected to image analysis. The estimation unit 92 analyzes the image using a trained model generated by executing deep learning, and estimates the number of objects captured in the image. The recording unit 94 records the estimation result of the number of objects captured in the image.

[0021] The trained model used in the estimation unit 92 may erroneously detect objects from an image. If an object is erroneously detected, the estimation result of the number of objects captured in the image may be incorrect. For example, when the trained model outputs a correctness probability of the estimation result in addition to the estimation result of the number of objects captured in the image, the estimation unit 92 adopts the estimation result when the correctness probability output from the trained model is equal to or higher than a threshold value, and when the correctness probability is less than the threshold value, the estimation unit 92 does not adopt the estimation result, and may adopt the result obtained by a user of the image analysis device 90 counting the objects captured in the image through visual confirmation.

[0022] However, when the correctness probability output by the trained model is incorrect, the number of objects may be erroneously estimated. Improvement in the estimation accuracy of the number of objects is required. Hereinafter, in the present disclosure, embodiments of an image analysis device 10, an image analysis method executed by the image analysis device 10, and an image analysis program, which can improve the estimation accuracy of attributes of an object including the number or size of the object captured in an image, will be exemplarily described.

[0023] (Example of the configuration of the image analysis system 1 related to this disclosure) As shown in Figure 2, an image analysis system 1 according to one embodiment of the present disclosure comprises an imaging device 2 and an image analysis device 10. The imaging device 2 and the image analysis device 10 are connected to each other via wired or wireless communication. The imaging device 2 and the image analysis device 10 may be directly connected to each other or connected to each other via a network. The imaging device 2 may be included in the image analysis device 10.

[0024] In the image analysis system 1, the imaging device 2 photographs objects contained in the sample and generates an image showing the objects. The image analysis device 10 acquires the image showing the objects, detects the objects from the image, and estimates the attributes of the objects, including the number or size of the objects shown in the image. The objects may be, for example, plants, microorganisms or viruses, particles, or oil droplets. Plants may include algae, etc. Microorganisms may include bacteria, harmful algae, aquatic microorganisms, or yeast, etc. Particles may include gum particles, protein aggregates, foreign matter, or fibrous particles, etc. The objects are not limited to the examples described above and may be various other objects. The sample may be a liquid such as water or a solvent containing the objects.

[0025] The following describes specific examples of each component of the image analysis system 1.

[0026] <Photography device 2> The imaging device 2 includes a microscope for observing objects contained in the sample. The microscope includes an optical system for magnifying and imaging the objects. The microscope may be, but is not limited to, a flow imaging microscope or a fluorescence microscope using laser light.

[0027] The imaging device 2 includes an image sensor that captures an image of the object formed by the microscope. The image sensor may be a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, etc.

[0028] The imaging device 2 may include a stage for holding the sample within the field of view of the microscope's optical system. If the sample is a liquid such as water containing an object, the imaging device 2 may include piping or a pump for transporting the liquid sample to the stage. The imaging device 2 may transport predetermined amounts of the liquid sample to the stage, photograph the object contained in the liquid, and generate images showing the object. The imaging device 2 may generate multiple images for a single sample. The imaging device 2 may transport the entire volume of the liquid sample to the stage and photograph the object. The imaging device 2 may transport only a necessary amount of the total volume of the liquid sample to the stage and photograph the object.

[0029] <Image analysis device 10> The image analysis device 10 comprises an input unit 11, an estimation unit 12, an evaluation unit 13, and a recording unit 14. The input unit 11 receives an image input from the imaging device 2. The estimation unit 12 estimates the attributes of objects, including the number or size of objects, that are captured in the image input to the input unit 11. The evaluation unit 13 evaluates the validity of the estimation results of the object attributes by the estimation unit 12. The recording unit 14 records the estimation results of the object attributes that have been evaluated as valid by the evaluation unit 13.

[0030] The input unit 11 includes a communication interface that enables communication with the imaging device 2 via wired or wireless connection. The communication interface may be configured to communicate based on, for example, a LAN (Local Area Network) communication standard. The communication interface may also be configured to communicate based on, for example, a mobile communication standard such as 4G (4th Generation), LTE (Long Term Evolution), or 5G (5th Generation). The input unit 11 may also be configured to communicate based on a serial communication standard such as RS-232C or RS-485. The communication interface is not limited to these examples and may be configured to communicate based on various communication standards.

[0031] The estimation unit 12 or evaluation unit 13 may include a processor or a dedicated circuit. The processor may include a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. The dedicated circuit may include an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), etc.

[0032] The estimation unit 12 or evaluation unit 13 may include a storage unit. The storage unit may store various information used in the operation of the estimation unit 12 or evaluation unit 13, or programs for realizing the functions of the estimation unit 12 or evaluation unit 13. The storage unit may function as the work memory of the estimation unit 12 or evaluation unit 13. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit may be configured integrally with the estimation unit 12 or evaluation unit 13, or it may be configured separately from the estimation unit 12 or evaluation unit 13.

[0033] The estimation unit 12 and the evaluation unit 13 may be configured as a single unit, or they may be configured as separate units.

[0034] The recording unit 14 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The recording unit 14 may be configured integrally with the estimation unit 12 or the evaluation unit 13, or it may be configured separately from the estimation unit 12 or the evaluation unit 13.

[0035] The image analysis device 10 may be equipped with a display to allow the user of the image analysis device 10 to recognize the estimated results of the object's attributes. The display may include, but is not limited to, a liquid crystal display (LCD), an organic electro-luminescence (EL) display, or an inorganic EL display.

[0036] The image analysis device 10 may include an input device for receiving input from the user of the image analysis device 10, such as operation input or data input. The input device may include, for example, a keyboard or physical keys, or a pointing device such as a touch panel or touch sensor or mouse. The input device is not limited to these examples and may include various other devices.

[0037] The image analysis device 10 may include a control unit that controls at least one of the components described above. The control unit may include a processor or a dedicated circuit. The control unit may include a storage unit. The control unit may be configured integrally with the estimation unit 12 or the evaluation unit 13, or it may be configured separately from the estimation unit 12 or the evaluation unit 13.

[0038] The image analysis device 10 may be configured as at least one PC (Personal Computer) or server. The server may be implemented in an on-premises environment or in a cloud computing environment.

[0039] (Example of operation of image analysis device 10) The image analysis device 10 acquires an image of the object from the camera 2, estimates the object's attributes, including its number or size, from the acquired image, evaluates the validity of the estimated object attributes, and records the estimated object attributes that are deemed valid.

[0040] The image analysis device 10 may perform an image processing method that includes the steps illustrated in the flowchart of Figure 3. The steps illustrated in the flowchart of Figure 3 may be implemented as an image processing program to be executed by a processor constituting the estimation unit 12 or the evaluation unit 13. The image processing program may be stored in a non-temporary computer-readable medium such as an electromagnetic storage medium.

[0041] The estimation unit 12 of the image analysis device 10 acquires images from the imaging device 2 (step S1). The imaging device 2 generates multiple images, each of which is a separate object in a single sample. The estimation unit 12 may acquire all of the multiple images generated for a single sample. The estimation unit 12 may acquire only some of the multiple images generated for a single sample. When the estimation unit 12 acquires multiple images generated by imaging for a single sample, it may acquire only a number of images corresponding to the amount of sample required to detect the objects contained in that sample.

[0042] The estimation unit 12 estimates attributes, including the number or size of objects in the acquired image, using an estimation model (step S2). If multiple images are acquired, the estimation unit 12 may estimate the attributes of objects in all acquired images. The estimation unit 12 may estimate the attributes of objects in some of the acquired images. If multiple images are acquired by photographing a single sample, the estimation unit 12 may estimate the attributes of objects in a number of images corresponding to the amount of sample required to detect objects contained in that sample. The image used as the target of estimation by the estimation unit 12 is also called the first image to distinguish it from the backup image described later.

[0043] The estimation model is configured to accept an image as input, detect objects in the input image, and output the results of estimating attributes of the objects, including their number or size. The estimation model may be configured to output the results of estimating the positions of the objects in the input image on a pixel-by-pixel basis. The estimation model may also be configured to output the results of estimating the coordinates of the bounding boxes surrounding the objects in the input image. If the bounding box is rectangular, the estimation model may estimate the coordinates of at least two diagonally opposite corners.

[0044] The object in the image may be an aggregate 27 containing multiple aggregate elements 21-26, as illustrated in Figure 4. The number of aggregate elements in a single aggregate is not limited to six; it may be five or fewer, or seven or more. The estimation model may be configured to detect each of the multiple aggregate elements when an aggregate is present in the image, and output the number of detected aggregate elements as the estimation result. When the image illustrated in Figure 4 is input, the estimation model may detect each of the aggregate elements 21-26 and output that six aggregate elements were detected as the estimation result. The estimation model may be configured to output the estimated position of each of the detected aggregate elements 21-26, or the coordinates of the bounding boxes surrounding each of the aggregate elements 21-26.

[0045] The estimation model may be configured to output results that estimate at least one of the object's color or texture as an attribute of the object. By the estimation model estimating the object's color or texture, the accuracy of detecting individual aggregate elements in the aggregate is improved.

[0046] The estimation model may include a trained model generated by performing machine learning. Machine learning may be performed using data that associates ground truth data of the attributes of objects in an image with the image as ground truth data. Machine learning may also be performed using images of objects as unground truth data. The estimation model may be generated by the estimation unit 12 or an external device performing machine learning.

[0047] The estimation model may include an image processing model. The image processing model may be configured to output an image that has been filtered or thresholded on the input image. The image processing model may include a rule-based model that identifies an algorithm for identifying the attributes of objects in the image.

[0048] The evaluation unit 13 of the image analysis device 10 evaluates the validity of the estimation results of the attributes of the objects in the image, which are performed by the estimation unit 12, based on prior information about the objects in the image (step S3).

[0049] The evaluation unit 13 evaluates whether the estimation result is valid by comparing the estimated attribute of the object in the image with prior information about the object in the image. The evaluation unit 13 may evaluate whether the estimation result is valid using an evaluation model generated by incorporating the prior information. As described later, the evaluation unit 13 may evaluate the estimation result as valid if the difference between the estimation result and the prior information is less than a threshold.

[0050] The evaluation unit 13 may acquire information regarding the size of objects assumed to be present in the image as prior information. If the estimation result of the object's attributes includes the object's size, the evaluation unit 13 may evaluate the estimation result of the object's size as valid if the difference between the size of the object assumed to be present in the image and the size of the object estimated to be present in the image is less than a threshold. Including information regarding the object's size in the prior information improves the accuracy of evaluating the validity of the estimation result of the object's size. As a result, the accuracy of analyzing objects in the image is improved.

[0051] For example, the estimation unit 12 may mistakenly detect the aggregate 27 as a single object from the image in Figure 4 and mistakenly estimate the size of the object as the size of the aggregate 27. In this case, the evaluation unit 13 obtains the size of one of the aggregate elements 21 to 26 as prior information, calculates the difference between the size of the aggregate 27 and the size of one of the aggregate elements 21 to 26, and can evaluate that the estimated size of the object is not valid if the difference between the size of the aggregate 27 and the size of one of the aggregate elements 21 to 26 is greater than or equal to a threshold.

[0052] If the estimation result of the object attributes includes the number of objects, the evaluation unit 13 may calculate the number of objects that are expected to be in the image from the prior information and compare it with the estimation result. The evaluation unit 13 may evaluate the estimation result of the number of objects as valid if the difference between the number of objects that are expected to be in the image and the number of objects that are estimated to be in the image is less than a threshold.

[0053] For example, the estimation unit 12 may mistakenly detect the aggregate 27 as a single object from the image in Figure 4 and mistakenly estimate the number of objects as 1. In this case, the evaluation unit 13 obtains the size of one of the aggregate elements 21 to 26 as prior information, calculates that the number of objects expected to be in the image is 6 based on the size of one of the aggregate elements 21 to 26, and evaluates that the estimation result for the number of objects is not valid because the difference between the estimated number of objects and the number of objects expected from the prior information is greater than or equal to a threshold.

[0054] Information regarding the size of the object may include the diameter (D) of the circle when the outer shape of the object is approximated by a circle, as in the cohesive element 21 of Figure 4. Information regarding the size of the object may include the major axis (A) and minor axis (B) of the ellipse when the outer shape of the object is approximated by an ellipse, as in the cohesive element 23 of Figure 4. Information regarding the size of the object may include the size of the bounding box surrounding the object.

[0055] If an image containing an object is cropped around the object, the size of the image and the size of the object in the image will be approximately the same. If the object is an aggregate containing multiple aggregate elements, the size of the image containing the aggregate will be approximately the same as the size of the aggregate. Furthermore, the size of the aggregate and the number of aggregate elements contained within it may correlate. Assuming that the area of ​​the aggregate elements in the image is uniform, the area of ​​the aggregate in the image will be proportional to the number of aggregate elements contained within it. From the above, it can be concluded that the size of an image containing an aggregate as an object will be proportional to the number of aggregate elements contained within the aggregate.

[0056] Therefore, if the object is an aggregate, the prior information may include the ratio of the number of aggregate elements contained in the aggregate to the size of the image in which the aggregate is depicted. The evaluation unit 13 may evaluate the validity of the estimated attribute of the object by comparing the ratio of the number of aggregate elements estimated to be depicted in the image to the size of the image with the prior information.

[0057] Furthermore, the prior information may include the ratio of the number of aggregate elements contained in the aggregate to the size of the aggregate. The evaluation unit 13 may evaluate the validity of the estimated attribute of the object by comparing the ratio of the number of aggregate elements estimated to be present in the image to the size of the aggregate shown in the image with the prior information.

[0058] Including information about the number of aggregated elements in the prior information improves the accuracy of estimating the number of objects in the image. As a result, the accuracy of analyzing the objects in the image is improved.

[0059] The evaluation unit 13 may acquire information regarding the shape of objects assumed to be present in the image as prior information, and evaluate whether the result of detecting objects from the image is valid by comparing the shape of objects detected from the image with the prior information. The evaluation unit 13 may identify objects present in the image based on the position of the object or bounding box or segmentation mask output from the estimation model, and compare the shape of the identified object with the prior information. The evaluation unit 13 may determine whether the shape of the identified object matches or closely matches the shape based on the prior information by performing pattern matching. If the evaluation unit 13 determines that the shape of the identified object matches or closely matches the shape based on the prior information, it may evaluate that the result of detecting objects from the image is valid, and evaluate that the estimation result of the attributes of the objects present in the image is valid. Including information regarding the shape of objects in the prior information improves the accuracy of object detection from images. As a result, the accuracy of analyzing objects present in images is improved.

[0060] Information regarding the shape of the object may include a figure that can approximate the outline of the object, such as a circle, ellipse, or polygon. Information regarding the shape of the object may include the aspect ratio of the bounding box surrounding the object. Information regarding the shape of the object may include the ratio of the major axis (A) to the minor axis (B) of the ellipse, i.e., the aspect ratio, when the outline of the object is approximated by an ellipse as shown in the aggregated element 23 of Figure 4. Information regarding the shape of the object may include the number of sides or vertices of the polygon that approximates the outline of the object.

[0061] The evaluation unit 13 may acquire information about the color or texture of an object as prior information and evaluate whether the result of detecting an object from an image is valid by comparing the color or texture of the object detected from the image with the prior information. The evaluation unit 13 may identify an object in the image based on the position or bounding box of the object output from the estimation model and compare the color or texture of the identified object with the prior information. If the evaluation unit 13 determines that the color or texture of the identified object matches or closely matches the color or texture based on the prior information, it may evaluate whether the result of detecting an object from an image is valid and whether the estimated result of the attributes of the object in the image is valid. Including information about the color or texture of an object in the prior information improves the accuracy of object detection from an image. As a result, the accuracy of analyzing the object in the image is improved.

[0062] As described above, the prior information may include information that identifies standard values ​​for individual attributes of the object. The prior information may also include information that identifies acceptable ranges for individual attributes of the object. The image analysis device 10 may accept input of prior information from the user via an input device. The evaluation unit 13 may use the prior information entered by the user to evaluate the validity of the estimation results of the object's attributes.

[0063] Prior information may be stored in the memory unit in advance as information combining the type of object and the value or tolerance range of at least one item of the object's attributes. In this case, the image analysis device 10 may accept input from the user specifying the type of object. The evaluation unit 13 may evaluate the validity of the estimated results of the object's attributes using the standard value or tolerance range of the item corresponding to the type of object specified by the user.

[0064] Prior information may be entered in association with the image when the image is input to the input unit 11.

[0065] Referring again to Figure 3, the evaluation unit 13 records or aggregates the estimated attributes of the objects that were evaluated as valid in the procedure of step S3 in the recording unit 14 (step S4). The evaluation unit 13 may record the estimated results that were evaluated as valid and the images from which those estimated results were obtained in the recording unit 14 in association. The evaluation unit 13 may aggregate the estimated results that were evaluated as valid among the estimated attributes of objects depicted in the images obtained for one sample as the estimated attributes of the objects included in that sample.

[0066] The evaluation unit 13 may calculate the total number of objects estimated to be present in each of the multiple images as an aggregation of the object attribute estimation results. The evaluation unit 13 may generate a histogram of the sizes of the objects estimated to be present in each of the multiple images as an aggregation of the object attribute estimation results. The evaluation unit 13 may perform various other processes as an aggregation of the object attribute estimation results, not limited to these examples.

[0067] The evaluation unit 13 discards the estimated attribute results of the object that it evaluated as invalid in step S3 without recording them in the recording unit 14, and excludes them from the aggregation.

[0068] After executing the procedure in step S4, the image analysis device 10 terminates the execution of the flowchart in Figure 3.

[0069] <Summary> As described above, in this embodiment, the image analysis device 10 has an evaluation unit 13 that evaluates the validity of the estimation results of the attributes of objects in the image, which are estimated by the estimation unit 12 using the estimation model. In this way, even if the estimation model incorrectly detects an object from the image or incorrectly estimates the attributes of an object in the image, the incorrect estimation result is excluded from recording or aggregation. By excluding the incorrect estimation result, the accuracy of the analysis of objects in the image is improved.

[0070] (Other embodiments) An image analysis device 10A according to another embodiment of this disclosure will be described with reference to Figure 5.

[0071] The image analysis device 10A further includes a re-estimation unit 15 compared to the image analysis device 10 in Figure 2. The re-estimation unit 15 may be configured to include a processor or dedicated circuitry, similar to the estimation unit 12 or evaluation unit 13. The re-estimation unit 15 may also be configured to include a storage unit, similar to the estimation unit 12 or evaluation unit 13.

[0072] The operation of the image analysis device 10A differs from that of the image analysis device 10 in Figure 2, after the evaluation unit 13 has determined that the estimated results of the object's attributes are not valid.

[0073] As one operation after the evaluation unit 13 evaluates that the estimation result of the object's attributes is not valid, the image analysis device 10A re-estimates the object's attributes using the re-estimation unit 15. The re-estimation unit 15 obtains the results of the user of the image analysis device 10A estimating the attributes of the object in the image for which the evaluation unit 13 evaluated the estimation result as not valid. The re-estimation unit 15 may display the image for which the evaluation unit 13 evaluated the estimation result as not valid on a display so that the user can see it. The re-estimation unit 15 may accept input of the results of the user's estimation of the object's attributes after they have seen the image from an input device. The re-estimation unit 15 replaces the estimation result evaluated as not valid by the evaluation unit 13 with the user's estimation result and records or aggregates it in the recording unit 14.

[0074] As an additional operation after the evaluation unit 13 evaluates that the estimation result of the object's attributes is not valid, the image analysis device 10A replaces the image evaluated as having an invalid estimation result by the evaluation unit 13 with a backup image, and the estimation unit 12 estimates the attributes of the object depicted in the backup image. The image before replacement corresponds to the first image. The backup image after replacement is also called the second image. The backup image is an image that can replace the first image before replacement, and is selected from among multiple images generated by taking images of the same sample from which the first image was taken, from images that have not yet been estimated by the estimation unit 12. If there are images that can be used as backup images remaining among the images input from the shooting device 2 to the input unit 11, the estimation unit 12 may select a backup image from the remaining images. If there are images that can be used as backup images remaining among the images that have not yet been input from the shooting device 2 to the input unit 11, the input unit 11 may accept input of an image that can be used as a backup image from the shooting device 2, and the estimation unit 12 may select a backup image from the input image.

[0075] The estimation unit 12 selects images similar to those evaluated by the evaluation unit 13 as having an unreasonable estimation result as backup images. The estimation unit 12 may determine that images with matching or nearly matching image sizes, or images where the difference in image size is less than a threshold, are similar images. The estimation unit 12 may also determine that images with matching or nearly matching distributions of luminance values ​​for multiple pixels constituting the image, or images where the difference in luminance value distributions is less than a threshold, are similar images. The estimation unit 12 may also select similar images using the distribution of color information, etc., for multiple pixels constituting the image, i.e., the image histogram, rather than being limited to the distribution of luminance values ​​for multiple pixels constituting the image. The estimation unit 12 may also determine that images with matching or nearly matching feature quantities, such as SIFT (Scale-Invariant Feature Transform), calculated from the image, or images where the difference in feature quantities is less than a threshold, are similar images. The estimation unit 12 may determine that images are similar if their feature vectors, which have multiple feature quantities as elements, match or nearly match, or if the difference between their feature vectors is less than a threshold. The estimation unit 12 may select similar images using the SSIM (Structural Similarity) of the images. SSIM is an index that quantifies the similarity of image structures. The estimation unit 12 may also select similar images using a pre-trained model for selecting similar images. The pre-trained model for selecting similar images may be a model generated by performing training using combinations of similar images as ground truth data. The estimation unit 12 may select similar images using a method that combines the various determination methods described above.

[0076] The image analysis device 10A may replace the estimation results that the evaluation unit 13 has deemed invalid with the results obtained by the estimation unit 12 in which the attributes of the objects in the preliminary image have been re-estimated, and record or aggregate these results in the recording unit 14. The image analysis device 10A may use the evaluation unit 13 to evaluate the validity of the results obtained by the estimation unit 12 in which the attributes of the objects in the preliminary image have been re-estimated. If the evaluation unit 13 evaluates the estimation results of the attributes of the objects in the preliminary image as valid, it may record or aggregate the estimation results that it has evaluated as valid in the recording unit 14.

[0077] The image analysis device 10A may perform an image processing method that includes the procedure illustrated in the flowchart of Figure 6. The procedure illustrated in the flowchart of Figure 6 may be implemented as an image processing program to be executed by a processor constituting the estimation unit 12, evaluation unit 13, re-estimation unit 15, or control unit of the image analysis device 10A. The image processing program may be stored in a non-temporary computer-readable medium such as an electromagnetic storage medium.

[0078] The estimation unit 12 acquires an image from the imaging device 2 (step S11). The estimation unit 12 estimates the attributes of the objects in the acquired image, including the number or size, using an estimation model (step S12). The evaluation unit 13 evaluates the validity of the estimation results of the objects in the image by the estimation unit 12 based on prior information about the objects in the image (step S13). The steps from steps S11 to S13 may be performed in the same way as the steps from steps S1 to S3 in the flowchart of Figure 3.

[0079] The evaluation unit 13 evaluates whether the estimated attribute of the object is valid (step S14). If the evaluation unit 13 evaluates that the estimated attribute of the object is valid (step S14: YES), the image analysis device 10A proceeds to the procedure in step S18, which will be described later.

[0080] If the evaluation unit 13 evaluates that the estimation result of the object's attributes is not valid (step S14: NO), the image analysis device 10A determines whether there is a backup image that can be used instead of the image whose estimation result was evaluated as not valid (step S15). For example, if there are multiple images taken for one sample acquired by the input unit 11 from the imaging device 2 that have not yet been used to estimate the object's attributes, the estimation unit 12 may determine whether there is an image that can be used as a backup image among the remaining images. Alternatively, if there are multiple images taken by the imaging device 2 for one sample that have not yet been acquired by the input unit 11, the estimation unit 12 may acquire the unacquired images at the input unit 11 and determine whether there is an image that can be used as a backup image among the newly acquired images. The procedure in step S15 may be performed by the evaluation unit 13, the re-estimation unit 15, or the control unit.

[0081] If it is determined that a backup image exists (step S15: YES), the estimation unit 12 estimates the attributes of the object in the backup image (step S16). The procedure in step S16 may be performed in the same way as the procedure in step S12. After the estimation unit 12 has performed the procedure in step S16, the image analysis device 10A may return to the procedure in step S13 and have the evaluation unit 13 evaluate the estimation result of the attributes of the object in the backup image.

[0082] If it is determined that there are no backup images (Step S15: NO), the re-estimation unit 15 performs a re-estimation of the attributes of the objects depicted in the image for the image in which the estimated result of the object's attributes was evaluated as not valid (Step S17).

[0083] The image analysis device 10A records or aggregates the estimation results that have been evaluated as valid, or the re-estimation results by the re-estimation unit 15, in the recording unit 14 (step S18). Specifically, the evaluation unit 13 records or aggregates the estimation results that have been evaluated as valid in the recording unit 14. If the estimation unit 12 replaces an image with a backup image and estimates the attributes of the object, the evaluation unit 13 may record or aggregate the estimation results that have been evaluated as valid from the estimation results of the attributes of the object shown in the backup image in the recording unit 14. If re-estimation is performed on an image corresponding to an estimation result that has been evaluated as invalid, the re-estimation unit 15 records or aggregates the re-estimation results in the recording unit 14. After executing the procedure in step S18, the image analysis device 10A finishes executing the procedure in the flowchart of Figure 6.

[0084] If the image analysis device 10A evaluates that the estimated result of the object's attributes is not valid, it may perform re-estimation by the re-estimation unit 15 for the image corresponding to the estimated result that was evaluated as invalid, regardless of whether or not there is a backup image. The image analysis device 10 does not have to perform re-estimation by the re-estimation unit 15 even if there is no backup image.

[0085] As described above, according to the image analysis device 10A, and the image analysis method and image analysis program executed by the image analysis device 10A, if the estimated result of the attributes of the object is evaluated as not valid, replacement with a backup image or re-estimation is performed. In this way, even if the estimated result of the attributes of the object captured in some images taken for one sample is evaluated as not valid, the lack of images necessary for recording or summarizing the estimated results of the attributes of the objects included in one sample is compensated for. As a result, the accuracy of recording or summarizing the estimated results of the attributes of the objects included in the sample is improved.

[0086] While embodiments relating to this disclosure have been described above with reference to the drawings, the specific configuration is not limited to these embodiments and may include various modifications without departing from the spirit of this disclosure. [Explanation of symbols]

[0087] 1. Image Analysis System 2. Imaging device 10, 10A Image analysis device (11: Input unit, 12: Estimation unit, 13: Evaluation unit, 14: Recording unit, 15: Re-estimation unit) 21~26 Cohesive elements 27 Aggregates

Claims

1. Estimating at least one of the attributes of the object, such as the number or size of the object, from a first image of the object, Evaluating the estimated results of the attributes of the object based on prior information about the object, Record or compile the estimation results that have been deemed reasonable. An image analysis program that instructs the processor to execute.

2. If the estimated attribute of the object shown in the first image is deemed inappropriate, the attribute of the object is re-estimated from the first image. Record or compile the re-estimation results. The image analysis program according to claim 1, which causes the processor to execute the following.

3. The image analysis program according to claim 1 or 2, wherein if the estimated result of the attributes of an object depicted in the first image is evaluated as not valid, the processor is instructed to estimate the attributes of the object from a backup second image that can replace the first image.

4. The image analysis program according to claim 1 or 2, wherein the prior information of the object includes information regarding the shape or size of the object.

5. The image analysis program according to claim 1 or 2, wherein the prior information of the object includes, if the object is an aggregate containing aggregate elements, the ratio of the size of the aggregate to the number of aggregate elements, or the ratio of the size of the first image to the number of aggregate elements.

6. The image analysis program according to claim 1 or 2, which causes the processor to further estimate at least one of the color or texture of the object as an attribute of the object from the first image.

7. The image analysis program according to claim 6, wherein the prior information of the object includes information regarding the color or texture of the object.

8. The way in which a processor executes Estimating at least one of the attributes of the object, such as the number or size of the object, from a first image of the object, Evaluating the estimated results of the attributes of the object based on prior information about the object, Record or compile the estimation results that have been deemed reasonable. Image analysis methods, including those mentioned above.

9. An estimation unit that estimates at least one of the attributes of the object, such as the number or size of the object, from a first image of the object, An evaluation unit evaluates the estimated results of the attributes of the object based on prior information about the object, and records or aggregates the estimated results that are deemed valid. An image analysis device equipped with the following features.

10. The image analysis apparatus according to claim 9, further comprising a re-estimation unit that, when the estimated result of the attributes of an object captured in the first image is evaluated as not valid, re-estimates the attributes of the object from the first image and records or aggregates the results of the re-estimate.

11. The image analysis apparatus according to claim 9 or 10, wherein the estimation unit estimates the attributes of an object from a backup second image that can replace the first image if the estimation result of the attributes of the object in the first image is evaluated as not being valid.

Images