Image processing device, image processing system and method

Abstract

To provide an image processing device and a method which can estimate a collection form of a collection object, and estimate a quantitative value of the collection object on the basis of the collection form.SOLUTION: An image processing device 100 which estimates a quantitative value of a collection object in an image comprises: an input unit 201 which accepts an input image; a collection object detection unit 202 which detects the collection object in the input image; a collection form estimate unit 203 which estimates a collection form of the detected collection object; and a quantitative value estimate unit 204 which estimates the quantitative value of the detected collection object on the basis of the estimated collection form.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing system and a method. [Background technology]

[0002] Analysis of images captured by cameras or microscopes is performed in many fields for the purposes of inspection, quality control, monitoring, etc. One common analysis method is to detect target objects in an image. In analyzing target objects in an image, there are situations where it is necessary to go beyond simple object detection and estimate additional information about the detected object. For example, there are many situations where it is necessary to estimate numerical information that enables quantitative analysis, and therefore there is a demand for technology that can estimate numerical information from images with high accuracy.

[0003] An example of a technique that requires detecting objects in an image and calculating numerical information related to the detected image is estimating the number of people in an image for purposes such as people flow analysis. Known techniques for estimating the number of people include detecting people using background subtraction or a machine learning model and then counting the detected people. However, counting detected people can be difficult to accurately count when people are densely packed in an image. Therefore, Patent Document 1 proposes a technology to improve the accuracy of estimating the number of people in an image. Patent Document 1 states that "the accuracy of the number of people estimation results using a learning model is improved by applying a learning model according to the human body size, eliminating erroneous detection results, and eliminating inaccuracies in the results due to inaccuracies in the application settings of the learning model."

[0004] Incidentally, when an object detected in an image is an aggregate object made up of specific elements, there are cases where it is required to estimate numerical information about the elements that make up the aggregate object. For example, there are cases where it is required to estimate the number of components of an aggregate object detected in an image. Anticipating such cases, there is a technology described in Patent Document 2. Patent Document 2 describes that "grapes and grape bunches are detected, and based on the position and size of the grape bunch, the grape bunch currently being worked on is identified from the detected grape bunches, and the grapes belonging to the grape bunch currently being worked on are determined." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2023-163079 [Patent Document 2] Patent Publication No. 2021-189718 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 uses a trained machine learning model to improve the accuracy of estimating the number of people in an image. Rather than counting detected people, regressing the number of people in the image is thought to make it possible to estimate the number of people in a cluster. However, the technology described in Patent Document 1 is expected to have difficulty accurately estimating the number of people using a uniform machine learning model due to differences in the size and shape of people. Therefore, Patent Document 1 improves the accuracy of estimating the number of people by dividing the image according to the size of the people and making the relative size of the people in the divided image closer to a constant. However, if people are densely packed in an image, the size of individuals within the crowd is not taken into account, and the image may be divided in a way that separates the group and the individuals within it. It is difficult to accurately estimate the number of groups or people that are cut off from the divided partial images. Therefore, Patent Document 1 is not suitable for images where many people may be densely packed or for objects other than people that may take the form of a group.

[0007] The technology described in Patent Document 2 estimates the number of grapes in each grape cluster in an image by detecting the grape cluster and the grapes separately. However, for aggregated objects such as grapes, it is possible that the grape shape may be distorted due to the high density of grapes, or that the boundaries between grapes may become unclear in the image. If the accuracy of grape detection is reduced for these reasons, the accuracy of estimating the number of grapes in each grape cluster also decreases. Furthermore, Patent Document 2 targets the detection and counting of grapes with uniform shapes. However, in reality, the clustering morphology that determines grape shape is not uniform, and it may be necessary to identify the variety and growth state based on the clustering morphology. Furthermore, since it is anticipated that varieties and other properties may need to be classified based on the shape and color of the grapes, it is desirable in practice to classify grapes by multiple types and diverse clustering morphologies. Furthermore, a method is needed to accurately calculate numerical information such as the number of grapes, which is a quantitative value, without relying on the detection of individual grapes. These issues exist not only for grapes but also for many aggregated objects.

[0008] An example of an aggregated object that can take on multiple types and diverse aggregation forms is blue-green algae, which are observed in river water quality surveys, etc. Blue-green algae are classified into many types based on the shape of their constituent cells. Blue-green algae are microorganisms that can also be classified by their various aggregation forms, such as spherical, chain-like, and sheet-like. In analyzing microscopic images of these blue-green algae, it is useful to detect them while distinguishing them by type and aggregation form, and to estimate numerical information that enables quantitative analysis of each detected blue-green algae.

[0009] However, because Patent Documents 1 and 2 do not take into account the aggregation morphology of the cyanobacteria, it is difficult to accurately detect individual cells densely packed within the cyanobacteria. Furthermore, the numerical information that can be estimated using the techniques described in Patent Documents 1 and 2 is limited to the number of elements in the aggregated object, i.e., the number of cells in the cyanobacteria. When the object type or aggregation morphology varies, i.e., when the visual characteristics such as the size and shape of the target aggregated object vary, a uniform method such as a machine learning model has the problem of making it difficult to estimate numerical information with high accuracy compared to methods specialized for each case.

[0010] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, an image processing system and a method that are capable of estimating the aggregate form of an aggregate object and estimating the quantitative value of the aggregate object based on the aggregate form. [Means for solving the problem]

[0011] In order to solve the above problem, an image processing device according to one aspect of the present invention is an image processing device that estimates a quantitative value of an aggregate object in an image, and includes an input unit that accepts an input image, an aggregate object detection unit that detects aggregate objects in the input image, an aggregate form estimation unit that estimates the aggregate form of the detected aggregate object, and a quantitative value estimation unit that estimates the quantitative value of the detected aggregate object based on the estimated aggregate form. [Effects of the Invention]

[0012] According to the present invention, the quantitative value of an aggregate object can be estimated based on the aggregate form. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 3] 1 is a flowchart showing an image processing method. [Figure 4] FIG. 10 is a diagram illustrating an example of classification of aggregate objects by object type. [Figure 5] 10A and 10B are diagrams illustrating an example of classification of aggregate objects according to their aggregate forms. [Figure 6] FIG. 10 is a diagram illustrating an example of quantitative values ​​of aggregate objects. [Figure 7] FIG. 10 is a diagram illustrating an example of a hardware configuration of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a display of an image processing result. [Figure 9] FIG. 10 is a diagram illustrating an example of a hardware configuration of an image processing system according to a third embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a display of an image processing result. [Figure 11] An example of calculating a quantitative value according to the type of aggregate object will be shown. [Figure 12] An example of calculating quantitative values ​​according to the aggregation form will be shown. [Figure 13] A graph of the quantitative values ​​of FIG. 12 is shown. [Figure 14] FIG. 10 is a diagram illustrating an example of a hardware configuration of an image processing system according to a fourth embodiment. [Figure 15] FIG. 10 is a diagram showing an example of a display of an image processing result. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will now be described with reference to the accompanying drawings. An image processing device according to this embodiment estimates the aggregate shape and quantitative value of an aggregate object in an image. The image processing device according to this embodiment estimates with high accuracy the quantitative value of objects of multiple types or diverse aggregate shapes by taking into consideration what types of objects are aggregated and what shapes they have. The image processing device according to this embodiment can analyze aggregate objects in more detail, even if they are made up of the same type of objects, by classifying them based on the shapes of the aggregate objects.

[0015] An image processing device 100 of this embodiment, which will be described later in FIG. 2, is an image processing device 100 that estimates quantitative values ​​of aggregate objects in an image, and includes an input unit 201 that accepts an input image, an aggregate object detection unit 202 that estimates the position, size, and type of aggregate objects in the input image, an aggregate form estimation unit 203 that estimates the aggregate form in the input image, and a quantitative value estimation unit 204 that calculates quantitative values ​​of aggregate objects in the input image.

[0016] According to the image processing device 100 of this embodiment, aggregate objects in an image that can take multiple types or various forms can be classified according to the form of the aggregate, and various quantitative values ​​can be estimated with high accuracy. [Example]

[0017] An embodiment will be described below with reference to FIGS.

[0018] <Hardware configuration of image processing device> The hardware configuration of an image processing device 100 according to the first embodiment will be described with reference to Fig. 1. The image processing device 100 is configured using, for example, a computer. The image processing device 100 includes, for example, an interface unit 101, a calculation unit 102, and a memory 103, and transmits and receives information via a bus.

[0019] Each unit of the image processing device 100 will be described.

[0020] The interface unit 101 is a communication device that transmits and receives signals to and from devices external to the image processing device 100. Devices that communicate with the interface unit 101 include, for example, an imaging device 110 such as a camera or a microscope, and a display device 111 such as a monitor or a printer.

[0021] The calculation unit 102 is a device that executes various processes within the image processing device 100, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The functions executed by the calculation unit 102 will be described later with reference to FIG. 2. The memory 103 is a device that stores the computer program executed by the calculation unit 102, the machine learning model used in the computer program, its parameters, the processing results, etc. The memory 103 includes a main storage device and an auxiliary storage device (neither of which are shown). Examples of the auxiliary storage device include a hard disk drive (HDD) and a solid state drive (SSD).

[0022] The image processing device 100 can be connected to a storage medium MM. The storage medium MM is configured as, for example, a memory device, a hard disk device, an optical disk device, a magneto-optical disk device, a magnetic tape device, or the like, and non-temporarily stores computer programs and data. The storage medium MM can transfer and store computer programs and data to the memory 103 of the image processing device 100. Computer programs and data can also be transferred and stored from the memory 103 to the storage medium MM. A computer program that realizes the functions of the image processing device 100 can be stored in the storage medium MM, and by connecting the storage medium MM to another computer and installing the computer program stored in the storage medium MM on the other computer, the other computer can function as the system 1.

[0023] <Functional configuration of image processing device> 2 is a functional block diagram of the image processing device 100. Each of these functional units 201 to 208 may be realized as a computer program that runs on the calculation unit 102, or may be realized as a module having dedicated hardware.

[0024] The image processing device 100 has, as functional units, for example, an input unit 201, an aggregated object detection unit 202, an aggregated form estimation unit 203, a quantitative value estimation unit 204, and an output unit 205. The aggregated object detection unit 202 has, for example, a feature extraction unit 206, an aggregated object position estimation unit 207, and an aggregated object type estimation unit 208. Each functional unit will be described below.

[0025] The input unit 201 accepts an image input from outside the image processing device 100. The input image is, for example, an image captured using a camera, a microscope, or the like. The input image is converted into a format that can be handled by a feature extraction unit 206 of the aggregate object detection unit 202 (described later), and is subjected to processing such as resizing, cropping, and brightness value correction as necessary.

[0026] The aggregate object detection unit 202 detects aggregate objects that have been set as targets in advance from the image passed from the input unit 201. An aggregate object may be any object in which one or more specific elements are densely packed together, and may be, for example, a microorganism that forms a colony such as blue-green algae, or a substance made up of multiple linked metal nanoparticles.

[0027] The aggregate object detection process performed by the aggregate object detection unit 202 includes a process of creating a feature map for the input image (step S12 in FIG. 3), a process of estimating the position and size of each aggregate object in the image from the feature map (step S13 in FIG. 3), and a process of estimating the type of each aggregate object in the image from the feature map (step S14 in FIG. 3).

[0028] The feature map is an output that summarizes the feature amounts extracted for each region in the input image. Below, the functional units 206, 207, and 208 that execute the processes related to the detection of aggregate objects will be described.

[0029] The feature extraction unit 206 extracts features from the input image passed from the input unit 201 and creates a feature map. For example, a deep neural network (DNN) using convolution of feature pyramid networks (FPN) can be used to extract the features.

[0030] The aggregated object position estimation unit 207 estimates the position and size of aggregated objects present in the image from the feature map passed from the feature extraction unit 206. The position and size of the aggregated object may be any information that can determine a rectangle surrounding the aggregated object in the input image. The position and size of the aggregated object may be, for example, numerical information representing the center coordinates of the aggregated object in the input image and the width and height of the aggregated object.

[0031] The aggregated object type estimation unit 208 estimates the type of aggregated object present in the image from the feature map passed from the feature amount extraction unit 206. The estimated type of aggregated object may be any information that can determine a predetermined classification label of the aggregated object, and may be, for example, the likelihood of each label used to classify the aggregated object.

[0032] The aggregate morphology estimation unit 203 estimates the aggregate morphology of aggregated objects present in the image from the feature map passed from the feature extraction unit 206. The estimated aggregate morphology of the aggregated object may be any information that can determine a classification label for the aggregated object based on a predetermined aggregate morphology, and may be, for example, the likelihood of each aggregate morphology label. The aggregate morphology may be, for example, a chain or spherical shape that cells in blue-green algae can take, or a simplex shape that indicates that the objects are not densely packed.

[0033] The quantitative value estimation unit 204 estimates quantitative values ​​for aggregated objects present in the image using the feature map passed from the feature extraction unit 206. The estimated quantitative value of the aggregated object may be any numerical information that can be estimated from the image. The quantitative value may be, for example, the number of cells in the blue-green algae, their density, length, area, or volume. In estimating the quantitative value, in addition to the feature map passed from the feature extraction unit 206, one or more of the following may be used: the position and size of the aggregated object in the image estimated by the aggregated object position estimation unit 207; the type of the aggregated object in the image estimated by the aggregated object type estimation unit 208; and the aggregated form of the aggregated object in the image estimated by the aggregated form estimation unit 203.

[0034] The output unit 205 outputs the positions, sizes, and object types of aggregated objects in the image estimated by the aggregated object detection unit 202, the aggregated forms of the aggregated objects in the image estimated by the aggregated form estimation unit 203, and the quantitative values ​​of the aggregated objects in the image estimated by the quantitative value estimation unit 204 as image processing results to the outside of the image processing device 100. These pieces of information may be output together or individually. The output destination may be, for example, a monitor display, a printer, or a virtual space.

[0035] <Flowchart> FIG. 3 is an example of a flowchart showing an image processing method S10 executed by the image processing device 100. Each step corresponds to each element in the functional block diagram shown in FIG. 2. That is, an input step S11 is executed by the input unit 201. A feature amount extraction step S12 is executed by the feature amount extraction unit 206. An object position estimation step S13 is executed by the object position estimation unit 207. An object type estimation step S14 is executed by the object type estimation unit 208. An assembly form estimation step S15 is executed by the assembly form estimation unit 203. A quantitative value estimation step S16 is executed by the quantitative value estimation unit 204. An output step S17 is executed by the output unit 205.

[0036] In the input step S11, an image input from outside the image processing device 100 is accepted. The input image is, for example, an image captured using a camera, a microscope, etc. The input image is converted into a format that can be handled by the feature extraction step S12 described later, and is subjected to processing such as resizing, cropping, and brightness value correction as necessary.

[0037] In the feature extraction step S12, features are extracted from the input image passed from the input step S11, and a feature map is created. The feature extraction is performed using a DNN or the like that uses FPN convolution.

[0038] In the object position estimation step S13, the position and size of aggregated objects present in the image are estimated from the feature map passed from the feature extraction step S12. Here, an aggregated object may be any object in which one or more specific elements are densely packed, such as a colony-forming microorganism such as blue-green algae, or a substance formed by the connection of multiple metal nanoparticles. The position and size of the aggregated object may be any information that can determine a rectangle surrounding the aggregated object in the input image, such as numerical information representing the center coordinates of the aggregated object in the input image and the width and height of the aggregated object.

[0039] In the object type estimation step S14, the type of aggregated object present in the image is estimated from the feature map passed from the feature extraction step S12. The type of aggregated object to be estimated may be any information that can determine a predetermined classification label of the aggregated object, such as the likelihood of each label used to classify the aggregated object.

[0040] In the aggregation morphology estimation step S15, the aggregation morphology of the aggregated objects present in the image is estimated from the feature map passed from the feature extraction step S12. The aggregation morphology of the estimated aggregated objects may be any information that can determine a classification label for the aggregated objects according to a predetermined aggregation morphology, and may be, for example, the likelihood of each aggregation morphology label. The aggregation morphology may be, for example, a chain or sphere shape that can be taken by cells in blue-green algae, or a simplex shape that indicates that the objects are not densely packed.

[0041] In the quantitative value estimation step S16, a quantitative value of the aggregated object present in the image is estimated from the feature map passed from the feature extraction step S12. The estimated quantitative value of the aggregated object may be any numerical information that can be estimated from the image, such as the number of cells in the blue-green algae, density, length, area, or volume. In addition to the feature map passed from the feature extraction step S12, the quantitative value may be estimated using one or more of the following: the position and size of the aggregated object in the image estimated in the object position estimation step S13; the type of the aggregated object in the image estimated in the object type estimation step S14; and the aggregated form of the aggregated object in the image estimated in the aggregated form estimation step S15.

[0042] The output step S17 can output, as image processing results, to the outside of the image processing device 100, the positions, sizes, and object types of the aggregated objects in the image estimated in the object position estimation step S13 and the object type estimation step S14, the aggregated forms of the aggregated objects in the image estimated in the aggregated form estimation step S15, and the quantitative values ​​of the aggregated objects in the image estimated in the quantitative value estimation step S16.

[0043] <Configuration and operation of each part> Below, we will explain in detail the operations of the functional units: the aggregate object detection unit 202, the aggregate morphology estimation unit 203, the quantitative value estimation unit 204, and the output unit 205. Although the subject of the operations is each functional unit, it is acceptable to interpret each corresponding step as the subject.

[0044] The aggregate object detection unit 202 detects aggregate objects present in an image passed from the input unit 201. As described above, an aggregate object may be any object in which one or more specific elements are gathered together, such as a colony-forming microorganism such as blue-green algae, or a substance formed by the linking of multiple metal nanoparticles. As described above, the aggregate object detection unit 202 includes a feature estimation unit 206, an aggregate object position estimation unit 207, and an aggregate object type estimation unit 208. The feature map output by the feature extraction unit 206 is passed to the aggregate object position estimation unit 207, the aggregate object type estimation unit 208, the aggregate morphology estimation unit 203 outside the aggregate object detection unit 202, and the quantitative value estimation unit 204.

[0045] The feature map extracted by the feature extraction unit 206 is extracted by a DNN using FPN convolution, etc., and maintains a positional correspondence with the image passed by the input unit 201 to the aggregate object detection unit 202. When a DNN is used for feature extraction, the network may include layers with functions other than convolution, such as a layer that performs upsampling to increase the size of the feature map, or a residual connection that connects an output at a certain point to a subsequent output, as long as the positional correspondence is not destroyed.

[0046] When a DNN is used, the parameters of the network used for feature extraction are optimized based on the outputs of the aggregated object position estimation unit 207, aggregated object type estimation unit 208, aggregated shape estimation unit 203, and quantitative value estimation unit 204, to which the feature map is input, and the loss calculated between these correct answers. In other words, by simultaneously considering multiple related tasks estimated by each functional unit, the feature extraction unit 206 can extract general-purpose features.

[0047] The aggregate object position estimation unit 207 uses a DNN to estimate the position and size of the aggregate object in the image from the feature map output by the feature extraction unit 206. As described above, the estimated position and size of the aggregate object may be any information that can determine a rectangle surrounding the aggregate object in the input image, and may be, for example, numerical information that represents the center coordinates of the aggregate object in the input image and the width and height of the aggregate object.

[0048] The aggregated object type estimation unit 208 uses a DNN to estimate the type of aggregated object in the image from the feature map output by the feature extraction unit 206. As described above, the estimated type of aggregated object may be any information that can determine a predetermined classification label for the aggregated object to be detected, such as the likelihood for each label obtained by classifying the aggregated object according to any criteria. When the output is the likelihood for each classification label, the label showing the highest likelihood may be determined as the type of the aggregated object.

[0049] Figure 4 shows an example of classification of types of aggregated objects, taking cyanobacteria as an example of an aggregated object. For example, cyanobacteria can be classified into those made up of spherical cells and those made up of cylindrical cells based on the shape of their cells. In this case, in image 400, cyanobacteria 401 may be classified as cyanobacteria made up of spherical cells, and cyanobacteria 402 may be classified as cyanobacteria made up of cylindrical cells.

[0050] The position, size, and type of the aggregated object may be estimated by estimating a feature map representing each piece of information using a convolution neural network (CNN) or the like, or the position, size, and type of each aggregated object in the image may be directly estimated using a transformer, etc. Furthermore, the aggregated object position estimation unit 207 and the aggregated object type estimation unit 208 may share part or all of the functional configuration, or may be configured completely independently.

[0051] The aggregated morphology estimation unit 203 uses a DNN to estimate the aggregated morphology of aggregated objects in an image based on the feature map passed from the feature extraction unit 206 in the aggregated object detection unit 202. As described above, the estimated aggregated morphology of an aggregated object may be any information that can determine a classification label for the aggregated object based on the aggregated morphology, such as the likelihood for each aggregated morphology label. The aggregated morphology may be, for example, a chain or spherical shape that cells in blue-green algae can take, or a simplex shape that indicates that the objects are not densely packed. If the output is the likelihood for each aggregated morphology label, the label with the highest likelihood can be determined as the aggregated morphology of the aggregated object.

[0052] FIG. 5 illustrates an example of classification of the aggregation morphology of aggregated objects, taking cyanobacteria as an example of an aggregated object. For example, cyanobacteria may be classified into single-celled cyanobacteria, spherically-shaped aggregated cells, and chain-shaped aggregated cells based on the way the cells are organized. In this case, cyanobacteria 501 in image 500 may be classified as single cyanobacteria, cyanobacteria 502 as spherical cyanobacteria, and cyanobacteria 503 as chain-shaped cyanobacteria. The aggregation morphology of aggregated objects may be estimated by estimating a feature map representing each piece of information using a CNN or the like, or by directly estimating the aggregation morphology of each aggregated object in the image using a transformer or the like. Furthermore, the aggregation morphology estimation unit 203 may share part or all of the functional configuration with the aggregated object position estimation unit 207 and the aggregated object type estimation unit 208 in the aggregated object detection unit 202, or may be configured completely independently.

[0053] 2, the quantitative value estimation unit 204 uses a DNN to estimate the quantitative values ​​of aggregated objects in the image from the feature map passed from the feature extraction unit 206. As described above, the estimated quantitative values ​​of aggregated objects may be any numerical information that can be estimated from an image, and may be, for example, the number or density of cells in the blue-green algae, or the length, area, or volume.

[0054] Figure 6 shows an example of a method for calculating the quantitative value of an aggregated object, taking cyanobacteria as an example of the aggregated object. As shown in the upper part of Figure 6, for example, the number of cells in the cyanobacteria can be set as the estimated quantitative value. In this case, the quantitative value of cyanobacteria 601 in image 600 is "8," and the quantitative value of cyanobacteria 602 is "4." In cyanobacteria 601, eight spherical cyanobacteria are gathered in a ring, so the number as a quantitative value is "8." In cyanobacteria 602, four cyanobacteria are gathered in a chain, so the number as a quantitative value is "4."

[0055] For example, the area of ​​the blue-green algae on the image may be set as the estimated quantitative value. In this case, the quantitative value of the blue-green algae 601 may be the area of ​​the region 603, and the quantitative value of the blue-green algae 602 may be the area of ​​the region 604.

[0056] In estimating the quantitative value, in addition to the feature map output by the feature extraction unit 206, one or more of the positions and sizes of aggregated objects in the image output by the aggregated object position estimation unit 207, the types of aggregated objects in the image output by the aggregated object type estimation unit 208, and the aggregated forms of aggregated objects in the image output by the aggregated form estimation unit 203 may be used.

[0057] At this time, each output may be subjected to shaping processing, such as conversion into a feature map format. For example, when the output of the aggregated object position estimation unit 207 is related to the position and size of the aggregated object, the accuracy of estimating the quantitative value is improved by using it in conjunction with the feature map output by the feature extraction unit 206. Similarly, when the output of the aggregated object type estimation unit 208 is related to the type of aggregated object, the accuracy of estimating the quantitative value is improved by using it in conjunction with the feature map output by the feature extraction unit 206. Similarly, when the output of the aggregated object form estimation unit 203 is related to the aggregated form of the aggregated object, the accuracy of estimating the quantitative value is improved by using it in conjunction with the feature map output by the feature extraction unit 206.

[0058] Note that there are many possible quantitative values ​​that relate to multiple factors such as the position, size, type, and aggregation form of aggregated objects. In such cases, the quantitative values ​​may be estimated by using the multiple outputs described above in conjunction with the feature map output by the feature extraction unit 206. The quantitative values ​​of aggregated objects may be estimated by estimating a feature map representing the quantitative values ​​using a CNN or the like, or by directly estimating the quantitative values ​​for each aggregated object in the image using a transformer or the like. Furthermore, the quantitative value estimation unit 204 may share part or all of its functional configuration with the aggregated object position estimation unit 207, aggregated object type estimation unit 208, or aggregated form estimation unit 203 in the aggregated object detection unit 202, or may be configured completely independently.

[0059] The output unit 205 compiles the positions, sizes, and object types of aggregated objects in the image estimated by the aggregated object detection unit 202, the aggregated forms of aggregated objects in the image estimated by the aggregated form estimation unit 203, and the quantitative values ​​of aggregated objects in the image estimated by the quantitative value estimation unit 204 as processing results and outputs them to the outside of the image processing device 100. The output may be, for example, a feature map for the input image estimated by each functional unit, or a list storing numerical information about aggregated objects, as images or numerical information. The output unit 205 may also draw on the input image shapes such as rectangles indicating the positions and sizes of each aggregated object, numerical values ​​or character strings indicating each label and quantitative value, and output the results in a format that is easy for the user to interpret. The output processing results may be filtered by the output of the aggregated object detection unit 202, for example, by removing aggregated objects whose object type likelihood is below a threshold. Alternatively, for example, when the overlap rate of rectangles surrounding aggregated objects is equal to or greater than a reference value, overlapping aggregated objects may be integrated and output.

[0060] As described above for the aggregate object detection unit 202, detection is performed on an aggregate object basis. That is, the aggregate object detection unit 202 searches for features of a predetermined object type in an input image when the predetermined object type takes on a similarly predetermined aggregate form. To this end, the image processing device 100 is trained using images of aggregate objects, position and size information that allows determination of a rectangle enclosing each aggregate object in the image, and, for each aggregate object whose position and size are indicated, information that includes an object type label, an aggregate form label, and a correct answer for the estimated quantitative value. In cases where elements of aggregate objects in an image are not in an aggregated state and detection and quantitative value estimation are required even when the elements are not in an aggregated state, a label indicating a single element may be included as one of the aggregated forms. The aggregate object detection unit 202 trained with the label of the single element can also capture features when the element is not in an aggregated state but exists alone.

[0061] According to the image processing device 100 of this embodiment configured as described above, aggregate objects in an image that can take multiple types or various forms can be classified according to the form of the aggregate, and various quantitative values ​​can be estimated with high accuracy. [Example]

[0062] Example 2 will be described with reference to Figures 7 and 8. In the following examples including this example, differences from Example 1 will be mainly described. Example 2 is an image processing system that uses the image processing device 100 described in Example 1 to accurately estimate quantitative values ​​of aggregate objects in an image.

[0063] 7 shows a hardware configuration diagram according to Example 2. An image processing system 1000 according to Example 2 includes an imaging device 110, an image processing device 100, a display device 111, and a storage device 1001.

[0064] <Operation of each device> The image capturing device 110 is a device for capturing an image of a target aggregate object, and is, for example, a camera, a microscope, etc. The captured image is input to the image processing device 100.

[0065] The image processing device 100 is the device described in Example 1. The image processing device 100 detects aggregate objects from an input image that is captured by an imaging device 110 and that has undergone processing such as cropping in some cases, and estimates and outputs the form (aggregate form) and quantitative value of the aggregate object.

[0066] The image processing device 100 may output, for example, a feature map for the input image estimated by each functional unit, a list storing numerical information about aggregate objects, etc., as an image or numerical information. The image processing device 100 may, for example, draw shapes such as rectangles indicating the position and size of each aggregate object on the input image, and numerical values ​​or character strings indicating each label or quantitative value, and output them in a format that is easy for the user to interpret.

[0067] The display device 111 is a device for presenting the image processing results output by the image processing device 100 to the user, and is, for example, a monitor or a printer.

[0068] The storage device 1001 holds information related to quantitative value estimation that is preset by a user. The information related to quantitative value estimation in the second embodiment includes information on classification labels based on the type or aggregation form of a target aggregate object that is preset, or the configuration and parameters of a machine learning model used in the image processing device 100. Although not shown in Fig. 7, the image processing system 1000 may include an interface device such as a keyboard, and may be configured so that the information related to quantitative value estimation recorded in the storage device 1001 can be rewritten by a user during system operation.

[0069] Figure 8 shows an example of the display of image processing results when cyanobacteria are used as the aggregated object and the number of cells contained in the cyanobacteria is used as the estimated quantitative value. In this example, the types of cyanobacteria are classified into those with spherical cells and those with cylindrical cells based on the shape of the cells. The aggregation morphology of the cyanobacteria is classified into those with spherical cells and those with chain-like cells based on the way the cells are gathered.

[0070] In the example of image processing result display in FIG. 8 , an image 1010 captured by the imaging device 110 is input to the image processing device 100, and the image processing result is displayed superimposed on the input image 1010. Rectangles 1011 and 1013 are rectangles that represent the position and size of cyanobacteria detected in the input image 1010. Window 1012 displays the analysis result for the cyanobacteria detected in rectangle 1011. Window 1014 displays the analysis result for the cyanobacteria detected in rectangle 1013. The analysis result includes, for example, the type, aggregation morphology, and the number of cells estimated as a quantitative value. These windows 1012 and 1014 are result display windows presented to the user.

[0071] In Figure 8, the type of cyanobacteria, aggregation form, and the number of cells estimated as a quantitative value are displayed in result display windows 1012 and 1014, but the analysis results are not limited to these and the likelihood of the type and aggregation form of the aggregated object may also be displayed. The analysis results may be displayed using numerical values, icons, colors, etc. that represent each piece of information instead of text. The result display window may not be superimposed on the image, but may instead display numerical information or text as a list of the detected aggregated objects. The position, size, type, and aggregation form of the detected aggregated objects do not necessarily need to be displayed in their entirety; only those necessary may be selected and displayed.

[0072] This embodiment configured as described above also has the same effects as those of Embodiment 1. The image processing system 1000 of this embodiment can estimate quantitative values ​​with high accuracy based on the classification results by aggregation form for target aggregate objects in an image, even when the aggregate objects can take multiple types or a variety of forms. [Example]

[0073] Third Embodiment A third embodiment will be described with reference to Figures 9 to 13. The third embodiment is an image processing system 2000 that is an improvement of the image processing system 1000 described in the second embodiment and that accurately calculates statistical values ​​related to aggregate objects in an image.

[0074] 9 is a diagram showing a hardware configuration according to Example 2. An image processing system 2000 includes an imaging device 110, an image processing device 100, an information processing device 2001, a display device 111, and a storage device 1001.

[0075] The imaging device 110, image processing device 100, and display device 111 are the same as those described in the second embodiment, and therefore their description will be omitted. As described in the second embodiment, the image processing device 100 detects aggregated objects present in an image captured by the imaging device 110 or in its internal area, and estimates and outputs the aggregated form and quantitative value. The operations of the storage device 1001 and the information processing device 2001 will be described in detail below.

[0076] <Operation of each device> The storage device 1001 holds information related to quantitative value estimation and statistical value calculation preset by a user. The information related to quantitative value estimation in the third embodiment is as described in the second embodiment. The information related to statistical value calculation includes a program for calculating statistical values ​​used in the information processing device 2001.

[0077] The information processing device 2001 calculates statistical values ​​based on the quantitative values ​​for each aggregate object output from the image processing device 100. The statistical values ​​calculated by the information processing device 2001 are, for example, total values ​​or average values. Statistical values ​​may be calculated for each classification label of the type of aggregate object or the aggregation form, or for each combination of classification labels of both the type of aggregate object and the aggregate object.

[0078] When calculating statistical values ​​for multiple terms as described above, the statistical values ​​of all terms may be normalized so that the sum of the statistical values ​​is 1.0, and the ratio of each statistical value to the whole may be calculated. If the calculated statistical value is highly dependent on the number of aggregate objects in the image, such as the total number of elements that make up an aggregate object, calculating the statistical value as the ratio of each term to the whole as described above makes it possible to calculate an index that can be compared between images.

[0079] When calculating the statistical values, a plurality of images captured by the imaging device 110 may be input to the image processing device 100, and aggregate objects may be detected from the plurality of images, and aggregate morphology and quantitative values ​​may be estimated. The image processing results for the plurality of images may be compiled by the information processing device 2001, and the statistical values ​​may be calculated. Furthermore, when the image processing device 100 uses a partial image obtained by cropping a part of an image captured by the imaging device 110 as an input image to detect aggregate objects and estimate aggregate morphology and quantitative values, the above-described process may be repeated for the plurality of partial images cropped from the captured image, and the results may be compiled by the information processing device 2001, and the statistical values ​​may be calculated. Since it is considered that a larger number of aggregate objects will be detected by expanding the range in which the image processing device 100 detects aggregate objects and estimates aggregate morphology and quantitative values, bias in the range in which detection and estimation are performed is reduced, and more average statistical values ​​may be calculated.

[0080] The information processing device 2001 may output the calculated statistical values ​​as numerical information or a list of numerical values, or may output them in a format that is easy for the user to interpret, such as a table or graph.

[0081] Figure 10 shows an example of the display of image processing results when cyanobacteria are used as the aggregated object and the number of cells contained in the cyanobacteria is used as the estimated quantitative value. In this example, the types of cyanobacteria are classified into those composed of spherical cells and those composed of cylindrical cells based on the shape of the cells. The aggregation morphology of the cyanobacteria is classified into those with spherical cells and those with chain-like cells based on the way the cells are gathered. In this case, in the example of displaying the image processing results in Figure 10, it is assumed that an image 2010 captured by the imaging device 110 is input to the image processing device 100, and the image processing results shown in Figures 11, 12, and 13 are displayed.

[0082] In the example of FIG. 11, the total value of the quantitative values ​​is calculated as a statistical value for each label classified by the type of aggregate object, and is presented to the user as a table 2015.

[0083] In the example of Figure 12, for each label classified by the aggregation form of the aggregate object, the sum of quantitative values ​​is calculated as a statistical value, normalized so that the sum of the statistical values ​​for all labels is 1.0, and presented to the user as Table 2016.

[0084] In the example of Figure 13, the total quantitative value is calculated as a statistical value for each label classified according to the aggregation form of the aggregate object, and the total statistical value for all labels is normalized so that it becomes 1.0, and is presented to the user as a pie chart 2017.

[0085] In FIGS. 11 to 13, the statistical values ​​are displayed as tables or pie charts, but they may also be displayed as numerical information, character strings, other graphs, or the like.

[0086] This embodiment configured as described above also has the same effects as those of embodiment 1. The image processing system 2000 of this embodiment can calculate statistical amounts based on quantitative values ​​with high accuracy for aggregate objects of interest in an image. [Example]

[0087] Example 4 will be described with reference to Figures 14 and 15. Example 4 shows an image processing system that is an improvement over the image processing system 2000 described in Example 3 and that can accurately determine the state of a sample.

[0088] 14 shows a hardware configuration according to the fourth embodiment. An image processing system 3000 according to the fourth embodiment can include, for example, an imaging device 110, an image processing device 100, an information processing device 2001, a specimen state determination device 3001, a display device 111, and a storage device 1001.

[0089] The image processing device 100, information processing device 2001, and display device 111 are similar to the devices described in Examples 2 and 3, and therefore descriptions thereof will be omitted. As described in Example 2, the image processing device 100 detects aggregated objects present in an image captured by the imaging device 110 or in its internal region, and estimates and outputs the aggregated form and quantitative values. As described in Example 3, the information processing device 2001 calculates statistical values ​​based on the quantitative values ​​for each aggregated object output from the image processing device 100. The operations of the imaging device 110, storage device 1001, and specimen state determination device 3001 will be described in detail below.

[0090] <Operation of each device> The image capturing device 110 is a device for capturing an image of the sample to determine its state, and is, for example, a camera, a microscope, etc. The captured image of the sample is sent to the image processing device 100.

[0091] The storage device 1001 holds information related to quantitative value estimation, statistical value calculation, and sample state determination that has been preset by the user. The information related to quantitative value estimation and statistical value calculation in Example 4 is as described in Examples 2 and 3. The information related to sample state determination includes statistical value thresholds used in the sample state determination device 3001, or the configuration and parameters of a trained machine learning model.

[0092] The sample state determination device 3001 determines the state of the sample based on the statistics calculated by the information processing device 2001. Any number of states may be set for the sample. For example, river water may be used as the sample, and three state labels—acidic, neutral, and alkaline—may be set for the river from which the river water was collected based on its pH, and the sample state determination device 3001 may select one of the three state labels. The state of the sample may be determined by performing threshold processing on the statistical value calculated by the information processing device 2001 or by using a trained machine learning model. When the statistical value calculated by the information processing device 2001 is a statistical value for multiple terms and is highly dependent on the number of aggregated objects in the image, such as the total number of elements constituting an aggregated object, the state of the sample can be determined with higher accuracy by calculating the statistical value as the proportion of each term to the whole.

[0093] For example, let us consider a case where the image processing system 3000 is used to determine the pH state of a river. Assume that three pH state labels—acidic, neutral, and alkaline—are set for the pH state of a river. First, a sample of river water is collected from the target river, and an image is captured using an imaging device 110 such as a microscope. The captured image is input to the image processing device 100, which detects cyanobacteria in the image and estimates the aggregation morphology of the detected cyanobacteria and the cell count of each cyanobacteria as a quantitative value. From the output of the image processing device 100, the information processing device 2001 calculates the ratio of the total cell count to the total as a statistical value for each classification label based on cyanobacteria species or aggregation morphology, or for each combination of classification labels based on cyanobacteria species and aggregation morphology. This ratio is input to the sample state determination device 3001, which outputs the likelihood that the sample is acidic, neutral, or alkaline, respectively. The display device 111 displays the output with the highest likelihood as the pH state of the river sample.

[0094] FIG. 15 shows an example of the display of image processing results according to the fourth embodiment. An image 3010 is captured by the imaging device 110, and is subjected to detection of blue-green algae and estimation of aggregation morphology and quantitative value by the image processing device 100. A tag 3011 indicating the determination result of the river pH state output by the sample state determination device 3001 is displayed below the image 3010. In FIG. 15, the label of the state with the highest likelihood is displayed, but for example, the likelihood and the likelihood of other labels may also be displayed. In addition to the state of the sample and its likelihood, statistics such as the ratio of each item used as the determination material to the whole may also be displayed as a table, graph, etc.

[0095] The present embodiment configured in this manner also has the same effects as those of Embodiment 1. The image processing system 3000 of this embodiment can accurately determine the state of a sample based on the quantitative values ​​and statistical values ​​calculated for aggregate objects in an image of the sample.

[0096] The present invention is not limited to the above-described embodiments and includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment can be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment can be added to, deleted from, or replaced with other configurations.

[0097] The above-described configurations, functions, processing units, processing means, etc. may be partly or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC card, an SD card, a DVD, or other recording media.

[0098] An aggregate object may be defined as an object formed by a collection of multiple elements of the same type. The method for estimating the number of elements constituting an aggregate object may be defined as having different technical properties depending on the type of collection.

[0099] The above-described embodiment describes the invention as expressed below.

[0100] (Representation 1) An image processing device that estimates a quantitative value of an aggregate object in an image, comprising: an input unit that accepts an input image; an aggregate object detection unit that detects aggregate objects in the input image; an aggregate form estimation unit that estimates the aggregate form of the detected aggregate object; and a quantitative value estimation unit that estimates the quantitative value of the detected aggregate object based on the estimated aggregate form.

[0101] (Representation 2) The image processing device according to Representation 1, wherein the aggregate object detection unit includes: a feature extraction unit that extracts features of the input image; an aggregate object position estimation unit that estimates the position and size of the aggregate object using the extracted features; and an aggregate object type estimation unit that estimates the type of the aggregate object using the extracted features.

[0102] (Representation 3) The image processing device according to Representation 1 or 2, wherein the aggregated form estimation unit estimates the aggregated form of the aggregated object using the extracted feature amount.

[0103] (Representation 4) The image processing device according to any one of Representations 1 to 3, wherein the quantitative value estimation unit estimates the quantitative value of the aggregate object using the extracted feature amount.

[0104] (Representation 5) The image processing device according to any one of Representations 1 to 4, wherein the quantitative value estimation unit estimates the quantitative value of the aggregate object using, in addition to the extracted feature, one or more of the position and size of the estimated aggregate object, the type of the estimated aggregate object, and the aggregate form of the estimated aggregate object.

[0105] (Representation 6) An image processing device according to any one of Representations 1 to 5, wherein the quantitative value estimation unit estimates the number of elements constituting an aggregate object in the input image as the quantitative value of the aggregate object.

[0106] (Representation 7) An image processing device according to any one of Representations 1 to 6, which uses a model trained by machine learning using information that provides the image with the position and size of aggregated objects in the image, the type of aggregated object, the aggregated form, and a quantitative value.

[0107] (Representation 8) The image processing device according to any one of Representations 1 to 7, further comprising an output unit that outputs the quantitative value of the aggregate object estimated by the quantitative value estimation unit.

[0108] (Representation 9) An image processing method for estimating quantitative values ​​of aggregate objects in an image by a computer, wherein the computer executes the following steps: an input step for accepting an input image; a feature extraction step for extracting features from the input image; an object position estimation step for estimating the position and size of aggregate objects in the input image using the features output by the feature extraction step; an object type estimation step for estimating the type of aggregate objects in the input image using the features output by the feature extraction step; an aggregate form estimation step for estimating the aggregate form of aggregate objects in the input image using the features output by the feature extraction step; and a quantitative value estimation step for estimating quantitative values ​​of aggregate objects in the input image using the features output by the feature extraction step.

[0109] (Representation 10) An image processing method according to Representation 9, wherein the quantitative value estimation step estimates a quantitative value of an aggregated object in the input image using, in addition to the feature value output by the feature extraction step, one or more of the position and size of the aggregated object output by the object position estimation step, the type of the aggregated object output by the object type estimation step, and the aggregated form of the aggregated object output by the aggregated form estimation step.

[0110] (Representation 11) The image processing method according to Representation 9 or 10, wherein the quantitative value estimated in the quantitative value estimation step is the number of elements that make up an aggregate object in the input image.

[0111] (Representation 12) An image processing method according to any one of Representations 9 to 11, further comprising an output step of outputting to the outside the quantitative value of the aggregate object in the input image outputted by the quantitative value estimation step.

[0112] (Representation 13) An image processing system for estimating quantitative values ​​of aggregate objects in an image, comprising: an imaging device for capturing an input image; an image processing device for detecting aggregate objects in the input image and estimating their aggregate form and quantitative values; a storage device for outputting information used in the image processing device; and a display device for presenting the output results to a user.

[0113] (Representation 14) The image processing system according to Representation 13, comprising an information processing device that calculates statistical values ​​regarding aggregate objects in the image using estimation results regarding aggregate objects in the image output by the image processing device.

[0114] (Representation 15) An image processing system according to Representation 13 or 14, comprising a sample state determination device that determines the state of a sample imaged by the imaging device using statistical values ​​regarding aggregate objects in the image output by the information processing device. [Explanation of symbols]

[0115] 100: image processing device, 101: interface unit, 102: calculation unit, 103: memory, 110: imaging device, 111: display device, 201: input unit, 202: object detection unit, 203: aggregation morphology estimation unit, 204: quantitative value estimation unit, 205: output unit, 206: feature extraction unit, 207: object position estimation unit, 208: object type estimation unit, 300: image processing method, 301: input step, 302: feature extraction step, 303: object position estimation step, 304: object type estimation step, 305: aggregation morphology estimation step, 306: quantitative value estimation step, 307: output step, 1000, 2000, 3000: image processing system, 1001: storage device, 2001: information processing device, 3001: sample state determination device

Claims

1. 1. An image processing device for estimating quantitative values ​​of aggregate objects in an image, comprising: an input unit that accepts an input image; an aggregate object detection unit that detects aggregate objects in the input image; an aggregated shape estimation unit that estimates an aggregated shape of the detected aggregated objects; a quantitative value estimation unit that estimates a quantitative value of the detected aggregated object based on the estimated aggregated form; An image processing device comprising:

2. 2. The image processing device according to claim 1, The aggregate object detection unit a feature extraction unit that extracts a feature of the input image; an aggregate object position estimation unit that estimates the position and size of the aggregate object using the extracted feature amount; an aggregate object type estimation unit that estimates the type of the aggregate object using the extracted feature amount; An image processing device comprising:

3. 3. The image processing device according to claim 2, The collection form estimation unit estimates the collection form of the collection object using the extracted feature amount. Image processing device.

4. 3. The image processing device according to claim 2, The quantitative value estimation unit estimates a quantitative value of the aggregate object using the extracted feature amount. Image processing device.

5. 3. The image processing device according to claim 2, In addition to the extracted feature amount, the quantitative value estimation unit using one or more of the estimated position and size of the aggregated object, the estimated type of the aggregated object, and the estimated aggregated form of the aggregated object, Estimating a quantitative value of the aggregate object Image processing device.

6. 2. The image processing device according to claim 1, The quantitative value estimation unit The number of elements constituting the aggregate object in the input image is estimated as a quantitative value of the aggregate object. Image processing device.

7. 2. The image processing device according to claim 1, A machine-learned model is used based on information given to the image, including the position and size of the aggregated objects in the image, the type of aggregated object, the aggregated form, and quantitative values. Image processing device.

8. 2. The image processing device according to claim 1, an output unit that outputs the quantitative value of the aggregate object estimated by the quantitative value estimation unit; Image processing device.

9. An image processing method for estimating quantitative values ​​of aggregate objects in an image by a computer, comprising: The computer an input step of accepting an input image; a feature extraction step of extracting features from the input image; an object position estimation step of estimating positions and sizes of aggregate objects in the input image using the feature amounts output by the feature amount extraction step; an object type estimation step of estimating a type of aggregate object in the input image using the feature amount output by the feature amount extraction step; an aggregate shape estimation step of estimating an aggregate shape of the aggregate object in the input image using the feature amount output by the feature amount extraction step; a quantitative value estimation step of estimating a quantitative value of the aggregate object in the input image using the feature amount output by the feature amount extraction step; An image processing method that performs

10. 10. The image processing method according to claim 9, The quantitative value estimation step, in combination with the feature value output by the feature value extraction step, the position and size of the aggregate object output from the object position estimation step; the type of aggregate object outputted from the object type estimation step; Estimating quantitative values ​​of aggregate objects in the input image using one or more aggregate shapes of aggregate objects output from the aggregate shape estimation step. Image processing methods.

11. 10. The image processing method according to claim 9, The quantitative value estimated in the quantitative value estimation step is the number of elements constituting the aggregate object in the input image. Image processing methods.

12. 10. The image processing method according to claim 9, The quantitative value estimation step may further include an output step of outputting the quantitative value of the aggregate object in the input image to the outside. Image processing methods.

13. 1. An image processing system for estimating a quantitative value of an aggregate object in an image, comprising: an imaging device that captures an input image; an image processing device that detects aggregate objects in the input image and estimates aggregate shapes and quantitative values; a storage device that outputs information used in the image processing device; a display device that presents the output results to a user; An image processing system comprising:

14. 14. The image processing system according to claim 13, an information processing device that calculates statistical values ​​regarding aggregate objects in the image using the estimation results regarding aggregate objects in the image output by the image processing device; Image processing system.

15. 15. The image processing system according to claim 14, a specimen state determination device that determines the state of the specimen imaged by the imaging device using statistics related to aggregate objects in the image output by the information processing device; Image processing system.

Images