Object classification device, object classification method and object classification system

Abstract

To provide an object classification device capable of accurately classifying the type or state of an object even when objects with various sizes, lengths, shapes and the like are mixed in an image.SOLUTION: An object classification device generates region unit information representing regional information of an object, classifies the object by discriminating the type or state of the object on the basis of a region unit feature amount extracted from the region unit information and a partial region unit feature amount extracted from a partial region of an input image, and presents classification results of the object to a user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to techniques for classifying objects in images. [Background technology]

[0002] When inspecting cells, materials, etc., inspection images taken with optical microscopes, electron microscopes, CCD cameras, CMOS cameras, etc. are often used. However, observing inspection images with the naked eye is very time-consuming and often requires specialized knowledge. Therefore, to support the inspection of cells and materials using inspection images, technologies have been developed, such as those that automatically classify objects in images.

[0003] For example, Patent Document 1 discloses a method of performing global classification using a fixed-size low-magnification image, and then performing local classification using a high-magnification image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-203949 Summary of the Invention [Problem to be solved by the invention]

[0005] However, depending on the object being inspected, the size and length of objects contained in the image may vary greatly, and methods using fixed-size low-magnification images may not be able to accurately extract regional information for all objects. For example, if the size of the low-magnification image is determined to match a small object, the entire image of the large object will not fit within the patch, making it difficult to accurately extract regional information for the large object. Furthermore, if the size of the low-magnification image is determined to match a large object, there is a risk that many other objects or debris surrounding the small object will be included, which may affect the accurate extraction of regional information for the object to be classified.

[0006] An object of the present invention is to accurately classify objects even when objects of various sizes, lengths, shapes, etc. are mixed in an image. [Means for solving the problem]

[0007] An object classification device according to one embodiment of the present invention includes an arithmetic unit and a storage device for storing a program executed by the arithmetic unit, wherein the arithmetic unit determines an object region of an entire object in an image, generates object region unit information based on the object region, extracts object region unit features from the object region unit information, extracts partial region unit features from each of one or more partial regions within the object region of the image, and classifies the object based on the object region unit features and the partial region unit features. [Effects of the Invention]

[0008] According to one aspect of the present invention, even when objects of various sizes, lengths, shapes, etc. are mixed in an image, it is possible to classify the objects with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating an example of a hardware configuration of an object classification system according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of a functional block diagram of an object classification device according to a first embodiment. [Figure 3A] FIG. 10 is a diagram illustrating an example of a method for calculating area unit information. [Figure 3B] FIG. 10 is a diagram illustrating an example of a method for calculating area unit information. [Figure 4] FIG. 10 is a diagram illustrating an example of area unit information. [Figure 5A] FIG. 10 is a diagram illustrating an example of a method for extracting a partial region unit feature amount. [Figure 5B] FIG. 10 is a diagram illustrating an example of a method for extracting a partial region unit feature amount. [Figure 6] FIG. 1 illustrates an example of an object classification method. [Figure 7] FIG. 10 is a diagram showing an example of a result of a majority calculation in the classification unit. [Figure 8A] FIG. 10 is a diagram illustrating an example of a method for presenting a classification result. [Figure 8B] FIG. 10 is a diagram illustrating an example of a method for presenting a classification result. [Figure 8C] FIG. 10 is a diagram illustrating an example of a method for presenting a classification result. [Figure 9A] FIG. 10 is a diagram illustrating an example of a GUI that displays the contribution of each feature amount to a classification result. [Figure 9B] FIG. 10 is a diagram illustrating an example of a GUI that displays the contribution of each feature amount to a classification result. [Figure 9C] FIG. 10 is a diagram illustrating an example of a GUI that displays the contribution of each feature amount to a classification result. [Figure 9D] FIG. 10 is a diagram illustrating an example of a GUI that displays the contribution of each feature amount to a classification result. [Figure 10] FIG. 2 is a diagram illustrating an example of a processing flow of an object classification method according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a hardware configuration of an object classification system according to a second embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of an evaluation rule for an image group according to the second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of area unit information. [Figure 14] FIG. 10 is a diagram illustrating an example of area unit information. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of an object classification device, method, and system according to the present invention will be described with reference to the accompanying drawings. In the following description and the accompanying drawings, components having the same functional configuration are designated by the same reference numerals, and redundant description will be omitted. [Example]

[0011] An example of a system for classifying objects in an image according to a first embodiment will be described with reference to Fig. 1. The object classification system 100 includes an object classification device 101, an image capture device 120, and a display device 121. The object classification device 101 includes an interface 110, a calculation device 111, a memory 112, and a bus 113. The interface 110, the calculation device 111, and the memory 112 transmit and receive information via the bus 113. The object classification device 101 is connected to the image capture device 120 and the display device 121 via the interface 110.

[0012] The following describes each component of the object classification device 101. The interface 110 is a communication device that transmits and receives signals to and from devices external to the object classification device 101. Examples of devices that communicate with the interface 110 include an image capture device 120 and a display device 121. Details of the image capture device 120 and the display device 121 will be described later.

[0013] The arithmetic device 111 is a device that executes various processes within the object classification device 101, and is, for example, a CPU (Central Processing Unit) or an FPGA (Field-Programmable Gate Array). The functions executed by the arithmetic device 111 will be described later with reference to FIG. 2.

[0014] The memory 112 is a device that stores the programs executed by the arithmetic unit 111, parameters, coefficients, processing results, etc., and is an HDD, SSD, RAM, ROM, flash memory, or the like.

[0015] The object classification device 101 may be a physical computer system (one or more physical computers) or a system built on a group of computing resources (multiple computing resources) such as a cloud infrastructure. The computer system or group of computing resources may include one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more processing devices.

[0016] When a function is realized by a computing device executing a program including instruction codes, the defined processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the computing device. Processing described using a function as the subject may be processing performed by the computing device or a system having the computing device. The program may be installed from a program source.

[0017] The program source may be, for example, a program distribution computer or a computer-readable storage medium (for example, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0018] The imaging device 120 is a device that captures an image of an object, such as a camera or a microscope. The imaging device 120 transmits the captured image to the object classification device 101. Note that the imaging device 120 may be replaced by a communication device that receives images via a network or the like, or a recording device that receives images by reading images recorded on a recording medium.

[0019] The display device 121 is a device for displaying the object classification information output by the object classification device 101, and is, for example, a display or a printer.

[0020] The object classification device 101 will be described in detail below.

[0021] 2 is an example of a functional block diagram according to the first embodiment of the object classification device 101. Note that these functions may be configured with dedicated hardware, or may be implemented by a computing device 111 that operates according to software.

[0022] The object classification device 101 includes an image input unit 201, an object region calculation unit 202, a region unit information calculation unit 203, a region unit feature extraction unit 204, a partial region unit feature extraction unit 205, an object classification unit 206, and an output unit 207. Each unit will be described below.

[0023] The image input unit 201 receives an image including an object to be classified, which is input via the interface 110. The input image is input via the memory 112 to the object region calculation unit 202, the partial region unit feature extraction unit 205, and the output unit 207.

[0024] The object region calculation unit 202 will now be described. The object region calculation unit 202 extracts an object region to be classified using an input image received by the image input unit 201. As an example of a method for extracting an object region, a case where a region divider is used will now be described.

[0025] Region segmentation is a technique for dividing an image into meaningful sets on a pixel-by-pixel basis. For example, pixels containing the object to be classified are classified as foreground, and other pixels are classified as background, and regions for each object are detected by detecting the combined regions of regions classified as foreground.

[0026] Examples of region segmentation methods include those using U-Net and SegNet. In addition, since objects may be densely packed in the inspection image, instance segmentation, which calculates region segmentation results for each object, may be used instead of normal region segmentation.

[0027] The following describes the area unit information calculation unit 203. Based on the object area information output by the object area calculation unit 202, the area unit information calculation unit 203 expresses area information such as the size, length, degree of deformation, brightness, and color of each object as one-dimensional or two-dimensional information of a fixed size and outputs the information.

[0028] An example of the area unit information will be described with reference to Figures 3A and 3B. The calculation example of the area unit information shown in Figures 3A and 3B is an example in which the shape of the area is expressed as waveform information (two-dimensional information) of a fixed size by plotting the distance from one point within the object area to each contour point of the area in polar coordinate format.

[0029] 3A shows region segmentation result 301, and FIG. 3B shows region unit information 302. Region segmentation result 301 is included in object region information output by object region calculation section 202. Object region 303 is an object region within region segmentation result 301. Point 304 is a point within object region 303.

[0030] Area unit information 302 is area unit information calculated based on object area 303. Point 305 is a point on the contour of object area 303. Deflection angle 306 is the deflection angle relative to point 305 when the vertical direction from point 304 (downward in the figure) is set to 0 degrees. Distance 307 is the distance on the image to point 305 when point 304 is the origin. In FIG. 3B, the horizontal axis represents deflection angle 306, and the vertical axis represents distance 307. Deflection angle 306 increases clockwise. The vertical axis represents distance 307, which increases downward in the figure.

[0031] The area unit information calculation unit 203 first calculates point 304 from the object area 303 included in the input area segmentation result 301. Point 304 can be calculated, for example, using the center of gravity or center of the object area 303. This makes it possible to obtain area unit information that enables more accurate classification. Points within the object area other than these may also be used. Next, the area unit information calculation unit 203 scans the pixels in the area segmentation result 301 in the vertical direction from point 304 and measures the distance from point 304 to the contour of the object area 303. The area unit information calculation unit 203 plots the measured distance in area unit information 302.

[0032] The horizontal axis of area unit information 302 represents the deflection angle when the vertical direction from point 304 is set to 0 degrees, and the vertical axis represents the distance from point 304 to the contour. For example, for point 305 on the contour line that exists in the direction of deflection angle 306 from point 304, if the distance from point 304 to point 305 is distance 307, then it is plotted at a point corresponding to deflection angle 306 and distance 307 in area unit information 302. In this way, the shape of object area 303 is expressed as waveform information of a fixed size, like area unit information 302.

[0033] Fig. 4 shows examples of the above-mentioned area unit information extracted for objects of various shapes. Object 401, object 402, object 403, and object 404 are examples of objects with different shapes. Area unit information 411, area unit information 412, area unit information 413, and area unit information 414 are examples of area unit information extracted from object 401, object 402, object 403, and object 404, respectively. Note that in Fig. 4, the center of gravity is used as the reference point within the object area. The method of expressing the area unit information is the same as that of area unit information 302 shown in Fig. 3B.

[0034] Object 401 is a circular object, and the distance from a point within the object area to the outline is the same in any direction, so area unit information 411 is expressed as a straight line. Object 402 is also circular, but is larger in size than object 401. Therefore, area unit information 412 is a straight line like area unit information 411, but is plotted at a lower position than area unit information 411.

[0035] Since object 403 has an elongated shape, area unit information 413 is expressed as a periodic wave. Object 404 has an uneven, distorted outline. The corresponding area unit information 414 also reflects this distortion. In this way, the area unit information described here is information that can express the shapes of objects of various sizes, lengths, and shapes as waveform information of a fixed size.

[0036] If we try to represent the area information of these objects using a fixed-size, low-magnification image, if we adjust the image size to fit a small object, there will be objects whose entire shape cannot be included in the image, and if we adjust the image size to fit a large object, the small object will contain many areas that are unrelated to the object to be classified.If the object to be classified is located in an area where other types of objects are densely packed, or near objects that are not to be classified, such as dust, there is a possibility that the shape information will not be extracted correctly due to the influence of the surrounding objects and dust.

[0037] In contrast, by using the area unit information described in the embodiments of this specification, even when objects of various sizes, lengths, and shapes are mixed, it is possible to represent the shape of a target object as information of a fixed size while suppressing the influence of other objects. As a result, by cooperating with the area unit feature extraction unit 204, partial area unit feature extraction unit 205, and object classification unit 206 (described later), it is possible to classify objects with high classification accuracy.

[0038] Note that the description here has been given of area unit information using only the object area information output by object area calculation unit 202. As another example, an input image may be received from image input unit 201, and area unit information may be extracted using both the input image and object area information.

[0039] 3A and 3B, the area unit information calculation unit 203 may calculate quantitative values ​​such as the average value and variance of the luminance of the input image on the line connecting point 304 to point 305 on the contour line, and may extract area unit information by plotting the values ​​against the deviation angle. The area unit information may be one-dimensional information, and may use, for example, an index such as the area of ​​the object area, circularity, or statistical values ​​of luminance or color. The area unit information calculation unit 203 may output multiple types of area unit information.

[0040] The above describes a method for extracting information such as the distance from a point within an object to each contour point of the area in polar coordinate format as an example of area unit information, but the method for generating area unit information is not limited to this, and any method can be used as long as it extracts area-related information and represents it as one-dimensional or two-dimensional information of a fixed size.

[0041] An example of obtaining area-unit information using frequency transformation will be described with reference to Fig. 13. Image 1301 is an input image and includes object 1302. Pre-frequency-transformation image 1303 is an image for frequency transformation in which only the area of ​​object 1302 is placed at the center of the image, and post-frequency-transformation image 1304 is an image obtained by frequency-transforming pre-frequency-transformation image 1303. Area-unit information 1305 is an image obtained by extracting a portion of post-frequency-transformation image 1304.

[0042] First, area unit information calculation unit 203 extracts object area 1302 from image 1301 based on object area information output by object area calculation unit 202. The extracted object area 1302 is placed at the center of pre-frequency-conversion image 1303. At this time, the luminance value of areas other than object area 1302 in pre-frequency-conversion image 1303 is set to a fixed value (for example, 0). Object area 1302 may be an image obtained by extracting the object area from image 1301, or may be object area information output by object area calculation unit 202 (an image in which the object area and other areas are expressed in binary).

[0043] The area unit information calculation unit 203 sets the size of the pre-frequency-converted image 1303 to the maximum size of the objects included in the input image. Then, a frequency transform such as a Fourier transform is performed to generate a post-frequency-converted image 1304. The post-frequency-converted image 1304 indicates higher frequency components as it moves away from the center, and larger components are represented by higher brightness.

[0044] Since low frequency components are important for the overall shape information of the object region, the region unit information calculation unit 203 extracts a region representing low frequency components from the frequency-converted image 1304 and sets it as region unit information 1305. In the example of Fig. 13, since low frequency components appear in the center of the frequency-converted image 1304, a partial image of the center is extracted with a predetermined fixed size.

[0045] Although the image before frequency conversion is also two-dimensional information of a fixed size, in the case of a long and thin object, for example, most of the area is the fixed value, which results in poor memory usage efficiency.By using the low-frequency components of the image after frequency conversion, the shape of the object can be efficiently expressed as two-dimensional information of a fixed size.

[0046] Note that when placing object 1302 at the center of pre-frequency-transformation image 1303, normalization processing such as rotating the object so that its major axis is vertical may be performed. Also, in the example of Fig. 13, low-frequency components appear in the center of post-frequency-transformation image 1304, so a method of extracting a central partial image of a predetermined fixed size has been described, but other frequency transformation methods may also be used, and only information related to necessary frequency components may be extracted depending on the transformation method. For example, in the case of a frequency transformation method that extracts multiple frequency transformation results with different frequency components from an input image, only the frequency transformation results related to necessary frequency components may be extracted and used as region-unit information.

[0047] Another example of area unit information will be described with reference to Fig. 14, in which the angle of the tangent at each point on the contour of an object is plotted and used as area unit information. Area segmentation result 1401 is object area information output by object area calculation unit 202. Object area 1402 is an object area within area segmentation result 1401. Object contour image 1403 is the result of applying image processing to area segmentation result 1401 to extract only the contour of the object.

[0048] Object contour 1404 is the contour of object region 1402. Contour point 1405 is a point on object contour 1404. Contour point 1409 is a point on contour 1404 adjacent to contour point 1405. Tangent 1406 is the tangent to object contour 1404 at contour point 1405. Angle 1407 is the angle between tangent 1406 and the horizontal line. Area unit information 1408 is the result of plotting angle 1407 at each contour point 1405.

[0049] First, the area unit information calculation unit 203 applies edge extraction processing such as a Sobel filter to the area segmentation result 1401 output by the object area calculation unit 202, thereby obtaining an object outline image 1403 and an object outline 1404. Next, outline points 1405 are placed on the object outline 1404. The number E of outline points 1405 is a predetermined fixed value. For example, multiple outline points 1405 are placed at equal intervals based on the value obtained by dividing the total length of the object outline 1404 by E.

[0050] Then, the area unit information calculation unit 203 calculates a tangent 1406 to a point 1405 on each contour, and finds an angle 1407. Here, the angle 1407 is set to a range of -90 degrees to 90 degrees, with the horizontal line being 0 degrees. Area unit information 1408 is obtained by plotting each angle 1407 on a graph. The horizontal axis of the area unit information 1408 corresponds to the position of the point 1405 on each contour, and the vertical axis corresponds to the angle 1407. The value range of the horizontal axis is from 1 to E, and the size of the area unit information 1408 is fixed regardless of the object size.

[0051] The above technique makes it possible to represent shape information about an object's contour as a fixed-size graph (two-dimensional information). It differs from the polar coordinate format in that there is no need to determine a center point and that information about the size of the object is not included. While area unit information 1408 is shown as two-dimensional information in FIG. 14 to facilitate understanding, it is also possible to represent it as one-dimensional information by using signal intensity (brightness in the case of an image) as angle 1407. While an example in which the horizontal direction is used as the angle reference has been described here, the angle may also be based on the tangent at point 1409 on a contour adjacent to point 1405 on the contour.

[0052] In this way, the area unit information calculation unit 203 calculates and outputs area unit information that expresses the area information of the entire object as one-dimensional or two-dimensional information of a fixed size.

[0053] The area-unit feature extraction unit 204 will now be described. The area-unit feature extraction unit 204 extracts area-wide features of the entire object to be classified from the area-unit information output by the above-mentioned area-unit information calculation unit 203. As a feature extraction method, a machine learning method such as a Convolutional Neural Network (CNN) or a Perceptron may be used, or a manually designed feature extraction method such as a Histogram of Oriented Gradients (HOG) may be used.

[0054] Furthermore, when there are multiple pieces of area unit information output by the area unit information calculation unit 203, feature amounts may be extracted from the multiple pieces of area unit information in an integrated manner, or feature amounts may be extracted individually from each piece of area unit information.

[0055] The partial region unit feature extraction unit 205 will now be described. Based on the object region information output by the object region calculation unit 202, the partial region unit feature extraction unit 205 extracts features relating to texture, brightness, color, etc. from the partial region of the input image output by the image input unit 201. As a feature extraction method, a machine learning method such as CNN or perceptron may be used, or manually designed features such as HOG, or quantitative values ​​such as brightness and variance within the partial region may be used as the feature.

[0056] An example of partial region unit feature extraction will be described with reference to Figures 5A and 5B. Figure 5A shows an example of object region information, and Figure 5B shows an example of an input image from which the object region information shown in Figure 5A is extracted.

[0057] Object region information 501 is an example of object region information for a single object output by object region calculation unit 202. Object region 503 is shown in white, and the background region is shown in gray. Input image 502 is the region of the input image corresponding to object region information 501. Object 504 is the object corresponding to object region 503, and in this case, the entire duct structure in which cells are arranged relative to the cavity is considered to be the object. ROIs 505 and 506 are regions of interest within object region information 501 and input image 502.

[0058] The partial region unit feature extraction unit 205 first sets the region of interest of the object region information 501 at the top left of the object region information 501 (ROI 505). The width and height of the region of interest are assumed to be determined in advance by the user. The partial region unit feature extraction unit 205 determines whether or not to extract partial region unit features based on the object region information within the region of interest.

[0059] For example, if the pixels belonging to object region 503 exist in the region of interest at a rate equal to or greater than a predetermined threshold, partial region unit feature extraction unit 205 extracts partial region unit features. For example, if the threshold is set to 50% or greater, in the example of Figure 5, there are no pixels belonging to object region 503 in ROI 505, which is below the threshold, so partial region unit features are not extracted from ROI 505. On the other hand, since 50% or more of the area in ROI 506 belongs to object region 503, partial region unit features are extracted from ROI 506.

[0060] After determining whether or not a partial region unit feature can be extracted from a certain region of interest and completing the extraction process, the partial region unit feature extraction unit 205 performs the same process while changing the position of the region of interest, for example, in a raster scan manner. After completing the process for the entire input object region, the partial region unit feature extraction unit 205 outputs the partial region unit feature.

[0061] The object classification unit 206 will be described below. The object classification unit 206 classifies objects based on the area unit feature output by the area unit feature extraction unit 204 and the partial area unit feature output by the partial area unit feature extraction unit 205.

[0062] An example of an object classification method will be described using Fig. 6. An input image 601 is an input image output by the image input unit 201. Object region information 602 is object region information output by the object region calculation unit 202. A partial region unit feature 603 is a partial region unit feature output by the partial region unit feature extraction unit 205. A region unit feature 604 is a region unit feature extracted by the region unit information calculation unit 203 and the region unit feature extraction unit 204.

[0063] A classifier 605 is a classifier for calculating an object classification result 606 in units of partial region based on a part of the partial region unit feature 603 and the region unit feature 604. A classification result map 607 is an example of the result of mapping the classification result 606 in units of partial region according to the extraction position of the partial region unit feature 603 input to the classifier. The processing of the object classification unit 206 is within the range enclosed by the dashed line in FIG. 6.

[0064] The object classification unit 206 receives inputs of a region-unit feature 604 and a subregion-unit feature 603. The region-unit feature 604 is represented by one feature vector for one object, whereas the subregion-unit feature 603 is composed of a feature vector for each subregion. The object classification unit 206 includes an internal classifier 605, which receives as input the feature vector extracted from each subregion in the subregion-unit feature 603 and the region-unit feature 604, and outputs a classification result 606 for each subregion.

[0065] 6, it is assumed that the object types (objects A to C) are output as classification results. The classification method used by the classifier 605 may be a machine learning method such as logistic regression or support vector machine (SVM), or a method using a manually designed classification formula. The classification results 606 per subregion are recorded in a classification result map 607 according to the extraction positions of the subregion-based features 603 input to the classifier 605.

[0066] After calculating the classification results for all parts of the partial region unit feature 603 and recording them in the classification result map 607, the object classification unit 206 calculates an overall classification result for the object based on the contents of the classification result map 607. One method for calculating the overall classification result is to take a majority vote of the classification results in the classification result map 607, for example.

[0067] Fig. 7 shows an example of a majority decision result calculated from the classification results in the classification result map 607. In the example of Fig. 7, since object A has the most classification results, the objects in the input image 601 are comprehensively determined to be object A. In this way, the object classification unit 206 calculates the classification result for each object in the image.

[0068] The output unit 207 will now be described. The output unit 207 presents the classification results from the object classification unit 206 to the user. An example of a method for presenting the classification results to the user will be described using Figures 8A, 8B, and 8C. Figure 8A shows an example of an input image. An input image 801 is an example of an input image accepted by the image input unit 201, and is assumed to be an inspection image of a granular material. The granular material in the example of Figure 8A is assumed to be of good quality if each particle is round and large and has no scratches or the like. Here, particle 803 is assumed to be a particle of good quality, and particles 804 and 805 are assumed to be particles of poor quality.

[0069] Fig. 8B shows an example of a classification result window. Classification result window 802 is a window for presenting classification results to the user. Classification result 806, classification result 807, and classification result 808 show the classification results for particle 803, particle 804, and particle 805, respectively. Fig. 8C shows an example of a quantitative value display window. Quantitative value display window 809 is an example of a window for displaying the proportion of each object in an image.

[0070] 8A, 8B, and 8C, the object classification unit 206 classifies each object (particle) into a good-quality particle and a bad-quality particle, and particle 803 is classified as a good-quality particle, and particles 804 and 805 are classified as bad-quality particles. In the classification result window 802, the classification result is visualized, for example, by color-coding the object region according to the classification result, or the visualized result is displayed by blending it with the input image. In this example, good-quality particles and bad-quality particles are represented by different textures.

[0071] The output unit 207 may also calculate quantitative values ​​such as the number and ratio of various objects in the input image and display the calculation results in a quantitative value display window 809 .

[0072] Furthermore, the output unit 207 may display, for each object, evaluation values ​​such as the contribution and effectiveness of the region-unit feature and partial region-unit feature to the classification result. Figures 9A to 9D show examples of GUIs that display the contribution of each feature to the classification result.

[0073] 9A displays a pointer 901 on the classification result window 802. The pointer 901 is a pointer that allows the user to select a classification result for an object in the classification result window 802.

[0074] 9B shows an example of a contribution display window 902. The contribution display window 902 is a window that displays the contribution of the area unit information and the partial area unit information to the classification result of the object.

[0075] Fig. 9B shows an example of an area unit information display window. Area unit information display window 903 in Fig. 9C is a window that displays area unit information. Fig. 9D shows an example of a partial area unit contribution display window. Partial area unit contribution display window 904 is a window that displays the contribution of each partial area.

[0076] After the object classification unit 206 calculates the classification result, the output unit 207 first calculates the contribution of each feature to the classification result. An analysis method such as GradCAM may be used to calculate the contribution. The contribution of each feature to the entire object region is calculated by finding the region-unit feature and the contribution of the subregion-unit feature for the classification result for each subregion and summing them over the entire region. The calculated contribution is presented to the user via a contribution display window 902.

[0077] As shown in Fig. 9A, after presenting the classification result window 802 to the user, the system waits for an input from the user. For example, when the user clicks an object in the classification result window 802 with a pointer 901, the contribution of each feature amount to the classification result of that object is displayed in a contribution display window 902, as shown in Fig. 9B.

[0078] The example in Fig. 9B shows that, among the area unit features, the contribution of "shape" is high, followed by the contribution of partial area unit features. The area unit information related to "shape" here is assumed to be information on the distance from a point in the object area to the contour in each deflection angle direction, as explained in the area unit information calculation unit 203. Note that information on the contribution of only one of the area unit information and partial area unit information may be presented.

[0079] 9C , the output unit 207 automatically displays, for example, the area-unit feature of “shape” having the highest contribution in the area-unit information display window 903. The display contents of the area-unit information display window 903 may be switched by the user's selection. For example, when the user clicks on the item name of the area-unit information in the contribution display window 902, the clicked area-unit information may be displayed in the area-unit information display window 903.

[0080] 9D, the partial region unit contribution display window 904 displays a map of the contribution for each partial region. For example, in the example of the partial region unit contribution display window 904, the level of contribution is represented by a brightness value (white: low contribution, black: high contribution). As described above, presenting information about the object classification results can help the user understand the results.

[0081] The details of the first embodiment have been described above for each functional block. However, the embodiment of the present invention does not necessarily have to be configured with the functional blocks of FIG. 2, as long as it can realize the processing that realizes the operation of each functional block. FIG. 10 shows an example of a processing flow diagram according to the first embodiment. Each step corresponds to each element of the functional block diagram shown in FIG. 2.

[0082] In image input step 1001, an image of an object input via interface 110 is accepted.

[0083] In object region calculation step 1002, an object region to be classified is extracted using the input image accepted in image input step 1001. The method for extracting the object region is as explained above for object region calculation unit 202.

[0084] In area unit information calculation step 1003, area information such as the size, length, and deformation state of each object is expressed as fixed-size information and output based on the object area information calculated in object area calculation step 1002. The method for generating area unit information is as described above for area unit information calculation unit 203.

[0085] In area unit feature extraction step 1004, area features of the object to be classified are extracted from the area unit information calculated in the above-mentioned area unit information calculation step 1003. The method for extracting area unit features is as explained above for the area unit feature extraction unit 204.

[0086] In partial region unit feature extraction step 1005, feature amounts relating to texture, brightness, color tone, etc. are extracted from the partial region of the input image accepted in image input step 1001, based on the object region information calculated in the above-mentioned object region calculation step 1002. The method for extracting partial region unit features is as explained above for partial region unit feature extraction unit 205.

[0087] In object classification step 1006, objects are classified based on the area unit feature calculated in area unit feature extraction step 1004 and the partial area unit feature calculated in partial area unit feature extraction step 1005. The method of classifying objects is as described above for the object classification unit 206.

[0088] In the output step 1007, the classification results calculated in the object classification step 1006 are presented to the user. The method of presenting the results to the user is as explained above for the output unit 207.

[0089] As described above, according to the object classification device and method described in Example 1, it is possible to accurately classify the type or state of an object even when an image contains a mixture of objects with various sizes, lengths, shapes, textures, etc. [Example]

[0090] In the second embodiment, an object classification system will be described that uses the object classification device described in the first embodiment to comprehensively determine the type and state of an object from an entire image or a group of multiple images.

[0091] 11 shows a hardware configuration diagram according to Example 2. The object classification system 1100 includes an imaging device 120 such as a camera or a microscope, an input device 1101 that accepts operations from a user, and the object classification device 101 described in Example 1. The object classification system 1100 further includes a quantitative calculation device 1102 that calculates quantitative values ​​such as the total number and proportion of each object from the object classification results output by the object classification device 101 and calculates a discrimination result for the type and state of the entire image or a group of images, and a display device 121 that presents the discrimination result output by the quantitative calculation device 1102 to the user.

[0092] The imaging device 120 and the display device 121 are the same as the hardware components in the first embodiment, and therefore a description thereof will be omitted. The input device 1101 and the quantitative calculation device 1102 will be described below.

[0093] The input device 1101 is a device for receiving operation signals from a user, such as a keyboard or a mouse, and is mainly used to select a group of images from a group of images captured by the imaging device 120, on which the quantitative calculation device 1102 performs quantitative calculations.

[0094] The quantitative calculation device 1102 calculates a quantitative value for one or more image groups from the object classification results output by the object classification device 101. The quantitative value may be, for example, the total number or percentage of each object, or the sum or average of the identification scores, and the quantitative value is calculated from the entire object classification results for the image group selected by the user via the input device 1101. Then, the type and state of the image group are determined based on the calculated quantitative value.

[0095] The flow of processing by the object classification system according to the second embodiment will be described using an example of evaluating the quality of a granular material. Quality refers to the state of the granular material. First, the imaging device 120 captures images from multiple locations on the granular material. The captured images are input to the object classification device 101, which then classifies each particle in each image into a particle of good quality and a particle of poor quality.

[0096] The user operates the quantitative calculation device 1102 via the input device 1101 to calculate quantitative values ​​from a group of images captured from a single material. The quantitative calculation device 1102 integrates the object classification results for all of the input image groups and calculates the ratio of good-quality particles to poor-quality particles. The quantitative calculation device 1102 calculates an evaluation result for the material according to the ratio of good-quality particles, for example, according to the rules shown in FIG. 12. The calculated evaluation result for the material is presented to the user via the display device 121.

[0097] This allows the user to determine the type and state of the input images according to the classification results of the objects present in one or more input images. [Example]

[0098] Another embodiment of this specification will be described. When determining points within an object region, the area unit information calculation unit 203 may use a method for suppressing fluctuations in the points. For example, a method may be used in which only pixels on the object are considered as point candidates, or a method may be used in which the distance from each pixel on the object to the background (distance image) is calculated and weighting is performed according to the calculated distance.

[0099] Although the area unit information calculation unit 203 has been described as two-dimensional graph-like information for the area unit information, it may be expressed in other ways, such as by representing the vertical axis in brightness or color. Furthermore, in order to improve the robustness of the area unit information, preprocessing or weighting may be performed on the input image when calculating the area unit information.

[0100] Furthermore, when calculating area unit information in polar coordinate format or from each point on the contour line, a starting point for calculating area unit information may be determined based on object area information in order to suppress phase shifts. For example, one method is to calculate area unit information from the point farthest or closest to the center point within the object area. In this way, the starting point can be determined based on the shape of the object area. This allows a more appropriate starting point to be selected.

[0101] Although the object classification unit 206 has been described as determining the classification result for the entire object by majority vote, the method for determining the classification result for the entire object is not limited to this. For example, the classification result may be determined based on the sum of the classification scores within the object region, or a rule may be used such as that a certain number or a certain percentage of specific classification results exist within the object region. Furthermore, weighting may be applied when determining the score or majority vote based on object region information.

[0102] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0103] Furthermore, the above-mentioned components, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned components, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0104] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0105] 100: Object Classification System 101: Object Classifier 110: Interface 111: Arithmetic section 112: Memory 113: Bus 120: Imaging device 121:Display device 201: Image input unit 202: Object region calculation unit 203: Area unit information calculation unit 204: Area unit feature extraction unit 205: Subregion unit feature extraction unit 206: Object Classification Unit 207: Output section 1001: Image input step 1002: Object region calculation step 1003: Area unit information calculation step 1004: Region-based feature extraction step 1005: Subregion feature extraction step 1006: Object Classification Step 1007: Output step

Claims

1. An object classification device for classifying objects in an image, comprising: A computing device; a storage device that stores a program executed by the arithmetic unit, The computing device determining an object region for an entire object in the image; determining a point within the object region, and generating object region unit information, which is a fixed-size one-dimensional or two-dimensional signal representing a feature of the object region, in a polar coordinate format with the point as the origin; extracting an object region unit feature amount from the object region unit information; extracting a partial region unit feature from each of one or more partial regions within the object region; An object classification device that classifies the object based on the object region unit feature amount and the partial region unit feature amount.

2. An object classification device as claimed in claim 1, The object classification device, wherein the calculation device determines a starting point of a deflection angle when generating information for each object region in the polar coordinate format based on the shape of the object region.

3. An object classification device for classifying objects in an image, comprising: A computing device; a storage device that stores a program executed by the arithmetic unit, The computing device determining an object region for an entire object in the image; generating information for each object region based on the object region; extracting an object region unit feature amount from the object region unit information; extracting a partial region unit feature from each of a plurality of partial regions within the object region; determining a classification result for each partial region based on the object region unit feature amount and the partial region unit feature amounts of the plurality of partial regions; An object classification device that classifies the objects based on the classification results for the partial regions.

4. An object classification device according to claim 1 or 3, the calculation device analyzes the object region unit feature amount and / or the partial region unit feature amount and the classification result of the object, determines an evaluation value of the object region unit information and / or the partial region unit feature amount for the classification result, and presents the evaluation value to a user.

5. An object classification device according to claim 1 or 3; an imaging device that captures one or more images to be input to the object classification device; a quantitative calculation device that calculates a quantitative value for each type of object included in the one or more images based on a classification result of the object classification device for the one or more images; An object classification system that classifies the one or more images based on the quantitative value.

6. An object classification method for classifying objects in an image, comprising: The device determines an object region for an entire object in the image; The device determines a point within the object region, and generates object region unit information, which is a fixed-size one-dimensional or two-dimensional signal representing a feature of the object region, in a polar coordinate format with the point as the origin; the device extracts object region unit feature amounts from the object region unit information; the device extracts a subregion unit feature from each of one or more subregions within the object region; The object classification method, wherein the device classifies the object based on the object region unit feature amount and the partial region unit feature amount.

7. The object classification method of claim 6, comprising: The object classification method, wherein the device determines a starting point of an argument when generating information on the object region unit in the polar coordinate format based on the shape of the object region.

8. A method for classifying objects in an image, comprising: The device determines an object region for an entire object in the image; The device generates information on an object region basis based on the object region; the device extracts object region unit feature amounts from the object region unit information; the device extracts a partial region unit feature from each of a plurality of partial regions within the object region; the device determines a classification result for each partial region based on the object region unit feature amount and the partial region unit feature amounts of the plurality of partial regions; The object classification method, wherein the device classifies the object based on the classification result for each partial region.

9. The object classification method according to claim 6 or 8, comprising: the device analyzes the object region unit feature amount and / or the partial region unit feature amount and the classification result of the object, determines an evaluation value of the object region unit information and / or the partial region unit feature amount for the classification result, and presents the evaluation value to a user.

Images